IBM Security Access Manager Appliance Networking Topics – 2019-06-06 webinar

IBM Security Access Manager Appliance Networking Topics - 2019-06-06 webinar



our topic today is appliance networking topics experts from the IBM security access manager technical support team will cover interface and IP configuration routing and management vs. application IPs also don't forget to join the IBM security community to continue the conversations we've had today I'd also like to tell you about the new IBM VIP Rewards program IBM VIP Rewards is an easy way for you to engage with IBM and be recognized for how you our clients add value to IBM in this program you'll complete fun challenges and be rewarded for interacting with IBM learning new technologies and sharing your knowledge learn about the new IBM VIP rewards program at IBM does slash VIP rewards and join the IBM VIP rewards for security at IBM does slash joined IBM VIP rewards security here's a glimpse of the new look of the security learning academy we want you to take advantage of the learning academy it's a free full service learning platform including videos and demos self-paced training hands-on labs webinars and open mic recordings and customized learning roadmaps go to security learning academy com2 see what you can learn about access manager today's speakers are Ellen Ewing and Daniel como they're both I am technical support agents they will be assisted in answering your questions by their teammates Anneliese clop Sten Drescher and Nick Lloyd I'll hand it over to Alan now hello everyone thanks for joining us for today's open mic my name is Alan Ewing and I'm a member of the IBM security access manager L to support team and I'll be your presenter today along with my teammate Daniel como so what is the goal of today's open mic today we will be discussing the subject of appliance networking the goal of this open mic is to help our customers become a bit more comfortable and familiar with the networking configuration within the appliance in addition to this we will provide helpful methods of troubleshooting the available tools that can be used to do so and how to address common issues that our customers may experience regularly on our agenda today we will discuss getting started with the network appliance the appliance network at I'm sorry how to add view or delete interfaces or IP dresses review static routing concepts configuration and policy based routing all before then moving to troubleshooting connectivity problems reviewing troubleshooting tools and some additional tips that dang is going to provide to you so let's get straight into it when first getting started with the configuration if I Sam you have to complete a wizard that would take you through the configuring of all the appliances based settings settings such as the appliance hostname the LMI login credentials and you know things of that nature doing this initial configuration of the appliance you are going to be presented with various networking configuration options some of those options being the configuring of an interface creating or deleting the VLAN and setting an ipv4 or ipv6 gateway as shown in the screenshot within these options you will configure your first interface and the network's management IP address you can see in the screenshot to the right the process of configuring your first interface and management IP address it shows me selecting the interface enabling it manually adding and clearing it as a management IP address the management IP address requires configuration to gain access to the local management interface what we commonly refer to as the LMI which is ICS graphical user interface by default the LMI only listens on port 443 and the management IP address you cannot use an application address to access the LMI but you can later alter the LMI listening port so what if your environment requires that more than one network interface card be configured from the CLI the command-line interface by running the network interfaces second man you can additionally enable other interfaces if you require them as you can see here on the right you will choose your new interface in my case this will be number 3 which is interface 1.2 you will then enable the interface select your configuration mode this is usually going to be menu then continue to set the IP address for this interface a reason you may want to add another interface would be to divide up your network traffic this is commonly done with our customers within using management and application traffic management traffic consistent of internal traffic usually not client-facing for example this would be a policy server LMI and ssh traffic application traffic on the other hand is generally what you would be exposing on the front end to the client examples such as reverse proxies and low balance from here you can run the network interfaces show command to view all configure interfaces this will tell you the configured addresses for each interface as well as this the interface is up and running additionally the output of the interfaces show command can be used to help confirm which interface maps to which network interface card so that you know which one to configure with which subnet this can be done by reviewing the IP version 6 address which contains bits of the MAC address so how do we view all of this configure all of this information within access managers graphical user interface from the main menu of the LMI navigate to the managed system settings network settings interface tab here we can see all of the networking configuration that we have done up until this point via the CLI now within the yellamma we now know where to find the networking configuration but how do we add additional IP addresses if we need them to add an IP address to an interface from the same interfaces tab select the configure interface and click Edit from here you would go to the ipv4 settings tab and click new click enable to make the IP active if the IP address is meant for management use then you'd also want to check the management address box as well from there save configuration and then deployed deploy the changes there's more than one way to also do this adding a new address to an interface can be done via the command line interface as well you will use the same network interfaces second man use when adding a new interface that I just showed you you'll choose option to add an address then provide the subnet and IP address this is the same place you would be able to also delete an IP address if needed via the command line interface as you may have heard or read this plenty of times before we often want our customers against spanning multiple interfaces with one-seven configuring the same subnet on multiple interfaces will result in the routing table containing duplicate routes one for each interface this being so requests might arrive over a particular interface an IP address but possibly be returned over a different interface an IP address in certain environments this prove this proves to be a problem because switches can reject response packets which return over at different IP address we generally able to identify this type of issue if some interfaces seem to become unresponsive or when a package tray seems to capture traffic only on one side as its profit proper networking we always recommend assigning each subnet to a new interface so we have taken extra precautions to prevent this type of configuration I see em now validates this type of networking configuration here's an example of me attempting to configure an overlapping subnet notice the system error presented it indicates the problematic route as well as alerts you that the interface would not be updated until the overlapping address has been removed I then click to override the overlapping subnet validation setting at the bottom of the screen and I am presented with a system warning which also advises against this type of configuration as well now if you were for during traffic in the same network you don't use the default gateway the art table is used to find the MAC address corresponding to the local IP address for which it is trying to communicate within the command-line interface by running the network ARP show command you have the ability to review all entries within the I am our cache table I provided an example of what this output may look like here so what if you need to deliver packets to systems with IP address is not on the same network as one of your appliances interfaces as an administrator you have to consider the IP addresses for all client success in the appliance services so there will be management LDAP reverse proxies authorization servers and all servers the appliance needs to communicate with LDAP a high-volume databases Junction servers things of that nature this is where static routing becomes a very useful tool as it provides you with more control over the path of the traffic the appliance maintains its own routing table that it uses to manage these static routes when initially configuring the appliance you specify default gateway until you configure others this will be the only entry within the appliances default routing table you may only need the one to fall route if your appliances IPS are all in the same Network having interfaces from different networks means additional routes may need to be to be recreated I'm sorry by default this table is used to deliver traffic when the appliance doesn't know what path to send it this will look similar to the screenshot below notice in the top left corner the default routing table is being displayed showing the only configure entry I have which is my default gateway and that the interface is leaving on 1.2 which is my application interface meaning traffic will arrive and leave on the same interface thus preventing what would appear otherwise is spoofing to configure a static route from the top menu select manage system settings Network settings static routes you would then specify the destination which would be the network address of the destination network the gateway address that will be used before that traffic the metric which assigns a cost to each route allows for the most cost efficient path to be chosen the exit interface through which the network traffic will pass through and lastly you have the option to leave a comment that note in the use of the particular static route that you are then creating from here you will save and deploy those changes after creating a static grou you will then see it populate the currently active routes table section at the bottom of the page now once the newly configured route doesn't move to active table at the bottom we get this question quite often if the route does not move to the active tables and it is not considered to be a valid route one thing to be sure of is that if IP address you are using is on the same subnet as the gateway IP address okay so the appliance supports multiple routing tables you can configure a specific set of routes for each IP that is configured on the plan this setting can overcome a single point of failure that from having a single interface and gateway for a particular subnet or from having a single default gateway this interface specific routes might also be required to solve some firewall conflicts in the appliance that has multiple interfaces the return path for a particular request might be different from that request path in certain firewall configurations this situation is seen as spoofing as I mentioned earlier and that packet will be discarded if a set of static routes is not provided for a particular IP stable or the static routes and IP stable do not result in a route for that IP address then the default static route table is applied if the ability to define different routes for different destination IP addresses is not required then that recommend then we recommend placing all require static routes under the default static routing table I will now be passing the mic to my teammate Daniel como to discuss some troubleshooting techniques thanks Alan yeah my name is Daniel come on also with the I am level to support team with Alan's and today I'm gonna be talking about some tools techniques and tips for debugging networking issues with ICM in the examples I'm going to go over I'm gonna focus on some issues relating to wet shield Sun activity since we see that was probably the most common but some of the techniques will also apply to to any type of networking issues so the first scenario I want to go over is a common one where you go through your I Sam you configure a new web scene and you notice that the server state for that Junction is showing is not running so that's telling us that the web shield can't contact that back-end server so how do we go about figuring out why so the first step I would recommend is using the test connection tools from the LMI which you can get at through managed system settings and this under the network settings the test connection tool has two different modes of operation you can run it in TCP or SSL mode the first mode the TCP just does a basic sense enact Act sense of the TCP connection to the server and then stops there if that completes a report that the connection was successful and if it can't reach it then it'll let you know that it's able to connect so that's pretty straightforward because it's only testing the TCP layer it doesn't matter whether you the server itself is expecting an SSL connection or not you could put port 443 in there and the test would work just the same because it doesn't go any further than TCP one tip to remember the ICM LMI doesn't have nslookup capability within the LMI you need to go through the command line for that but if you want to check the name resolution quickly to make sure things are working correctly if you use the pass connection tool in tcp mode you'll see that it does report you the IP address so this is a useful test to do rather than having to open up an ssh session or a console session there's also the ssl mode which takes the test one step further once the tcp connection is established it then goes ahead and performs the SSL handshake with that server to let you go through and see the SSL details in this particular slide i've checked the advanced parameters to show what's available there they're not necessary for basic troubleshooting but if you want to do some specific SSL type debugging like for example you can selected use TLS 1.2 mode in the SSL handshake the test to make sure that server supports SSL 1.2 starts our TLS 1.2 or if you wanted to test SSL version 3 to make sure that's been disabled on the backend server for security reasons for example you could run that test there or if the backend server is expecting a certificate let's say from webseal you could test here where you specify the key file in this certificate to present to that server so just additional testing options that are available if the ssl connection lets you run the test you'll see connected if it connects successfully as well as other details of that ssl connection so in this case we can see the certificate change and by the backend server which in my case was just the one certificate it's just a self signed certificate but if it sends more you could see an intermediate certificate or a CA certificate as well the output also shows us the certificate that we received from the backend if you take that section between the beginning certificate and certificate you can copy and paste that into an online certificate viewer or load it into Windows and you can see the details of the certificate that was sent you can also see some of the protocol parameters that were negotiated during the handshake such as deciphers and things like that so all this information is useful if you're suspecting problems with the SSL itself so getting back to our are failing scenario where we can't connect to the back end server the first thing I would want to do is go and try the test in TCP mode so this will confirm that the wet seal or the appliance can't connect to that back-end server and it also tells us that IP address so the first thing you want to do is make sure that IP address is the one that you expect it's possible that someone's made a typo while setting up the DNS or someone was doing some testing added a host file entry in the appliance and forgot to remove it or something like that so you want to check the basics before going too far then once you've confirmed you know where you need to go the next step is figuring out how you're going to get there which is where you get into the IP routing as Alan mentioned with IP if the system is directly on your same subnet as you are then you can send to them directly you don't need to worry about an intermediate router but if the device is on a different subnet then we need to look at the routing table and figuring of how we're going to get there in my particular case I only have one subnet on the appliance than when they need to do one six eight not one and only one route so I know that I'm going to need my default gateways one 92168 dot one dot one so having verifies my configuration and my settings I can then talk to the networking team and make sure that this is the proper route to get to where I need to that there's no other router other router that I need to use to get to that specific destination one particular thing that you want to keep in mind when you're looking at the routing table is you want to look at the bottom section which is the active rows it's possible that you can go in and add a route which the ISO Mathias beams is not a valid route so it won't activate it so for example in my case because I only have a one 92168 dot one subnet if I go in and try to add the route to use router 10.10 dot in the LMI but it will not show up as an active route because I can't send to it directly I can only use rotors that are on my local subnet so that's just one tip to keep in mind as well if you things aren't working the way you should validate that the active routes are what you expect there so that all looks correct and the next step is to figure out what's actually going on on the network and for that we have the packet tracing tool which is also to manage system settings and network settings also including a link to attack note that we send to customers which details all the steps to enable the packets race that 9:06 we do have some improvements to the packet tracing capability so I'm going to go through that because not everyone I know has played with 9:06 yet so this is just for the awareness the for 905 before 906 rather this is what you see when you go to start a packet trace there's only three options the interface the filter and the maximum file size the problem before 1906 is that once that maximum file size is reached the packet ratio will stop it stops capturing so if you're not paying attention you may not realize that the packet trace is stopped so if you're running a test you need to keep an eye on the packet trace make sure that it's running if it stops before you've been able to reproduce the problems you need to either export or delete the old trace start it again and keep doing that until the problem has been captured with the trace running with 9:06 we now have the ability to to configure a number of trace files and how many we want to keep the maximum files parameter so what this does once the first trace file fills up and we roll over and start a second one and then the third until it reaches the maximum number of five so if I have that set to five files was it reaches five files it will get rid of the oldest one to make room for a new one so it'll always keep cycling between the five most current trace files so that's very useful you don't have the trace topping unexpectedly and causing you to miss capturing the problem that you're trying to capture we also have the option to specify a snap life and what this snap length is is the amount of data that's captured in the trace file from each packet let's say you're trying to troubleshoot an intermittent problem on a very busy system let's say that the problem is just seems to be network related like an Internet Mitton glitch on the network like connectivity type of issue you really don't care of all the contents of each packet going through especially since most of the time they're all encrypted anyway what you're really interested in is to be able to see the flow you want to see the packets whether they're sent properly whether they're acknowledged whether they were lost and retransmitted that sort of thing so in the case like that by reducing the snap length to say 80 bytes you still capture enough to see the IP and TCP headers and be able to watch the flow but not have to capture all this encrypted traffic that you're not interested in any way that would result in the smaller you know though it eases the load on the system as well as keeps the traces smaller so that's another useful feature that we have at 9:06 a couple of tips for all the versions when dealing with packet tracing the interface specification if you leave it blank it captures all traffic and all interfaces if you specify then you're limiting the capture only that one interface unless you have a reason to try to limit it we recommend just leaving that flanked it's not uncommon that we've seen people with traffic where traffic goes out one interface and comes back on another depending on the routing and stuff on the other systems and on the network so if you limit your captured only one in your face you're only capturing half of the conversation and that's not gonna help us debug your problem so recommend leaving that empty also the maximum file size defaults to one megabyte which is too small for pretty much anything recommend setting that to say a hundred megabytes before you start so that you don't end up missing the problem it's better to have too much not enough say a capture filter these are the same syntax as the TCP dump commands so I'm just providing some examples which might be useful for you in your debugging one common one that we use but no arms they should be captures everything but the ARP traffic so it's a good catch-all if you're not sure for example what type of problem you're having it could be DNS it could be connectivity also if you need to capture for example he's not sure if it's an issue with the LDAP or with the backend or whatever this captures everything and then you can filter that afterwards if you need to try to reduce the capture size because you get so much traffic going through you can for example specify I will only want to trace TCP 46:36 traffic in limited to that or four four four on one specific holster or even to specific hosts so there's just some examples that you may find useful and finally remember that when you're submitting data to IBM the packet race is not collected as part of the support file so if you're sending us the support file you'll also need to manually export the packet trace and upload that separately it's not part of the support file so once we have the packet trace we can load that into a tool like Wireshark what this does it allows us to see the packets on the network see what was happening and then from there understand the problem better in this particular case we can see this the webseal server sent a syn packet to try to set up the network connections or the TCP connections to the backend it didn't get a response so waited for a bit another one still no response since a third one and it eventually gave up so in this case you know we can't say why those packets weren't responded to but we now have the details that we can get the networking team involved and show them this and say okay tell me what's going on you know so they're gonna want to know the source IP in the source port and the destination IP in the destination port so that they can validate their firewall rules they can check their butters routers or load balancers or maybe even the problem with the end system so this is the what you're gonna need to troubleshoot this type of problem now the second scenario is a little bit more complex where the problem it doesn't happen all the time you have a junction that's set up and it's working fine everyone's using it but every now and then you get the user saying you know I got this messaging third party server's not responding but when you go in you take your Junction status everything's looking good so how can you troubleshoot this type of problem when you can't actually see it when it's happening so the first place that you want to look is in the webseal log file in there you see some entries like the ones I'm showing here where we can see that the webseal has lost contact with the junction server and then shortly after regain contact with the junction server so this is going to be our starting point a lot of these types of problems tend to be network related but there are some cases for example on the BG system if you actually run out of network Forks where some of the tuning on the I am and help I'm providing a link to some TCP tunings that we recommend customers implement to prevent any types of these these port exhaustion problems so that being said obviously if the problem is on the network you know I am support can't fix it but we can help debug it because we are the ones reporting the error so we're in the best position to to provide the details as to what is going on so if we if we want to report the issue to the networking team obviously there's not enough in our website log file to for them to be able to go on it's just basically saying we couldn't contact this server but if we want to give them specifics we're gonna need IP addresses and port numbers and for that we use the PD web snoop trace but in WebGL that'll give us those IP addresses and port numbers that we need to be able to report event and finally with the packet trace running at the same time we can see the complete picture of what was going on exactly at that time that caused the problem the PD web snoop trace I'm sure everyone's used that before so I'm not going to go into details of how to enable that but I did provide a link to the pack note that we send out to customers with the steps on how to enable that and say with the network trace I was already gone through that so I'm not going to spend more time there so once this nuke and the network trace are enabled at that point we just want to wait for the problem to happen once the problem does happen they remember to stop your traces so that they don't overwrite themselves and you lose the data then we go back to the web see a log file and we can find the entry for when the problem happens in this particular case we can see again this is server back-end ducky c-calm that we can connect so once we have that entry there's a few things that we're gonna need to make note of obviously the time stance so we want to know exactly when the problem happens the server that we were trying to talk to in the thread ID that's right IDs that's a big text number that I've got highlighted in red and I've also converted that to decimals because we're gonna need that next in this new trace so taking the timestamp and the thread ID we can then look through this new trace to find the particular thread and what happens that triggers that error in this tree in this example we can see that the webseal received to get requests from the client and then it failed to connect to the server and then close the socket so now also with that is we can see the the IP address and port number of the back-end server the sixth one 80216 8.1 that's 61 480 and the local IP address and port for the web still tried to use to make that connection which is the dot 125 for three 874 if you try to look through the network trace or that port 80 traffic because it's a server you may find 100 connections going on at the same time so the key here is that ephemeral port of the client for that web still used because there's only going to be one connection at that time from that IP address using that ephemeral port so that's what we need to look for in the network trace and loading that into Wireshark we can see that in this particular scenario webseal sent us in packets to the backend server and then somebody sent the reset back so we can't tell for sure whether that's the firewall or bounce or the in system itself but we know that you know this isn't the problem with the ice hand side so the next step obviously is to take all this information you know and send this to the networking team and let them debug and figure out why that reset was coming back so after going through that example I just want to also discuss some of the tools that we have on the ice I'm appliance through the command line through the ssh interface or the console so if you log into the admin and type in tools and then question mark you can see a list of the different tools that we have the first one is the connect tool this is just the same as the test connection tool from the LMI run in tcp mode it just does a TCP connection if it completes says it's successful if it can't complete they'll say it wasn't able to connect the next tool that's useful is the connections so with this you can actually see the the live report of all the listening sockets on the appliance and the established connections or other connections so for example if someone saying you know I'm trying to connect the web shield but it's not connecting or I'm trying to connect to the LMI and I can't you can go in and look for the listing socket for the IP address of the web feel or the LMI in the specific port and make sure that the process is running and listening on the socket if it's not perhaps the process is crashed or something like that so this is a good tool used in debugging the nslookup tool is also available this is a common tool for name resolution I'm sure everyone's used that the ping tool is also there the pain until just a couple of things to remember ping is checking the operating system of the destination host so it uses ICMP so you don't specify a tcp port or anything like that so it's not worrying whether your web server is up and running or anything like that it's validating that that system can be reached but because it uses ICMP you need to be aware that not every firewall out there is going to allow the ICMP traffic through so in your particular environment you may see pain failing even though there's nothing wrong on the network there's nothing abnormal it's just the firewalls are blocking it so that's just something to be aware the session tool is also similar to the test connection in through the LMI in that it is a connection to the server either in tcp mode or an SSL mode the difference is that well when you're doing it through the command line it actually keeps that active session open for you so for example if I do a session to a web server as in this case here we can see that it was connected and then after all the SSL output you're going to actually find yourself in a prompt within the session tool itself because you're actually sitting at an active connection to the server so for example if I wanted to test to make sure that servers active in what I think it is I could type in what I put there in yellow to simulate a client request going to that httpd server and I will see the response coming back from the server so this is one way to test basically you know if that is the proper server that you think it is if it's responding the way you think it is it's not a tool that you're gonna use very often but I just wanted to mention that it's there in case it comes in handy sometimes finally the last one is the traceroute tool this is a common tool again it's used to see the various hops that are used in the network to get from point A to B a couple of things to keep in mind with with the traceroute command the way it works it sends out the UDP requests and it's going to trigger each hop along the way to send back an ICMP response or an ICMP message back to you to let you know that it was there so because it uses UDP and ICMP we could again have the same issue as the ping command we're not every firewall may allow this through so something to be aware and also because of the way it works because it's relying on each host to send back an ICMP message when each host does that they will choose whatever source IP address they deem appropriate to get back to your address based on their own routing on the system so that doesn't necessarily mean that it's going to use the same IP address that you sent to it that you're expecting so as an example in my case here I only have the one router the one they D 2 1 6 8 1.1 so naturally I would expect to see that router in all my trace Road output but in this example if I do a trace route that 10.10 to attend up to which is just another IP address on that same router I see that it chose that address of the source address to reply back so my trace route is showing the 10.10 defend us to address instead of the 182 1 6 a 1.1 that I expected there's nothing we're on with this but I just wanted to point it out because if you're not aware of this it may be confusing you may be thinking that it's not going where it's supposed to go while in reality it is using the same system that you told it to so that's just a little gotcha that I wanted to bring up there so that that's all I have for the troubleshooting section we also have some additional links that we've included in here where you can find additional information feel free to your browser and use that's it for me thank you very much all right and it looks like we do have a question that's come through and that question is are there plans to force static routing to use specific IPs instead of GW yeah for that question I guess I'm not clear what what you're looking for the way that the routing works I mean you can specify a route which will use a specific interface if you want otherwise if you don't specify which interface to use for that route it will just pick the yeah whatever one it deems most appropriate so I'm not sure I can reach out to you afterwards and see what what exactly it is that's you're having trouble with and see ya route goes to a specific IP which is the gateway so it what we need to know yet what do you mean by a specific IP you point a route at a specific IP which is generally a gateway now does it necessarily have to be a gateway no you know in the concept it can do it has to be an IP that can accept the traffic and do something with it or do you mean you want to make sure that a route has a specific IP as the source IP you can do that at several different ways but yeah we need to know more about what what you mean by exactly a specific IP all right and it looks like we had a follow-up from the same attendee saying some interfaces have multiple IPS that have been configured correct if a route is if you have a static route that's going to use that interface it will use the source IP will be the first IP in the list that you look at in the LMI I think that may be what you're asking and the only way to change that is to reorder the IPS you can make policy based routing for that IP but that only affects traffic that comes in and out if it's traffic that's instantiated on the appliance such as a connection to LDAP or a connection to your Junction server it will it will use the default static routes and is there's also an option if you're doing the only component that can override all that is a webseal junction you can't pick the local address to use it's an option for a junction all right and it looks like that wraps up the questions that we had for today so I would just like to say thank you to everyone that attended today and I hope that everyone has a great day you

Leave a Reply

Your email address will not be published. Required fields are marked *