Cisco SD-WAN Issue Resolution Best Practices

Oh Jumping on jumping onto our today's agenda, we'll be covering a basic overview to just give you understanding of.00:00Of SD WAN architecture, and then we'll dive into control plane or control plane issues and.00:20A small demo, then we'll be going over to data plane issues and demo and then we'll be going over overlay routing and what are some issues that we see. We have limited time on our hands so we'll not be diving deep deeper into lot of issues we'll not be covering all of the issues.00:28We are covering the basics, right? And laying the foundation on what you could check before even like if it's a really complex problem before going to TAC, what we we what we could look at.00:48So we'll be covering all of that and we'll be covering some of our troubleshooting tools which which will help us determine all of these issues. So, alright, jumping into overview as everyone everyone already knows, right? SD WAN contains of all.01:05Plane, control plane management management plane and data plane. Our management plane is basically our V manager through which we get a single glass, a single pane of glass.01:25Through which we can manage, monitor, do software cycles on all of our edge devices. Our then comes the control plane which basically is our controller.01:40Just to put it out there, right? We have new nomenclature or new names for our controllers. I have listed those down here at the bottom, like we have vmanage, which is now called as SD WAN manager. We have catalyst SD WAN validator, which is called as V Bond.01:56One, and then there is SD WAN controller which is best. So I'll be using or switching between these two, so.02:16Yeah, don't be confused. These are the exact same things. The going to controller plane, controller plane consists of our which is basically our brains of an overlay, which is responsible for our policies and traffic engineering. Then there is.02:26Which consists of our validator.02:47Which is responsible for authentication of any edge devices that are coming into our and then there is data plane rate which is our edge devices through which our actual traffic is flowing. So we'll be covering all of these aspects and what are some issues that we see while we are.02:49Going over each of the spleads. So 1st diving into.03:09Control plane issues. Control connections, by control plane is basically control connections if you, what is control connection? Control connections are basically your DTLS TLS connections that are used for your control plane in SD WAN overlay or solution. These are very important.03:14And without our edge devices forming control connections, the edge device cannot come over to the overlay and form the IPsec tunnels or BFD tunnels and go get all the feature out of SD WAN. So.03:34As I said, control connections are used for pushing the configuration, updating updating the policies, sending alarm information, gathering statistic data in the much more, right? So how does this work? Like what is the primary thing that happens when we are trying to form the control.03:55Every edge device, it tries to communicate with it's, it's gonna try and communicate with the validator, right? So any edge device that is being onboarded to SD WAN overlay initially will come to the validator and validate.04:15Itself. Then the validator basically gives the information about its other controllers.04:34To the edge devices and that's how, it receives that information and will try to create the control connections. So, here to just go through the nodes that we have put in, it will, what it will do is it will give the T lock information.04:41Of V smart and we manage to the edge devices. That is how the edge devices will reach out to your vsmart and.05:01Be smart and we manage. Also, it acts as the, it acts as the net facilitator where it basically gives the T lock information. Let's say your edge device is basically sitting on a private IP and it it's.05:11That particular information that is given.05:33Advertised in our overlay by V bond. So that has an additional key part that it plays in our overlay. Alright, now we know about why control connections are con control connections are important. So we.05:37Wanted to cover basic.05:56Basic control connection down issues, right? So when when we are running into any control connections down issues, right? What, what are our main main commands that are gonna help us to troubleshoot what is going on in our network, right? Those are gonna be show as the control connection.05:59Which will give you information about your control connections, right? If they're forming over the colors that you have to determined. And 2nd one is show as the control local properties, which gives you information about.06:20Your system parameters, if the DNS resolution was done correctly, your information, your certificate information, what type of NAT was used and your WAN interface colors or VAN interface information as well, right? In addition to that here we have put.06:36In some.06:56Put in some debug debug commands as well. These can be enabled on your controllers and edge devices as well. With C edges, you could, you'll have to use the SD WAN keyword in between to enable this commands. This will give you information about the.06:58The background process, right? We didn't process sending out sending out messages about control connection.07:18Alright, here is an example of show shows the control connections. It gives you gives us a lot of information right to which controller are we communicating to or connecting to through which color, like here we see it's connecting over this internet and public internet.07:30Internet, and with the manage it is connected over the internet. So it gives you information about.07:47Through which color the control connections are forming and what exactly is happening. It gives you a public peer IP, and private IP as well. The segate command, second command is show as the control local properties. I had as I had mentioned all earlier.07:56Earlier, right? It gives you information about system configuration that was done. It gives you information about yourself certificate that was installed on the router, right? It gives you information about your V bond, right? It gives you information about number of active.08:17Interfaces that are present, right? So all of this information, you, if, if you are running through this particular output, right? It, it will give you a lot of information on based on the error that you're seeing, right? We'll be going over the next section when you are actually seeing the control connections down, how would we.08:36You how would you look at this?08:56So, let me go to the next slide. Yes. So when when we are seeing an issue, right, with control connections, one of the major major commands or main key commands that we can run is through as the control connections history. Let's say we had a.08:58Issue with our control connections, it went went down and came back up again and we are trying to figure out when did it go down, what was the error?09:18What was the error due to which it went down? So this command gives you information about what exactly or how exactly your control connections are behaving. If you look at the V bond, it gives me, discover or discover V bond, VBO, which basically is.09:27Disconnect V bond after registered reply and its status basically a tear down. As we know, once the device once the edge device.09:47Reaches out to the validator, it authenticates itself and shares the information about the other two controllers. It is automatically gonna tear down the connection. Here, these particular this local error right is is the column that we are looking for.10:00Let's say right now this was disconnect V bond, right? It could be any of this error.10:20This particular and let's say we saw this system IP change here, right? That means there was a system IP change due to which the control connections went down, right? Most common you would see is deconf failure. I'll get to that.10:28But this is our main main main column to look at. So most common issue that we have seen, right? Is what why could we see this? This could be due to dropped off filter packet.10:48Filter traffic for our control connection traffic, right? So how would you determine what is causing this or what, what would be your troubleshooting steps in this particular scenario? The things to check would be let's say.11:09For when you're forming the control connections, if it's your primary one, right? Like if you are forming the control connections, we go with four, 1st is the system configuration check, right? Where four parameters or four sections of parameters are very important, which contains of system, system.11:28System configuration, our default route, your tunnel configuration, and your SD WAN tunnel configuration, right? So all that, once that is configured, you check all of this, right? We usually check next top reachability, check if the ARP is resolved.11:49Ping the default gateway, see if the DNS resolution is working. Check if the V bond is reachable, right?12:09Even if our traffic is being sent out of our edge devices, there might be some middle hub that is dropping our traffic, right? If the DNS is not resolved for our controller, our, our traffic will not be forwarded. So that is also a key key part here. And then.12:17This route to be bond to find out which hop is basically dropping the traffic, right? Let's say you have gone through all of this process and you are still figuring out why exactly you couldn't find any solid.12:37Solid evidence on what exactly is happening. You could run a capture on the controller side with this particular command, as you guys maybe familiar with Linux, this is a pretty common Linux command, right? TCP dump, you're mentioning your VPN zero, you're mentioning the interface.12:49And you're mentioning your host IP and here are some verbose flags and all. So when you run this, right, basically you will see your traffic that is hitting your controller. Let it be.13:09We smart, we manage V bond. All of that, and the vhs basically are all based on same same platform. So TCP dump works on those. So here is an example, right? When you run this particular packet capture on your controller.13:25Are edge devices VH or controller devices, then you will see traffic coming and going between it, right? Let's say your traffic is not reaching this from your edge devices, then you will see only.13:45Single direction traffic you would not see bi directional traffic here. So that that is the 1st way to determine after checking your primary things that if your packet is being reached out or not. For I know vhs are.14:00And most of our customers are using ChS, right? How would you do a packet capture? There are multiple ways through we manage, you could do a packet capture, right? For this particular session, we are showing you through the CLI.14:21Because I know most of the tag uses this particular way and these commands are not out there so much. So we are trying to show you that here today. Example of when you're doing a packet capture on a CH, you'll have to have an access list which is con considered.14:39Listing of your source IP, destination IP, and it, it would be bi directional, right? Here that's why you will need two sequences here. And then you have to enable.14:59EPC, this is similar to what we did on ISXC regular isxs, right? The packet capture commands are seen which goes about this monitor capture commands, right? Where you determine access list.15:11Call out the access list determine, define the interface, define the buffer size and start stop it, right? Here is a screenshot for your reference. When we create an access list, this is how it would look like and once we configure that particular monitor.15:28Cap on our edge router. This is what you would see, right? Here the statuses inactive you'll have to start it, right? The direction is both buffer size is ten MB, access list which we configured on the edge device.15:47We are not configuring that today, but these commands are pretty pretty useful, so I would recommend you guys to keep this in mind, like this can be run on this. What are some show commands that can be run once you have that packet captured?16:04You could run it's similar to isxe, right? One cap, show monitor commands. One of the things or one of the commands that I have a screenshot is for this show monitor cap cap buffer, right? It shows you the parameters that we have configured.16:25On it. And 2nd one is basically showing me the traffic that is the control traffic for which we we created or created this packet capture part. So when we run show on cap buffer brief it will show us the packet capture or packet capture.16:43One key key thing to note is the DSCP values for your control traffic right here it shows us 48. So for your control traffic, it uses DSCP value as 48. This will be pretty important in one of our next slides, so keep that in mind.17:06Alright, now going to a few other other prominent errors that we see while forming control connection, right? One of the other ones is assert verification failure, right? This is the flag that you would see control verification fail or we.17:26Verification field.17:46Why does this happen? This happens due to expiry if like expiry of the certificate or there is a root cert miss mismatch or corruption of the root cert. So that these could be reasons due to which you could see the re see this particular error.17:48The one more reason is if the serial number is invalidated, let's say you go onto the V manage, go to your devices or certificates and invalidate the edge devices, push that to the controllers. If you have active edge devices that was up and running, it, it, it will basically throw this.18:09How do we, how do we basically, how do we, how do we take care of that particular issue, right? How do we approach or resolve that particular issue? 1st thing is.18:321st thing is to verify if the certificate match on the controllers on and on the edge devices, right? So you could run few commands and match the binary of the certificate and see if those are similar. If not.18:51Then you could basically re recopy the certificate from your controller, any controller Vbond V smart or vmanage, and basically reinstall it on the V manage. That is how you would be able to resolve that particular issue. Here is a copy command and.19:09And basically this is not under root, so any admin user on any of the controller will have access to the root certificate. This is the path user share with tell for all of your controllers. On the, on the CS side, you can install that on.19:29There is a particular file that is not under root.19:50Which I'm showing here, that is basically a/boot boot slash/home or SD WAN internal, that is the one that basically boot flash SD WAN is the.19:54Is the path which basically is not under root. So here is an example how you can copy the root certificate directly from the to any of the devices if it had control connections previously and the system IP is already there. You can use the path.20:13Public IP here as well and directly transfer the certificate to the edge devices. To install that certificate, you can run this particular command, request platform software, SD WAN, root searchain, install and give it the path. And it will reinstall the certificate.20:32And basically, you will be able to resolve that this particular issue.20:52Now going to another one of the common issues that we see. If you do not have a system configuration correctly added to the device, you might see this particular problem, right? If you have not correctly added added the organization name, you would see.20:58This control G MMS issue, which is basically organization mismatch mismatch issue.21:16Key important key thing here in an any overlay while you're onboarding the device is to have that device work in time synchronization. E.g., if your controllers and your edge device.21:31Is not in is has not does not have nt NTP configured or is in unsynchronized, you will most likely run into control connection issues. You will run into certificates not being installed issues. So I would always recommend.21:46Before onboarding or while onboarding, always check the clock because from my past experience working intact for four more than four years, and most for me. We have seen so many issues around this where if we do not have.22:06Time synchronized, you will not form control connections. You have everything correctly configured and your time is not synchronized, you will not form control connections. So I'll hand it over to Charita for a demo. Thank you Charita.22:25Thank you Tushar. Let me go ahead and share my screen. Alright, can I get a thumbs up or verbal communication if my screen is visible?22:42It's blank right now like I see your screen, yeah. That's not good. Let me share my screen again. Okay, this should do the trick. Alright, what about now?22:59Give it a second. Yes, I see the Yes. Perfect. Thank you so much and thank you to participants who have also been confirming the same. All right, so for the control connection issue, keeping time in mind and the fact that there's a lot of ground to cover, we'll be focusing on the most common issue that we've seen with custom.23:15Cases, is decon fail as to rightfully pointed it out. Anytime we start troubleshooting control connection issues, the 1st place that I would start if I were an admin is understanding the timestamp and details of the control connection failure, right? So if you go to monitor logs, you can.23:35See we have two options here. You have alarms and events. Now, if you've already set up alarm notifications for events like BFD flaps or control connection occurs, then you're already ahead of the game. You're already being notified when the issue is happening. For the purpose of this lab now, I'm going to be relying on the events section to understand when my control connection.23:55Flap happened, right? And the router I'll be working with is CH site one West R02 router. I've been made aware that there have been some control connection issues that have happened on this router. I could click on the three dots next to this particular event entry, click on device details to get some additional information about what the.24:15Flap, where the flap is occurring. Is it occurring on? Which specific T lock, which router, what time, what's the state and things like that. Now, something I do want to point out here is in a lot of customer cases, most likely you will have.24:35Two or more van circuits, which means you may not have a complete control connection failure, right? And that is also the case today. Right now this particular router that we're working with, it has two control two T locks, which means the control connections are lucky for me. They're only down on one T lock and it's up on the other one. So I want to take advantage of.24:51That and I want to show you what are some troubleshooting steps.25:11General steps that you can take from the dashboard and what are some troubleshooting actions you can take from the CLI of the router. Now, of course stack is here to help you out, but at the same time, any information, anything that you're able to resolve on your own will just make things that much more efficient for you. So what we're gonna do 1st is we'll go into monitor and.25:15And devices.25:34I'm going to go ahead and pull up my router that we're working with CH site West R02. If you scroll down through this menu, we should have an option control connections. And as you can see, it looks like even though we have two T locks.25:36Which I'll show you in just a bit. The control connections are up only on the public internet circuit.25:56Now, 1st thing I wanna do is make sure my biz internet circuit is up and running, right? So if you scroll up a little under the van section, you have an option called T lock. You can click on that and it should give you information about your T lock circuits. Now, as a matter of fact, you are able to move through the timeline. You're able to change it to seven days, 24 h.26:02I will probably move it to real time right now because I know I want to know exactly what's happening at the moment, right? So it looks like both the public internet and biz internet are looking up based on the T lock state and a green upward arrow. So that's looking good. So what I'm gonna do now is I'm going to go into the real time section of the monitoring and in the device option.26:22We have some.26:42Options that we're able to select, right? This is equivalent to you logging into CLI of the router and running commands like show SD WAN control connection or show SD WAN control connection history that Tushar shared with us earlier. Now we will do that today as well, but I also wanna show you some of the options we have from the dashboard. So let's.26:44Go ahead and do control connections. All right, so we see the control connections are up only on public internet. I also did see that I have an option for control connection history, so I'm gonna go ahead and select that as well.27:04Okay, looks like we have some control connection attempts being made to V bond, which is listed under the peer type and it gives us information about the public IP of the V bond. And the most important piece of information are these three columns, right? It tells me that the issue is happening on the biz internet. It gives me information about the local.27:24Now here it is not giving you the abbreviated format, but this DTLS connection failure is nothing but decon fail. So let's quickly hop over to our west router CLI.27:43And I'm just gonna run the same commands just to show you what the output looks like, ok? And we can see the connections are up on public internet. However, the connection to V bond on this internet is in connect state. What this essentially means is the router is attempting to form a controlled connection to vbond. Now.27:58Are known as orchestrator. And we know that is the 1st connection attempt that happens, right? 1st, it's gonna try forming a control connection to V Bond, and then Vbond will orchestrate the control connection to the rest of the controllers. So let's take a look at the control connection history once again.28:18Okay, so this is definitely my favorite part, a legend of all the errors, so if you see something in the local error, you don't recognize, you can always refer back to this index, but yes, here you can see that for biz internet V bond, it currently says decon fail. Now just to refresh our knowledge of what Tushar shared on decon fail, it's usually the case when there's.28:37Activity failure to the controllers. And this could be caused by multitude of things. It could be instability in the underlay, it could be a packet loss, maybe if it's a new onboarding that's happening, maybe there's a firewall on the part that has not been configured to allow this traffic. So one of the easiest things that we can do 1st.28:57And foremost is verify whether or not our router is forwarding those control packets to the controllers or is it dropping it because of any misconfiguration. Now in preparation for this demonstration, I have already set up a packet trace which is also known as Fiattrace and generally from my experience with.29:17The working intact Fiattrace also known as packet Trace is a quick way for us to verify if there are any drops happening on the router level itself. So we will share a document in the chat in just a second here on how to use packet trace for troubleshooting. But just know that I've already set up the trace and I will.29:37Show you the conditions that I've set up. You can run the command show platform conditions, and right now I'm.29:57Filtering for traffic going and coming to this IP address, as you can see the direction is set to both and I'm capturing this on all interfaces. However, you can drill it down to a specific access list or a specific interface or a specific IP address as well. So I'm gonna go ahead and start debug platform conditions start.30:03To start my packet trace. Now the control connection attempts will be made pretty frequently, right? So we don't have to wait for too long to start seeing some data. So I'll do show platform packet-trace statistics, and surely we're already seeing some packets get matched.30:23What concerns me though is the packets that are showing up in the drop. Now, just in the interest of time, I can tell you this, right? We can see these packets are being dropped and good for us, it already tells us what the reason is. It says IPv four ACL. Now to simulate the situation of decon fail, I did introduce an ACL into the lab.30:41So now I know why those packets are being dropped. In fact, once I go ahead and remove that ACL, my control connections will come up. But the idea of this demonstration is for you folks to understand what are some quick steps that you're able to do to collect some information before you engage stack or maybe even resolve it yourself, right? Now, in.31:01In a live environment, in a production environment, things maybe a little bit more complex and a little bit more gray. So it maybe possible that your router is forwarding all of the packets and maybe you're not seeing anything drop, which is good. Your router is doing everything it should. Now there are two possibilities that remain. One is packets are being dropped on the path to the controller or maybe they're being dropped.31:21For some reason on the controller. While that is not part of the demonstration, I do want to quickly take us back to the TCP dump command that Tushar shared with us.31:41So you could run that command on the controller, let's say V bond, and you can filter it for the router's public IP address. So now you're able to confirm on the controller side whether these control packets are being received or not. Now let's say on the controller side you're not receiving it. Then there you have your answer. So somewhere on the path it's getting dropped. You'll be surprised to.31:50Know how many cases we've had where we have seen ISP drop the packets and this is especially the case in newer deployments, right? It could be due to selective UDP filtering. We're not sure, but now at least we've narrowed down to the underlay being the issue here. So this is what I wanted to cover for control plane issue demonstration. With that, I'll hand it back.32:10Back to Tushal and he'll walk us through the data plane issues and what are some common issues faced in the data plane.32:30Tusha you maybe on mute. Oh, thank you. Thank you, thank you for an insightful demo. Can you guys see my screen? Yep. All right. All right. Thank you. We'll jump into our next section, data plane issues. I'll be.32:38Covering little bit of.32:58Basic architecture and what are our primary protocols that we use in our data plane. 1st thing is we use BFD, our overlay tunnels after the automatic overlay, overlay tunnels or IPsec tunnel formation, right? One of the key components of.33:00Protocol that is running on it is used for link link performance, it automatically runs on every overlay IPsec, right? And it cannot be disabled. BFD continuously monitors and measures the critical metrics.33:20Such as packet loss, latency, and loss, which is basically used in your application of the routing policies, data policies and all that. So this is a key key protocol to know. Second, second protocol is OMP overlay management.33:39Protocol basically which is used for, used for forwarding or sending all the, all the routing updates.33:59Like routing update, it basically operates over that DTLS TLS connection which is formed right between your controllers and your edge devices. The OMP peering, right? That happens is basically between vsmart and vsmart and from vsmart to VH.34:12There is NO peering happening between your edge devices. That is to, that, that is a key thing to keep in mind because I have seen a lot of confusion happening around now. And on that particular IPsec tunnel that is built.34:32You will see BFD packets being transmitted on. So OMPD is the process running in the background which is responsible for overlay management protocol. It basically requires control plane information which can be exchanged all.34:51All of this information is exchanged at vsmart, right? What happens is once your control connection forms, then, after, after that this OMP peering happens, right? And during that all the exchange happens about your T lock information, T lock is.35:11Basically your van interface which has color in SD WAN world, that particular van interface you are defining it with a color and that particular interface is called as T lock. That particular information, you're directly connected.35:31Directly connected routes, let it be BGP, OSP static routes, all of that information is basically sent out to vsmart. V smart then acts as a route reflector route reflector and sends out that information to all the overlay. Let's say we have hundred devices in our in our overlay. One.35:50Oh, one for one device, once the OMP update is received it will basically send it out to all of our overlay. Alright, now going into some of the 1st we'll see how we validate.36:11Our control plane or data plane is formed, right? How our IPsec tunnels are formed BFT tunnels are formed, how, how the OMP peering is happening, and then we'll go into a few common issues that we usually see across this data plane. 1st thing that we always validate, right?36:31Like is.36:511st part is control plane, right? If the control plane does not have control connections does not happen, the process does not go forward, right? There will be NO OMP peering. There will be NO update update going between the devices and.36:55Edge devices and the controllers. So I just wanted to real quick go over the process. 1st, what happens is your control control plane form or control connections form, then the OMP OMP peering happens. After that, T lock advertises our.37:15What advertisements are sent?37:34Once the vsmart receives that T lock advertisement, then it sends the, then the reflection happens, right? It is gonna directly send it over to your overlay. And then you have those BFD tunnels, right? Once the other end device receives the information about your one VH.37:36Now it knows that, ok, the other, other, let's say CH one has this T1T lock.37:56Great this particular information is known by C CH two and other way around and that's how after that information is exchanged, then they will start the IPsec formation or the tunnel formation process.38:03So, 1st we'll have to validate all these basic parameters, right? When we run show local properties, it gives us information about interfaces, the color assigned to that interfaces and state, right? This is basically our T lock.38:19Then going to control connections, it gives us that, ok, control plane is up, your controller connections are up. Then, then basically we can look at our show SD WAN OMP peers, right? This basically.38:37It gives us information about if there is peering between that particular edge device and the V smart. If that is correct, then all the information is basically sent out or received. So if you see, see it here, it gives us information like route received, route installed and route sent.38:56Then this particular command, right? So gives us information about what is being exchanged, right? What has been exchanged with the V smart. So anything from peer 000 means our router is basically sending out.39:19Its own T lock information to, to the vsmart. And from PR 3.3.3.1, basically this is our which is sending us information about other T locks present in our overlay.39:38And once that is done, we can run through when gives us information about all the BFD sessions that are happening or forming between all the edges present in our control. So this is the verification part.39:55One other thing that I wanted to mention is if you run the packet capture, there is a tool on the V manage. You can run now you can, you know how to run the packet capture on the device itself. You can export it into PCAP format.40:13Once you open it, right, you can, as I had mentioned, right, it uses DSCP 48 value. The control traffic is basically using BSCP 48 value. So here, when I look at our BFD session, right, this particular BFD session, if you look at the source IP and destination IP, it is.40:28It's 311-642-2242, right? If you do a packet capture for this, you can actually capture the packet packets that are exchanged for that BFD BFD between that particular edge devices or for that BFD tunnel, so.40:48Oh, yeah, just throwing out throwing it out there. Alright, now we'll jump into our troubleshooting scenario one. Let's say I know we have seen our customers usually see a lot of BFD issues. We'll be covering two scenarios here but the approach of the things would be similar.41:07Here similar for any kind of issues you see related to this. So 1st scenario that we are covering here is that.41:27We have two colors on two edge devices. One is business internet MPLS and the BFD session is down on MPLS. This is a hub and spoke edge routers where sending where we are sending traffic to each other, but they are not receiving anything any packets on MPLS categor.41:37This can be used for any topology, like even if it's full mesh, the control connection the process would would look similar. The 1st thing you would verify if you run into this kind of issue is basically those control connection look at the BFD session.41:58So when we run show as the BFD session session, right, what we see is that particular session isn't down, right, between MPLS to MPLS. We can actually check the statistics, right? For that particular tunnel. If you run show SD WAN tunnels.42:17Statistics section and in my scenario I only have four or five tunnels, but in your scenario you might have hundreds so this is a good way to filter out that one particular set.42:37So when you run this multiple times, right, it gives us information about, ok, what is the system IP, local color, remote color, tunnel MTU, it gives you transmitted and received packets, right? This is basically telling you, ok, fire edge device is trying to send.42:54Send traffic or trying to reform the connection, right?43:14Is it like that will tell you if the issue is on our device or the like basically spoke or hub device, right? The same thing, how, if you are seeing this, right? The packet is being sent out, right? And what we are doing is we can run a packet capture, right?43:18Which basically shows us, let's say this particular number even if it's incrementing, there could be an ACL, there could be a rule that is basically a firewall rule that is basically blocking that traffic on, on the same router that we have, right? So for.43:38Just to determine that we can run fiatres on the, on our edge devices, right? Similar to what we did for our control traffic, this can be done on the edge devices. These are the commands once we run that, right? And once we look at the summary, what we see is that our traffic is being forwarded.43:58By that what it means is we do not have anything that is blocking the.44:18Traffic, traffic for your BFD tunnel, right? So it basically tells you that. So this is from spoke side. You'll have to do same on the hub site, right? If I look at the hub site for the similar situation similar commands show as we are doing.44:26Every session tunnel statistics, I'm seeing similar things.44:43It is in downstate, I don't see any uptime and I see a transmitting packets are incrementing, but there is NO received packets. By that, what what it tells me like after doing on the both the routers, right? And it, it sees that it is.44:47Forwarded. By that what it means is that there could be something if it's not in the router that is being blocked, right? Then there could be something on the underlay that is blocking that particular traffic. You might see this issue for a particular circuit, a particular BFD tunnel.45:07Because every underlay part taken by the BFD tunnel, right? The formation we do not see we only see the endpoints, but we do not see the underlay path over which the IPsec tunnel is formed. So the key component here is gonna be the underlay part. The vmanage has as we know the the controllers the.45:26With every new release we are bringing in new feature and we have this underlay troubleshooting tool now from the V manage which basically gives you gives you information about your underlay path.45:46For each of your, for each of your con edge devices and you can basically set up local color and remote color as well. So this is a pretty, big, pretty, good tool. I know there has been a lot of struggle where we do not know what, what.45:58Underlay path is taken by our IPsec tunnel. So this is basically gonna give you that information. So here I'm not seeing any issues. Let's say my 3rd half or 4th half had any issues and it has been dropping my packet.46:18I would see a hundred percent being dropped and then I can directly log in into that particular hub and see what is happening to my traffic. All right, this was our 1st scenario where the VFD tunnel might be down or controllers are being sending traffic, our edge device.46:31Are trying to form that tunnel right exchange information, but there's there might be something in between that is dropping the traffic. 2nd scenario is pretty common as well where let's say we see control connections and all of the VFD sessions flap.46:51Right? So 2nd scenario that we have taken here is control connections and BFD session flap on the business internet car carrier. What, how would your approach would be? So 1st would be always looking at.47:08From the V manage itself, right? Trying to find out what is happening with that particular route. Going into vmanage UI, going into monitor network events. It gives you information about all the major events happening on the edge router. So if you filter it out or filter it out with control connection.47:23Questions BFD, you will find all of this information. Here, when we drill down or open that particular message, it gives us the reason, right? Flap reason. It is basically timing out. By that it means is like our device is trying to reach out to 2nd device, but there is NO response back. As we know, there is a.47:43The BFD timer, right? And there is a threshold so every like there is I don't recall the exact timer, but it keeps on trying to form or sending the BFD hellos.48:03And seeing if it receives the response back. And that is how it's gonna basically determine the liveliness of the path and keep it up, right? So by timeout, that means the traffic is not basically reaching the endpoint. What we do is basically look at our 1st phase, now here.48:15What we can take a look at is from your control plane, right? Remember what, what I said.48:35It is very important for edge device through each con each color to form control connections, right? Once it forms control connections, it received the T lock information. It forms the timer. So there is a graceful restart or graceful restart timer, right? Which is 12 h by default.48:41Which let's say your control connection forms and it drops.49:01Your will be up for 12 h, right? If it does not and it retries forming control connections. If it does not form connections, then it's basically gonna break it down. So here, the 1st thing we usually check is.49:05If the van interface is up, I see that Van interface is up, but if I see at VSVM, right? I'm not seeing any control connections, right? So when then I run show as the control connections I see that it's trying to connect to the V bond, but it's not able to.49:23There could be something happening, right? Like there could be a NAT issue, that happened. There might be some configuration issue that happened, right? When I do show control connection history, I I see what I see is basically a D decon failure and.49:42VM time VM timeout right for the timeout. But by that it means it is trying to communicate that with the V smart but it's not able to. I see the repeat count here as well, right? Twelve five five it always start keeps on reconnecting, right? So, and here if you see the public.50:00Private port, these are pretty important as well. And when I look at the show as demand BFD session, I see that it is basically a down.50:20What, what I could look at is after that look verifying the control connection, control plane information, then I go into that particular tunnel, right? When I do shown statistics.50:31What I'm seeing is that TX packet and RX packet are zero zero. By that what it means is our edge device is not initiating that BFD hello excine between the endpoint or so, right? And that is why basically it is down. You can drill down into QFA active paths, right? Which basically.50:45He gives you the information about your IPsec tunnel. If you look at this, right, it is in downstate and it's in in it state.51:05Right it has not initiated like it is initiated but it's not forwarding traffic. Like there is like three stages or 4th stages to it for any IPsec tunnel rate it goes from init state to full state, so you you see that here. So.51:13When looking at when what what we have seen previously is the previous, if we run the command shows to local properties, right? When interface list is a additional filter to it which basically gives you the van interfaces. You can look at the.51:30The port that it is using now. So on the previous command it was using, 12346.51:49Right, but now it has changed the port to 12386. Our previous control BFD session rate is also gonna use the same port. So due to this, it saw this particular issue. So once that particular, if you look at the show as.51:57You see that it is previously using another, another.52:18Port. So it is between for some reason it had to port up, by default it port hops between five or default ports and if you look at like if we clear the control connections, right, for that particular interface or clear.52:25That particular IPsec tunnel, right? It will basically form control connections with that new port. So.52:44Ideally, ideally this should not happen, but sometimes there if there is a cache of cache of that NAT translation save for some reason you would run into this kind of issue. Again, the underlay is gonna be very helpful here. Underlay troubleshooting tool is gonna be very helpful.52:52In any data plane troubleshooting issues. I'll hand it over to Charita for a quick demo.53:12Thank you Tushar. Let me go ahead and share my screen ok.53:21Alright, so I'm gonna assume that my screen is visible. I'm starting to get some thumbs up, ok? Great. So for any troubleshooting, I personally always like to start off from the events page and I'll do the same for the BFD troubleshooting as well, right? So for the purpose of this demonstration, we'll be focusing on the scenario.53:35Area one that Tushar talked about, which was BFD session showing down and.53:55Now I'm gonna go ahead and come to monitor events page. I did a little bit of a preparation here already. So I selected the advanced filters. There were a lot of events and I didn't want to go through all of those searching for the ones of interest. I selected the object type as device, I selected the device information, I set the severity level to all and I set.54:01BFT state change as the only event that I want to take a look at, right?54:22The reason I wanted to bring some attention to the screen was because if you click on the device details information, it gives us a little bit more information about the BFT flap. It tells us to which remote router the flap happened, it also tells us to which remote T lock the flap happened, but most importantly it tells us the flap reason. Now while.54:26The Flap reason by itself is not very verbose. It doesn't really give us, it just tells me timeout. It does give me a little bit of an understanding of what maybe happening, right? A quick refresher if already covered it, but with flap reason listed as timeout, that generally means that east router in this case, is sending out BFD hello packets to the.54:46The West router, but it's not getting any response back. So it goes ahead and times out that BFD session. Similarly, you may also see some other flap reasons like let's say remote down.55:06In such situation that's usually seen when the remote router maybe had some issue with the, that particular T lock on its end, so it explicitly sends out a message saying that the BFD session is being pulled down. So we go ahead and list that BFD event on our end as a remote down. Now, another flat reason that comes to my.55:17Mind is BFD deleted, which we've seen cases where if maybe the control plane is pulling it down for any routing change or any T lock related reasons. But all these reasons give us an understanding of how we want to start troubleshooting. While the initial steps will remain same.55:37I believe that it helps us figure out where we want to start troubleshooting from, right? Now, now that I have some information on the timestamp of the BFD flap on the flap reason timeout itself, I'm gonna go ahead and go into monitor devices. I'll go ahead and pull up the CH site East R two router that I'm working with.55:54And I'll go into real time because I want to take a look at some of the BFD related logs. So we know the drill under device options, I'm gonna search for BFD and I see an option called BFD sessions, so let me go ahead and pull that out. Now, in this case, it's a lab environment, so I don't have a lot of BFD sessions. However.56:14As Tusha pointed out earlier, in production environments, you maybe looking at hundreds of sessions. So it would be vital for you to capture the remote system IP from the event information and you can use that to filter here. Now here I can see one BFD session is already showing down. System IP points to 1010 81, which.56:34Which is my west R two router. It tells me the T lock biz internet on both ends.56:54And it gives me source IP and destination IP information and the encapsulation. Now, let me go back to the reason flat reason showing us timeout, right? Now this, like I said, this generally happens when we're sending BFT hello packets, but we're not getting anything back in return. So now I know in such situations, one of the 1st things that I.56:59I wanna take a look at is the tunnel statistics. Primarily what I'm looking within the tunnel statistics is the TX and RX packet count. TX meaning the packets I'm sending out RX meaning the packets I'm receiving. Now on this particular router.57:19Let's see. This is the session that we're working with because I remember the destination IP address, and let's see, we've got the tunnel empty here. We've got TX packet count and RX packet count. So this is currently what it is, right? Currently 9702 TX and 303 RX packets. Now, on this particular.57:34Lab, the BFD hello interval is set a thousand milliseconds, which means every second I'm sending out a BFD hello packet to ensure the liveliness of the tunnel. So let me go ahead and refresh that because I'm curious to see what happens to TX and RX count. So again, it looks like the TX count went up, but the Rx count remains the same. I can refresh it.57:54Again and if the same behavior persists, which it is, this generally means that this router is sending packets, but it is not receiving anything on this particular T lock from the remote router. Now I'm curious how things are looking on the other end. So let me quickly go over to the west router.58:13It's automatically going to take me back to the page. It's already run the device option command. Excellent, and the 1st session is what we're working with, right? Cause we're working with 1020 81 which is our east router and this internet is the T lock. So here I can see we have TX packet counter, RX packet counter currently showing one.58:321791 and 1046. So let's go ahead and refresh it.58:52And looks like both the TX and RX counter are incrementing here. So let me go ahead and do that again. Again, they are incrementing. So this tells me that this router is receiving packets from East as well as it is sending back its own BFD hello packets as well. So clearly there seems to be a situation where either the other router.58:57Is not receiving the packets that West router is sending it.59:17Or it could be possible something on the path is dropping those packets. Now in the interest of time, I'm not gonna run another fear Trace but I can tell you there is an ACL in place we've used to simulate the situation. So once I remove it, the BFD session will come up. But it is possible that in a live environment that may not be the case. Things maybe a little bit more murky, right?59:21Maybe you are seeing the same situation you have done Fiertrace, you're not seeing any drops, you have confirmed with monitor packet captures that you are seeing unidirectional traffic on the east router. So what you're gonna do then is.59:41With packet traces, you can confirm if there are any drops happening. If not, and if you feel like there's something in the underlay that maybe causing the issue because believe it or not, a lot of BFD flaps tend to be because of instability or issues on the underlay itself. As Tushar pointed out, I want to call out the underlay discovery tool. So if you go to the trouble.59:56Shooting section from the same page. Click on underlay discovery. I believe this is a feature that was part of the whole, it's a it's pretty mouthful so I've made a note of it, it's called underlay measurement and tracing services. So it's something that came from 1710 2010 onwards, and here I can select, let's say I wanna do public internet.01:00:16That I'm going to select east R two router.01:00:36In public internet again. I'll go ahead and start. So it usually takes a couple of seconds to couple of minutes, but it should give us a hop by hop information between the two BFD tunnel endpoints, and at each endpoint or at each hop, it also gives us information on latency and loss. So e.g., if I hover over, let's.01:00:40In 2nd half, it tells me the RTT and loss. Now, maybe you're dealing with some underlay loss here, right? This becomes extremely vital in situations where you're dealing with let's say, not a simple BFT down issue, but a BFT flapping issue, which is a telltail sign that there is some loss or instability in your underlay circuit. You can always come back to underlay.01:01:00Covery and review these parameters to understand where exactly in your you know in your path from BFD .1 to .2, are you having an issue and you could further continue your investigation at that point.01:01:20With that, I'll go ahead and hand it back to Tushar now to continue to walk us through the policy and some issues that we tend to face with policy deployment. Thank you so much. All right, we have very less time, so I'll try and cover as much things I can cover.01:01:35Alright, just to give you overlay overview of what, when it comes to policy what which particular component is responsible for what, right? From our vsmart, it is responsible for all of our overlay routing policy enforcement.01:01:54He is basically responsible for sending all the updates and everything. Our managers are responsible for creating policy definition.01:02:14Then our our is responsible for basically receiving more overlay prefixes and also sending out centralized policy updates to the edge devices.01:02:24Well, yeah. So alright, going to jumping into policy architecture, right? The SD WAN policy architecture consists of two main, two main parts or two components. The centralized policy contains of topology and we VPN membership which.01:02:41Contains a control policy, right?01:03:01As you know, there is, there is UX.two and UX.one. The legacy way of configuring policy is called as UX.one UX.two consists of policy groups which contains all of the, all of the policies, right? From Your data policy, security policy.01:03:06Your security policy, any integration, all of that is part of your policy groups. Whereas on legacy or one, it was different, right? Your localized policy was part of your device template.01:03:27Whereas that is not the case currently. So how do you do you are basically defining your centralized policy and through netconf, it is basically pushed to your vsmart, right? Policy configuration for your centralized policy basically sits on your vsmart and all of the.01:03:46All of the actions or traffic engineering actions that we have pushing that basically propagates to your edge device, which is basically stored in your policy rib and zone based firewall rules and all of that.01:04:05All right, what are some building block blocks of centralized policy? You will see a group of interests which consists of prefixes, five, sites, T locks, VPNs colors, and SDA, SLAs, sorry. These are basically your building blocks. The policy definition is where all of.01:04:22All these blocks come together, where you are setting your control policies which will basically affect your overlay routing, right? How your traffic is flowing. AR basically stares your traffic basically based on your SLAs, whereas if you see a loss, send your traffic to another circuit.01:04:41That is basically our data policy is basically a VPN level policy based routing where you're directly setting rules to forward the traffic based on a particular path on on device level itself.01:05:02Then, policy application, right? Policy applications like if it's inbound or if it's all all all services transport service side or transport side, like that is where basically where you actually are enabling your policy, right? What action are you?01:05:16Taking, on the policy. Is it inbound? Is it outbound? All of this will come together and form your centralized policy.01:05:36Alright, so I'll directly jump into a basic workflow of troubleshooting. There, there will be two parts to it. One part is troubleshooting on your right? You define your centralized policy, if you are defining your policy.01:05:47See it will not be pushed to the.01:06:05V smart until that V smart has a template attached to it. If that template is attached correctly then there are some pre checks already being on the V manage itself. If you have any problems, if you have any conflicts it will point that out if you have invalid.01:06:08Parameters that are put in, it will point that out. So.01:06:29That is the 1st part. And then second part is basically on your vsmart and where you are actually going to the V smart and seeing if the policy that we had defined is basically correctly propagating to the vsmart. If the V smart has.01:06:35Correctly receive the policy and if it's sending like what routes it's receiving and if it's correctly filtering it, if it's sending the correct action to the vantage. We'll not be covering a lot of this in detail today because we are we have a time constraint but let us know and feedback if you.01:06:55Like to have a in detailed session on any of these three topics.01:07:14All right, so troubleshooting 1st we'll dive into troubleshooting policies from the managed perspective. So as I had said 1st, you'll have to have a template assigned to the vsmart to forward any, to.01:07:21Enable any control policy.01:07:41So, once, once you have a, once you have a template attached to it, verify if we have actually enabled that control policy, right? In, in, in legacy or uX.one it was easier.01:07:43To see if it it if it is activated or not, but in UX.two.01:08:03Now you have two different, two different ways. So now in part, now in policy groups, right, you will see two different two different scenarios or two different ways. All of your centralized policy and control policy is bought together by.01:08:08Work together on policy group and there is an independent field for your topology, whatever custom topology that was part of your centralized policy, right? Where you would build hub and spoke or full mesh, whatever.01:08:28Now that is an independent part. You have to define that and basically then deploy it once that is deployed, then only it is basically forwarded or trickled down to the edge device. Yeah, like if you do not attach a template to the V smart, you will see this particular error.01:08:42Let's say you had some duplicate values let's.01:09:03The branches or so, you, it will basically point that out in case you have some kind of localized policy that are being pushed through uX.one then verify that if they are actually called out in additional templates.01:09:07Alright, this is pretty important as as I was mentioning right like as we know, the Cisco SD WAN is moving towards UXR two, there is a big effort going on where customers are mo are moving towards configuration groups.01:09:27It is relatively easy, on that right what I wanted to see is as, as we knew, right? The, the localized policies were basically configured on templates, right? Device templates. That is not the case now it.01:09:45All of that comes under policy groups. The, the downlier architecture rate where your data policy or localized policy which is sitting on the router is seen. It's just that the grouping of all the policies has been bought together for ease of configuration and ease of understanding.01:10:03While deploying this, you have to to to two steps, right? You have to 1st associate the device, Associating is not deploying the policy for associating the device you have to deploy it, which was not the case in legacy.01:10:25Yeah, here is another example when I, when I was when we are trying to attach a device and we had a incorrect value put in on one of the parameters and you can see, that.01:10:44The V manage pre check automatically gives you that information. Next big ne next slide this particular site is policy preview. Keep in mind that when we are defining any rules, right? Always try and verify, what rule was.01:11:03Applied through the policy preview. This is very important that shows you, ok, whatever you are trying to do is actually being forwarded or configured or put in, and that way it just forwarded or.01:11:23Goes down to your vsmart. Next next part is basically your vsmart troubleshooting and device troubleshooting will not go into deep but I'll just cover the basic overview of this.01:11:38Before that, right, most important things, most important things when we are creating centralized policy creating centralized policy or pushing a centralized policy, right? There is there is like inbound action and outbound action, right? So in on.01:11:55Inbound action, what happens is it is applied to all the routes that are advertised, right? All of the routes that are being received from a particular site, that particular inbound action will take effect on all the routes received.01:12:14And it will basically send out the updated updated route filtered or let's say you are receiving XX route from site list, all of the site list, you are putting in a preference on it, it will be advertised.01:12:31To all of your edge devices or reflected to all of your other edge devices. The outbound direction, right, is applied after when the route is being advertised, right? To the edge devices. So example, if some of the received routes are not best path for a particular site.01:12:51It can be restricted withoutbound policy, so keep keep that in mind.01:13:11The workflow for troubleshooting control policy from the perspective is basically 1st check the policy is correctly committed. We can check that using show configuration commit changes. You will see if you do show configuration commits, it will show you the number of the commits. You can.01:13:19And look at the recent commit number. Then look at the OMP pairing, like verify OMP pairing center.01:13:39Which confirms the control connection between active. If peering is not active, then this particular updated policy is not gonna be propagated to our edge devices. Then is identifying the applied policy and direction.01:13:46As I had mentioned, right, like direction is pretty important. It I know it sometimes confuses us.01:14:06Okay, it would be inbound or outbound, but it is very important to verify that once that is pushed, if we are seeing any issues with our policy, then validating the CLI preview, what is basically trickling down if the V manage has correctly sent the policy to our.01:14:12We can do that using this command show run apply policy, if it's a particular section you have a sub you can basically filter down to particular control policy, data policy, all of that. After that, you can.01:14:33You can simulate a policy match action with this particular test policy match, right? It is gonna, it is basically give you give will give you a execute it will give you a virtual virtual trace to see exactly which sequence is.01:14:50Talking or catching a particular route or what, what is it editing? So this is a pretty cool command to have in a.01:15:10With us. And then if everything matches out after this, then we'll have to look at routing, right? On the edge devices, what exactly is happening there? With routing, with overlay routing troubleshooting, it could be.01:15:23We can go with the SD WAN devices does not have set, right? It has, it has RIB routing information based it does not have set like so with with the 1st command you are basically verified.01:15:41Find the rib to see if the actual actual policy has been trickled down to the edge device and if it's making changes logically on the device itself. Then you can check the OMP table, basically it confirms if the control plane received the route from the.01:16:01Smart and if it's here but missing from the RIB. We see lot of scenarios where OMP receives it but it's not installed in, so then you can look at it basically inspecting the next hop parameters like system IP color encapsulation.01:16:21If T lock T lock is invalid, right?01:16:41Are unresolved, the route cannot be trusted. So, if T lock is down or so, that particular route will basically that is why one of the reasons could be that route is being. Then local policy filtering, right? Like with.01:16:45So what you're trying to do is basically verifying the local configuration like route maps, policy list to see if local redistribution filters, right? If they're.01:17:05Well blocking the traffic or blocking the routes that are being advertised down to the edge devices. It could be route maps, it could be ACLs. Then you can also look at V smart routing table.01:17:19But to see if the brains of the network, right? We smart is the brain of our network. If it's successfully run learning a particular route and if it's advertising the T lock advertising the route and the T lock from the advertising the route and the T lock correctly to the.01:17:36Yeah, that, that is what that is how you will determine or try and troubleshoot any issues related to policies. Alright, I'll hand it over to Charita for a quick demo of the tools that can be used to.01:17:58Update some of these things.01:18:17Sounds good. Thank you, Tashar. I know folks, we're almost at the end of the session here, so I promise I'm gonna go ahead and wrap this up. Just need couple of minutes here. We will be going in the future sessions. We will be going more in depth on policies based on the interest we get from you guys and likely we will go more in depth into.01:18:19The NWPI and all other tools in the future sessions. But for now before we go ahead and wrap this up, I just wanted to quickly call out NWPI once again. And in this demonstrations context, I'm only going to be focusing on how NWPI can be used to validate the.01:18:38Policy being deployed, right? In the sense, is my application traffic taking the path it is supposed to take as designed or as intended by the policy? Now, let's quickly hop onto this lab. So while I'm pulling this up, what is NWPI? It is essentially giving us end to end visibility into how the app.01:18:59Application is performing and it can also be used as a diagnostic tool. Essentially, you will go ahead and click on new trace, you will set up the name of the trace, mention the site on which you want to conduct the trace. You can select the service vrfs, you wanna capture.01:19:19You are also welcome to put in source destination address if you must, but for me, my particular favorite is selecting the application of interest here. You have few more options here, which we'll go into much more depth in the future sessions. Once you've created a trace, now in the interest of time, I'm gonna hop onto another lab which already has an existing trace because I just want to.01:19:35Give you a visual of what I'm talking about. The GUI will be slightly different because this is on a much more different build. It's on a higher version. But yeah, so let's go down to flow number six, right?01:19:55Let's say we are dealing with a application, let's say Webex and we're sending the traffic out on, let's say biz internet to the data center and out to the internet and then the rest of the traffic will simply leave through local internet breakout. If your customer or your end user tells you, hey, I'm having performance issues with the application.01:20:09Without NWPI, you would be setting up packet captures on the branch router. You'd be setting it up on the data center router. Maybe even on the multiple different routers in the path. But with NWPI, once you set it up on your vmanage dashboard, the rest of the process is kind of automatically done, right? 1st router will add a metadata to the SD WAN header.01:20:29The corresponding the next router in the hop, if it receives that packet, it's gonna look at the metadata and it's going to be like, oh, I need to send the flow related information to be managed dashboard. All that information that we managed dashboard receives from various hops in the path is collated to give us this, a holistic view of what is happening to the flow.01:20:49So I'm gonna scroll down to this advanced view and if you click on expand all features, you will see that it tells us information on the policy that it is matching to. It gives us information about the sequence, the SLA class that it needs to be at and the SLA class it's actually compliant to it, and it tells.01:21:09This is the reason as to why the traffic is being sent out on MTLS in this case, it's because that's what sequence 71 dictates and.01:21:29If you also click on view here, whatever you're seeing information here, it gives you enough flow chart in a visual way to understand how the routing decisions are being made. But this is a great tool to have in your pocket when you are dealing with any application related issues or any issues that you face with policy post deployment. With that said, I know we already covered.01:21:37Heard a lot of information today.01:21:57I'm gonna go ahead and hand it back to the team if they have any closing remarks to share, but thank you so much guys for all your time and patience. Thank you. Thank you everyone for your patience and patiently listening to the session. I know we are at the end of the session. Please remember to fill in the.01:22:00Responses and the feedback and please let us know what you would like to see us bring bring to this sessions. Thank you so much.01:22:20