Insight in Action: SD-WAN Monitoring & Troubleshooting

Hello everyone, Welcome to today's expert insights series session where we'll be talking about SD WAN monitoring and troubleshooting.00:10So for the next hour or so, what we're gonna do is look at some of the built in monitoring and troubleshooting tools within our SD WAN manager and also how we can use them to better understand what we have going on in our WAN. So just to level set, if you're new to Cisco SD wan.00:19And.00:39If you've used it a little bit or if you consider yourself an expert or well versed in it, this session is for, for all of you. So really the goal that we want here is, like I said, not just to show you where these things or these capabilities, where these tools are in the dashboard, but also help you understand how they how they work.00:40So, we won't necessarily be deep diving into.01:00Any troubleshooting actual issues, but rather again show you how you can use the tools that you have at your disposal to help you properly troubleshoot and monitor your network. So with that, let's go ahead and get started here. So here are just some upcoming sessions that we have for you. We have.01:05One coming up next week, next Thursday for Cloud on ramp seamless SaaS and multi cloud access. And we also have one the following week for streamlining your SD WAN from templates to policy groups. We definitely recommend that you attend those as well.01:25And if you need to register for those, you can go into the same portal that you use to register for today's session. So what we'll do, we'll just do quick introductions. My name is Paul Dunden. I'm a customer success specialist on the networking experiences team focused on.01:41On SD WAN, and joining me is my colleague Sergio. He's a customer success specialist on our team as well focused on SD WAN. So our main roles as.02:00Customer success specialists is really to help our customers understand, how to use the SD WAN platform effectively, not just how to configure it, but really how to optimize it and get the most out of it to, to better streamline your deployments. So throughout this session, we have some panelists, we have Edgar and Tushar as well.02:12On standby here, so they'll jump in, help answer any questions that you may have.02:32So with that, I'll just kind of jump in a little bit about myself. So I've been in enterprise networking for a little over ten years now. I spent eight of those in support services, have my CCNP and SD WAN certifications and I've been at Cisco for a little bit over six years supporting the.02:37This portfolio mainly catalyst center.02:57NSD WAN, so Sergio, i'll pass it over to you. I think you're muted, so if you wanna go ahead and unmute. Thank you very much for noticing Paul, and yes, as I was saying, thanks for passing the ball and yes, I.03:00I've been here in Cisco for a couple of years already. I have been working mainly with enterprise network, but I do have also experience on the professional services area. I have two ccmps and also a little bit of devnet. So that's pretty much it guys.03:20Awesome, thank you Sergio. All right, so we'll go ahead and get started with our agenda topic items for today. So we have several items that we're gonna cover in this session here and mixed in between, we're gonna have some demos, so we're not doing all PowerPoint. We're gonna try to put some of what.03:37That we show you here into actual practical application and and show you where to go to access these particular tools and capabilities. So we'll start off, we'll talk about SD WAN monitoring, give an overview on that. Then we'll get into analytics and insights with a little bit of the device three 60 view. Then we'll get into some of the troubleshooting tools.03:56That we have available that are built in, and then we'll talk about how we can view and utilize the logs, alarms, events, and the built in reporting. And then at the very end, we'll touch upon how we can use some of our support case management tools.04:16So we'll get started here with the SD WAN monitoring overview. And when we look at SD WAN, more specifically when we talk about the monitoring piece, we're really talking about how we have this centralized historical and real time visibility and view into all of our performance and the.04:31The Health of our overlay and all of the underlying transport links, right? So what we're doing is we're leveraging some of the built in capabilities, some of the advanced analytics and integrations, e.g., like thousand eyes, so we can make sure as, as engineers that we have optimal application use and user experience. So they're just some other.04:51Other considerations that we want to keep in mind here when we talk about monitoring our, our WAN, we want to make sure that we identify what devices we want to monitor, because I know a lot of us here work in very large businesses, very large networks where we have hundreds, if not thousands of devices, and sometimes it's just not feasible.05:11Able to keep up with all of them. So sometimes we have to pinpoint what devices do we want to target, do we want to monitor? We also want to know what the frequency is that we want to set for doing any pulling or pushing or collecting of any data from our devices and endpoints as well as choosing what.05:31Protocols we want to use when it comes to monitoring. And then also we want to make sure that we know and understand what our thresholds are, right? And that kind of ties into the next point that we have here when it comes to monitoring KPIs, so our key performance indicators, and.05:48Some of this maybe very familiar to to you all on the call, but when we talk about our KPIs, we're just really looking at our specific measures that we want to use to identify our our network performance. So we want to kind of quantify our performance, if you will. So for KPIs really to work when it comes to monitoring.06:06We have to follow a few simple rules here. So we have to have a baseline established, right? So if we don't know what our normal looks like, we're never gonna know if things are actually going wrong or going well, right? We have to know what our baseline is. Then we have to find a way to measure our impact. So.06:25Is this metric that we have? Is it actually affecting the business or user experience? If it doesn't matter, then really and truly it's just noise to us. And then we want to make sure that we have a clear measure. We have to have a good KPI to, to understand. We don't want anything that's too complex for us to understand or use. And then also we.06:44Want to make sure that we clearly highlight the problem and then we can address it pretty quickly and easily, right? So some of this might be some review. Nonetheless, it's something that we do want to keep in mind when we talk about monitoring and as we go about looking our, looking at some of our capabilities and tools here.07:04One other thing too, when it comes to our monitoring aspect of our WAN, we have to make sure that we identify the devices that we wanna monitor, and this was mentioned just a little while back. That's really gonna help us pinpoint and give us the best insights into the status of our network.07:24Then we also want to define our our telemetry, right? And it's really not just about monitoring our devices. It's about making sure that we pull the right information and view the right information, right?07:42And then also we want to make sure we establish a cadence for our information. Do we want to collect this, do we want to collect this information? We want to collect our statistics daily, weekly, monthly, how often do we wanna collect that? And then also how do we wanna, how do we wanna use that? How do we want to integrate that? We might want to integrate that into an existing animal.07:54This process, right? So really these are just some, some points and some questions that we wanna ask ourselves especially when it comes to monitoring.08:14So when we talk about Catalyst SD WAN, assurance monitoring and analytics, we have kind of this unified monitoring and analytics approach. So we have a lot of different blocks here, if you will, and we're gonna cover some of these more than others in today's session where we're not gonna.08:23Talked so much about the security and multi cloud aspects. We will have more in depth sessions on those and how we can use some of those tools and capabilities, but strictly speaking for today's session, we're gonna cover more of the monitoring troubleshooting, app optimization, little bit of the AI system at the end and then some of the reports and the alerts.08:43So let's get started really into our our 1st real topic here, which is gonna be our analytics and our insights. So, when we look at our WAN, and really just our our network landscape in general, we really know that things have have drastically changed over the years.09:06More recently the last few years and we know things have changed so fast because we're dealing with a lot more users. We're dealing with a lot more devices and there's just been this huge explosion of applications popping up everywhere. So just when we think we only have.09:26You know a certain set of applications we have a lot more than we thought we had. So with that being said, it's really not just the sheer volume of of applications and users and devices that we're dealing with. It's really just the complexity. So what we've done is we've shifted away from this simple static connections to more.09:44More of this mix of internet and multiple transports for us to have our connectivity. So.10:03Really with all of that, keeping track of everything has just become a major challenge. So today really a good experience doesn't just depend on the network itself. It's gonna depend a lot on multiple factors as well. Anything from our app performance to the path that our application takes. So.10:09So that's exactly what we're talking about here when we talk about how things have changed and just the need for us to monitor our stuff and get the best user experience.10:29So analytics and insights, so how do we actually really get the most out of analytics? So if you recall, if you've been using SD WAN for a while, we used to call our analytics platform V analytics, now we call analytics and what we've done is we kind of converged or merged to two. So we have our.10:42SD WAN analytics dashboard merged in with our SD WAN manager. But before we get to that a little bit, we just.11:02Want to discuss a little bit these three core tenets or pillars, if you will. So we have this total visibility piece when we talk about our visibility into our network and app performance. So this is our way to get a clear view of what we have going on in our network and how our applications are performing. So.11:10We could see things like historical trends, do things like automated reporting. So all of that kind of rolls up into that total visibility or visibility pillar. Then we also have this troubleshooting, as we see here, troubleshooting with extensive traffic analysis. So.11:30Whenever we get a problem that comes up, we are able to get this end to end path visualization and traffic flow analysis inside of our analytics platform. So we have all of this built right in for us to view and Sergio is gonna take us through some of this in our demo coming up here shortly, but.11:47But nonetheless that is.12:07Our other core pillar, our, our deep troubleshooting. And then we also have our adaptive and predictive networking. So I, we kind of like to call this our smart prediction pillar. And this is where things really get, get powerful, right? So we're actually using predictive pathing, where we're using machine learning and AI for us.12:09To do things like do bandwidth forecasting anomaly detection, so we can actually get ahead of and catch issues before they even.12:29Impact anybody on our, on our network. So really it's just moving from what just happened to what do we do next? So we're just getting the insights so we are able to stay ahead of the curve. So.12:37Our comprehensive monitoring analytics piece, so this is just a little bit of a review of some of the different items that we covered earlier. So we have anything from our app and network where we're monitoring historical data all the way to using AI and machine learning to kind of crunch that data and make sense of it. We also have some trouble.12:53Troubleshooting tools available, network on path insights, where we're able to kind of correlate all this information, make sense of it so we can take action on it. And then we also have things like our reporting where we could set reoccurring reports to view any information and statistics that we collect, we could get that right to our email or we could do that via webhooks.13:13And then also we have native integrations with thousand Eyes and Splunk as well.13:33So I mentioned earlier about the convergence of our SD WAN manager and analytics platform. So right now we're able to get all of that into our single dashboard here in, in our SD WAN manager. So from 2015 version onward, we now have this kind of converged so we're.13:41Able to view things like our applications dashboard, our sites circuits dashboard, all inside of our SD WAN manager user interface. We also have some other.14:01Dashlets as well, which we'll cover here in the upcoming slides like our clients widget, which is really helpful, which actually leads us to a good transition to cover some of the, the widgets that we have built in. So tunnel health.14:13This is actually really important because this the tunnel health capability that's built in, that falls into our total visibility part. So if you have hundreds of tunnels, maybe you're a large bank or telecom and you just want to see what's performing poorly, it's just a matter of a click and we get a quick snap.14:30Shot of what some of our poorest performing tunnels are based on loss, latency or jitter. So we could come in and sort and filter out that way. And as you can see, and Sergio will show this in the demo, we could zoom in and click on more view details and that'll pull up more information regarding that specific tunnel. And then we can.14:49Sort this by time frame. We could do 1 h 3 h 6 h 1224. So really whatever timeframe we decide to choose, we can go back and view our tunnel health for that period of time. We also have the ability to check if we have any SIG or SS.15:10Tunnels. We can view the tunnel health on those as well. So as you see here, we can also sort and filter by site. So this becomes especially helpful if you got.15:30Hundreds of tunnels, thousands of tunnels. Maybe you just wanna look at a specific site, we could click our site and then view the tunnels for that specific site and then view our tunnel health metrics. So what our tunnel health is, as you see here, we have a QE score.15:41As well as our status, and that's gonna determine whether or not our tunnel is deemed as good, fair or poor health. And then also, like I said, we got our SIG tunnel health view, so this will give us stuff like our tunnel group name, our tunnel name, and also what our endpoints are and what site they're associated with.15:57As well as whether it's IPsec or GRE. So we have all of that in our dashboard here and then we could see that in table format. We also have the ability here to do the heat map view. So what we can do is we can drill down.16:16And look at the tunnel health at any given time. So if we see here, we got a grid view of our our tunnel and each block represents a period in time. And what we could do is we could click on each individual block or square, and then we'll get additional details pop up for that specific point in time.16:36For that tunnel.16:56Then our applications dashboards. So with our applications dashboard here, it makes it really easy for us to see and to track our app performance. So we get clearer insights into our QoE scores, our traffic trends, our app distribution across our network.17:01And now since we have the converged dashboard here, we can go back even as far as one month to get historical view on our app performance. So this is where we have this trend analysis built in here. So it's kind of on the right of our application experience-here.17:21And what we could do is we could click on that and we could zoom in and look at.17:41Things and usage for a specific period of time or going back say one week or a month. And that may help us determine if there's maybe been a spike in the usage of a particular app or maybe a specific time of day that we were unaware of where app usage has gone up. We could go look and use.17:48This trend analysis feature to kind of help us pinpoint and get those questions circulating in our minds. Hey, maybe we need to go take a deeper look at this. So.18:07That is our applications dashboard. And again Sergio, he'll take us through some of this in more detail in just a few moments here. So you're probably asking yourself, ok, I see this QoE score, how is it actually calculated? So.18:17Our QoE score, what it is, it's just a measure of the deviation in our loss latency and jitter from established application thre thresholds that we set for each of of the network KPIs. So what we have is we have an application family and then we have app thresholds set for that and.18:34Then based on those thresholds, we'll compare that to what the actual loss latency and jitter experienced is for that application and then we'll be able to calculate a QoE score, which is what you see here. So it'll be based out of ten. Anything eight nine and ten is gonna be deemed as good. And then anything below that will either be deemed as fair or poor depending on.18:54And what that score is. So that is what our QoE score is.19:14In a nutshell and how we get that score calculated. We also have our circuits dashboard here, so this is just gonna give us pretty much the full scoop on our circuit availability, what our color usage is, the split between our DIA and tunnel traffic. We're also able to track.19:18Bandwidth utilization over any time range so we could keep a good eye on our circuit performance. So like I said, we could go back 1 h, one day, one week, right? And this becomes really useful, in my opinion, especially if you have sites we could.19:39Go in, we could expand our time frame, and we could track and see the trends or patterns of downtime or spikes in bandwidth utilization for a specific site. And then we also have our clients widget built in.19:56So this is pretty much just gonna display our top clients in our overlay, and any usage of data or any changes in the usage from the last time period that was captured. So what we're doing here is we're using our source IP of the client and the overlay, and then we're gonna display what our top three applications are that are being used.20:12By that client, and then all of that.20:32Data usage is gonna be ranked by our bandwidth. And then here I think this is one of the most powerful parts of our analytics, and this is gonna be our predictive networks, and this is, this is one of those areas where we can spend an entire.20:35Our session talking about predictive path recommendations, bandwidth forecasting, and anomaly detection, but we'll just give a little bit of a.20:55Little bit of a sneak peek on this here. So when we talk about our predictive networks here, more specifically our predictive path recommendations, which is what we see in our top right image here. What we're doing is we're just pretty much leveraging our.21:03Our historical traffic data and we're using our statistical modeling so we can forecast any future performance issues, and then what it'll do is it'll recommend us what an optimal path is for us to take so we can have a very seamless and uninterrupted application experience.21:21Experience. So what we see here.21:40I wish I could zoom in a little bit, but e.g., we can see what our recommended actions are, and that's gonna include a suggestion for us for a path to take or an alternate path. It's gonna tell us what the qualities are of the current path and the estimated path and what, how much gain we would get.21:45Get out of that in terms of performance, and then also it's gonna allow us to see how many users are gonna be impacted. So if we see here under Salesforce, we see impacted clients for, so if there are gonna be any changes and we need to switch over to a more optimal path or an alternate path, we would see the.22:05Impacted user count. And then we also have closed loop automation built into that. So if we want to apply a recommendation, we could do so by simply clicking and the policy will be applied.22:25Then we also have our bandwidth forecasting and our bandwidth forecasting, what we're doing is we're collecting or gathering about twelve weeks of our historical data so we could predict out up to three weeks what our bandwidth is gonna be for that particular, for that particular link, right?22:38So that's just gonna really help us with capacity planning. And then we have this anomaly detection, which is relatively new here. So that's gonna.22:57In a way, be like our, our watchdog or our proactive watchdog. So it's gonna detect any spikes that we have in latency jitter and and usage. And what that's gonna do, it's going to flag that and and kind of distinguish, hey, what's, what's a persistent issue and what's more of like a one off.23:06Blip. So it's gonna flag that and it's gonna bring that to our attention so we can look into that deeper.23:26So then we have our device three 60 and just a few more here before I turn it over to Sergio, we have our device three 60 dashboard for our particular WAN edge device, and in there we could look at different things like our interface stats and QoS as we see here, we could check things like our interface drop.23:32Queue stats and we could view all this in chart or table format. As you see here on the right, we can click and highlight.23:53Each individual queue to see the statistics for that queue in chart form. And then we also have our real time, our real time view here. So what our real time view does is it allows us to run a CLI style of show commands, but from from the graphical user interface here.24:01And, and one thing that we really like about this is that we're able to filter out any any of our output, so if we want to get rid of any excess output that we don't need or care for, we could, we could just leave that off and we could just view what we think is relevant for us.24:21And then we have our troubleshooting tools here, and I think this is a good segue into our next section, and we'll cover troubleshooting tools in the next slideshow section. But before we do, I'm gonna pass it over to Sergio so he could cover some of the items that, that we talked about here in the slides. So Sergio I'm gonna stop sharing and pass it over.24:39Are you here?24:59All right, thanks Paul. And let me know when you can see my screen. I think you can see it now. Yep, we see it. Alright, so yes, as Paul was saying, I will be demoing a little bit of the features that he just discussed and.25:01What I have here is the landing page of the catalyst as the manager and well right here we already see some of the metrics that we can get from this, from the manager. By the way I am using.25:21Version 2018. The recurring recommended software version is actually 2015, however, because of the availability of our labs I am using 2018. But then again everything that I'm going to show you here is still available on 2015.25:39So having said that yes, we have the monitoring section here. This is the landing page. I am of the overview monitoring tab. We can see the one edge health, we can see the top CPU load, the CPU load for the devices, we can also filter out for the good devices. We can see we have a couple.25:57The Overloaded CPUs in this network. We can see the tunnel health, you can see that most of my tunnels are healthy. However, some of them are also in fair health and some other on poor health. These are the top poor tunnels that experience that are experiencing too much latency.26:17This two over here and down here we also have divided in different circuits. You can see that.26:37All my sites are available. However, we can also filter out by the poor circuits that are performing not so well in my network and some others that don't even have any metrics. That's because we don't have that telemetry available on those sites. Also we have the site health. You can see.26:44How available my sites are and we again we can filter out by good fair or poor performance.27:04And if you need any more details on this, this is just like the overview, but let's say as Paul mentioned, what if I want to see my tunnel health? Well, I can see the overview here, but what if I need a little more? You can either click on view details or go straight to the tunnel tab at the top.27:12Both things will take you to the same page and as you can see here, you have a list of every single tunnel that you have, and I am filtering this out by the last 24 h. So in the last 24 h, these tunnels have been performing like this. This is the average for that those 24 h and you.27:32You can see that this tunnel here at the very top is not performing very well. It's still up, but it doesn't have a very good health. Why is that? Well.27:511st of all, I have a QoE of two to 21. This is a scale from one to ten, so two weeks pretty low. And the reason for that is because let's take a look at the metrics, we have a latency of 220, which is too much for SD WAN. We also have a packet loss above 3 %.28:02Three.twenty three and the jitter well the jitter is quite decent, but these two values are more than enough to take down the QoE. That's why this tunnel, even if it's up and running because it's a state, it's up, we still consider this tunnel to be on poor health.28:22Poor health is considered anything below five QoE periods more than five, but lower than eight and everything with eight QoE is considered to be all good. Alright, so that's for tunnels, but.28:41Paul also mentioned that we can get all a lot more statistic than this. Let's say that I want to run the tunnel tunnel health. We can see the heat map. Where do I go there? Well, you can go over here.29:00And click on tunnel health. Let's see why this tunnel is not performing so well it will take a little while, but it will generate a graph, it will.29:16Generate this graph as you can see here about in the last 24 h yesterday this tunnel was experiencing five.seventy 1 % packet loss and the other one the blue, the public internet circuit just 01:20 percent. And as you can see, it has been flucked to.29:32Waiting for the past 24 h and that's the reason why this tunnel is not performing so well. It has.29:51Too many fluctuations. What else can I see? I can only not only see packet loss, I can also generate this graph for latency and jitter. Jitter, we just have a spike in here, but it's not too bad, it was just.29:59Oh, one time, but it has been fluctuating and the latency, it's also fluctuating between 204 hundred milliseconds, so that's not healthy for this kind of tunnels, we are looking for less than 150.30:16But as you can see, this one has been experiencing a couple of issues every now and then. So, that's, that's a kind of, of tunnel health that we can get from the manager, but let's go back and also talk about applications. So there's a landing pa.30:32Page and if I want to see application, there is an application tab over here or I can also see the application experience right here. You can see that my top.30:52Performing applications this binary over HTTP, so HTTP is, the biggest application that I have, but if I go to the tab, I can also get some details on the other applications that I have on the network Wait for it.31:02It's.31:21There it is. I have 78 applications in this network and they have been consuming 70 gb in total in the last 24 h. You can filter that over here at the top and well this HTTP traffic is actually looking pretty good. Most of my applications are in.31:23In good shape, so let's take a look at the 1st one. We also have some other minor traffics like secure socket layer, Apple services, Jahoo, and that kind of stuff Dropbox.31:43So let's take a look at this one. And if you click on this application, you can get more details on this certain application. This in good state for three of my sites and only one site has NO data that this site does not have application.31:56Some Visibility available that I'm not collecting information in there. However, in the other three sites.32:16Let's let's take let's see the list view. The application is performing perfectly. We have QoE of ten and it has been like that for the past 24 h and NO change. This one is experiencing a little bit of packet loss, but it's very low. It's zero.one almost NO packet loss, very good latency, very good jitter.32:21So this application is performing pretty well.32:41And then if we go back over here, we also get this nice table just just so you can see which applications are performing the best and which which applications are performing the worst. This is a lab, obviously, because everything is looking so fine. However, if you do run this test.32:45On an actual network, you will see that some applications are not performing so well or if you have like scavenger apps, you.33:05You have considered this in your QoS policy, you will probably see those applications performing not so well, which is not a bad thing. This needs to be in line with your QoS policy, so not every application is going to perform well.33:12Moving on, we also mentioned the QoE score. Paul mentioned that it's constituted of the packet loss latency, and jitter, and all that mixed together will give us a QoE score. That's why some of your applications might look.33:29Near perfect like this one something with ten QoE or if you see something else like in yellow or in red, they will probably have a lower, they will most certainly have a lower QoE. Anything between five and eight, it's considered fair and everything below five, it's in red.33:48Or poor.34:08So circuits, let's talk about that this that section over here. We have this circuits tab that will tell us, well, 1st of all, the total uptime that we have for the circuits on this network takes a little bit to load, but as you can see in this network, most of our circuits have been up at 97.34:10Percent of them have been up for the last 24 h and most of the usage for the circuits, it's on the tunnel side, so it's communication between every single site and just a little piece of it. Eleven.eight percent it's going through the internet through DIA. You can get this statistics of.34:29Over here and you can also see how much bandwidth you have been consuming. These are the top.34:49Consumption for all your different channels, you can see that this site branch nine is the one that consumes the most. However, we also have branch four, branch ten, branch three, and the other the redundant branch three but in in a different transport. You can see how much bandwidth it have been consuming or also you have the same view over here on.34:55List if you got, if you want to see the one that has consumed the most or has transmitted the most, you can just sort these things out.35:15It takes a little while, but it will tell you which circuits are consuming the most bandwidth. Also, right here you can see packet loss, latency, jitter, and all the interesting things that you can get from those QoS metrics. You can also see.35:23TX utilization, RX utilization, which color does this tunnel is this tunnel using, that kind of stuff. So this is good for analyzing your circuits or also let's talk about predictive networks. Where is that? So.35:40To see that, you can, you can go to the analytics section, see the predictive networks, and we have the predictive network bad recommendation. This, this is a feature that SD one will allow you to analyze all your traffic and see some of the.36:00Of the applications that you're running and it will tell you which kind of recommendations do you have. This, e.g., this one, this it's a bad quality improvement. You can see a little more details on it. The current path is good.36:17However, we can make this like a redundant one. So what it means is that you currently have only one part that is working pretty fine. However, it would be nice to have the recommended path just to have some redundancy. You can get some more details over here. The bandwidth forecast, it can be seen here. This is good for planning for the future.36:34Based on your current utilization, this will tell you this red.36:54Section over here is for the future. It will tell you what is the maximum expected bandwidth for all your sites, which which is the minimum forecast that you could could plan for. Also, important section here, the anomaly detection. This will.36:59Based on the behavior of your network, it will make a baseline. However, when there is a change, let's say there is a failover between one of your circuits and now you're consuming instead of MPLS circuits, you are moving to LTE, it will detect that anomaly.37:15And it will tell you that you are consuming too much bandwidth in LTE in a couple of sites. Like in this case, I have this data center over here in San Jose. Right now it doesn't have too much going on, but it has been consuming this much.37:34And that's pretty much it for this section. So, let me just finish by going to the last thing that Paul mentioned, which is the device 260. And to do that, you need to go to the device section.37:50Let's wait for it to load. Yeah, so here it is. We go to the device section pick any device, doesn't matter which one, I'll pick the 1st one I see. And this is the device 360 view. You can see right here the CPU utilization, memory utilization.38:10How many times have the device been rebooted in the last 90 days, temperature sensors, bands, that kind of stuff?38:31Also, if you want to run commands, if you want to see other statistics, you have the interface section which it will tell you, how much bandwidth and consumption due to have consumption you have per interface. You can also see errors packet drops, utilization packets per second, that kind of.38:40Have stuff. And you can filter out the interfaces down here. I am currently just checking for tunnels, but you can also check the LAN interfaces and their statistics.38:59Moving on, the real time section, which is this one over here. This is just for you to run kind of like CLI commands, but instead of doing the CLI and trying to copy and paste these things into a TXT file, you can do something as simple as an IP rack, e.g..39:11You can filter this out if you want. I'm not gonna apply any filters so I want to see everything. If you remember this command needs to be filtered by the BRF, but here we can see everything in a single, in a single go. I can see VPN one VPN eleven VPN the default one. I can see six.39:30This, 65500 208I can see everything in here and filter out, sort things out by next hop, maybe search for a certain destination so you can see how easy it is for us to filter out and you can see the metrics, that kind of stuff. So it's much more powerful.39:49And with that said, I will hand over the session back to Paul so we can continue with the theory. Yeah, thank you Sergio. Alright, so I'll go ahead and grab the ball back here.40:09Okay. All right. Thank you for going through all of that. I know that was a lot especially for the analytics piece and then going through the device three 60. Like, like we said earlier, there's just so much to share for a lot of those capabilities and those tools that we could almost make an entire session out of each and every one of those. So thank you again Sergio.40:27So let's let's go in a little bit on the troubleshooting tools. So we have a lot of them built in for us to use. We'll cover most of these here. We have things from basic ping and trace route to being able to run a speed test, do an underlay discovery.40:47Do you simulate flows, run packet capture, and then use network wide path insights, which is actually a really powerful.41:08Troubleshooting and and analysis tool that we have available to us now. So, let's talk 1st about the speed test. So we know we got ping, we know we got trace route. So those are are pretty straightforward to use inside of the the dashboard. But let's talk about speed test here. So.41:17When it comes to the speed test.41:37We have two types. We got our site to site speed test and we also have our internet speed tests. And when we're running our speed tests, as we know, what we're doing is we're trying to measure what our upload speed is from our source device to our destination device and then measure the download speed in return, right? So if we focus a little bit more on the speed.41:39Test.41:59Before we get to site to site and the internet tests themselves, just a couple of things that we do want to keep in mind when it comes to the speed test. So if we do decide to run a speed test, we don't need anything specific to be configured on the actual wayne edge itself. We just need to make sure that we enable our data stream and the way we do.42:01That is we go to our administration settings and then under our data collection and statistics dropdown, we'll see data stream.42:21As we see here. So we need our data stream to be enabled for us to run a lot of our troubleshooting tools. And what we're doing is when we enable data stream, we're pretty much opening up a netconf connection to our WAN edge, and then our WAN edge is going to initiate.42:29A troubleshooting command and then the edge is then gonna establish an HTTPS connection to our SD WAN manager and then it's gonna close that connection once our troubleshooting tool is done with what it's doing. So as you see here right below, we have our IP address type and we have a couple different options there or three different.42:45The options. We have system transport and manager. Our system that's gonna be our our default setting there, and that's where our server internally configures data stream to use the system IP of our SD WAN manager that our particular device is connected to. And then the transport and management options, I know management is kind of blocked.43:05Here by that little pop up, but our transport management options are where we can enter our transport and management IP of our of our SD WAN manager, so transport meaning VPN zero and our management is our VPN 05:12.43:25So just make sure that we have data stream enabled. And then when we actually get to running the speed test itself, so if we're gonna do a site to site speed test, we're gonna choose our source circuit and then our destination device and our destination circuit as we see here. And then we're just simply gonna click start and then it'll take maybe just a few moments to run and then.43:39We'll get our download and upload speed results returned to us. So that would be our site to site test.43:59And then our internet test in a similar fashion, we run our source circuit, which we select, but for our destination device, what we're doing is we're gonna be selecting the internet option from that dropdown. And then from there, we're going to enter in an iperf three server so we can, we can.44:06Put one in or we can leave it. If we leave it, then we're gonna use one of our built in iperf servers.44:26And then from there, we could hit start test and then we will get our download and upload speed results returned to us. So as you see, we have a list of several different public IP three servers that we have that are that are built in to use if we don't specify one.44:33So then we have our next tool here, which is packet capture. So this is one that we've had for a little while now, and what we can do is we can now have the ability to capture traffic passing through an interface or for the control plane. We could capture.44:54The traffic in both directions bi directional, so.45:13So tune from, and then what we could do is once we're done capturing that, we can download our packet capture, we could download the file, and we could run it offline. So Wireshark, e.g., for further analysis. So we could actually do that straight from the dashboard here and Sergio will show us where exactly we could go to.45:17Do that once we get to our our next demo part. We also have our underlay discovery, so what we can do is get visibility into the exact path that a tunnel is gonna be taking between say a remote, you know, a local and a remote WAN edge.45:37Through our underlay. So really that's gonna be what our physical devices are underneath, and then what we're gonna see is the path, and that's gonna include all of the nodes in between those two devices that that we have. So we'll see things like our round trip time, we'll see our loss for each.45:57For each particular, hop that we see here. And we also have our simulate flows tool as well. So we like to call this more of, of say a diagnostic tool that's designed to help us especially if we're building out, say, app route policy.46:16To help us kind of validate how our traffic's traversing through our fabric. So when we run this simulate flows, tool here.46:37What we're getting is, we're determining what the next hop that packet with the specified headers that we, that we have here is gonna take. So really this is gonna help us not only kind of validate but also troubleshoot and verify what our forwarding path is for our traffic flows.46:46Because we're gonna be simulating our packet forwarding decisions here. So this is just another tool that's really, really helpful especially if you are looking to validate or do some basic troubleshooting.47:06Then we have NWPI or network wide path insight. So network wide path insight. This is gonna give us information about our all of our operations, right? So we can use this for things like getting real time app performance, network wide.47:19Path visibility, you know, policy network design validation, and also troubleshooting nonetheless. So, it's a, it's a really neat capability that's built in. This is something that Sergio and I and Tushar and Edgar, this is something we always try to advocate to all of our customers that we work with.47:39It's very powerful and provides a lot of information as you see here, all the different use cases that we could use this tool for.47:59So with all that said, how does it, how does it really work? What is the theory behind it? So 1st what we do is we kick things off, we start a trace in the manager, then from there, our manager's gonna tell that 1st router in that path to add our NWPI metadata to our SD WAN header of that packet and then the metadata is gonna.48:07Include things like our site VPN and any sort of filtered filters that we apply. And then from there, our packets are gonna travel through the other routers along the path as we see in this depiction on the right. And then each router is gonna use that metadata to send flow details back to our manager. And then from there, our manager is gonna gather and collect all that information.48:27From the routers, it's gonna put it together in one clear easy to understand view for us, and then that's gonna allow us to perform troubleshooting or analysis from there.48:47So here's just a s a quick glimpse at what we can look, look to what we can look forward to in NWPI. So what we see here is we can create a new trace here, we could do a new trace or an auto on task. So if we do an auto on task, what we.48:58What were able to do is automatically trigger a trace based on a specific network event. So this could be helpful if maybe we got an intermittent issue that we're trying to troubleshoot, we could run a new auto on task and have that captured as soon as that particular event is triggered. As you can see, we have a lot of.49:18Advanced filters and monitor settings and parameters that we can fill in based on our particular use or need. And then once we run a trace for whatever the duration is, we could run 10 min 60 min, that's when we start collecting that information and collecting that data. So we'll be able to see things like what our completed flows are.49:37What the app performance is, why we have drops e.g. or NO server response, who's impacted. So really the who, what, where, when, and why.49:57So, I'll pass it right back over to you Sergio, to talk about NWPI and some of the other troubleshooting tools that we discussed. Alright, so let me share my screen once again.50:10I think you can see it and well, we were just here a couple of minutes ago. We were on the real time tab. Now I am just going up one tab, and this is the troubleshooting tab. As Paul mentioned, we have a couple of troubleshooting tools over here. We have the ping, the.50:28Is which is probably not that interesting. However, he mentioned one of the most interesting features, which is the speed test. I saw a question on the chat window regarding the Iperf server. Yes, the iperf server can be reached. However, to do that, you need to click over here on the internet and then provide the IP.50:47Server. If you have a site that has the Iperf server connected to the internet, then it should be reachable even if it's on one of your sites. So.51:08The answer is yes, but it needs to be reachable from the internet. So having that said, let's say that you want to run a test for another site. This is usually, something you might want to do. Let's say that you have this data center, data center one and you want to see your best.51:18With connection, how much bandwidth do you have to data center to, let's say. And we will use.51:38Public internet circuits. So you start the test and after a couple of minutes or probably just seconds, you will start seeing this bar going up and you will get the downlink and uplink that you have. How much bandwidth you can consume on.51:43Those circuits. As you can see down here while that thing is running.52:03I ran one test yesterday for the public internet and well the this destination for the circuit was default and my maximum downlink was 251 and the maximum uplink was 348. You will see a new entry over here at the top.52:08Once this test finishes. It takes a little while, you can see it has already finished the downlink direction, so I have a downlink of 227, almost 28 mbps, and now it's running the uplink test.52:28Moving on, it will take a little while, so I'm just going to skip that and let's go back. Oh, it's finished already, so, 251 for uplink and you will see the new entry over here. So it took just a couple of seconds. So going back to troubleshooting, we also have.52:46The underlay discovery. Why is this important? Well usually you have tunnels for every circuit in your network. However, this tunnel will not allow you to see everything that it's in between. However, if you use this tool, the underlay discovery, you can see every jump.53:04As you do to reach that destination. So in this case I'm going from data center one to branch, branch five, and let's see how long it takes. We have, we can see every jump in the middle between these two because usually you just see a tunnel.53:23A tunnel that is going across your origin and destination and you don't see anything in the middle. But with this tool, you can see every single jump that is in between and how much delay your jitter or which percentage of the packet drops are contributing to this tunnel. You can see all that over here.53:41Here. It's populating and it will take a couple of seconds, but you can see we have like a little delay, let's make this bigger. We're here of point.sixty two milliseconds and hasn't finished analyzing but the the.54:01The corners, you can see that this is.3. 37 milliseconds added and it it still takes a little while, but this way you can analyze which are your packet drops and how many.54:19How many milliseconds of delay you have in in between? Going back here, we also have the packet capture and simulate flows. Packet capture 1st you need to select the VPN that you want to capture in, then select the interface and then click on start.54:36You can apply filters to that. Usually this is the best way to do it. It will take either 5 min or until the file reaches 5 mb or until you click the stop packet. Once you do that, it will start preparing the file, and once the file is ready, you will be able to download this PCAP file and open.54:55This with example.55:15There it is. You can click on download and it will download the file. Moving on, we also have the simulate flows, which is a great tool for testing application aware routing. Once again, you pick the VPN, pick a source IP address and interface, a destination, let's say that you want to.55:17Test internet connection and if you want, you can also if you want test application aware routing, you can select any application from the list or you can also leave it blank.55:37And click on simulate. That way you will see how many interfaces can reach this IP address. In this case I have two internet connections, so I have two possible transports for reaching 111. And since they are showing over here, that means that I can load.55:48Good Balance across these two interfaces.56:08So, last feature, let's talk about NWPI to see the NWPI oh go on. Hold on. Yeah, sorry about that. So to see the NWPI that is over here under tools and network wide.56:12Network insights and if you want to run a test, if you want to create a new one, you can just click over here on new trace. However, for this example, I will be using an old one. You can see every single trace over here, but you can create a new one over here. You can see things like the.56:34Different applications that were running at the time?56:53In this case I have an unknown foreign known applications but most of the flows, not applications flows. We have six flows for an unknown application and and these others for the test app, which is a custom application we have at this lab, and a couple of events. If you scroll a little further down, which is the interesting part.56:56That, you can see every single flow in the network and which are performing good and which are not.57:16E.g., let's take a this one that has a green check means that it is reaching its destination. It's going down the upstream direction is using the local data center two. It's going through the public internet via nat. So it's trying to reach its destination. It has NO NO packet drops.57:24So it's performing very well. And if you click over here you can see.57:44The advanced views over here, you can see that it's receiving this on the SD one section. It's being received on gigabit Ethernet four and well if you keep on analyzing this, let's say.57:49It has the destination port 53, so it's probably DNS, and the egress you can also see where it's going to. So it's a UDP it's a GDP port number 53. So again, this is DNS it's not, it's using.58:06Net so it's going through the internet. These are the IP addresses, this is the translation that it's been done. It's been sent via via the DIA interface, so it's going directly to the internet. However, let's see one of the red ones like this.58:26This one, it's an invalid remote color, which means it's been dropped. That's why we see this in red and it's not it's not reaching its destination. However, we can analyze that over here and we can see that NO route was actually found for this destination. So we have NO any.58:41Three.59:01For this IP address, which means we will drop that packet. And you can use this tool to troubleshoot your network. That's why it's so powerful. A lot of tech engineers usually go to this tool to troubleshoot and to analyze what problem might be. So I present this to you so you can also use.59:01This same tool. Very powerful, very interesting, and well, that's it. I will hand over the session back to you Paul. All right, thank you Sergio. All righty.59:21So that's gonna lead us to our next couple of topics here. So 1st we're gonna talk a little bit about the logs, alarms, events, and reports. So when we talk about alarms, one thing we could do is we could view our alarms, which is pretty much.59:41When anything interesting happens on an individual device inside of our, our network, what we could do is correlate all of that.01:00:01And then it's going to put all that into our, our alarm section, so pretty much it's gonna consolidate all of our major and critical events and put them in as alarms, and then from there we can set up alarm notifications. We could do that via email or we could do that via webhook.01:00:13Just know if we configure email for any alarm notifications, we have to go into our settings and configure our SMTP server details.01:00:32Then we have our events, so events pretty much again anything that happens within a particular device that will come into our SD WAN manager for us to view and to see. And what's really neat here with the events page is we're able to quickly narrow down.01:00:43Drill down into relevant events. So we have multiple filters that we could use, which is our advanced filtering button that we have here along the right hand side. And then from there we can do custom filter conditions, so that's gonna be based on the org condition. And then we could always, we could also always.01:01:03Download all of our events as a CSV, so we could export that for further analysis and further viewing.01:01:23And then again we could, we could go back into a particular point in time. So if we look closely here, this is based off of the last hour, we could go back to 24 h or we could go back to a time period before that. And then we also have our audit logs. So for our audit logs, we can see.01:01:32See a lot of details here, we can see things like our authentication events showing whether authentication attempts succeeded or failed for a particular user. So here we see admin, authentication succeeded, and then we have the source IP and we also have the time stamp as well as any.01:01:51Any fails as well for a specific user trying to access the SD WAN manager. We also are able to get audit logs for any sort of multi cloud activities. So we're not covering multi cloud in this particular session, but if you're configuring a cloud on ramp.01:02:10The multi cloud say for Azure or AWS, we can do a periodic audit, we could get that configured, and then we can come to our audit logs here under our monitor section and see whether or not that periodic audit.01:02:30Went through whether it's started, completed or failed. We're also able to see our, our packet capture, start stop and download sessions as well. So Sergio just ran one if we were to log in and go to our monitor and log, audit log section, we would see where Sergio just.01:02:46Ran a packet capture.01:03:06And then also device config edit created deleted. So if you're doing configuration groups and you create a parcel or edit a parcel or delete a parcel, and someone's wondering who did it, we could come into our audit logs here and see exactly when that took place. So.01:03:08So.01:03:28And we can drill down a little bit further and click audit log details and that'll give us a little bit more details on that particular that particular log entry. And then we have reporting, so we can use reports and run reoccurring ports directly to our mailbox. So that'll give us snapshots of our.01:03:28Environment so things like link utilization, executive summary report, our link availability, our SLA report and and several others as well. Like I said earlier, we'd have to go into our settings and configure our SMTP.01:03:49And then we'll be able to generate reports and have them run at a recurring basis and delivered to our email box. And then APIs, just wanted to take a moment to mention APIs as well.01:04:03We use APIs to collect and fetch info for us. So we've seen customers run reoccurring scripts where they collect information on, apps, devices, interface statuses. So we thought it'd just be a good way for us to come in here and just mention, since we have.01:04:20Have clear documentation and schema included, so if you go to the SD WAN manager IP address forward/API docs, you could go in and try it out in our swagger UI. We can, we do have the ability to use APIs to fetch information for us.01:04:40And then integrations. So for us to really optimize our SD WAN fabric in our network operations, we offer three key integrations that enhance both our visibility and and also our security as well. So we could have.01:04:58We could leverage Splunk, so that's gonna allow us to have that, that deep visualization and analysis for our security and our connection logs while thousand eyes agents provide us some of that real time and to end performance monitoring across our our network paths. And then we have secure access.01:05:17Our integration with secure access really converges our security services. So we're using REST APIs so we can seamlessly distribute our policy information across our our SD WAN devices. So although we're not covering all of this today, it's definitely worth noting as we're seeing these become more and more prevalent in.01:05:36In deployments. So definitely stay tuned. We should have some sessions around these particular integrations coming in the very near future. So our last topic here before I pass it back over to Sergio to take us through more is support case management tools. So.01:05:56Now if you've ever, if you ever had to go into the CLI of your device to generate an admin tech, we now have the ability to come into our SD WAN manager and do that. So this has been around for a while now, but in case you didn't know, we could come into our tools menu and then we.01:06:15We can go into our operational commands and then we can generate an admin tech for our SD WAN manager or any of our controllers or our WAN edge device. And what we do is we could come in and we could select either our logs or core or our tech files, and then we could generate that and then that'll.01:06:35So.01:06:55Be available for us to download, and then from there we could upload that to a TAC case. So speaking of TAC cases, you know, if we ever reach a point where we're using all of our troubleshooting tools and we decide, hey, we need to get further investigation.01:06:55An analysis by Cisco tech. So we know we already generated our admin tech logs.01:07:15And we want to upload it to a TAC case. What we can do is come into our maintenance menu and then click our TAC cases menu to bring us to our TAC support cases dashboard here, and then we could select our particular wan edge device that we want to open a TAC case for, and then we hit open a case and then that'll bring us to.01:07:21Our TAC case or a support case manager where we could then input details for.01:07:42For TAC to troubleshoot, and then from there we could upload our, our logs. Just know that we do require cco ID and also a valid support contract to use this. And then last but not least, we have our AI assistance, so this is still in beta, but it's worth a mention.01:07:48It is.01:08:08It provides us the ability to come in and do specific prompts or queries where we can get information back on our SD WAN environment. So anything from what's new where we can ask and say, hey, what's new? And it'll give us some details and information on what's new with the current SD.01:08:09And release or we could say help me troubleshoot and it'll give us a list of commands that we want to help us with troubleshooting some, some basic things or it'll help us gather.01:08:30Link us to some contextual documentation so we could get further help. So as you see here there's some prerequisites for this. This is again still in beta, so as of twenty.fifteen, we will have the ability to.01:08:44Work with this AI assistant piece. And then also for optimal streaming and UI experience, we do recommend using 2018 two, and then also too, we have to enable analytics for troubleshooting queries. So that's a little bit of the AI assistant.01:09:00So now Sergio I'm gonna pass it over to you so you can show us a little bit more on some of the items that we just covered. All right, both, thanks for sharing that information and also before moving on, I just want to highlight there was a question regarding the server. You can manually input.01:09:17Something over here. If you want to use a different hyper server than the one over here that if you want to.01:09:36If you don't want to use France, you can just write down which hyper server you want to reach. And also regarding the security question, you will see that more about that on this tab over here. This is the best way to check that the firewall filters and kids, you can see that, but you will learn more about that on the security.01:09:42This is the presentation. Moving on at let's go back to our alarms. So.01:10:02This is the alarm section. You can see right see right here on the monitoring section and click on the lock step, you see the alarms, events and outlets. As Paul mentioned, you can filter out things over here. We have the advanced filter or you can just search for something. Let's say you want to search.01:10:09For every data center one, which was the one that I was looking at. We have four alarms related to data center one and that way you can see the severity over here. If you want to see a little more details, you can click come over here and see what are the possible causes. In this case we have none, but there is a.01:10:29This is a policy related alarm.01:10:49And you can get more details over here. Also moving on the same thing, it's true for events. You can again search for a certain event. Let's say that you want to check triple AAA events. So type in triple AAA and you can see that we have the system AAA login fail.01:10:53Over here. This is an event that occur on this side with this system IP and if you want more details, once again you can create click on the three dots and click on device details and you can see that this was the user and someone input the wrong password for this guy.01:11:13And all the logs, same logic, you can still use the search table and search for a particular log that you are looking for in case you are trying to troubleshoot something. Like e.g. this one, disabled session, you can search for almost anything. Every column over here.01:11:31Can be, used in the search table. So if you're looking for a particular, let's say that you want to see only out of locks for this IP address, then it will just filter out and only show you anything that it matches this.01:11:51Regular expression. Moving on, and that was we also need to talk about our reports so how do I generate a report? To do that, we go to the report section.01:12:09We have a couple of templates here and a couple of reports that have already been generated. These are the options, link utilization, executive summary, link availability, this kind of stuff, and all of them can be generated either on PDF or on a CSV file.01:12:26So let's see an example of a report, have a couple completed over here. Let's say, let's see this one. This is a PDF you can see the actions over here. You can either run it now or edit or see what has been done.01:12:41Previously, so let's see if we can downloads. Sorry, my UI is slow today. You can click over here and if you click over here, you can actually download the file.01:12:57That was the result of this of this test or you can just click over here and run it now. If you do that, yeah, you will see this being generated right now. It is generating and it will result in.01:13:16A PDF file that we can later see. So this takes a little while, and as you can see this has a recurrency of every seven days. You can change this to 24 h 30 days or something that you decide, that this is manually configured. We will come back to this later because this takes a while.01:13:36But we also need to discuss the how to get the admin tech generation. To do that, that's under tools, and this is especially useful when opening a tag case, I am under operational commands. Right here you can do much more than just generate another.01:13:57In tech, you can run a couple of commands here. Here is the generate admin tech section, but you can also run operational commands just to reload or factory reset or to reset an interface. You don't need to do that on CLI since we don't allow those CLI commands when you are on SD WAN mode or you can do it from here.01:14:17Also, very important you can generate the admin tech for the manager or for any of the devices that you have available. Let's say also be sure to have a reachable device, let's say if you want to generate one for this guy, you just click over here.01:14:37Wait for it to load and select what you want to include in this admin tech. By default it will it will tell you to include the logs and the tech itself, you can generate and you can add custom commands to it. I didn't mention that but that was the text box over here and it will start generating the.01:14:52That.01:15:12That admin tech. If you go to show admin tech list, you can see which are in progress, and once this one, once it's finished, you can download it from here. Last but not but not least, we have the AI assistant, and this is a prompt. I decide to run this before.01:15:13Presentation today, I ask the AI why does the site data center one in San Francisco has health issues because this one has a low health score and it starts analyzing everything. It takes a couple of minutes, but in the end it will result in this table and it is telling me well that this site.01:15:33Right is up and running. It's a hundred percent available. However, it has been experiencing anomalies.01:15:53And one of the anomaly, the most critical one is that it has on the secondary data center router, it has one of the circuits down and this is forcing the traffic to be rerouted to other circuits, which are these other three, and even if the traffic is up and running is.01:16:00Experiencing sixteen.sixty 7 % of packet loss and the root cause analysis and the recommendation here is to.01:16:21Further analyze why this circuit is there. So you can use this for troubleshooting. You can probably just ask questions as something as simple as a network overview. Tell me how my applications are performing or tell me the general health of my sites. You can see many different ways of using.01:16:29The AI to your advantage. So this will help you understand your network and troubleshoot it much faster.01:16:49And with that, I hand over the session to Paul once again. All right, thank you Sergio. Okay, so with all of that.01:16:57We are winding down here, so we'll do a little bit of a recap since we did cover a whole lot here the last 75 or so minutes. So what we did when we started out, we reviewed the purpose of SD WAN monitoring. We also looked at what some kit considerations and processes are that we need to.01:17:13Follow as well as KPIs, what they are and why we need to keep those in mind. We also explored SD WAN analytics and some of its uses. Sergio took us through a demo of the analytics dashboard.01:17:33And then we also navigated the device three 60. Then we walked through some of the built in troubleshooting tools. So we had, we had our speed test. We also had our simulate flows. So some of those built in troubleshooting tools, and then we showed how NWPI can be used for.01:17:48For on demand and to end tracing of our traffic flows. And then we covered what details alarms, events and audit logs can provide to us, and then we looked at how we can leverage these support case management capabilities that we have inside of our SD WAN manager dashboards, so that's generating admin tech, support cap.01:18:08This manager, through the menu and then also.01:18:28And then also some of the other, the AI system as well. So with that being said, we do have some links and resources to share with you once we post the recording and the slides. So some of these are Cisco document.01:18:32Documentation so NWPI monitoring and maintaining predictive path recommendations, a link to our SD WAN configuration guide as well as a link to our devnet site, our developer.cisco.com.01:18:51Page for SD WAN. So we highly, highly recommend that you continue your journey with us. We also have links for our smart license using policy for Cisco SD WAN. We have a link for our solution overview and our SD WAN documentation as well as some other webinar recordings as well.01:19:06So with that being said, we'd like to thank you. I thank you for joining spending time with us. Sergio Edgar and Tushar all appreciate you joining. And if you have any other questions, please let us know in the Q and A and the chat. We will stick around here for another couple minutes to, to answer those. So thank you everybody and we hope you have a good.01:19:27The Rest of the day.01:19:47