Cisco NSO observability in focus

Hello, good morning, good afternoon, everyone. Today, we have our.00:103rd session of expert NSO expert insight series, and we have Fernando and we have Cassandra that they are going to give give us all of the information about this session today. Fernando is a customer success specialist focusing service provider automation technology and we have.00:16Cassandra. She's from.00:35Delivery from, software consulting engineer and she has a lot of experience implementing different projects in NSO, so today she will share some of her experience with customers driving us through the demo that we have. So saying that, I will pass the ball to Fernando.00:38Over to you Fernando? Thank you David, and thank you all for coming here for this to this session. So for today we were talking about the network services orchestrator and the observability.00:56In focus.01:15As David just mentioned, my name is Ferrando Vira I'm customer success specialist and Cassandra Barrettos, which is a software engineer that is going to join us on the jo on the, on today's journey. Now, 1st of all, before we begin, let me give you some advice for all the people that joined today's session. So you can always ask your questions on you on our Q and A.01:18Panel where our experts will be more than glad to answer any inquiries that you have. The only the only thing that you need to do is just raise your hand or you just can go directly into the Q and A and write down your questions. Everything that you are telling us, we we are going to be pending on how is it.01:37Which is the best answer for you. Now, let's roll in. Now the goal for today is to have a simple and starting with an overview of the automation world and what it is necessary.01:571st of all, what we need to know is the introdu it's introduction to NSO observability and metrics. Now, after we establish this solid base on how the NSO can reach with the observability and metrics, then we will go to the open telemetry and exporter.02:09This is a package that it is included into the NSO which is going to give us the advantages on use open telemetry tools such as Grafana and many others. I don't want to spoil to to have more spoilers. But this is how we are going to wrap it up. And at the end, we will have an integration with NSO.02:27Observability with the external tools. We'll see how can we interact from the point of view of the NSO on all the observability tools that we currently have. Now, 1st of all, let's start with the introduction to NSO and the observability.02:46And metrics. 1st of all, we need to do something that is really useful. What is observability and what is monitoring? Maybe we are doing already doing monitoring or maybe we are doing observability, but we do not have that idea and solid based. 1st of all.03:04Monitoring its answering to this question. Is the system working and the observability is what is my system doing? Monitoring can tell me tell monitoring is going to tell me that something went wrong.03:22An observability is going to tell me why something went wrong. While monitoring it's more reactive in nature, which is typically focused on the shorter term or the moment that it's having happening now. Observability will run in a proactive way.03:38Which is in a long term. Also another difference is that in monitoring we have enabled the quick response when an incident occurs. While in observability we are reducing the duration and impact of incidents over time.03:58If I am using just monitoring, I will known up front like 5 min CPU average should not be on certain threshold. But if I am using observability, I am collecting more data and I'm gonna be able to query it after the fact. So I like to think of it like.04:16We want to be able to form a hypothesis about the system.04:36And then query the observability data to prove or disprove our hypothesis. If you can do that, you're good to go to to go to the observability data. Now, the main top, the main goal of this on having this differentiation is that the observability is monitoring its part.04:40Of observability. It does not mean that one it's out of the other. No, it means that they complement. But if we are on a really short scope, then we need to open ourselves to the observability. Now, once that we know what is the observability versus monitoring.05:00What is NSO observability? So, the NSO observability exporter package allows Cisco NSO to export observability related data using software industry standard formats.05:19And also the standard protocols that they are on out there. Now such as open telemetry protocol, OTLPP, we need to keep that in mind. Now it supports the export of progress traces using OTLPP.05:36As well as the export of transaction metrics based on the progress trace data into an influx database. Now, this may sound like a big noise to you, but just as the diagram I am showing you here, I have the NSO. I do have my observability exported.05:53And I need to focus on the observability related data. Now you can ask me here, Fernando, what are these type of observability related data?06:13These are the types that I am just talking about. We have logs, we have traces, and we have metrics. Now each of the data types serves a different purpose. Metrics allow you to get a high level view of whether the system behaves in an.06:24Expected manner.06:44E.g. NO or few failed requests. That's something that we can get from the metrics. Metrics also help identify the load on the system, the typical KPIs, CPU usage, number of concurrent requests, et cetera etc. But they do not inform you what is happening with a.06:46Particular request or transaction, e.g., the one that it's failing, a metric is only going to give you a number. It's only going to give you.07:05Like the name, just metric. Now tracing on the other hand shows the path and the time that the request took in different parts of the overall system. Perhaps the request failed because one of the subsystems took too long to provide the necessary data.07:14That's the kind of information that the trace is going to give you. However, to understand what took a specific subsystem a long time to respond, you need to consult the relevant logs.07:33As you can see, these type of observability data types separately, they are going to give you the information but you need the conjunction of all of them to now start talking about the observability.07:48Now that we already know what, how observability looks like in the NSO, we need to talk about the open telemetry and exporter, which is something that we already previously saw in this diagram. If I am going to do some zoom in into the previous diagram that I show you, the.08:05This is what you are going to find.08:24Here we have that also on the NSO we have the observability exporter, and then we are going to have two fluxes where we can just go depending on on what am I going to do? Does it depend that I am going to use one on an or another? No. In fact, we will see.08:28That most of them, they need to be living together in order to give you the real observability experience in it. So as you can see, we have a split of the of the OTLP technologies here.08:47On the upper side we have the OTLPP collector, and then we have the influx dB on the bottom. We can just see here that we have Jakers, Splunk, Grafana. These are some of the OTLPP collectors that we currently have into the NSO.09:04Export observability exporter. Why are we using this? Because as we saw previously, it is an open telemetry protocol. So we can just work and interact not just with a proprietary vendor, not only with a type of tool. No, it is open to multiple selection of tools.09:24Now, this is a question that you can tell me here. Why open telemetry Fernando? Why why are we going to use it? Well, 1st of all, because it's an industry coveraging on OTLP, ok? As you can see we have here open tracing for CNCF and open census out of Google.09:43It's not only part of the event of of our of Cisco that using the open telemetry. It's already open to s multiple vendors and multiple tools that we currently have. And also open telemetry is based on three signal types. Those three signal types we both.10:03Very scene which is traces.10:23Metrics and logs. This is why we are using now the open telemetry. Now, just to start giving you, how am I going to use it? So, 1st of all, to configuring exporting tracing data.10:26We will need to use the OTLP container. Now this is a presence container that controls whether export is enabled or not. In the container, you can define the target host and the port for sending the data.10:44As well as the transport is used. Now the influx dB export is configured and enabled using the influx dB presence container where you set the host to export the metrics too. Here we can see just how am I going to attach this type or how am i.11:00Going to enable these add on if we are just using in the style or in a Java style. And we can see on the left and on the side we have most of the.11:21Most of the metrics here we have enabled influx dB host local host, what we were saying, we need to paste the username, the password, the database where I am going to collect my data my my data and also.11:33The OTLPP host and the transport that I am going to use. Though all of this information is we need to enable in order to start integrating my NSO observability with the external tools. So far are we having any.11:49Comment, any questions? If you so, let's remember that you can just start asking any question that you like on our Q and A section. Now, this is where the really important things for the fun or the fun is beginning.12:091st of all, I will need to start talking about NSO and checker. This is an example that we were going to that I will guide you little by little on how am I going to use these multiple OTLP?12:26Tools. In this case, I am using a tracing example for checker. 1st of all, what am I, what's the 1st thing that I need to do? I need to be aware that my observability package exporter is already running.12:43If I am going to start using the observability exporter, I need to be aware that it's already on my NSO deployment. 1st of all, that's the 1st thing that I need to do. Then I will be I will go and download a.13:02I need to download and run and reason all in one binary from the from the website. And once I am downloaded, then what I am going to do is just to go to my console and start Jagger all in one.13:18Collector.OTLP enabled. This is what I am going, this is 2nd step if I am using Jaker. Now the the the 3rd part that I am going to you to do is to Keeper running and from another terminal entered to the NSO CLI.13:37Try to enable the OTLP data export because let's remember.13:57I know that I am already, I already have the package, but I also need to start running my OTLP export package. If we see on the, on the commands that we have.14:02We have config, progress export OTLP and then commit, which is going to running the, the data export. And then on this part, then we, the should now be receiving the transaction traces.14:19How am I going to do this? I still I'm still on my NSO I'm still on my NSO and then what I am going to type is session idle timeout and then the time response that I want. And after that I just type commit.14:38How is it going to look like in the, in, in the interface? Well, the simple thing that I can just do is if I go to the web UI and then I start to exploring the data.14:55Now in the search pane, we need to select the NSO service and click on find traces. This is how is it going to look like if I am just using NSO and Jaker. As you can see, this is the 1st, this is one of the of the examples that I am just doing.15:12Specifically focused on tracing.15:32For the NSO. Now, if we are going to take a look at the metrics, then we can use influxdb, which is something that we currently currently saw previously. Now what's the 1st thing that we need to do? I need to be aware that my package of.15:35Observability exporter is already there.15:55Then the next thing that I do is to set up an influx dB instance, ok? In this case I do have the command here for a Mac OS system, blue install influx dB, influxdb CLI.15:59After I just I just created this, I will just create an organization named my org and a bucket named NSO. Now after I just did this, I will need to verify that it's, that my ID and my name have just been created. If I just created my org.16:16And I just created inside my org a bucket name. So this is the output that I am going to be looking for, ok?16:36In this case, this is the one that we are using, but in your case it's going to be different, different. So NO worries, there is NO nothing to be worried about. After I just did this, I will need to create a username and password for my API access.16:45This is, this is why we were just installing the influx dB CLI because the interpreter is going to take all the instructions from influx dB and then start using their commands. We can see here that I have authorization create, organization, my organization, username and so the.17:02The password that I that I want and I'm going to write the bucket on the ID that I just created. Now it's time to connect the NSO CLI and configure the influxdb exporter to use this instance.17:22So I need to go into the configuration mode and then I will add progress export influxdb, the host local host, the username, and the password, and then I commit. All of these it's going to match into the previous step that we were just.17:36Adding the the for the influx dB organization that we just created. After I did this, I will just need to the user interface of influx dB and this is how my data will be visualized.17:58So, we can see here on the left side that we can just load data. We have the data explorer, which is the screen that we are just watching, and at the bottom we can see that we have the NSO, we have the filters that we can just start using them. And this is the information that we have.18:14So you can tell me here, Fernando, quick one. So am I do I need to turn off something on the NSO or if I want to take a look at my data, I need to go to the NSO? No. Once that you have the exporter working on your influx dB, then you can start filtering.18:34Or you can start recovering the data that you want. You can save it on dashboards and also you can just customize them.18:54Now, last but not least, let's talk about Grafana. So, this is an integration that has been tested, tested and it's also a tool that is open and it's widely used by many vendors just to take advantages of the observability.19:01The Capabilities that has. Now, the 3rd part that the on this specific site, what is the 1st thing that I need to know that my observability exporter package is up and running. Then I will need to.19:20Set up an influx dB instance. Why am I going to use the influxdb? Because that's where I am going to pull my information into the grafana. 1st of all, I need to install influxdb and influx dB CLI, and then I.19:41You'll just go and do a blue install grafana. Okay, now that I just installed Grafana, the next thing that I need to do is just to start and running Grafana. Got it? Now, once we've done that, I am I'm already running influx dB and I'm already up and running Grafana.20:00The the the 4th steps that we have here is to configure the Grafana organization name. So I just go and I just send this group with all the information required just to start the Grafana organization name. Then I will need to add influx dB as the data source in.20:20Grafana. Why doing this? Because if I am already having my configuration of influx dB, I am not going to target Grafana to the NSO. I am going to target into the influx dB instance that we have.20:40Then what I am going to do is just to do and install the JQ command in line tool, which is the step that we are just performing right now. After that I need to locate an NS an NSO dashboard JSON file.20:58So this dashboard is the one that I am going to take a look into Grafana. With this curl, of course that you are going to change it with your own setup that you, that you have. This is where we are going to extract the, this is where we're going to extract the.21:13The local dashboard that we are currently building in jason. As you can see here, we are the inputs that I am expecting is the DS influxdbeam.21:33Also my URL, the consent and the value. Now, last but not least, some of the optional things that we need to do, we can just set the NSO dashboard as a default dashboard in Grafana. It's just for you to make the things easier.21:46Now, all these final steps, everything is connected, everything is just aligned. And now what I doing is the previous step that I was just that I just did into influx dB. What am I doing this? Because I am already having the sent the information sent.22:05Centralized into influxdb and then I am just and then I am just sending this into Grafana.22:24All these steps we already noted because it's export influxdb, the host, the port, the username and the password. That's it. After commit, what I am expecting now is my dashboards here in Grafana. This is a tool that we that if you are not familiarized it.22:30It's really easy. It's.22:50It is a tool that most many of the members of vendors are using because of the capabilities to implementation into observability tasks. Here just are some couple of snapshots that we have from the graphics or that we can just start retrieving.22:53This means that every time that you are having any new graph or if you are having anything into the into your NSO, this is how you are going to show it. And as, and as I told you before, in influxdb, you can create new dashboards, you can save them, you can start creating.23:13New ways on how do you recover the data on Grafana you are performing the same. Now, everything looks good, looks really nice, but the main goal here is that we are going to have a demo where.23:36My colleague Cassandra is going to show you more in deep the capabilities of the NSO and the observability tools. So now I'm going with you, Cassandra, if the floor is yours. Thank you very much Fernando. I will start sharing my screen. Please let me know.23:52Whenever you can see it, give me 1 s.24:12I'm I'm having some issues for sharing my screen. No worries, NO worries in the meantime, we can check the questions that we have here. Oh, I see that we're asking that this observability exporter a new feature. It was already answered by Robert. So yeah, the observe.24:31Ability exporter package was releasing NSO six.one. So after six.X version is where you can start finding this observability exporter package.24:51To include in your NSO, in your NSO instance, right? 1 s, let me check if Cassandra are you ready to share the screen?25:04Okay, here we have. Perfect. Can you hear us Cassandra? Hello, hello? Sorry I was having some technical difficulties.25:27No worries. Just make you a presenter, so you should have the rights to share. Okay, please let me know when you can see my screen just in case. Yeah, now it's clear. Okay. Awesome. Thank you very much for the introduction. So this is our demo that we are going to have today.25:45And basically we'll just discuss a few of the topics that Fernando just mentioned, but most importantly, we are going to show show them in a real life scenario. So this is a small repository that we build just for the sake of showcasing this demo. However, as Fernando has.26:05Detailed, this observability exporter has very powerful capabilities, so of course this means that we have some customers that have been using this observability exporter already. So it's a quite powerful tool and we would like to show it to you.26:25In this live demo. So the 1st thing that we are going to check here in this repository is the main configuration that we have. But before that, I would like to show my NSO CLI, right? And why is that? Because as Fernando mentioned before.26:45It's very important that we have the observability exporter package imported into NSO, right? So if we don't have it, then it won't be possible to analyze all of these things that we'll see in a second. So before.27:06Doing any of that 1st I'll verify if the package is in there, right? So for that I will just perform a sh packages command just to see if the package is right in there. So I'm just going to do a.27:22Show packages package, right? And I'm going to perform an upper status. So the reason why I'm doing this upper status is just to make sure that the package is up and running, so as you can see here, we have multiple nets.27:37A custom package, a custom service package and our observability exporter package. So all of these packages have, up status, so this means that they are working perfectly. So I think that with this, we can now check the configuration that our repository has.27:58So 1st of all, I want to start by showing you the observability exposure config file that we have for this particular demo. So as you can see here, we have this XML file that includes all the necessary configuration to connect these multiple components that Fernando mentioned.28:19E.g., influx dB, the OTL collector collector, so these components are going to be right here. And as you can see, I'm going to select it with, with my prompt, you'll see that each component is running in its own port. So that, that is going to enable.28:39Us to interact with these multiple components and of course this is how they will also talk to each other as well. So in the case of influ influxdb is running in port 8086 and in the case of the auto collector it will be running in port four three.28:59818. So.29:19Here it's very important to notice that these configurations are enabled and the OTLPP will send traces to the OTL collector over HTTP. Aside from that, also influxdb will send the this NSO.29:21No instance will send derived metrics directly to influx dB. So that's also a very important part that we should mention and.29:41When the OTLPP metrics are enabled, the OE will push these counter metrics over the OTLPP to the collector which will be exposed to Prometheus, right? So that's something that we should also notice here. And aside from that, I would also like to show you.29:52The OTL configuration as well. So that's also part of our repository, so here in the config section I will open this file called otelco.30:12Dot yaml file, and here you'll see that the configuration for my receivers and exporters, right? So we have the OTLP running in port, 4317. We also have the jigger.30:25Exporter, right? So the let me just show it right here. This is the Diagar exporter configuration and we also have as well the Prometheus endpoint. So it's also important to point this out. So now we can also take a look into the.30:43The Prometheus configuration file. So please just note that this is a a demo repository.31:02And here's our Prometheus configuration file as well. So as you can see here, we are seeing that the OTL collector is currently running in port 9464. So what is this file doing? Well, it's a screw.31:10Typing.31:30The otl's collector's Prometheus Prometheus endpoint, right? So that's something that we should also take into consideration. So as we mentioned before, all of these components will be talking to each other at the same time. So I think that we can start with the demo and 1st of all.31:30Well, I would like to show you, a very quick example.31:50Life so then we can actually see how this works. So what I'm going to do is to open my NSO CLI and I'm going to go to the configuration mode and now we'll configure a quick service instance into NSO. So for that.31:57I will just perform this command services loopback demo. So this is just a very quick and simple, service package that will allow me to configure loopback interfaces in a device. So I'm just going to.32:16Put this comment out and then I'm going to perform a comment. So as you can see here NSO is telling me that the comment has been completed, right? So now I will visit Diagar UI and why is that? Because as Fernando detailed before, Diagar will actually allows us to find.32:33Traces. So we want to inspect the comment that we just did a few seconds ago. So whenever we commit something into NSO, Jagger is going to show us the trace of that commit in this simple UI.32:53So if I, please take a look into this panel right at the left part of the screen and here I'm detailing that I want to check the traces for my NSO service, and I would like to find the traces.33:11Off the last 5 min, so if we were doing this with a real NSO instance, we would probably have many traces, right? So Jager has this nice capability of actually allowing you to filter out based on on time and duration, of course, so.33:26NSO transactions can be longer, right? So we could also filter out by this length or max duration time. But for now, let's just use the last 5 min filter. So look back last 5 min and I'm going to click on find traces. You, so as you can see here, there is a transaction.33:46Question that happened at four.34:0603:00 P.M.. Yes, so that was like 2 min ago, and if I click on it, you'll see that we'll have all the information related to this trace in particular and what I want you to notice here is that this trace.34:09The transaction has multiple phases. So these spans or faces are arranged as a timeline waterfall, and basically all of these spans, as I said before, they represent a phase of a transaction. So we have multiple different transactions and as you.34:28You can see here, e.g., I have this right start transaction and all the sub faces that are part of this right start. So in this case we performed a commit, right? So let me just click on the, on the commit trace.34:48And here you'll see that I have these reading config and transform input phases as well, right? And it's also important to notice out, and let me go back to NSO, is that we performed this service.35:04Instance to this iOS zero device, which is a regular Cisco iOS device. So we configure the service instance on this device with this specific IP address, and if we go back to Yager, we'll see that all this information resides in the.35:22The transaction as well. So if I click on this reading config, you'll see that it's detailing the specifics of the device to which I performed this comment in particular. So e.g., let's see, let's look at this package field. You'll see that it's pointing out to the.35:41Net that we just saw before, Cisco iOS, and of course.36:01It's also detailing that the subsystem in which this comment was performed was the device manager. So as you can see here, the observability exporter is telling us exactly what is happening with this transaction. So it's telling me, hey.36:05I ran this commit into this device and this device has this specific net or package, right? It's also detailing the name of the device as well and the configuration that we pushed into the device.36:24So as you can see here, we have also the duration of the spans, so as you can see here, it says that this transaction in particular lasted more than 1 s. So that is, that is quite nice. But if we had a much bigger transaction, e.g. 20 min then of course we could see the entire.36:41Spend time writing here in this pane right here that I'm selecting. So it is it also allows me to filter out by the time. So as you can see here, if I selected this other time, e.g., let me just drag my mouse over here. You'll see that it will only show me what happened in this specific.37:01Timeframe. And also something important to to point out here is that.37:21We also have a parent and child hierarchy, so that means that there's a nesting, right? Because there are multiple phases in my trace. So we have multiple phases like waiting to apply or validate. So if you take a look here, e.g., for the validate phase, you'll see that many things occurred with.37:27Thing this phase as well, right? So that's why we are seeing multiple.37:47Things going on in here. So, of course, if we wanted to check a very specific facing here, then we would, we would have to take a look for it in here. So as you can see here, there are multiple children for this validate parent, and this is something that it's quite nice because.37:52We can actually check what's going on with that transaction if there's something hanging out in a specific part of the transaction, we can definitely see it here. As I did tell before, we have several tags.38:12So we actually were able to see the device information and the subsystem in which this transaction happened. So of course these we can see this by having these tags, right?38:27Alright, so now I think that we can take a look into the Prometheus UI because as we mentioned before, it's also quite important and at the end, Prometheus will expose what we just saw in NSO to Grafana as well. So if we take a look into Prometheus, we'll see.38:42That we have.39:02This UI, but in particular, I want to see the targets. So here, we have the targets that Prometheus has, and you'll see that we have a very important target called NSO metrics, right? So here is.39:04Where all the traces are going to be exposed, right? And here.39:24These metrics are exported by OE via the OTLPP metrics as well. And I want to use a specific filter. So what I'm going to do is that I'm going to click on this graph tab right in here, and I'm going to select this command.39:31That I'm gonna.39:51Pasting here. Well, it's more like an expression. So I'm just going to, add this expression called NSO metrics, and that will show me all the available transaction counts that Prometheus did for the specific NSO transaction that we completed a few minutes ago for in our.39:52For coming.40:12Right, so I'll here click on execute and you'll see that these are all the available transactions or metric names that are appearing here in Prometheus. So here, as you can see, there are multiple things. We have CDB subscribers, we have transactions.40:13Counts. And of course, as I mentioned before, all of these come from the OTLP metrics pipeline.40:33Here we also see the exported job name, which is NSO, the instance, which is the OTL collector in this case and the job name which is NSO metrics. So basically it's the, the same name that we have as our target in Prometheus. So now I think that we can.40:41Also explore the metrics in influx DB. And here we just need to use our password and login user to access.41:00So in this case, my user is very simple, right? And I'm going to click on sign in, and here we have this tab called the Data Explorer. So here.41:13We can verify the span and duration of our metrics coming from NSO. So here I'm just going to select on monitoring and spam and also duration.41:28And here, in the filter name, you'll see that I'm able to select a specific part of my NSO transaction. So e.g., if you take a look here, I can select comment. If we perform, e.g., a connect command into the device, then I could just click here in connect.41:46But for now, we want to see the information that was traced during our commit.42:06So here I will just click on commit applying to transaction and I will click on submit. So here it's not really showing anything. So let me just modify this. Alright.42:131 s and not sure why it's not appearing here. Let me just unclick this and for some reason I cannot see the information.42:33Of my transactions for the last 15 min. So give me 1 s, spend transaction, duration, and.42:49And give me 1 s because maybe it's because we created a service instance, so.43:09I'll not sure why it's not appearing. So let me try with this other filter that I have.43:18I'm not entirely sure why it's not really showing the data, but here we should.43:48See the information of our comments so maybe I will just try to look at another one with a different time spent of different, transactions that we could already have here.43:54In our NSO instance. So let me select here case ban.44:11Duration, and here you'll see that I have transactions here in the Data Explorer tab. So my apologies, I'm not sure why the latest transaction in our NSO instance did not appear, but here we have.44:27Some information about other traces in my NSO instance. So if you take a look here in the Data explorer, you'll see that there were multiple transactions since yesterday March 31. So this is very handy here in influxdb, so it's still.44:44Me what's going on within a specific time span and it's also telling me what a specific phase of a transaction happened right in here. So if I click on into one of these lines, you'll see that we have multiple phases. So e.g., right here.45:04If you take a look into the root spam, it's telling me that we were applying a transaction. So there was a transaction applied into NSO on that specific time frame. So if I wanted to increase the timing into.45:25In which I wanted to look at then I could click on the past seven days, and here would, it would show me the transactions that happened during these seven days as well, right? So of course since it's a quick demo, well, we don't have much information, but basically here in influxdb, we can.45:43Also take a look into each phase of every single transaction that happened in my NSO. We can also take a look into how many concurrent operations we have in a given phase, and we can also see the specific transaction time that it took.46:03We can also take a look into the lock contention or how many transactions we're waiting for a lock in my NSO devices as well. So this is the informative thing.46:24And I think that we can now get through the most important part or at least to me the most interesting, which is the Grafana dashboards here. So, this is my Grafana dashboard and let me just.46:38Navigate you through the principal page of Grafana. So this is how the main page of Grafana looks like. And if I wanted to see my NSO specific dashboards, then I would just click here on menu and I would go into dashboards. So let me click on it and you'll see that we have.46:55Have this one called NSO metrics. So this dashboard in particular is going to have all the specific NSO dashboards that we have for our NSO instance. So let me click.47:15On here, and you'll see that I have multiple panes over here. So we have this one called overview and here it's showing me all the different transactions that we have in our main main NSO instance. So something important to point out as well is.47:30Is that Grafana also allows us to select a time frame. So e.g., if I wanted to see the last transactions in from the last 30 min, then I would click on here.47:50Latest 30 min and you'll see that here these specific pain called transactions is showing us that there was a transaction happening in NSO at 43pM. So that corresponds to the comment that we performed earlier, right? But let me go.48:02Back to the latest two days or seven days. And here you'll see that the here it's showing us the different transactions that we've had over the last seven days, and it's telling me that we have three of them.48:22Here in the Spain called transactions, but now let's take a look into this other pane called longest transactions. So basically what it's going to happen here is that this is going to show us the longest transactions that our NSO instance had. So of course.48:39In this case, it's showing us very short transactions because this is just a demo instance, but in a real life scenario, of course, we could have much longer transactions. So in production systems, there are transactions that even last more than 2 h.48:58So we could definitely see that here. E.g., as an example I can say that reconciliations or service reconciliations and redeployments can take a long time. So in production systems, we've actually saw that type of transaction writing here. So they can last more.49:18More than 30 min and of course that can also be concerning especially for the performance of our instance. So as you can see here, this can really help us identify when a transaction is taking longer than usual. So an an extremely nice benefit.49:38Of having this Rafana dashboard is that we can also take a look into what's going on with not only with the transactions but with other aspects of our NSO instance and all of course our services as well, right? So here, if we take a look.49:58Here, if we go down, you'll see that we also have these two dashboards called transaction log and queue length and transaction duration. So here this is just a different way of looking into our transactions, right? So if you take a look here.50:17This will show you in a graphic manner the time and this pain with different colors of the transaction or the longest transactions but here in transaction duration at the right side of the screen, this is displaying us the same information but in a different type of graphic.50:34Right? So again, if I wanted to see the transactions that happened in the last 5 min, then I could definitely just change my filter here.50:54And that will show us the information, right? So let me go back to last seven days. And here we also have more information. We also have a dispaned called transactions details, and basically it describes.51:03The specific transaction phases that we've had in a commit, e.g., and of course how long each of these phases took as well and how long the transaction log took in my NSO. So of course this information is quite specific, but it can definitely.51:19Definitely help us to understand what could be happening with a very specific transaction operation that we could be doing with any of our NSO services as well. So here it's also telling me, hey, in the service pane, I have this service.51:39Modification or this service modification happened into my NSO instance and here you'll see that it's telling me the longest service create span. So that means how long creating a service instance took into a device. So here if we.51:59Take a look into this specific.52:19Line, you'll see that here we are detailing or at least the Grafana is detailing that we performed this service instance called test one into one of our devices and it's telling me or it's telling us that it took more than.52:23Six milliseconds, right? So as you can see here, Grafana is very specific with the duration time of a service and a trace, of course. Here it we can also see the trace ID of the service instance. So this is also very handy as well because Grafana is allowing on.52:42Just to see what's happening with our services as well. And in some other cases, we can also take a look into which service in particular is using more disk space and memory as well.53:02So here in the devices pane, Grafana is also detailing us some information, some very specific information about my devices. So here it's describing the device logs that we've had.53:20And why is that? Because some of you might already know that whenever we perform or we push configuration into a device, there is a lock in the device. So here it's detailing me the specific device locks that we have, and it's also detailing us the longest device connection that we.53:35We've had, so as we mentioned before, sometimes devices can take longer to respond to a comment or a sync from as well. So it's here, it's also detailing me how much time it took to perform this connection.53:55Into the device as well. And it's also showing us the concurrent device operations as well. And let me check here. So it's telling me, well, this concurrent device operations, it's telling me how much or how long it took the device log.54:12Into to be held. And of course here in the left side, you'll see that, again, as we mentioned before, Grafana also allows us to detect the the amount of time that a sync from into a device took. So here you'll see that, e.g., on March.54:3230 at 1150 there was a device sync from to a device that took, this amount of time or this amount of milliseconds, right? So again, this is very handy especially because now we know that we do not.54:49Only can see or check what's happening with transactions or with my NSO services, but we can also take a look at what could be going on with a specific device. So this is very nice especially for troubleshooting. We've encountered this in, in production.55:09Systems, so of course this is very handy. So just so you know, there is one specific customer that with the help of Grafana, we were able to see that NSO, the NSO instance of the production system.55:29Reached ah reached to 1 million transactions in NSO, so that's, that's a lot. Of course, here, as I mentioned before, we are just seeing a very small amount of data, but in a real life scenario or production system, of course, we would definitely see way more information, but.55:46At the same time, this would also allows us to troubleshoot any issue that we might face not only in our NSO services but also in our devices as well. And of course.56:06The transactions of our NSO instances or when we are applying any sort of configuration. So I think that that's pretty much all that we can we could cover in the demo. So let me get back to you, Fernando. Thank you very much.56:19Thank you Cassandra. So now just to wrap up our session, we do have the documentation for NSO Observability Explorer. It it is already into.56:38NSO-dot cisco.com where you can just go and take a look at how is these new observability capabilities we have and how you can take advantage of them. Last but not least, it would be really nice for us if you can just take couple of minutes just to.56:55Answer the survey and any feedback that you have, we'll be more than glad and we value a lot everything that you, that you write to us. From my side is everything. Thanks a lot for, for your time. Thanks a lot for attending and remember to keep on on this series, we will bring more info.57:15Mission not only on NSO.57:35Also with other service provider products. So thank you very much. David coming back to you. Thank you very much Fernando, thank you Cassandra for a very nice demo, very explanatory, and we were answering a couple of questions in the chat, so if you have any questions, you can drop us an email. I think you you can see your emails there.57:37Or in the community page that I just shared in the section of questions and answer you can take from there.57:57This video of this session and the previous two will be posted there so you can, you can check it at any time you want. You can even ask questions and open a conversation in that community page of NSO, right? So let me put it here in the chat for the ones that didn't have the time to see in the.58:03In the questions and answers?58:23Okay, so this is the link to the community page that I just put here in the chat. Again, thank you very much for being here in the in the session. I hope it was very useful for you as you learned something new, something interesting that you can apply with your NSO and see you next time. Goodbye.58:26