Cisco Network as Code: AI Agents for Network Infrastructure

Oh, so basically, for today, we're actually going to have a NO slide session.00:00So, the session we will be delivering between myself, and Ed, I will be kicking it off. And the goal of the session is to kind of show.00:20How AI and network as code can work together or complement each other. And of course there are multiple scenarios that we can think of but we cannot cover them all. So we kind of created a couple of of.00:35You know hypothetical scenarios using some demo labs that we have available and.00:55Through that, we're gonna try to kind of show you, you know, the change done through a user, e.g., in my part and then the change done through using AI or a model or a chat agent. And then Aid is going to take that even further and he's gonna start to go.01:03More deeper into the different types of personas that will do different changes and how AI is being leveraged or can be leveraged to facilitate the, the, the work that those personas will do.01:23So I hope this will help. I think it's the 1st time when we're trying something without slides. Please let us know what do you think and you know if this is a good mean of, delivering a session like this and we'll take that as feedback.01:41And incorporate it into into the next into the next sessions.02:00Okay, so, for my part, so yeah, before we start again, both being Ed as well as David, we are part of the customer success team, part of customer experience in Cisco. Our role is focused on adoption. So basically what we do.02:06We work with customers to help them once they have invested into Cisco solutions, to help them understand the product, you know, work them through the product capabilities, introduce them to new capabilities as new versions come in, and also, you know, to, to help them adopt and understand, you know.02:26What is needed in order for the use case that they have invested in to become a reality. So that's the main part of of our job. At the same time, as part of this, we are running this expert inside series on multiple topics, so.02:46We had for other products across all the architectures in Cisco. They are all recorded and you know David, if you can, if you can share the link to the community in case people are interested, they can go back and then seem some of those recordings or maybe even register for future future such such, sessions.03:05So, with that being said.03:24Let me share my screen. Okay, let me just prepare here just a second. Okay, so this is live. Hopefully everything will work properly. If, not, yeah.03:28I apologize in advance, but you know as it is with all live sessions.03:47Challenges might might rise. Can you guys see my screen real quick? Can I get the thumbs up or Yes.03:56Here we go. Thank you very much. I really appreciate the engagement. Okay, so, we already had two sessions on network as code as part of 2nd series. I call this for me session 2.5 because it builds a bit on the previous sessions.04:10Yeah, I just realized that the changing here did not save properly, ok. Why was that? Let me just check because I did change this, but it didn't work, ok, anyway.04:32Okay, so this is going to build on the, on the previous sessions, specifically on the 1st session. There are four scenarios that we have in here. It's a simple use case, which is built on a lab in Cisco modeling labs CML.04:52In the four scenarios.05:08We're gonna have a an engineer, a junior engineer that is tasked to to configure a VLAN on a, on an access switch, and there is a very simple mistake that can happen for from the command perspective, right? Just.05:10Missing an ad can basically wipe or overwrite or delete all the VLANs and just keep that the VLAN in question. So that can bring us to something which is impacting the the revenue generating services for, for a company.05:302nd is where we gonna fix that by leveraging a pre built pipeline within GitLab, leveraging NAC validate, you know, and some of the network as code capabilities developed by, by Cisco, which are part of that pipeline.05:50And then we will try to do the same thing by going through, through the pipeline, and we will use NAC validate to actually block that command has not been valued, right? Based on a condition that is preset in the pipeline, right? So basically, we're learning that there is a.06:08Mistake and then we implement that into an automation process that can prevent it in the future, right? So maybe in the future our junior engineers, you know, will NO longer.06:28Get us into such a situation. The 3rd and the 4th scenario is where we introduce an agent. And here I'm gonna use a a local setup with Olama and a model.06:41From OpenAI, and I'm gonna try to run exactly the same the same scenarios where I'm gonna, you know, ask it to tell me the VLANs, I'm gonna ask it to I'm gonna, we're gonna start to interact with it. We're gonna ask it to do a change.07:00And leverage the the pipeline. There are two scenarios. Why? Because in one scenario, we're gonna just go through, you know, through the very let's say custom NCP that just goes to the device and fetches the, the VLANs.07:20And it's gonna just give us the, you know, the answer. We can also do the same implementation if we want like you know generating the same command for the VLAN, and create the same, same error, but we just we're just gonna keep it at the visualization point. And then the the the 4th scenario is where instead of going.07:39Going through the the MCP directly through to the device.07:58It will actually go through the pipeline, through GitLab pipeline. So it's gonna leverage the same tool set like in the case of the user, and hopefully we'll be see how regardless what it comes, the ask, the.08:03The implementation is going to be either blocked or validated, ok? Good. So those are the four scenarios. Now let me let me just go real quick to, to the lab just to show you a bit how the lab looks like.08:20Okay, so what we have in here is basically we have three services, checkout staff in IoT, we have an access switch, we have a distribution switch, we have a number of probes which keep, you know, everything going through the VLANs active, and also.08:39Active probe which generates traffic. All of this is helping us build some dashboards, so data comes into Splunk and this is one of the dashboards. Let me see. Let me go here. Dashboards.08:59Okay, so let's go and let's use the S two dashboard, ok? So in here we see the the three services that run that are linked to those three VLANs, everything right now it's up and running. We have NO alarms.09:18On the device here we're gonna see what changed and then a view of the of the pipeline as well as the pipeline ID.09:38Okay, once we will get to that step, but just to show right now, you know, all the services are up and running. So now let's go and try to implement the change by going directly into let me just.09:46And this, let's look at the configuration. So this is going to be the uplink, right? That goes to the distribution and on this uplink we have these three VLANs, right? So the three services, they're active, right? So the ask is for the junior engineer is.10:02To, you know, configure the IoT right without knowing that that one is is already active, but I chose to keep the IoT.10:22Service just in case so that we can see the difference on on on the graphs, right? So let's go on the interface and let's dot trunk now on the 30.10:30So this command, what basically it does, it's gonna overwrite, right? So it's gonna delete VLANs ten and 20 and it's gonna keep only 30. The error is that actually what needed to happen was to have an ad so we can add to the existing configuration, right? So this is.10:50You know, a human possible mistake, right? That can happen, and right it's good to to try to prevent something like this, right? So if we go back, we can see that we only have VLAN 30 applied.11:09If we go to the dashboards, let's see, we can see that the two services went down, the service is still is still going on, ok? We can see also who, how the change was done, right? And who was the actor in this case.11:27This is, you know, this is the username that I'm using to log into the switch and it's called the Cisco and we can see also the command.11:47That was done and that it was done via console. So something that usually does not easily show up in different monitoring tools, right? But we do have that visibility in here. Okay, so now let's go into GitLab and.11:54Fix this because what I have in here is a defined state of the network. That switch needs to have those services defined like they are. No additions should bring down any of the existing running services, right? So that's a state that I I.12:11Locked into, through the pipeline inside of that is the value that, you know, network as code brings us. It allows us to take the state of the network and externalize it and put it into files where we can track it and version it, right? And go back as needed.12:30So let's try to run this pipeline real quick.12:50So if we go on main, say a new pipeline, now it's gonna start to run it. 1st it's gonna validate the existing file where we have.12:54A switchboard with the three VLANs, right? So basically it's gonna send that command back so we can bring back the the existing services. So if we go in here, let me see if we can manage to find it.13:11Okay, so here is validating as they did the one for a second, go to pipelines. So we are in 2nd stage. Now it's gonna stop. Purposely, I built this in because this would be a stage where.13:32You know, before you automate everything and let it run by itself, you want to have a person that would review this, right? So what they would do, they would go into the plan, right? The validate and then the plan, have a look, and then it will decide if it allows the apply of what was decided or not.13:50All right, so in this case I'm gonna just apply it.14:10Okay, so it's gonna move forward. Then it's gonna move into the stage of testing, yeah? Okay, and in here I actually implemented NAC test as well.14:16So we have NAC validated at the beginning, we have KNAC tested later on So basically what we're doing now is we're.14:31Easily bringing back the two services, right? That went down. We already had a notification in Splunk, sorry, that someone did this change, right? So this can go into another that can come to Webex or Teams or an email or whatever you.14:47Which will tell you that, you know, these two services went down. If you would go into a network monitoring tool then everything would look fine, right? Because that command is valid, there is NO problem with the trunk. There is NO problem with the VLAN.15:08It it so from a networking perspective, things work from a service perspective, your customers will not be able to pay to do the checkout, right? To basically buy your products or your your staff won't be able to basically work because they won't be able to leverage the tool.15:25So let's go back to the pipeline so it's finished. Okay, so if you go on refreshing here, we should be able to see now that the services went up, ok? So easy fix, we can see that now it went through Git apply, we have the root and the actor as root.15:45Device alarm still zero, right? So you have NO alarm because the network is fine in both situations. We can see the pipeline and then the name of the pipeline, right? And I can cross launch from here directly into GitLab if I need to go and see that pipeline. So depending on the persona, some will use.16:06This dashboard and some would use, you know, other tools to monitor the, the, the network and the services, ok? But we can do the cross launch, we can make it we can make it easier. Good, so we've seen the problem, we've seen the fix real quick now. Let's do exactly the same thing.16:23But, the same error, but what we will basically do is we will have it run through network as code, so through the pipeline. So what will happen now is there is a rule which is enforced.16:43And that rule says that you are not allowed to delete any of the existing services, ok? So if you go to the repository, let me see.17:00Rules. Here we go. So this is the rule and basically that's that that is what is it's telling us. So if you go back, yeah, I'm not gonna show everything inside this because we need to move forward, we can see that the validation failed. However, the pipe.17:18The entire pipeline was validated at the end. Why? Because in here there's a condition, if the command is not valid, NAC validate will stop it and then we send a message into into the chat. In our case, it's Webex, and it's telling us exactly.17:37The value and the commands that we were supposed to, that were supposed to be sent, right? So at this point it was stopped it was prevented, ok? So if we're going to validate and while that's loading, we can go back to the dashboard and we can see that.17:56But, you know, everything works properly, ok? We can see the new pipeline that was run. We can see that it failed and failed at the validate stage and everything else was skipped, right? And it's, it was skipped because, yeah, there was not necessarily to, you know, continue within the pipeline because the command was a danger to the existing.18:16Services, ok? Good, so this is basic automation. Now, let me go back to this. You can see in here what it's applying the, the rules.18:36You can see that list must include services 1020, and 13, ok? So this is the rule that it was infringing and because of that it got prevented and then you know the admins were notified. Okay, good. So.18:55Let me go back to code. So we went through the 1st scenario, we went to 2nd scenario. What this is saying, this is outage and then we repaired it and in this it was refused, so prevented, ok? So now we're gonna go to the assistant.19:11So let me try to load the model. So we're gonna use GPTOSS 120000000000. Okay, so now let's see if this one actually is gonna work.19:28Because yeah with well with AI it's a, it's a unpredictable model, right? Or it's an unpredict unpredictable situation, it's a statistical situation, right?19:44And the goal is to make it as deterministic as possible and from here on, it's actually, you know, what can be done to make it more deterministic by leveraging network as code and the capabilities that automation introduces into the network or into, you know, different deployments. So what we're gonna do.20:04Who is gonna ask.20:24The VLAN list on this specific interface. Now, what it will do, it will go to, it will use its tool, it has an MCP yeah ok.20:26So it's going to go to, it's gonna use an MCP server that I configured in that MCP server it has the, what, what it needs to be able to connect to the device, and then it has a tool defined which will, which tells it how to go and get the.20:46The list of VLANs on that specific interface.21:05It's gonna take a while because it's running on a local server. So again, this is a local local setup. In general it's maximum, you know, 50 s at the beginning of while the model is loading.21:12Okay, let's see, so one that is running.21:33Let me go back to, this one is to make sure services are still up. Okay, so everything is still going on. Good. Let's go back to quote and I see this one is still.21:52It's taking a lot. Okay, I did run it before the session just to make sure that we're going through a warm startup and not the old startup and let's see if, if it's gonna work hopefully.22:11It would work yeah ok this one is gonna take too much reasoning. Okay, so this is still loading. So for the next, what we will do once that is done.22:30In theory it should work faster. We're gonna ask it to do the change.22:50Okay, and if it's gonna continue, maybe.22:56Okay, it's taking a while. Yeah, it should not go this long ok maybe while this is running.23:13It looks like it didn't start from a work start, it's a cold start. Maybe I pass it to you Ed and then come back at the end and have them run and then I just worked for them. Yeah.23:28So I think maybe what we're seeing right now is kind of the importance of model selection and how that kind of applies to a lot of the infrastructure that we've got. And so, there was a question I saw Charlie asked about, you know, how do we kind of see, our comfort levels with how we're using AI at the moment and I guess a transition towards.23:44Will be a gentleps in the future and I guess the the the difficulty seems to be is that it's not exactly like we do a release and then you have six months to catch up and then you go to the next stage and you know you can kind of progress in that manner. There's a lot of variables that's kind of in place and there's a lot of changing factors as well. And what you saw with razor's screen.24:04And then is I don't think anyone would kind of say, you know, waiting at a screen to do it or to understand a simple kind of network change like that is ok to sit there for a minute and kind of watch it, right? You know, that's not what we want to kind of do. So what we want to kind of do in all of these areas is to give context specific knowledge or awareness to some of these tools.24:24To try and give us the right outcomes that we want. That's really, really tough to do when we look at very broad things. So, I'll I'll start this just by explaining what my setup looks like and then we'll go into some of the different scenarios that we have as well.24:44So, I'll just stop sharing. Hopefully you guys can see this one, ok? Similar to what i'm using a Git environment as well and so this is just a open source kind of project we're working on where we're using a self service portal and then we have GitLab and that's in place. We run.24:59Pipelines through a runner with there's two different images, I think there you've got Ansible and Terraform tools alright so fairly standard kind of setup for doing networkers code. And what we're gonna be doing today is we're gonna be having a look at how this kind of applies to agentic development, I guess and how we're gonna use some of these models.25:19In place as well. So if you joined us for last week you saw this page where we we looked at how we can have different interfaces to making network changes. So in this context, we're quite specific in terms of how we work.25:39We say fill in this form and then we're gonna be able to understand exactly what you want. And we can use deterministic tools, things like and Terraform formats and some kind of linters in place in order to make sure that the values that we see inside here are ok. But again, I kind of look at this and going well.25:52It's kind of a bit of a pain to have to do all this kind of stuff. I don't really want to to to leave this for people to do because you also enter a bit of an issue where, you know, how are you hundred percent confident that you put the correct IPV four address in, you know, are you hundred percent confident that everything looks ok? And the the reason I I worry about this is that deployed.26:12Quite a lot of ACI environments in the past and other Cisco kind of architectures and I found, you know, mistakes that I'd made during deployment like a year later and you'd go back and you're like, oh.26:32You know frustrated almost to some of the stuff, but when we start to build automation in place to this and we're trying to be deterministic, we really want to to guarantee that actually what we've deployed is what we, what we want as well. And so the next kind of step that we're gonna see in this space is how we can start to use natural language to do this. So in this context, we're doing a.26:43Very similar thing before and I'd argue actually we're making it more difficult for people asking them to fill all of the information in like this. But what we're gonna try and do is to make it easy as possible in order to be able to, to interact with that infrastructure.27:03And there's a few mechanisms that we can actually use in order to to control how we're doing some of those things. And the the mechanisms that we're using are actually very similar to the, what we should be using to how we can control what agents should be able to apply into our environments as well. So in this context, we're doing multiple checks. So I.27:18I and I'll go through how we actually do that, but just so you kind of you understand at the beginning is that we're able to kind of validate the the configuration that they're applying. We try and say actually this is the structure that we expect afterwards, then these are the kind of configurations that you can change and then afterwards we want to validate that as well.27:38So withinside this repository, and again, if you guys want a copy of any of this kind of stuff, just hit me up, I'll send you some of the links afterwards as well. What we have are these different descriptions of how we want some of these models to work. And in this context we have one called author reviewer and I'll go through these individually, but just to show.27:58Show you the difference, these different kind of tasks that we're expecting them to perform. So the 1st one that we're actually having a look at is this author one, which I think is also I've called builders sometimes as well. And in this particular use case, we're hard wiring it just to be looking at a loopback configuration. Now reality is when you're gonna be doing this in your environment.28:18So you're gonna have different types of configuration, you're gonna have different files here in terms of what it looks like in the end. But the the concepts are the same, right? We say this is how I want the configuration to end up looking within the environments. This might be how I want you to check that it's was successful.28:38And if we want to try to do validation, we don't just want to be using agents in order to do that. We also want to use the terministic steps as well because we want to try and reduce the the number of hallucinations we might have and, you know, improve the overall reliability of what we've got.28:54So in this 1st kind of step, what we have is a user makes a change into the portal, and then as part of that we have an API call where we render the configuration, and then get goes into GitLab. That's the manual kind of way of doing it. But we also have another one where we can interact directly towards Alarma on our local models.29:10As well, that then goes into GitLab, which then goes into our runner. So again, I I really want to stress this. We're not saying actually we're giving all the responsibility for different agents to make that change and we're not saying that we we want to manually kind of code all of this stuff as well. It's a bit of a combination in between.29:30So if I just, let me just pull this one up. So just to kind of show you, this is actually I'll go one step back. So this is a topology that I'm using just to be really clear, it's a single VM that's got a load of RAM in it. We're running an XRD containers on that as well as GitLab and some other tools like alarma.29:48In order to be able to run some of those models. So if you are trying to build up your own setups, you can make quite comprehensive tests now without having to have loads of dedicated hardware or necessarily loads of compute. This is running on a 64 gig VM, which is again quite a lot of RAM, but it doesn't need to be that big, you know, you can go smaller and.30:07It'll be a bit slower, but it'll do the same job.30:27Now when we go and have a look at the particular models, let me just pull this one back, and yeah, so just as explained, we're gonna access via GitLab, we have alarm runs and runners in place that's what's kind of useful. Now in terms of the the models that we choose, they're really important when we're focusing on this and again I saw Charlie mentioned the Taris earlier on and that.30:30That's one of the concepts that we we're really trying to nail down on because it doesn't make sense, in my opinion, to have very large and expensive models that can do absolutely everything when we want them to do a very specific thing all the time. And we've kind of seen over the last six months, maybe a bit more of a resurgent in small language models where we can use some of them in specific.30:50Specific use cases. So the Microsoft I think is released to five models, which are really quite comprehensive. So if I just.31:10And I can't write it all. And so and what they have when they're having a look at some of these is you have absolutely tons of different models that you can use.31:29I think it was this one?31:39But I can't find this specific page I found earlier on but essentially what we have with these kind of models and we have them as well within Cisco and everyone's kind of looking is we have very specific roles that we're gonna perform. So when you looked at the one at the beginning we had a role that we were looking at for triage. We had a role that was looking at for creation of configuration.31:41And a role that was responsible for reviewing air configuration. So what I think we'll see more and more is that we'll see specific models that we're gonna use in place to do some of these functions. We don't want to pay those ridiculous high kind of costs, we don't need to necessarily, you know, have these massive.32:00Context to work with, we just want it to be reliable and focus on particular areas as well. So in this environment, we're actually running everything using Gemma four. So I will just pop back to the screen. And what you can see here, this is just the output I'm looking at from alarma, with the different tags that are available to us. And so we can see here we have.32:15XRD reviewer and we have XRD builder and the idea essentially behind these is we might want to use different models underlying, so you can see here the parent model that's associated to it. They're both using Gemma four, but we could see maybe one of them we're gonna use as five, one of them uses and things like that is in combination because our goal should really be.32:35So we want it to be as fast as it can be and as deterministic as it possibly can be as well.32:55And also one of the things that's important here, you can kind of see the different capabilities that the model's gonna have. We might want to give different tools to some of these models as well. In a case it's not kind of clear about the difference between models and tools and how this applies, models are essentially, I'd like to give the example of, you know, if you were transported to a town in the middle of nowhere.32:59You know, could you figure out where to go? And if you put sherlock comes in that environment, you'd probably figure it out quite quick. You know, maybe he'd look at the language and the signs or you'd kind of have, some kind of things that I wouldn't think about too easily, he'd probably be able to find out really well. Now if you put me in that environment, maybe I would struggle, you know, to kind of do some of those things. I'd be your non.33:19Model I think in that example. But if I had the right tools in order to be able to do some of these things, I probably would, you know, we talk about MCP quite a lot at the moment how we have model and context protocol where we can connect to different data sources that if we can give our kind of tools the right, sorry, if we can give our agents the right kind of tools and access to.33:40The right information, you know, they can become much more you know clever, I guess in terms of how we would view them. So we don't necessarily need to use her.comes in every case. And classic example maybe it's just here before I kind of show some of the demo, I asked the question earlier on this was in general 2.5, so like a bit of an older model. And I said.33:59Can you tell me about iOS troubleshooting? And it actually fell, you know, it didn't work for this one. And so I used a more modern one I think this was rock two.six, and it did work. It gave me a nice comprehensive answer, but unfortunately, because I didn't define that I wanted Cisco iOS, it gives me a lot of information about Apple iOS. So it doesn't necessarily matter about how smart some.34:19All the models are in place, if we don't have the right kind of structure around some of the requests that we're doing, we're not always gonna get the answer that we want. And that's where we kind of use skills and rules in that place, you know, to kind of to force some of those things. And I'll show you some of those as well later on. But when we provide the right context a lot of the time to some of these requests, that's where we can start.34:39Getting, a lot of useful information. So it's important that the data that we've actually got is relatively structured as well. We don't want to have to go and look at 20 data sources to figure something out. It should be in a relatively.34:59Refined way. So that's always the bit that makes me feel a lot better because we've looked at automation for the last ten years, but actually the foundation of automation is really, you know, can you try and make, data as as simple as possible and easy to read and all that kind of stuff. And then you make your processes as standardized as you can. All of those things are gonna really benefit us when we.35:13Start looking at how we can use AI in in our infrastructure as well.35:33And to give you an example of how this works, I asked it a question earlier on, can you tell me how who's prompts to shape the behavior of the agents in this pipeline and it correctly defined actually how we do some of those things. And it explains all of the elements that I have with inside of this project to pretty high.35:37Accuracy, you know, I was quite happy with with what I was able to come out with. And this is the same model to be fairly, it's the same foundation as what we used for the question before that it got completely wrong. We just got that context to kind of help us understand that as well. So to give you one more example before we start having a look in, I wanted to show you how we use some of the sub agency.35:55In place here. So one of the architectures that we're gonna see when we're gonna talk about in the next series of expert insights is how we can start using agents in combination with each other. So how can I use one agent to maybe orchestrate what's happening and use another agent to be the technical like support persona, all of these different types of roles.36:15And we'll talk a little bit in that and in those series about how we use agent to agent communication instead of mtps and all that kind of stuff. But just to give you a really visual way of how it works, here's six that I use quite often, so when I'm coming up with new plans or new things that I want to kind of work on, I don't like having to think about all of the aspects.36:35Of what I want to create in one place. And so what I create in this context is a persona of a dev team that I work with. And I named all of them. Like I I live quite far away from the the Cisco office, so they all have names, they have like analytical Anna and efficiency Ellen and Innovative island. He, he's quite a cool one, and all what their responsibilities for is essentially.36:55Actually to to look at a problem from different dimensions and to give us that output. So when we look at the the pipeline that Raz was showing earlier on, maybe someone in that example needs to be checking operational kind of data. So is there anything else that's happening at this time which means you can't make a change? And so you have a risk score person, are you introducing any.37:15Security vulnerabilities by making this change. And so we can have all of these different roles that have different approaches to to how we might do something and then we need to coordinate all that together. And that this is a really powerful next step, you know, I think it's the the next evolution for where we're going. So in your mind, when, when you look again at some of these demos, don't think that's what we should be aiming for.37:35It's a transition state, so we start from maybe NO automation to some automation with AI in elements and then we might end up in the position where we're looking at fully you know autonomous environments, and that's where we started to say and honestly that we're we're looking at agentic ops. So just to show you one of these again quickly, I'll give you an example.37:55This is Anna, e.g., she has it's a markdown file, so you can kind of see here, it's just normal kind of text file and then withinside of that we define how is it that this particular agent should operate. So what is it that they should kind of do? How we should kind of use it in what context, what the output I'm expecting of the.38:15Off the back of it, e.g., and what do I want to see? And so this is very different, so you can see e.g. she's only got read only access into this environment, she's responsible for a subset of things and I give her access to these tools. Planning e.g..38:35Honeypole also is read only, but he's responsible for essentially invoking some of the other agents that we have in this space as well. So again, they've got different capabilities, different projects. And the way that I use that is I am call all of them from a command.38:52And in this example it's called dev team and we can execute them in this individual flow. So what when we start to think about the demo that we're looking at today is that we were gonna use our triage agent for when there's a mistake that happens basically we're gonna monitor the state of the pipelines. We're not gonna use our builder one to do.39:09The plan review, we're gonna have context kind of specific applications as well.39:29So if I just pop down into this one so you guys can kind of see, we've touched on the fact that we've got different personas that they kind of exist. So let's do the 1st one. So if we open up the pipelines, so I have here similar to what Raz was showing before, I have a very kind of.39:34Straightforward pipeline in this stage, so we just have a quick look and we have validate plan, a review, application, saves of the reports and then we're finished. So one of the things that we're kind of doing, which is maybe the next level of what we saw earlier on is that you want a manual gate, you know, for support for all of these applications and.39:52Some of the time, you know, we don't want to kind of to to say you have to wait for a human to kind of be involved. And those are scenarios that we're calling human in a loop scenarios, and some of them you you really might want those scenarios. And so before what we did was we had static rules that quantified the risk.40:12Associated to certain things and one of the nice applications I found for for doing some of the work that I'm using is can we try and use agents in place to kind of do some of this review? So in this case, we have a quick look at the plan. We're removing some configuration. So if if you're not familiar with network as code.40:31Or Terraform in general, when we do execution of a change, we're able to kind of say, what is it that's gonna change?40:51In the current state that we've got, based on what you're proposing. And we can see here that essentially what we're doing is all destroy kind of elements for these loopback interfaces. So we had loopback 213 and loopback 214, which I was using for some.40:58Unit tests and kind of smoke test for this lab, and we can see here that we're gonna take away all of this configuration. Now, you could set up a, an example which says anytime we want to destroy some configuration or change or remove the configuration from our devices.41:14That we want someone to review this and maybe you've screwed something up there and we want to check. And so what we can kind of do is we can build some of those, that logic into how we view risk, and you can see the output here and I'll explain how we do it in a second, but hopefully you can see that, I don't know.41:29It's just not wrapped very well, but what you can kind of see is we've used this model here, this XRD reviewer in, in place, which is underlying before to be clear as well. And then it's given us a risk score based on this. So we said, ok, well, your validation passed you know well done. There doesn't seem to be any errors or any failures with.41:46The tests in place, which that usually would mean green for us, right? You know, we've passed all of our individual tests. The configuration looks ok. In most cases in an automated workflow, that means go. But in reality here, you might want to pause. And so it says, however, looking at this, it can see it's destroying two loopback interfaces. They're not defined anymore.42:06Or While it's kind of looks ok, it's disruptive and so we need a human to review this. And so immediately what we're able to kind of do is have some kind of awareness over the context of the change that we're making in these environments. So when you're creating things, e.g., and we want to say again hypothetically, we want to create a new.42:26Loopback address or whatever configuration you know you would normally do very regularly, we want to allow that in. We want the risk profile for that to be considered low. So in those cases actually we don't want a human to be involved there. We want to automatically try and do the deployment for that. And you can see, we have this logic here.42:46So let me just visualize this into apply sorry I actually simplified this. I removed the logic and made it a single job. What I had before was a deploy in a deploy auto stage, so that deploy was a manual kind of change. Now that logic is built into the job. So it's it's.43:05Essentially if the risk is low, pause if the risk is high, go ahead. And we can kind of see that on some of the different environments that we've got.43:25So, so far what you've seen is the plan reviewer and they were able to kind of review what was going on. I automatically triggered the deploy. So actually maybe I just run one of these and you guys can see this one.43:31I'm gonna change this ton and I'm gonna change the address also to this and here go. And as long as my, this might take a little bit of time because we're gonna process with the local model as well. So I'll let that run so you can kind of see what's happening.43:48The last stage that actually happens before we do ironically do the 1st one is that we we want to have a look at the the state of triage, so if something actually happens at the end, do we want to go and kind of review the failure that exists? And the reason for that, and again, there's another application of how we can use agents in this context.44:06Is, when a pipeline fails, the way it works today for me at least I usually get a message within Webex and then I can go and log in and have a look and see what's going on. Now some of the time what we actually want to try and have is a retry policy. So if the problem, e.g., it failed connectivity to a to a device, maybe you want to retry the pipeline.44:26Automatically in that kind of case. And we can build that kind of rules into some of the logic that we have.44:46But it's much easier to kind of do that using some of these models as well. And so in this case what it we're having a look at is that it looked at the entire job that existed. It ran a little Python script here which reviewing the change collecting information about all of the different jobs and that you can see why it.44:50And so it says here that we failed the validation stale stage because you failed this rule.45:11What should you try and do next? And I think this is a really powerful thing that we can look at as as the next kind of use of the technology is how do we try and get recommended steps maybe that, you know, we want to say does a human want to try and review this and have a look at the change? Do we want to actually to provide this information to another agent in order to be able.45:16To trigger a new change. And that's again where we kind of see this autonomy kind of creep in. Are we expecting that, you know, very initially, is it a human that's reading the output of this pipeline? Is it now an an individual agent that's been able to read what's going on here and give some recommendations? And then the next stage is actually how do we act upon.45:36Right, so if some of these things could be, if failed because of network connectivity, maybe the next step is to trigger another one or if it's failed because security violation, maybe it's important to to go and.45:56Yeah, have a particular output, you know, in that space as well. So what I want to kind of be clear is, is that we've we're still using agents, we're using slightly different ones in this context and well the application for what we're doing for them is a bit different. So let me just open pipelines again, see if it's running. Okay perfect. So it passed, so.46:08We have a look here and I think you're going to.46:28It did apply. Okay perfect. So it said that the risk was low. So let's just do plan review. Yeah let me just check that. I might have made a mistake in my pipeline. I hope not, have a look.46:31No, ok, perfect. It was showing us this in progress state because it's a child's pipeline of the parent pipeline. But essentially what you could see there is let me just pop back actually so you guys can see easy access went through that quite fast.46:49So same as what we saw before. It was a change that was happening, got a deletion and creation of some kind of interfaces, everything kind of looks ok, but you you want to go and have a little look about what's going on. Okay, so we're saying it's a medium risk that we've identified. We open up the deployed.47:07You said a child line for this and then based on this, is it doing a manual action or is it doing an immediate action, right? And we can see actually you need to do it straight away. So we hit run on this one in this context, it's gonna go trigger off the deployment for this stage and it's gonna run through the same kind of pipeline that we saw before.47:24If we're creating a new one, I would hope that actually, it does one without does it in an automatic kind of way. So, let's kind of talk maybe about some of the differences. How can we kind of restrict what people are gonna do? Because if you can go to this interface here.47:44So you can see actually that it was able, let me just show you this 1st. It was able to take the configuration that I had as part of this, it mapped it into a structure that we wanted to kind of see. And then we have some particular guardrails and things around it that we're going to prevent it from doing certain things. So.48:02I'll just pause now and we'll go and have a look at how is it that we can kind of prevent it from doing some of those things. So the information that we have with regards to what is how it should work was described here. So you can see this is what I want the kind of the structure of the data to kind of look like. This is what I want you to, to respond.48:20And you know if there's been a particular problem, how should you do that? And then this is the.48:40The rules that we're going to apply and based on this. So in this example, we're using a single node, and so we're saying actually you can only make changes here. So back to the question about, you know, what can we do in terms of limiting the scope of what some of these things could do? You could say actually that maybe this agent can only work in.48:44Your dev environment, e.g.. You know, we don't have access necessarily to production, but we can start to kind of add some of those instructions into this as well. And this is used with an alarmer as the contract for how the the different roles should operate, but there are also different ways that we can, we can prevent this as well. And so what you can see here, and this is.49:03Yeah, this is generated for the purpose of this demo, right? It's.49:23And not gonna necessarily be something you you copy directly, but what we can kind of start to build are some of the rules actually for how some of the requests are made and what we're gonna do. So you can see here we limit the total number of characters that might exist, you know, we have some kind of manual check to see whether we're referring to loopbacks. And then same thing here.49:27But one of my tests was could I change the host name of a device? And that's obviously not something that we want to do. So we can add these kind of checks when we're looking at the text that's applied as part of the change, right? So we can have different levels of how we.49:46Restrict the different scopes that we provide as some of the agents to perform or what people can actually try and request through some of those tools as well, which is a really kind of, I think it's a nicer way than what we do today. It gives us the the ability to kind of catch a lot more problems than what we did in the past. The the challenge.50:05Times that we're actually trying to face is that we've we're giving people more ability to be able to make changes. So that kind of, it almost cancels itself out in my mind from a risk perspective, but it's gonna be interesting I think as we see the different ways that we can apply some of this AI defense and some of the observability around what our agents are doing in.50:22To that as well.50:42So let me just pull this out. This should have built. So we looked at author, we looked at reviewer and then we looked at triage as well. So in terms of some of the things that was interesting just to mention about triage as well is that we can tell it what stuff are we gonna be interested in? You know, what information is gonna be relevant, what isn't, and then we can kind of use some.50:44For that to.51:04To to guide some of the decisions that we want to make as well. So, what you should be able to kind of see as part of this is we haven't gone too far away into some of the tools from what we're talking about here. You know, we still have the structured data within networkers code, we're still accessing or configuring devices using.51:05Terraform within this context.51:25But what we're doing here is we're augmenting part of the pipeline to make it easier for people to use and to be able to give us more information about what we want to do. And a really simple use case that we've done off the back of one of these for another customer is they have very large terraform plan outputs and you know for if you're not a big Terraform person that's.51:27Quite an intimidating thing to do. And so what we do is we use an agent for them they have their own one in their organization and they summarize what the change is gonna be because about two, three years ago when we worked with them, we worked out that the Cisco part of their change made up less than 10 % of the total effort. You know, most of the effort was actually.51:47To change approvals and you know getting tickets raised and approvals, all that kind of stuff. And whereas actually if we can try and summarize what we're doing in advance, we get a massive improvement in productivity because we have a nicer interface to be able to generate some of that configuration, but then also once it's created.52:07We've got the tools and the resources available to analyze what that.52:27Planned change is meant to be, and then we can provide it to people who aren't necessarily networking people, but we can summarize up what it's gonna be. So can I try and quantify the risk associated to a change? Can I tell you the scope that might be involved as part of it, and can I try and give it to you in a language you might understand? So instead of, you know, 5000 lines of Terraform plan out there.52:33But can we give them a paragraph just explaining what the, the idea of the change is meant to be?52:53So this all kind of sounds like in in my head at least I'm kind of waiting for the but you know what's the the thing that sucks about this whole kind of approach and that the thing that we we really need to kind of focus on and I think the one is changing the fastest, is how confident can we be on the output of what we see with some of these AI agents? And the the.52:58The reality is, is that we're becoming more accurate in terms of the outcomes that we want. But as you saw earlier before, if we don't provide the right context or we don't provide the right kind of guardrails around how it works.53:18We can get outputs that we don't want, you know, in this case. Like I I know the classic example was always, you know, can you try and give me the cleanest code or environment you have, and yes, you can remove all the files with inside of that location, it's very clean, you know, that's not what we want to have. And so we now have the ability to really prevent those kind of cases.53:30Occurring.53:50So just to reiterate maybe from my part and then happy to kind of take any questions as well is I think what we're gonna see more and more of is we're gonna see more engagement between different agents with different awarenesses and these kind of processes and then we can start to integrate them into what we're doing. And in terms of the things I want.53:51People to start looking at today is we need to still look it towards standardization and how we simplify operations and the environments that we've kind of got because the more complexity that we have in our environments, the more the agents need to know, the more that, you know, kind of checks we need to add into things which slows the whole process down. And so even if we're not quite sure many.54:11Maybe about what's the correct way that we want to guard against every attack or the correct kind of application that we've got. There are still really positive things that we can do that's gonna improve how we how we work with this infrastructure. And.54:31Last example to kind of explain what the main benefit that I kind of see at the moment is if what we can kind of do is start to describe this infrastructure in a standardized way, you know, we're not just looking at full text based kind of configuration for individual devices or we have lots of different kind of formats. We can build these kind of.54:46Schemas in place that really, really helps us in that space as well. So yeah, combination of deterministic controls.55:06Try and use the non deterministic tools with agents where we can do and then build it into a process that we kind of see that's you know the people in our organizations are happy with for now. Tomorrow, you know, maybe that's gonna change but for for the next six months I think that's about it.55:14So that was everything I wanted to discuss. I don't know if you wanted to to show the outcome of your demo as well. Yeah, I think it's, we can take questions for the moment. It looks like it's not collaborating. No problem, so we have some direct questions.55:30Okay, I'll answer the direct ones as to one and then Charlie I hope that kind of answered to me your question with in regards to comfort. I I absolutely can see that we're gonna start off with some of these things with only the read only access, you know what.55:51You were seeing before with some of the different sub agents that I had in my own environment to give some of that context, right? You know, we could integrate the the triage and the reviewer kind of personas there without having any right access at all, you know, just trying to give us some more awareness of what's going on. I think that's a really powerful kind of use case.56:11And then I think as people get more and more comfortable for some of this stuff, that's where they're going to, to move towards the, the right based.56:31Roles as well. We we purchased an organization called asterisks a little while ago, all about identity for agents that that's a really interesting topic at the moment. So if you're, if you're looking for something else to go and read up again after this, I'd probably start with, A to A or how we kind of do with multi agent architectures and then have a look at agent identity after.56:37Towards is gonna be really quite powerful as well.56:57And then I think that was it. So I think for your question hopefully that answered what we were talking about. Essentially a generic ops is gonna be how much autonomy we give to the different agents in these environments to be able to to run the things that they want to. Like what you saw with environment, in my opinion is is still AI ops. It's, it's saying standard operation.57:01In procedure, I'm augmenting it a little bit with some of the AI elements. If you then took mine and started to to add automatic remediation e.g. into the pipeline, I think that's where we're gonna start to see some of the things that we we consider agentic, but it's not really there yet. It's somewhere in between. We're gonna be running another session probably about February.57:20Time where we look at some of these more advanced topics where we have a look at multi agent and, you know, really deep dive into some of the the ways that we can control what it's doing as well, to look out for that one, same kind of format you should get or should see as a LinkedIn or if you have a look at the community pages, I think there should be some signups there. But if there are NO more.57:40Questions? Nope, then, have a lovely day everyone. It's, it's been a pleasure and thanks very much for your time.58:00