Cisco Catalyst SD-WAN fabric upgrade workflow and demo_1

Hello, good morning, good afternoon, everybody. Welcome to our expert insight series on catalyst SD WAN upgrade best practices.00:09Today I am joined by my Colleaguez and we've got panelists Astasia and Michael and my name Isaf. You are not sharing, can you share this slides? Yeah, now it's, we can see. So it's me.00:18I've been around for a while. I've got 15 years of industry experience and currently based out of Krakow.00:38I've been a part of TAC team in different domains, be it networking or smart services and recently been doing for almost six years. I've been part of several papers on cisco.com and I'm an active Cisco live speaker as well.00:45With me, It's. He's a networking professional specializing in enterprise domain and SD WAN being his expertise. He focuses on helping customer design in in with deployments and with optimizing the modern WAN architecture. He is also in Cisco for, well, for almost about 15.01:02If I'm not wrong.01:23And he has great deep technical insights and real world deployment experiences. So both of us today will be walking you through the the upgrade process of your catalyst SD WAN fabric. But before we move forward, we would, request you to join this Slido. This would give us another.01:24Understanding of what kind of audience we have and there are few questions at the end as well that will help us better understand the overall interaction and transaction that we are gonna have today. So just scan the code. This is the passco scan the QR. This is the passcode right here.01:44So we are waiting for the joinees to come here, the QR as well, ok. That's great.02:04Alright, so more or less we are comfortable, but we also have the audience who generally need some help. What major version of Catalyst SD WAN code are we currently running today in the network? All right, it's.02:332015.02:53That's good. We got still few on twenty.nine. Okay, great, so.02:54Majority are on 2012, few are on 2018, that's that's great. So today's session will will give us you know a better understanding in terms of how we can move if you are planning to move from twenty.twelve to 2015 or eventually to 2018.03:14And what flavor of the SD WAN manager you have in production? If it's it's a single instance, single tenant, single tenant cluster or a multi tenant cluster?03:32Okay, you've got a single tenant cluster, so it's gonna be a big deployment. Alright perfect. So it's it's majority is single tenant and single instance or rather more of them together.03:54All right perfect. Thanks. Thanks for this insight. And now comes the other important question is what are the major challenges you face while upgrading your catalyst SD WAN fabric? More or less we do understand that the process is.04:10It's is somewhat intuitive, we would say, right? But still we do land up into different kind of problems. For me when I was, you know, in my initial days working with the SD WAN manager, I mean for me it was more about the logs where I would see the logs when I'm trying to upgrade.04:27And if it fails.04:47Where I should start seeing for any issues, right? I've seen few customers who had problems in, in, in where what's the best place to store the images, right? If you've never done it before, that's absolutely fine. And today's session is going to help you a lot to understand how you can go about doing it.04:49Yet, right?05:09Yeah, I I can totally relate. It's it's more about if I'm in production, I've got a maintenance window. Will it be over during that certain time or not, right? And if it fails, then what would happen? I do not have somebody from Cisco tag with me on the call. They're on standby. I have to, you know, go and.05:12Call the cisco tag for getting you know support so absolutely what is also there. Right, right. So we are actually, and I'm glad to see a successful upgrade as well. And eventually that's what the idea is. That's what we want to do.05:32Right? Absolutely. Great. Thank you. So what we are going to do today is, all of around those those concerns that we have, we are, we, the whole reason of why we have this session is to address such concerns.05:48Right, so what we are going to cover today is more from these aspects where we'll start off from the upgrading your manager I mean, so we've got a small flowchart to see that this will give us a better understanding in terms of what we should do when.06:04Right, and then as we move on with the slides, we'll get into each and every detail, each and every component of your fabric that how you go about upgrading, so it would be more about more for the people who've never done this, kind of a revision for people who are pros and who are, you know, who got an experience working with the solution, they know how to upgrade, but then.06:20How it looks on the very newer version. So the demos that we have is on a twenty.eighteen version, right? So there are few enhancements that that have been done. There are few workflows that have been, you know, added in this upgrade scenario as well. So.06:39We'll cover that aspect as well and what we'll also do is we will talk off a cluster upgrade as well, right? So not just a single tenant upgrade, so we will also have a cluster upgrade or not a single instance, but a cluster upgrade and along with disaster recovery and if you do not have disaster recovery, then what we should you should do, right? So we're trying to cover all.06:56Aspects of different kinds of flavors that exist.07:16And hopefully it would answer a few of your questions. By the way, in terms of questions if you have, please post your questions in the Q and A sections at the bottom right of your Webex's and our panelists will take care of answering those questions. So starting off with the, with the flow chart.07:19Right, so the 1st thing that we want to understand is do you have a disaster recovery or not? Because the moment you have disaster recovery, there is a separate step that you need to do take a small detail and take certain actions, but if not, you need to decide if you've got a cluster or a standalone instance. So if you have a disaster recovery on.07:38Then the only thing that you need to do is to go to your administration settings, disaster recovery and verify the replication status. If it's successful, I mean it should in your normal stable environment if it's it should be success. If it's anything else, in fact, we'll look into these details in a little bit more detail when we we turn to the disaster recovery.07:57Upgrade section, but if it's the whole idea is if the disaster recovery is running.08:18Pause the recovery, you know? So pause it and then you come back to your upgrade process. So if it's standalone, I mean you would follow the process that we will see today. And if it's a cluster, then there are certain series of steps that you need to follow based on the version that you are in.08:23Right, so there are certain specific things that you need to keep in mind when you are moving from one version to another, right? And or from one minor release in that major release, so there is that twenty.nine is the major release, twenty.9.one is the minor release, so if you are moving within the train, then there are certain things that also you.08:41To keep in mind. So my colleague Vacas will walk you through the cluster process.09:01With or without disaster recovery and I will take care of the standalone upgrade process that we will just look, right? So most of the processes would kind of like overlap, right? But still we wanted to make sure that we are coming up with.09:06You know the the clear steps and the guidelines that you need to follow based on the type of deployments you have, right? So you might still see certain things that are common between both, right? So, but still for the previty of the discussion, we have put them here.09:22So before upgrading your catalyst as the, what you should do is to stop and take backup 1st, right? So somebody did mention about snapshots, there is a, there is a small little typo. So it's like a real level snapshot, so either you can take a snapshot out of it if it's your.09:39On prem managed SD WAN manager, it's your responsibility to take that snapshot on a regular intervals, but if it's cloud hosted or or like a Cisco cloud hosted, then Cisco cloudops team do take regular backups. They do, they keep you know a series of ten snapshots.09:59Shorts on a daily basis and then it rolls over and it happens around midnight each day depending upon where that particular vmanage instance is being hosted in which region or you can also take a configuration DB DB backup, right? So.10:19I'm not sure if you know this, but it's like you can also have a script to do that kind of daily backup creation and then you know we can export that out to your external servers just in case you would need for getting your new instance built. So we always recommend that they.10:36And take both.10:56If not, you know, we cannot take snapshots on that regular frequency, you can take a configuration DB backup as well from the we manage command line as well, right? So either you can take a one odd backup or you can just do enable daily backup, right? So it would just continue to keep I think three copies on the, on, on the part that you would.10:57Mentioned and daily around around midnight again.11:17And, but it would not export. So for exporting either you'll have to do this manually or you can create, as I said, automate this daily backup creation and export. So as the manager upgrade matrix, so this is one of the initial things that we suggest our customers to follow, right? So whenever you are planning.11:21You have an upgrade of your SD WAN manager, you must check.11:41That from my current release, can I go on to the target release directly or there has to be a step upgrade, right? So always and always check that 1st. So here you can notice that I'm trying to move from 2012 to 2015 and it says that there is a direct path from the current release, and there are few upgrade notes that we highly, highly recommend you to go through.11:45Right, but let's say if you are trying to move from 2012 to 2018, it will tell you that please 1st go to 2015 and then go to 2018, right? So always and always check the upgrade matrix and this is specific to your SD WAN manager, and here is the link for the upgrade matrix tool.12:042nd thing that we want you to suggest to check is to have the compatibility between your SD WAN manager/the controller and your edge devices, right? So it should not be that your.12:22SD WAN manager is let's say on 2018 and you've got devices that are on 17 dot, let's say nine or seventeen.ten or eleven. So these are primitive versions in comparison to the twenty.eighteen SD WAN manager version, right? So we always and always suggest you to 1st.12:39Do the SD WAN manager version check that how you can move from one version to another if it's a step upgrade or a direct upgrade, and also that the kind of the, the, the, the products or the vantage product line that you have within your network.12:59The versions on that particular edge device, is it compatible with my target release or not? If not, then you need to, you know, plan according to, to the to the version as well. So again, here's this link that link to this tool that you should go and you know run this check before you are, you know, actually executing your upgrade.13:14So these are all the, the.13:34We check best practices in terms of even before starting your upgrade that you should do. So starting off with the upgrading your SD WAN manager in a standalone environment. So the 1st thing we all know is to have the images added to the repository.13:35Right? Yeah, ok, so I was just checking few questions on the top. So we will put the links in the chat. Yeah. All right. So you need to go to maintenance and go to software repository on your SD WAN manager, and if I can zoom.13:55The very important thing that you need to keep in mind is selecting the right image because that's something that we kind of have the, the, the questions regularly from the customers that I'm not able to find this image for upgrade. I'm not able to find that image for the upgrade. So, sometimes it's like even choosing the wrong image for an upgrade, so we.14:12Should be mindful that which version and which image we are looking for, and we need to put up the right image. If not.14:31The V manage will give you an error saying that the image name is incorrect and you need to find out the right image. So once you've done that, you know which image you've downloaded, right? So there are three different ways that you can shift the image or transfer the image.14:39On the target device, right? So you can have a remote server or you could have got, you know, a manager option or a remote server manager option. So remote server is more around the fact that you are not storing the images locally on the SD WAN manager, but you've got an external repository from where you'll be transferring your the image to the end device.14:54Right, and for that you need to do certain changes on the device itself, and those changes are, you know, around the implicit/the explicit ACL changes and.15:14It's it's mentioned here pretty clearly that you need to either create an explicit ACL opening the FTP or the SCP ports for the image download or you can just do allow service all under the tunnel interface of your VPN zero interface, right? So you need to make those changes just for the.15:27This file transfer once it's done, you can revert back those changes.15:46You can also store the images locally on the manager, which most of the times we do in the in the lab work or if let's say if the image is small enough, we do that because it actually it takes over your control connection. If the image is huge, it's it's big, we would not follow the manager.15:50Option because what happens it's it's it's kind of like there is a throttling that happens on the edge device that might not give you the direct or you know the the live status on your SD WAN manager in terms of where my upgrade is because there is throttling happening on the same link where you've got your DTLS connection towards your manager. And on that same.16:09Link you are transferring the image to your edge device, right? So we want to kind of avoid that situation. So in that case, if you do not have a remote server externally, what you can do is make your SD WAN manager as a remote server where you only need to enter the VPN 05:12 IP address on the top and you'll do the configuration and it would be actually.16:29Copy to your device or an out of band connection. So quick note from a twenty.eighteen.one perspective is like when you are uploading a certain image to the SD WAN manager, it would check that inside that particular directory right here, open/data. If it's more than 80 %.16:49It would give you an error saying that I do not have much you know space to.17:08Upload this image, please clear some space. So it's always best advice you reach out to tag, clear some space out and then start off with, you know, adding the images to your repository. So I just quickly want to show you that how it would look like from a repository perspective. So here you see, you can add a remote server.17:15You need to enter the server name, IP address, protocol port and everything. And once you've done that, you will have to upload the software images. So under software images, you will either add it on a remote server, on a local manager or an local manager in an out of band environment.17:33So this is how it would look like. Quickly from, I mean, this is just like from a flow perspective, you just need to drag and drop the image here, but this is for your remote server manager where I was saying that you need to enter the IP address. And once you've done that while activating the image, this is the place exactly from where you'll be.17:50Transferring the images to your end devices. Same, you know, option goes for putting a virtual images or firmwares for your CH devices where let's say for virtual images is basically for your enabling your UTD features like IPS IDS, all those features, right? So you would need a virtual image.18:10For that matter. So they also have to be in the repository. So once you've added the image to the repository, now we are talking from our NO disaster recovery option, the process is is very straightforward. You just need to go to maintenance.18:30Software upgrade, control components, manager action, right? So right here maintenance, software upgrade, you click on the controller components, click actions right here, and then there would be this dropdown where you would need to select upgrade, right? The moment you would click upgrade.18:46It would ask you to select from where you want to upgrade, so in this case it's the manager and since you already have the image in the repository from the dropdown, you can select which version you want to, you know, upgrade to, click upgrade. So at this stage there would NO, there would not be any reboot. It is just installing this particular image.19:03On the since this is locally for your manager, so it doesn't really need to transfer the image anywhere. It's locally there, so it would just activate and come up with a new partition with this with this version. Once the upgrade part is done, that means you've installed that image, you have to, you know, inactivate that particular image. Now activation.19:24Is an intrusive process where it actually would reboot.19:44The device, right? So again, you come have to come back to actions, activate and select which version, and this version would be the version that's already there. Hit activate. Now the moment you would do that after like two or 3 min, you will lose access to your GUI because the SD WAN manager has reloaded.19:48And that's the time where you can actually go to the CLI to see what's happening or you can just wait for it to come back up. Now it can, on the newer versions it can take some time because there are a lot of dependent services that are waiting for the, you know, the NMS services to come up. It might take up to 60 min also.20:08I mean in my experience, I have not waited any time in a expected or, you know, a stable and we managed to wait up to that long but what the view says it's like it can even take up to 60 min. So once you've done the, you know, activation, so right here you've done the activation.20:25And done. So it's like once the V manage has come up with the newer version, you have to come back again and always and always set the default version. I've seen many customers missing this step, and for some reason if the V manage reloads or you know resets for some reason, it will.20:44Always and always come back on the default version. Now what happens is you upgrade it to a newer version, you made configuration changes, all the configuration changes would be specific to the newer version. The moment you fall back to an older version.21:04The configurations would be lost. The configurations that you did on the older versions would become active, right? So it's always advised that you always come back under actions and set the default version on the latest version that you made an upgrade.21:18Right, so this is an important step that you need to follow. With disaster recovery, so all the steps that we just saw were with without disaster recovery, but now if you have disaster recovery enabled, what you need to do, as I said, is take a little deeper before starting off with that upgrade process. It is just that you have.21:35To come back to administration disaster recovery and check the status. If it is anything like not success, if it is like import pending or export pending or download pending, anything other than success, you, need to, you know, make sure that you need, you have to pause that, right? And if it is for some.21:54Reason not in the right status and it is not pausing as you would expect, please open attack case, you know, and 1st clear this state, this this this transitioning state, make sure that your DR is properly paused.22:14Only on only then follow the upgrade process. If it's standalone, you follow the standalone process upgrade process. It's, it's a cluster, you would follow the cluster process that we will see in just a few slides. Once the upgrade is done, come back again here and start the replication process.22:33Unpause the disaster recovery and, you know, make sure that it lands in success again and that's where you would want your finally your disaster recovery as standalone manager to be at. So that's the only little thing that you need to keep in mind when working with a.22:49Disaster recovery enabled setup. From a validator standpoint, the steps will not change. Again, you will have to do installation. You need to make sure that the images are there in the repository. Once done, you will have to come back here for the installation.23:05Under validator and controller, you can select both of them together to make an upgrade, right? But if you will do that, the V manager will do the validator upgrade 1st. Once it is successful, only then it would move to the controller upgrade. Here we are only doing for validator. Again, you'll have to do an upgrade 1st.23:24Click upgrade, you can actually activate and reboot at this stage itself. If you are in a maintenance window and you are, you know, you have a go ahead to reboot your validator, you can do that right here. But if not, you can just click upgrade, it would.23:42Install the image on the on the on the validator and not activate but install the image on the validator and then you'll have to come back here and activate the image that you've just installed on the.23:57On the validator, right? So you just have to come back here, hit activate, select the image, click activate, and this is again going to be a reboot of your validator. Once done, again, you'll have to set the default version because the concept remains the same if for some reason your validator resets.24:14It will come back again on your default version. So you can check what's my current version, what's my available version and what's the default version right here. Always send the default version to the the latest version that you have. Same goes for controller, but there are certain guidelines and the best practices that you need to follow. Because Cisco recommends that you should.24:31To be upgrading half of your controllers 1st. If there are let's say four controllers upgrade two and wait for at least 24 h.24:51To observe the stability of your network, everything is ok. You go about upgrading the rest of your devices. And here we I mean same as in validator and your few of the cases in manager, we always expect you to upgrade via GUI.24:59Certain prerequisites that we recommend that you should do when you are planning for an upgrade of your controller. Controller in previous terms is V smart. If if we were to put it that way, you always should monitor the, you know, capture the CPU and memory utilization of your controller. Check the OMP summary.25:17Well, you can do it from the manager or from the CLI OMP peer state notification for any peer state down notifications if you have. Back up the configurations. I mean even though it would be more or you know from from a government from a template.25:36From the managed configuration should not be a NO issue, but still always take a backup of your configurations. Check the OMP and control connection status before making an upgrade, and what we should also do again it's recommended that you take ten or 15 of your managed devices and check the OMP state there. That after you made an upgrade.25:53You should also check, ok, what was the state before and after making an upgrade, what's the state now if it matches or not? So these are certain things that you should always keep in mind because if you make an upgrade and then say ok after an upgrade, this OMP peering is not coming up or this BFD session is not coming up. Why? Because certain T lock I did not receive from.26:13From from the from the controller. So all these things happen, but if we know exactly what was the state before my upgrade.26:33Okay, probably I was never receiving that T log and that's why I was never creating an IPsec session towards that peer, even before the upgrade, so always be and post check validation is important. From the action side of things, nothing changes, you know, upgrade.26:40Activate and install activate and set default. These are the three processes that you would also need to do for your controllers. So just putting it into context together. So this is from where on software.cisco.com you will be downloading the image. So.26:55Go to the SD WAN software update section in order to download images that you can make use of while you want to upgrade. Sometimes what we do is we go to the SD WAN software or sorry, the vmanage software for downloading an upgrade image.27:13That would not be there. So you'll have to come to the software update page, select the right image that you want to download. Let's say e.g. I'm looking to download twenty.eighteen and.two version and here you would see the version that you see is for your.27:28Controller and validator and the V managed version is for your SD WAN manager, so download both of them together if you can and add it to your V manage repository, the way that we just saw, right? So here in this case I'm using on the software version side, I mean locally on the SD WAN manager.27:43So I'll just, you know, here I selected a wrong image. It's a quecom image for your bare metals versions. So I will have to select the right image and the moment I can do I can just upload both of them together, right? So once this is done.28:03By the way, important thing is right here, please do not refresh your page. Even if it's all blue here, it would say, please wait, the managed server is processing the image. So, please wait for it to be successfully uploaded. Once done, it is ready to be used and, you know, activated on your end devices. So I will.28:19And now just go to control components and here in this case 1st the manager.28:39And you just have to select the software image action, you are upgrading, you select from the manager, select the version and start upgrading. What I want to show you here is the is this log section, right? So where from where you can check the logs.28:44Is right here under action, the small little button, if you click, it would open the logs section and I am really, you know, excited to show you the, the, the details of the logs that you would see while making an upgrade. So it is it is just starting the upgrade from 20 1542 to 2018. So this is a direct.29:01Upgrade, right? So.29:21It it is it is very verbose and it does all the prechecks and the post checks in in in great detail, right? So here we've just done an upgrade in terms of installing the image, but we will now come back and activate the image, right? So once we've done the activate so I'm just quickly skip.29:24I click the activate button and on the SD WAN manager, as I said, you can check for the logs, so how you can check the logs is show log TLF under bar log NMS v managed server.log. So this is the place where you would see all the logs. And what I want to quickly show you is the.29:44The logging on the SD WAN manager, right? So all the logs activities that are happening in the background will be shown up here and see here. So it is executing a pre check.30:03Script where it does a lot of checks in the background if and if there is an error at any stage, you know, it would, it would either try and resolve that. If not, it would stop the upgrade process right here.30:23So this is very detailed and when let's say if you land up into any kind of issues while upgrading, take the copy of these logs and share with Cisco tech I mean that would greatly help. So, once it is done, as I said, it would take some time for all the services to come up, so you can always, you know, check.30:35Like the status of the services running is with the help of request NMS all status command, right? And the main app server that should be running for your GUI to be active is the application server and it should take at least be stable for at least for 02:50 odd, seconds. Only then you will have.30:55The access to your application server. Now, the moment it will come up, we will see another very interesting thing that has happened on the newer versions. By the way you see for the software type of versions, the image is active, the newer versions, but it's still not default here.31:14So what I want to show you is when the application server is up and running, right? For let's say about 50 60 s, and if you would refresh your manager page, you will see an option that, ok, system is booting and you can actually get into the diagnostic center for checking more logs just in case your V manage is taking a lot more than.31:3160 min what we've stated, you can get into diagnostic center and check what's happening.31:51So you can enter the username password the same as you have for your net admin username and password, you log in and you can check the logs there on the diagnostic center itself. So you can check the overview, you can check the different services that are probably stuck because that's why you are coming into diagnostic center to see.31:56And you know probably you can export the status of the the errors as well. You can export service status, system error status and share it with Cisco tech, ok, this is the problem that we see while making an upgrade. So this is something really cool that we have for you to check while, you know, you're making an.32:14Upgrade and that's not going as as intended.32:34So, once I come back in, I'll have to make the software version as default. So that is something that we know and same goes visual validator and controller, if you see here I had made the upgrade together, but the V manager is waiting for the validator to come up 1st, only then I will go for the controller.32:37Upgrade. So once this is done, I mean the logs are here in again in very verbose fashion, so once it is done, you can always come back and see what's the stable version, what's the stable state where I can see the default version, the the the the upgraded version all should be in the.32:56Same version train, not like my active version is something else, my default version is something else. So CH upgrade, that was from the controller side, from the CH side of things, I just also quickly want to show you that.33:14How you can go about doing it. So it's like from a maintenance upgrade, vanage, select your, you select your device, click software image action and then these actions are intuitive enough to tell you what you want to do because the concept remains the same. If you just upgrade, it would just install the image on the end device. It would not reboot, but the moment you hit activate.33:31It would reboot your edge edge device. And again, all the logs that we.33:51Can put up, you know, these are again on the V manager space. We can you can check those logs for any issues. But what I want to, you know, bring your attention to is from a compliance and conflicts perspective. So this is something new that we've added up where we always recommend you to go and check for any compliance and conflicts that are present on the.33:55Manage, so you go to monitor and click on compliance and conflicts and you will see that are there any compliance errors within your fabric? So here I've got one compliance error where the CH is on you know a software version that does not match or is supported from the SD WAN manager.34:16So it is on 1715 and zero four and apparently I'm on 2018 to one. So it is throwing me an error here and this is the compliance thing that I need to 1st resolve or resolve so that you know I do not land up into unexpected issues.34:33Another thing is upgrading using workflows. This is also something new that we've added. So you go can, you know, manage your upgrade via workflows both for your CHS and for your control components. So you go to workflows, search upgrade and you will see either firmware upgrade or device software upgrade or a controller component upgrade. So let's.34:50Say you select a device software upgrade.35:10You select your device, you select the version, and the the the best part is you can schedule an upgrade now. It's not just you have to do an upgrade right right now. What you can do is schedule your upgrade when you want to do and based on the device time or I mean you can also select devices based on the network hierarchy or tag.35:12But more importantly from a device time standpoint, you can.35:32Where that particular device is in the geography, you can actually schedule an upgrade and, you know, stay free from in, from from from taking a certain action at certain time. You can just schedule it and it would, it would make that upgrade whenever you are scheduling that particular software upgrade. So this is something.35:36Interesting that I want to, you know, highlight here.35:56So that's from the standalone V manage side of things and the CA side of things, but now I'll pass it on to start off with the catalyst upgrade from a manager cluster standpoint. Thank you, thank you yourself. So can you go back to the previous slide, please?36:00So I just want to start with the one statement that like SD van, does not fail because of the van. It fails because the controller can't keep it up. Next slide please.36:20So when the vantages scales and traffic explodes, your SD WAN manager gets hammered, especially when you enable DPI and C flow telemetry. And that's not an optional in the real deployment. That's a baseline.36:35And and a single node becomes a bottleneck fast. CPU spikes.36:55Database lacks and delayed visibility. And suddenly your centralized controller is not a centralized anymore. So clustering is not a luxury.37:01Yeah we cannot hear you, maybe you can Can you hear me now? Yes yes. Okay. So what I was saying is that so.37:19So in so clustering is not a luxury, right? So it's a survival, right? So when in a clustered environment, the workload is distributed. So that gives you scale performance, stability and not a guesswork. So, but here's when people get it wrong, right? So the cluster is not.37:33Only as strong as it communication, right? So you must have a dedicated low latency interface in VPN zero for inter cluster communication, not shared, not overloaded. If that link suffer, your entire cluster becomes inconsistent. And now you have created.37:53Your own outage and then it comes the backbone of it all.38:13Quorum. In cluster quorum is very important. This is why you always deployed an old number of nodes. Minimum three, because the system only works if the majority agrees. Three nodes, you need two alive.38:18In five node deployment, you need three. So if you lose quorum, then your database go into read only, so there will be NO rights, NO changes, and NO control. So the takeaway is simple. Clustering is not about adding nodes, it's about ensuring your SD WAN control plane can.38:34Scale, stay consistent and survive failure. So without it, you only, you're, you are only one spike away for failure and only a one bad design decision away from losing the the control of your entire overlay network.38:54Next slide, please. So designing a cluster is easy, but keeping it stable during upgrade is when most customers face. So we're gonna walk through how to upgrade in the right way using the CLI.39:13Can you move to the next slide, please? So we have we we divided the clustering upgrade using CLI into four steps. The 1st step is like you 1st you have to take a snapshot of your.39:32We manage cluster nodes because if you plan to do the upgrade, you have to take a snapshot 1st. So if something bad happened, you have something in order to restore your the connectivity of your controllers. You also need to take a backup with a configuration database.39:46And we're gonna go all these three all these steps which we are right now discussing in step one in detail. And we also have a Ora tool, a sure tool, which is basically a CLI tool that you can run to just to see if your nodes in the cluster are in healthy state and they are ready to upgrade into their.40:03Desired version which you have chosen.40:23And then once you got the order report ready, then you can copy the image into all the nodes in your V managed cluster, and then you can validate in the cluster nodes that the image has been copied successfully and it is available on all nodes. So once you have done that, you can move into the step two.40:25In the step two, when you decided that on this day you want to do the upgrade of your cluster nodes, you have to perform some prechecks. Those prechecks includes to verify that the have the control connections with the, with the controllers, including the all the nodes in the cluster that you have.40:45Up the OMP pairing with your controllers is up. You have all the applications that is being visible in the cluster node environment in your V manage deployment. So this is the step two. This is just to take a.41:05Snapshot before doing the upgrade. So once the upgrade is being done, then you can verify and after doing the same steps again which you did in step two after performing the step three. In step three, what you have to do, you have to perform some CLI commands to perform the, the upgrade of when you have all.41:21Already copied the image into the node in the in the in the cluster deployment.41:40So once you perform the CLI commands and the nodes is being upgraded in in the cluster deployment, you have to make sure all the services which is being available on all the, on three different nodes when we talk about three node cluster deployment for the V managed, you have.41:46To make sure that all these services are being running on all the nodes.42:06Next slide, please. So in step one, so we have already talked about that like you have to take a snapshot and this is this command like when you want to take a configuration database backup, you have to take it on the node where the.42:11Configuration database is being service has been running on that particular node. You have to run that command on that node, take a backup and save that configuration database into your some server.42:30Then you perform the Ora health check. So the Aura health check is basically the CLI tool which is being developed by Cisco to perform 30 checks for the standalone vmanage deployment and performs 36 prechecks for the cluster deployment.42:42That, make sure that like your nodes are ready and if there are any risks or then it will also give you the recommendation how you can fix that if if the tools find some issues so that when you.43:01Try to perform the upgrade, then the upgrade goes smooth and failure free. Once you got the Aura health check report ready and it gives you the green signal, then you can copy the image into that particular node, into the home admin directory and you have to copy that image into.43:18On each we managed server node, then you go to that.43:38Home home admin directory on each we manage node to make sure that the image is being copied correctly. You can also perform the MD five check on each and the on the image itself just to make sure that there is the image is not corrupted. So once you copied that, you can also run that.43:45Show software command on each we managed node in the cluster to make sure that there are NO more than two image available on the, on that particular we manage node. If there are more than two, then it is the recommendation from us is that you remove one of them to just to keep.44:04Two image node on that we managed node so that the upgrade is goes smooth. Next slide please.44:24So, so in step two, we perform the pre upgrade validation checks. We check the control connections, we check the routing on the SD WAN controllers, we check the OMP pairing, so all these you know the normal show commands just we just do we verified or.44:33Validate that we have the control corrections available with the V manage with the V smart and the OMP pairing with the controller is up. So we take all these show commands on on the controllers and also on the V manage just to take a snapshot before performing the upgrade.44:53Next step next slide please. So in, in step three we because in step one we have already copied the image on all the V managed nodes in the cluster. So now we install that image. So once we install that image by using this request software install.45:10Command. So this will make.45:30Make the image available in the show software command, meaning that now we can use that particular image and in that command request software you know activate and then we can provide that image. Copying the image does not make that image available for.45:33The upgrade, you have to install it. Once you copied it, you have to run that command request software install and then you can run that command to activate that image for that particular node. So once you activate that image on all the nodes, then the all the v managed nodes in the cluster will reboot it, and it will take around 15.45:53Minutes to to come up and all the services will be available. You have to make sure that when all the notes comes alive, you have to and and the services are available. You have to run that command request software upgrade confirmed, you know, and it usually after 15 min, the, the, all the services.46:13Are available and then the, then this command will make sure that the upgrade, the all the nodes in the V managed cluster are upgraded successfully. But in the cluster environment, once the image is being upgraded and the and you run that command, you will find out that the app server and the configuration.46:33Database server database server is does not start automatically. You what why why is that? Is that it you have to upgrade the database manually. Like if you are upgrading from twenty.9.two to twenty.twelve or twenty.fifteen, you have to upgrade the.46:53The configuration database manually by using that command, request NMS configuration DB upgrade. So this will upgrade the database because we are moving from one major release towards the other major release. So if you are upgrading from twenty.9.two release to 20.47:13Three.nine.five or twenty.twelve so or twenty.fifteen or twenty.eighteen, you have to run that command manually to once the configuration database is being upgraded successfully, then you can see that this app server and the configuration database self.47:33Service will be will start and it will be available for the, for, for the V manage nodes. Next slide please. So once, so once this, the, all these config all the services are available, then you can.47:51They Perform the, the these show commands or run these show commands to make sure that the control connection is being established between the controllers and between the vhs, the routing is correct and you are receiving all those routes or the T logs and the and the IPsec tunnel is being established between the devices.48:11Next slide. So this is, you know, so these are the steps you have to follow if you perform the CLI based upgrade in the V managed cluster environment. So basically CL CLI gives you the control, but it also gives you the ways.48:32To break things because you are doing the CLI manually. So now we're gonna look how the upgrade, you know, has been done through the V managed GUI. So in cluster, in, in a clustered SD WAN managed deployment, your critical services.48:50Are basically the configuration database coordinating service and the messaging because all these services are all distributed because when we deploy the V manage in a cluster environment, we specify that which service will be run on which node, right? When these are all.49:10You know, very important services in a cluster deployment, even in a single node deployment also because the configuration database service will hold all the configuration which we have which we done through the vmanage controller. So that means upgrades are NO longer a single node activity, they impact the entire cluster. So.49:30That's where the upgrade coordinator service comes in, which is basically introduced for standalone starting from twenty.ten and extended to cluster starting from twenty.twelve.four. It is basically not an upgrade tool. It is basically.49:50It a service which runs in a containerized environment on the V managed node. It's basically controls it it is you can say that a control layer that runs prechecks, validates readiness, and tracks every step of the upgrade.50:10Right across all nodes on your behalf. So now.50:30You have to make sure that like starting from twenty.twelve.four, we have this upgrade coordinator service which do all those prechecks and validation on your behalf, right? When you are performing the upgrade through GUI. And Cisco also recommends that.50:34That you perform the upgrade using this using the GUI or UI based because it enforced the validation reduced human error and gives you the visibility you simply don't get when you do it through CLI.50:54So post upgrade, the system does not assume success, right? It verifies it. If something breaks, you cannot trouble, so you can troubleshoot, right? So retry and if needed, the system rolls back automatically in activation and after activation failure. That's your safety net. That is being done by the upgrade coordination.51:11Service, on the starting from twenty.twelve.four. But here is the you know the risk During the upgrade, nodes can run in a mixed software version because you are doing the upgrade in a cluster environment and that's a temporary state, right? So, but it can.51:31Introduce a service inconsist inconsistency if you are not careful and when things go wrong and they, and they will at some point your source of truth is the upgrade coordinator logs because that is a service which is doing all those control to.51:51To make sure that all the nodes in the V managed cluster you know upgrade successfully. So you can go into the shell, check the upgrade-service.logs across all the nodes in the V managed cluster, and that's where you will find the real story. What is happening in your cluster, why the cluster is not being upgraded.52:11So the bottom line is the upgrade coordinator is not an optional service that you can ignore. You, you can check that file to see why your cluster is not being upgraded, why it is failing, why a particular service is not being you know installing. Next slide, please.52:31So, so from the GUI perspective, you go to the maintenance, you upload that image, right? So in this environment or in this, you know, example, we are upgrading from from one major release to the, to the minor release, meaning that we are upgrading from.52:49Twenty.fifteen.two to twenty.fifteen.three.53:09Next slide. So before starting an upgrade, we mentioned that even we are doing it using the GUI method or the UI based approach, you have to take the configuration database backup by yourself. It is not being the process.53:14Added in in in the in the GUI approach or not added in the upgrade coordinated service, you have to run that command, you take a backup, you store it somewhere safe. So if something bad happened during the upgrade, and the upgrade coordinate service is not able to roll back, then you have something in not.53:33You have to restore your managed cluster, so so that is the 1st step you have to take it even if you are doing it the upgrade using the good UI based approach. Next slide.53:53So during the cluster upgrade, based upon the GUI, you install the software, right? So you install the software using that on the user interface, right? And you just install it on one single node, any single node, if it is a three node cluster, you can just.54:03Pick one and install it there. And then we manage is smart enough that will automatically install across all the nodes that is part of that vmanage node. So you don't have to install it on every single node in the, in the V managed cluster. You just pick one and then we manage install it on all the chrome.54:23And then you, you know, pick up the, and version which you want to install. Next slide.54:44And once it is being installed, you can see that it is being installed on all the nodes in the in the all the V managed node in the cluster. So we just picked it before and the one node and now it is being installed on all the nodes in the cluster.54:53Next slide. So you can also verify that the like once the the software image is being installed, you can go to that maintenance and the software upgrade in the vmanage GUI to see that that particular version is now available. You can also run that show software command on all the V managed node that is part of.55:10That we managed cluster to see that that particular version is available or not.55:28Next slide. So once the vmanage, have that particular version available for the upgrade, you can just go to that maintenance software upgrade for that particular version and then activate and then you choose that particular version which you want to upgrade, right? So once you.55:35Select that particular version, then the upgrade coordinator container runs in the background on all the cluster nodes to manage the upgrade process. And now we're gonna talk about how what what happened. So here you, so next slide.55:55So now here you can see that once we, you know, activate the version, we can see that this upgrade coordinator service is being activated and it will now perform all the prechecks and and and do and and do the upgrade for all the nodes in the, in the cluster.56:11Can you go to the next slide, please? So now, you know, they, this is where this upgrade coordinate upgrade coordinating service gets interesting. So during the cluster upgrade, the V manage or the system automatically elects and one V.56:30Manage node or a SD WAN manager node as the upgrade coordinator leader.56:50That node becomes the brain because it tracks progress and orchestrate the upgrade progress process across the entire cluster, but don't assume stability, right? The activation order is random. Even the sometimes even the leader, a leader itself.56:56Health can get upgraded 1st and if that happens, then the cluster does not panic. It simply elects a new leader and continues until that builds in resiliency, right? So if that particular leader got upgraded, then the the remaining two will elect the leader and then they will.57:16Yeah, you know, keep track the proxies. So now if you want a real visibility, you don't guess. You go to the logs, every node has the upgrade service log file available and you can access it you by going into the vision. But here's the key insight. Not all logs.57:36Are equal. The leader node gives you the full picture, detailed task level visibility across the entire cluster. So follower nodes, only partial node specific details will be visible. So if you are troubleshooting, always find the leader. That's where the truth is.57:56Next slide please. So here you can see that, you know, you can find all the logs about, you know, for the upgrade service log when the upgrade we when we activate and if you go to that you know the next slide please. So this is where this container management.log, if, things don't.58:17Come back, e.g., the upgrade container does not start. You can see that in the V managed GUI. You can go to that container management.log file to see why the container is not being started, you know? So bottom line is this file will give you the full visibility about what is happening with your, the cluster upgrade, why the.58:36That particular upgrade you know coordinating service is not being installing and if it is being running why the which you know on the leader node what what is happening with the other, you know, followers in the upgrade cluster for the for the upgrade process.58:56Next slide please. So here you can see that when the cluster is being upgraded successfully, we can have this success message. Next slide please. So don't forget like once the cluster is being upgraded, you have to, you know, make the default version of.59:14The version which you select for the upgrade because this is very important for some reason any node got reset or crashed or, you know, got rebooted it if it is this not being set as a default version, then it will go back to the old version and then it will give you the lot of, you know, inconsistent inconsistency in your cluster.59:33The deployment. So next slide.59:52So now we're gonna talk about like if we have you know the V managed cluster deployment in a disaster recovery, so in a disaster recovery scenario, we are responsible for the, for both the active and the standby manager node. If you miss one.59:56And you have just created, you just created the inconsistency in your cluster. So next disaster recovery, don't assume it's fine. You have to verify that. As mentioned before, you have to make sure that the the disaster recovery status reflect a success you know.01:00:14Status in the in the in the if it reflects apart from success status any other state, meaning pending or or you know the download pending or export or import, you have to 1st fix that like the the the disaster recovery reflect the success state. Once.01:00:34It is being showing the success states before doing the upgrade, you just have to make sure that you pause a disaster recovery and then you follow that, you know, the the CLI based approach to upgrade your one you know the active cluster and then the standby cluster one once both the cluster has been upgraded, then.01:00:54You can, you know, resume the disaster recovery again. So.01:01:14The key point is you have to remember that before doing the upgrade, you have to pause the disaster recovery. Once you upgrade the the all the nodes in the cluster both active and the standby, once it is being upgraded successfully, then you can, you know, resume the disaster recovery. Next slide.01:01:19So in the multi tenant deployment or environment, the, the, the upgrade, we have to make sure that and we have to understand that the multi tenant environment responsibilities are clearly split, right? So we have any multi tenant we have this provider and the.01:01:39Tenant admin, right? So the provider admin owns the controllers and they handle the upgrades of the manager or validator or the or the or Cisco SD WAN controller or the core infrastructure. Tenants don't touch the controllers, right? So the tenant admin admins on the other hand are responsible for the.01:01:59Their own edges to perform the upgrade process. But the only using the software image provided by the provider, right? So the provider has to upgrade to our.01:02:19To copy the image into that Cisco SD WAN manager and the tenant admin can use that images to perform the the upgrade. So before that, the workflow stays the same, same steps, same process, just one restriction that is separate based upon the role.01:02:30So the provider can do the upgrade, can perform the upgrade of the controllers, and that the the tenant admin can only perform the upgrade of the vantages. So there are, you know, so that's, you know, how you can perform the upgrade if you have this, you know, we managed cluster in a multi tenant environment.01:02:50That's all from my side. So now it's time that you can, we can have some feedback from you guys. The next slide. Yeah. Thanks. By the way, apologies for spilling 5 min over.01:03:09So we got a lot of cover cover you know different topics that we need to, you know, highlight. So, we try to, you know, compress it into a 1 h session, so we hope that it made sense. So please tell us what you think, how we did today and.01:03:25We would, we would take this feedback and try and improve on the next ones. We've got few more important questions to ask. By the way, the survey would be up and running even after the session is over, so you can always come back and you know give us the feedback.01:03:42What we would also like to know is in terms of automation, do you leverage any kind of automation within your SD WAN deployment and from the operation side of things? Because the next series of of of expert inside series sessions we are planning to, you know, discuss around those aspects.01:04:02Next as well.01:04:22Okay, so would you rate your organization's interest in adopting as a code automation principles for your SD WAN environment? So do we see that as, you know, it's not a priority maybe today, but are you in a learning phase or exploring phase or it's a high.01:04:27Priority and you think that yes, we, we need to work on.01:04:47On on on automation or as a code principle. So if you do, we would, we would like to know that and if you think that a Cisco customer success specialist should reach out to you to discuss SD WAN as a code benefit in some detail.01:04:52You can let us know there and if yes, please fill in your email address here and the the respective customer success specialist will reach out to you for.01:05:12Starting of the discussion on SD WAN as a code. So with this, we don't want to hold you back any further. We've got few references that you can always, you know, go back and and refer. We also have, you know, a next level of discussion from SD WAN upgrade scenarios. So if that session is gonna happen on 1 April, so you can go and.01:05:28And register it.01:05:48From this QR or from the registration link. So thank you for taking time out and apologies again for spilling over an hour. I hope the discussion made sense and we'll see you in the next one. Thank you and have a very good day ahead.01:05:50