Cisco ACI Upgrade, Conformance & Software Strategy

Hello, good morning, everyone, and thank you for joining the Cisco ACI upgrade conformance and software Strategy Expert insights.00:11My name is Don and I will be your host for today.00:20We will be using Slido with today's webinar and we'd love you to participate with us. You can join the Slido by scanning the QR code shown here or by navigating to slido.com and entering the code ACI six and the passcode ACI six. Joining us on Webex, you can also simply click on the three dots at the.00:23The bottom of the right and select Slido or select the ats option in the bottom and select Slido from that list as well.00:43Today our data center experts, Irene Alberson and Dave mccarlan will be leading the session. Irene has experience working in the data center field specializing in Cisco Nexus platforms and Cisco ACI technology. She is a customer success specialist with Cisco's customer experience center and her current technical focus.00:52Is on Cisco ACI.01:12Dave joined Cisco IT operations as a server and storage administrator and moved over to AS as an NCE specializing in SAN and then customer support EFT beta team working with customers and the BU on Nexus five K and UCS. He spent six years delivering product training and consulting on ACI UCS.01:15In MDS and other data center products throughout APJC.01:34And he also does hold a CCIE in both SAN and data center. This is the 1st in our expert insights series for the quarter all about software strategy mastery with ACI and Nexus dashboard.01:40Here's the schedule of upcoming events. We do encourage that you scan this QR code here and register for the other sessions. I will also post the link to do so in the chat real quick.01:56So feel free to scan that QR code or click on that link and register for the rest of the sessions of this quarter. As I mentioned earlier, we are gonna be using Slido. So I'll display the Slido info one more time. Again, join us@Slido.com with the code ACI.02:13Six and the passcode ACI six and then again you can also join via Webex by clicking the three dots at the bottom and selecting Slido. So let's get warmed up with our 1st Slido poll and see where everyone is joining us from today.02:33Again, you can participate by scanning this QR code or and or selecting Slido under the apps in Webex or visiting slido.com and entering the code ACI six and passcode ACI six. I'm gonna go ahead and.02:52Share my screen so we can kind of all see where everyone is joining us from today. We've got Washington, Maryland, Tracy. Is that florida? Hey, I'm in Florida. Very cool.03:07Is that Washington DC or Washington state? Got a couple more people. Reston Virginia, very cool. Maryland, so a lot up in the Northern Virginia, Maryland DC area, Washington State, very cool.03:25Los Angeles out on the west coast. Awesome. One more person typing, we'll go ahead and let them finish and then we will proceed North Carolina. Okay, very cool. So people from all over Minneapolis, all over the the country, very, very cool.03:42Oh, from India. Awesome. Welcome in San Antonio. Very, very cool. All right, so we'll go ahead and proceed. I will now turn things over to our experts with the all that housekeeping out of the way.04:01So Irene, I'm gonna go ahead and pass you the ball and let you take it away. Thank you so much then once again, hello everyone, thank you for joining session. So today's webinar is divided into three parts, so we're just, you know, divided into three parts for better.04:19The in a structure. So 1st we are going to talk about ACI upgrade process, right? We are what steps we need to conduct for successful ACI software upgrade. Then we'll switch to a demonstration part, right? Show me how my colleague Dave Dave mcfarolling will take us through the demonstration.04:39Part of this webinar. And then with time left, right? So we'll dedicate a session at the end of this webinar for your questions. We'll answer your questions and then we'll talk about 211 consultation offers here.04:59You know, available with our services. So let's go ahead and get started, ok? So the 1st what we're going to address is preparing the ACI upgrade, right? So what steps we need to take to.05:15Approach this project.05:34So starting with Cisco ACI version six.zero, Cisco stopped differentiating, differentiating between a short lived and long lived releases. So instead, all major releases are treated equally with a consistent life.05:37Cycle models, so as we can see here on the slide. So the software release trains are categorized into two models, feature releases.05:56F releases, right? These releases contains new features and maintenance releases were indicated by M M releases, right? Those contain only bug fixes. So the life cycle for each major release spawns about.06:07Four years or so, including development, maintenance, extended support and tax support phases. And this approach replaces the previous model where releases were explicitly designed as either long lived releases, right? Which supported for a.06:27To approxim approximately 18 months with frequent maintenance and the or short lived releases, right? So we supported for about six months with limited maintenance. So the takeaway from this particular slide is that the differentiation between short lived and long believed.06:47Releases NO longer in practice, starting with Cisco ACI release six.zero.07:07So, there's also a concept of a BU recommended releases also referred to a start releases, as we can see it here on the slide. So let me.07:14See that was trying to do. Okay, so as we can see here on the slide, right? So we can see that six.one.5eM, right? Maintenance release start and six.zero.nine EM started releases.07:28Also that be you recommended. So when you go to Cisco software download page or repository, right? So when you navigate to application.07:48Application centric that infrastructure location, right? So you will be able to see those suggested releases. Ok. So now how do we choose a release, right? So we as I said, when you navigate to cisco.com, you can see.08:00See there's a list of releases. So the question is how do we choose? And of course the 1st step in conducting successful upgrades, right, is to choose that ACL software release that is right for your environment, right? For your particular environment. So in.08:19And to help with this step, Cisco put together several online tools that are available@cisco.com or, you know, you can search search or, you know, we'll post that.08:39Links on chat. So, and, those tools as I said available@Cisco.com for customers just like you, right? So you can use that or refer to those tools during the preparation of during the upgrade and then for post verification as well. So the 1st tool it's.08:53We call it tool as well, right? So it's a checklist, upgrade checklist, and you can see that the URL, HTTP column format/CS.com forward/ACL upgrade. You can also, you know, simply google it and it will take you directly to this particular tool.09:13So I have it, available on my screen and let me share my screen real quick.09:32And I'll show you that screen, right? So the 1st the 1st tool, right, that way checklist, so I can also share that so you can follow along. I think you can you should be able to.09:38Oh, I probably not be able to share it right now because my chat window is unavailable. But anyway, so you can google it right and it will take you directly to this particular checklist. So what is the checklist, right? So once we navigate it here.09:54This, you know, this checklist right to list all the steps we need to take during the, you know, preparation for ACI. So you can see there's the checkboxes right for this particular table and it's not necessarily inter interactive, right? Checklist is not you cannot click it.10:14But you will be able to print it and just, you know, check each checkboxes you know you move down this particular.10:34A list. So and you can see here right so step number one pick your target epic and ACI switch version. So both appics and ACI switches must be upgraded to the same version. So I will not be able to, I mean, I will not read the entire table, so you can just, you know, navigate this checklist, you know, make a.10:41Bookmark of this, but you can see here for each step, right? So the recommended steps what you need to do, you can see that links to those documentation is provided or available and you can click on those links, right? And this will be updated up to you know to.11:02Days date.11:21So that's the checklist, right? So now, 2nd, right, 2nd, let me stop. So 2nd tool, it's appic upgraded downgrade support matrix. That's actually answered the question how we're going to choose software and which software.11:22That we need to select. So go ahead and share my screen one more time.11:42Right, and this tool is that is Cisco Epic upgrade downgrade matrix and now provide this URLs, direct URLs after this particular session. So here when we arrive at this tool, right? So we see that matrix which will help us to.11:48Select software. Let's just let's just, you know, go from e.g. here. So, let's just, you know, assume that we have a5.three.two, right? The existing software right now or release running in your environment. And let's just say that you want to go.12:07To recommended or the latest release, six.two.two e.g. or six.two.one?12:27So when you select target release, right, the tool will tell you, ok, so here's the current release, your starting release, that is the target release, and then recommended path, and you can see here that we have intermediate, intermediate hub as well. So that is a recommend recommendation. So if you want to go.12:35From five.three.two to six.two.one, so you will need to go through the intermittent hop as well. So show all paths, right? And you will see alternative paths as well. You may ask why do I have so many alternative paths, right? Why you you know that you know how do I select which one? So the.12:56Opportunity pass here as well.13:15So you can select because each customer environment is unique, right? Even though you run the ACI solution, but you might have different hardware, maybe new different features involved et cetera. So each customer environment is unique, so therefore, you know, those we have alternative paths as well available, and it's so.13:17Are also based on, you know, e.g., open, issues or open caveats. So here's alternative paths as well. So you can go through each.13:37Release intermediate release and check for release notes right for for each and see if there's any open caveats or something like this or which one is which one fixed or which one is still open et cetera. So you can navigate directly to release notes, right? We can go to from here, it will take you to release notes.13:49The two here as well. So, but from the dropdown menu.14:09You can see all those releases available, right? So you don't have to navigate through a URL to each release, you can just, you know, from dropdown menu, you can select release you, you know, you would like to learn more about, you know, open cases et cetera.14:13So let's just say which one we selected the recommended six.one.five. Let's just stick with the six.one.five. We navigate to release note, we select six.one.five, that is our intermediate intermediate hub, six.one.five.14:33And we can read. So that's you know the step, it's very important step to read release notes for each, for each release, right? Even though if you you might have one intermediate hub or you might have two, right? Depending on your existing software release and target release you're.14:51You wanted to accomplish. So you can read here, right, for release introduction, right? Release overview and then you change behavior. This is important as well, resolved issues, open issues, and then you can just, you know, go through open issues and see what is expected.15:11So it's something like in the known issues as well. So even though you see them listed here, it's not necessarily that will apply to your environment. You can read them and see if this is something, you know, it's applicable in, you know, maybe not those features is not available in your your environment you have.15:31Not configured yet, so this is in the endpoint security group is on demand mode and NO zoning rules are deployed for the endpoint security group. So this is the feature you were.15:51Option you configure right so for next level. So that's not necessarily could be, you know, applied to your environment, but if it's applied to your environment, you can, you know, read more about this bug ID and whether or not it's, you know, could.16:04Implicate your project. Same here for ACL leaf switch experience crashed due to event management process consuming, right? So you might want to, investigate this further etc. So that's the recommendation, so when you navigate to this upgrade, downgrade matrix, right? And you select your existing software.16:19And target software, right? So you just read those, release notes. That's very important. So you release read release notes for intermediate hub and for the target release as well. So, and as I said, the same tool you navigate to here.16:38And just, you know, select the release from a dropdown menu and read all those information. So you can of course you can approach it you know individually, right? So you can approach your if you're going to do this project AC upgrade by yourself.16:56Right, so that's, you know, tools available for you. We also offer, you know, professional services which will be able to assist you with this with this project, right? And professional services will take in consideration your specific environment, right? And based on available switches, hardware.17:16And, you know, existing.17:36Existing environment, right? We'll be able to prepare you for successful AC upgrade. But as I said, this approach you can take, you know, if you are going to you know complete this project with your team. So we can also see here direct only, right? So if you want to see.17:39Direct only and you can see that from from a5.three.two, right? So those available paths for direct only. So once you identify which one is, you know, for you right for your environment, you know, you can proceed with the next step. So.17:58Let me go back to my presentation, right? So we have, checklist, right? Easy upgrade checklist, right? So we're gonna go through that check checkboxes. Then upgrade, downgrade support matrix, right? We're going to refer to this tool so we can select paths, the correct pass. It could be.18:14One intermediate hop, it could be two intermediate hub, right? Depending on your existing software and target release how you want to approach it. Okay, so release notes, right? So the next step with and we already covered that one, right? So once you select intermediate hop over your target.18:34So you will read, you will need to read release notes for each.18:54In a release in your particular path. So we already covered that too. So the next tool is AC upgrade preparation, best practices in troubleshooting. So let me share my screen one more time. So I already have it up in here.19:00So release notes we already covered. So, change behavior will cover. Okay, so this tool, right, or through other article, ACI upgrade preparation best practice and troubleshooting, and this is available@community.cisco.19:18Dot com, right? So this article is written by Cisco subject matter expert, right? And, you know, post it on the community, Cisco community. So in this very thorough article, so you can read about them read best practices, right? So everything we will talk about today.19:38Upgrade preparation, known issues, behavior check overlapping VLANs, check rogue endpoint feature etc. So this is very useful article, right? And we recommend to get familiar with this article, right? So you can read about, you know, every step.19:58Which, you know, needs to be taken during the ACI project upgrade. So review behavior changes in the targeted version clear all faults, so we'll talk about this in upcoming slides as well. Check NTP status confirm AP classes fully fit, right? So everything what we are going to talk about today in the.20:18Written in this particular article. So make a note of this article, right? And.20:38Just refer to this article prior to your project and we'll share that link to this tool as well. Okay, so we covered upgrade preparation best practices and troubleshooting. The next, is the Cisco applicant.20:44In upgrade downgrade guide. So why this, it's also as.21:04Documentation, right? So this is a documentation share ok so this is the documentation as well ok so this is so what is what what is.21:12You know, useful about this cancel. So what is useful about this?21:32Article, or documentation, right? Upgrade to downgrade guide, because when you navigate to this article, right? Or documentation, it will cover specifics for each release. E.g., you can see here session section upgrading or downgrading this epic.21:39Releases prior to four.x, right? And this is something if you want you know that train for that X, then upgrading and downgrading this epic releases four.X to five.zero, right? So you can, if this is something in your range, you can read this section as well. So upgrading and down.21:58Downgrading with releases five.one or later, right? So, this is something you know covers specific for those particular releases and upgrade or downgrade with appic release six.two or later using graphical user interface. So you can find here, you know, a lot of useful information.22:18You know about upgrade to downgrade guides, but as I said, this will also cover specific for each software training.22:40So this is very useful in an article and we also recommend to read this article as well. Okay, and then so we have another tool ACI virtualization compatibility matrix, and this is something.22:48Especially if you have ACI virtual environment integration. So when you navigate to here, you can see that ACI virtualization compatibility matrix and you can navigate throughout this page and you can see some, you know, vendors.23:09Applicable here. So of course most popular or, you know, common virtualization vendor is VMware, you can, you know, navigate to VMware and read that you know that matrix support for your environment, specifically for your environment. And you can see here, you know.23:27Which one is supported, which is not supported and what you know steps will be needed will need to be taken in order to support upgrade if this is something, you know, in not in your path.23:47So this is VMware support integration, ok? So that's, you know, some of the tools you know recommend to you you know to have it on your check, checklist. And that's of course, you know, listed on the checklist, so that's why we recommend to use this checklist and refer to this checklis.23:58List for all those links and tools available@Cisco.com.24:18Alright, so the next, it's a tool or you know location of this particular tool, right? So that's the ECI behavior changes. So this is very important to check, right? This is as I said, the standalone tool or reference where we can read more about software behavior changes. And.24:26And this is relevant especially if you are moving between the software trains, right? From four.five or from five.six and you know why is it important? Because sometimes in feature releases some features maybe deprecated or the opposite enhanced, right? And either way behavi.24:45Favor of software releases maybe changed, so it is important to keep that in mind as well.25:06So navigate to this tool as well, right? So that would be the software changes it's right here, right? So you can also.25:11Navigate to this particular standalone tool or reference through that checklist we shared at the beginning. So in here you can select your software, let's see if we're going through five.three.two and then the six.zero.one and you can see, ok, so in the six.01, we have some base functionality change in the beginning with.25:26This release, the online help has been removed from the GUI, et cetera. And you can read that of course you know for each release, you are, either it's, you know, your target release or your intermediate release.25:46Okay, so next pre upgrade validation script. So this script is not strictly mandatory, right? It is a beneficial tool to use in preparation for the upgrade, however, it's not mandatory to use. So this tool is available on gitHub.com.26:03Right, repository, which is, you know, that particular repository is managed by Cisco system. However, it is considered as an external external tool. Therefore, your organization infosec policy may not allow using this in your environment. However.26:23We have this available.26:43For customers to use. So, and this is you can see this is a live Python script to be uploaded and executed on appic, right? So you will need to, as I said upload it to appic along with target software. So that's the tool script will run on epic, right, against that.26:46Target software and, you know, in the existing environment. And it's provided well at the end right as an output it will provide validation checks and you know some output something like success fail or required manual checks etc. And this is something, you know, we'll take it look at and see.27:07Through the through the online.27:26So there's NO operational impact on however, as I said that this is something, you know, you might want to to discuss with you in infosec policy as well. So, this script is recommended to be run at least two weeks before the maintenance window to.27:29To Allow time to address any flagged issues, so it can be automatically downloaded by Cisco Intersight as well or.27:49I mean inter site if your app it has connectivity with internet or connectivity I'm sorry internet with inter site or you can manually download it from GitHub or cisco.com for air gapped environment. So I'll navigate to this.28:00Navigate to this tool very quickly, right? So here's the when you navigate to the GitHub, so you can see here data center, right? ACR pre upgrade validation script and you can you know read more information about this tool, right? Introduction.28:17And you can navigate to this.28:36New location, they see a pre upgrade, right? And you can read all the instructions, you know, they've written specifically for that pre upgrade validation script. So introduction, you can see how to get started usage, right? And you can see that example output, something like that, right? And you can see that's whether it's passed or out.28:40Warning, upgrade failure, check required etc. So as I said, this tool is very beneficial and it's you know recommended to use but once again, because it's considered as an external tool, so there's something you know your environment or your organization may not allow it. But this is available for you to use.29:01Okay, so, let me go back to my presentation. Okay, so now we have talked about epic firmware upgrade best practices. So of course NO number one epic cluster must be fully fit. I mean, this is.29:20A mandatory must you must comply with this particular requirement.29:40So and how you can check that you navigate to graphical user interface, right? And 1st you log in, so you will see that information on your graphical user interface or you can CLI into app, you can run show controls and then we'll see, you will see that the state, right? Whether it's fully fit or not.29:46So in fully feet must be indicated for all three apex if you have three apex in your cluster, right? So for all, for all numbers fully feed. If you see something less than fully feet, right? You have a one node failure or something like that, right? So that's.30:06You know, you will need to postpone your project until you will fix controllers, right? So must be fully fit, that's number one. Then you have to make sure the time of synced with NTP, right? Your epics must be able to communicate with nexus, with NTP.30:25And the time must be synchronized. Prepare CMC access to KVM console just in case right and CMC we know it stands for stands for Cisco integrated Management console, right? That console we use to access server itself when the ACI solution is running.30:43So we need to make sure that this is available for us if you may need to connect to for KM. Then so next one if that script is allowed, right, preparation P upgrade script is allowed in your environment, so to review that pre upgrade validation script, and then of course recommendation open.31:02And this will create a request for a standby engineer. So a tech engineer may or may not join the maintenance window right away, so this is something you know depends on your contract, but the request for standby tech engineer will be logged and if anything goes on.31:24During the procedure, right, upgrade procedure, you can engage TAC engineer much faster. So that's a recommendation, open proactive case with Cisco TAC as well.31:43Okay, so firmware upgrade groups. So ACI, as we know, is a solution that allows administrators to manage multiple switches as if they were a single switch. So Epic do not participate in the traffic so they don't disrupt the traffic flow. So what we see here.31:54On the slide, right, is that the Cisco ACI extends that modular chassis design principle of the Cisco nexus, 7000 series which is in the distributed fabric architecture, right? So in this model, we see that spines, which is conceptually operates as a fabric module.32:14Lifts which is functioned as a distributed line cards and controllers provide that centralized supervisory and policy control.32:35So what this slide illustrates is that when we, you know, humans or admins, we need to separate switches in the maintenance group, right? The fabric will automatically choose appropriate switches for non disruptive traffic flow, right? So we need to make sure that.32:43Our, you know, fabric, we don't push that upgrade upgrade steps to all switches in the environment, right? So we need to make sure that we separate on on groups.33:04And so highlighted in green, right? So we can see here spine switches, right? Spine switch in two leaf switches in one maintenance group and highlighted in red is another maintenance group, right? So one spine and two leaf switches in another maintenance group. And we'll take a little bit to.33:17Talking a little bit more about maintenance group in the upcoming slide as well.33:36But so as I said the takeaway from this slide, right? So we need to make sure that we configure maintenance groups for our traffic through ACI when we select the groups and the push upgrade, then traffic will automatically move to another, you know, available switches.33:40So here's about firmware upgrade groups, right? So of course create at least two update groups per pod in even switches, and of course, you know, this will, the number of maintenance group will depend on how many switches you have. So if you have smaller to medium environment, you know, from ten, maybe to.34:0250 switches or something like this, this could be considered the you know midsized environment. So you might you know have just two maintenance group, right? So half of the switches in in one group and half another half of the switches in another group.34:23But if you have a large environment and there's something like 300 switches or 500 switches or something like that, right, then you might want to create additional maintenance groups because, you know, you want to have a better control if anything. So especially if it's a habit traffic, you don't want to push that havid traffic from one.34:41You know to one group and you know while another group is.35:01Being up upgraded. And once again, of course you might not have that, have a traffic, you know during the maintenance window because, you know, most of the time maintenance windows happen during the night hours. But once again, you know, here's a considered the size of your fabric. So if it's small to medium.35:08Same size, right? You might, it might be enough to, to make this group right odd and even but if you have a large environment, so you might want to consider add more.35:28Groups. Then always allocate the following, you know, parameters under separate update groups, right? So leaving leaves or leave switches following the VPC pair. So you want to separate this VPC pair. You want to put one switch in one group in another switch in.35:40One group, right? And the reason for that, you know, of course we, you know, we're familiar with VPC so we don't want to have that VPC.35:59Pair you know to be out of service when it's reloaded. However, in the later releases later releases, right, so we will be able to, even if you know VPC pairs or VPC pair added on the same group so ACI is smart enough to understand that you know.36:08VPC pair is added in the same group and the ACI will not push configure updates to a second, to one of the list, right? So if a VPC pair ended up in a one maintenance group, so ACI will not push upgrade to one leave.36:28Leave well, you know, another leaf is being upgraded. However, you know, best practice recommendation, even though, you know, ACI will be able to recognize the best practice recommendation to separate leaf switches in a VPC pair in separate groups. Then leafs connected to the same epic, right? That's also best practice.36:48Like this recommendation to separate them in a different group. Of course spines, right? Especially if you have two spines, make sure that spines are not in the same group because you know if you put them in the same group, of course you don't want to end up in outage. So you need to make sure that spines which is which you know serve as a route reflectors and.37:08And points or something like that, right? So you will need to make sure that they are in a separate groups.37:28And spines connected to the same IPN and we know IPN stands for inter part network. This is something relevant to inter multi part right so then make sure that spines connected in a or separated in a separate group as well.37:33Okay, so this may not be relevant to your environment or or you know maybe relevant to your environment. So effective six ACR six.one.two app center is NO longer supported. So if you're familiar with app center, let me share my screen very quick.37:52Right, so, right. So this one is Cisco DC app center, so we used to have some you know applications like on top of you know this epic or ACL when we, you know, be able to provide some.38:14Additional services or additional functionality. So that Cisco DC app center will NO longer be hosted on epic. And you can see here starting with ACI release six.one.two app center and its application can NO longer be hosted on Cisco Appic, right? So that's, you know, keep in mind that will not.38:34Be available. So, but if you have something, you know, if you're on a releases prior to six.x, right, of something like you know five.something, then that maybe irrelevant to your environment at this time. So the recommendation, right, if you do have some applications running, so disable them.38:53For the purpo for the reason of the upgrade. So once your EC upgrade is finished, then you can enable them. But keep in mind that effective six.one.two does NO longer be in practice.39:13So the reason for them, right, so some of the application right now is just, you know, being natively integrated directly into Appic software itself and you know while other applications have alternatives or standalone versions outside of the apic. E.g., several.39:25App centers applications such as ELAM, assistant, and the Nexus Insights Cloud Connector are now either integrated into Appic or replaced by Nexus dashboard applications. In Nexus dashboard, if you're not familiar with Nexus dashboard, we do have sessions we offer sessions dedicated specifically for Nexus.39:45Dashboard and all you know features and the functionality of Nexus dashboard where we're going go in details about that product as well or that solution. So Nexus dashboard now, you know, offers all those, you know, applications. So as I said.40:05If you do have this, you know, relevant to your environment, so make sure that you disable this application. But once you on the six.one.two, so that application center is NO longer applicable to your environment as well. Okay, so multi pod, so if you are in a multipod environment.40:20And so be aware of this known behavior during spines which is reloaded. So when a spines which reboots right during the because of the upgrade, it should not be used for forwarding traffic immediately after its routing adjacencies come back online and to prevent the.40:40This ACI advertises a higher routing metric for the 1st 10 min after the spine returns to service. So this allows the time for the Coup database in you know in the fabric control plane to fully converge before traffic is forwarded through the switch. And we know.41:00The kube database rate which is located on spines, which is the database contains all the information reachability information about endpoints. So that's just important to keep in mind, right? So if it's kube is not converged yet, but which come comes back online and ready to pass traffic, but it's not, you know.41:20Synchronized yet, and.41:40And not, you know, router not available. So that can create outage. So, in the recommendation for that, right? Because OSPF that mechanism is OSPF that's working fine, but ISIS, which is, you know, by default.41:44The underlay protocol for ACI ACI may not, you know, or not is not able to provide that functionality.42:02Right away. So that's the best practice recommendation to, you know, mitigate that particular issue is to assign metric for ISAS lower than default 63. So as you can see here on the slide is 32, right? So you will navigate to that.42:15Right fabric, fabric policies, pot policies, pop up policies, and then ISIS policies default and adjust that particular metric for lower than the 63.42:34Alright, so now we are at the next question. So we'll pass the ball to Don. He will, he will conduct this sorry very quick. All right. Thank you, Irene. Yes, so I've made this question live now.42:53So, you should be able to vote. Now let me share the screen and we'll see.43:15What, what some of you have to say on this one? So, what is the primary challenge preventing you from upgrading your ACI environment more frequently? So right off the bat, it looks like we see lack of dedicated testing or lab resources. Now we've got finding difficulty maintenance windows.43:24I understand that here of instability understandable as well. Difficulty finding maintenance and windows and downtime, yeah. Well, at least it's not because it's a complex process. It seems it's mostly.43:44Be unable to get a maintenance window and then the fear of instability of new code, which if you know if you do have that dedicated testing lab or or resources can help with that fear.44:04But not everyone has that, has that luxury, right? So we got nine votes in, we'll give it another 10 s if anyone wants to sneak their vote in. If not.44:19We'll go ahead and stop the voting here and oh there we go, one more, and I will stop sharing and looks like we have some good information here.44:37So I'm gonna stop sharing now and pass things back to Irene to continue with her presentation. Alright, thank you very much, Tom. So now we are the performing the ACI upgrade. So we already prepared for, for our upgrade, we can created our plan, how are we going.44:54Going to proceed. So now we have, we'll go through that upgrade itself. So what you can see here is ACI upgrade steps, right? We can need, we need to take here nine steps. So step number one, right? Perform epic backup, and this is something you know of course we can take it prior.45:14Two maintenance window. And this is prerequisite, right? So that's always the best practice recommendation to have a full backup stored somewhere on external external location.45:34Then step two perform SIMC upgrade, right? Cisco integrated management cannot console, right? So we need to upgrade to SIMC and that is also recommended. So that's the procedure itself, right? So the upgrade not necessarily happen, I mean will not necessarily be happening.45:46The during the same maintenance window. So you can separate this maintenance maintenance window, you can upgrade simc 1st and then you can proceed with ACI. If you do have the time for, you know, for, for one.46:06The maintenance window, so if you want to perform SIMC and the AC upgrade at the same time, so you can do that as well. However, you know, the recommended recommendation is to make sure that version is compatible with ACI target release as well. So that's the.46:23Number two, make sure that you.46:43You have that covered. Then steps three prerequisite as well, download software, right? So you can download software from a Cisco software download repository, not to your, you know, local machine or somewhere where you're going to upload that image to appic. Then I.46:47Upload software to Appec as a step four. Step five schedule control upgrade and node upgrade, right? So always upgrade controller 1st.47:06Oh, that is the best practice recommendation, and this is something just you know all the, not only recommendation but it's in your prerequisite you will need to upgrade controllers 1st. So once you upgrade controllers, then upgrade the cluster of Cisco.47:17So with the cluster of Cisco, right? So with schedule control upgrade, That's five, we schedule control upgrade and note upgrade and then you know we upgrade the cluster of Cisco apex. So step five of six, right, could be.47:39Combined. Then once we have controllers up upgraded, right, then verify the fabric is operational and the cluster is fully fit before processing proceeding. So that is very important, right? To make sure the clusters cluster in fully fit state. So once we finish with controllers.47:53Then we upgrade 1st group of switches, then we upgrade after that we verify that fabric is operational, then we upgrade 2nd group of switches, right? And then verify the fabric is operational. So if we have intermediate second intermediate intermediate hop, right? So we need to proceed with the controller.48:12No agreed. So let's just say you have from a5.three.two, you need to jump to six.two.two and you have two intermediate hub so you will perform upgrade to 1st intermediate hub, right?48:32Appic and then switches, then second intermediate hub, you will jump to second intermediate hub starting with controllers and then switches and then your target release. So as I said, if you have two intermediate hub, then you will need to repeat those steps starting with the controllers. So controllers 1st then switches.48:47Then once again controllers 1st and then switches or switches groups. That is the procedure.49:07Now, so starting with Cisco ACI only six.X it is important to keep in mind that uploading 64 bit on 32 bit images to Nexus switches in ACI mode is mandatory, right? So we must make sure that we have 64 bit and 32 bit.49:15Images.49:35Available. So the reason for that is that the existing switches in your environment may only accept 32 bit or only 64 bit images due to sites of available memory as you can see it here. So to avoid possible upgrade failures or delay, right? Upload, upload those booth images and.49:37The Appic will select the correct image for each of your switches. So you need to make and to make sure that you have that available for Appic and Appic is you know smart enough to recognize which image needs to pull to upload to a particular switch.49:56Alright, so from from where this is something we know we're already covered very quick so make sure that is compatible so you can also check it online@cisco.com right and it's available on this ACI checklist, right? So when you go down that list on this ACI checklist, right? So all that checkbox.50:14Boxes, you will see there's also make sure that CMC firmware upgrade is compatible, so you can navigate to that link and you can see which version is applicable to particular ACI release. So right now we can see that CMC upgrade is required when upgrading to Cisco ACI six.one.50:33And recommended release so we can see here for Epic M3L3 and M4L4.50:53So you're gonna also see here the CMC important, right? P sorts announced on the April 2024, right? So therefore you can also verify that you know information there as well. So there's another reason why we need to make sure.50:59That CMC is compatible and the upgrade is upgraded to a recommended release, you know, for that piece of fix.51:19Okay, so backup configuration, right? This is a step number one, right? So we need to backup configuration on external devices. And we can see here turn AES encryption on before creating the backup. So encryption what it does, right? It saves all that sensitive information such as, you know, passwords.51:30So we need to make sure ensure that encryption is on. And what is important here, right, to make sure that's on so you make a backup this encryption on and then if you need to restore, you need to make sure that that encryption is on as well. So keep that pass passphrase when you were created passphrase, right? So you.51:51You will need to make sure that you keep it somewhere in a safe storage, right? Because if that path phrase is lost, right? So you made you take a backup with AS encryption.52:10And if a passphrase is lost, right? So you will not be able to recover that passphrase from anywhere, so you even TAC will not be able to recover. So if that mean you back up configuration with encryption on and if you know that passphrase is lost or something like that, you know, something is not, you know.52:26Not recoverable, so then you will not be able to log in to your ACI environment, Nobody will be able to recover it and basically right you will need to redeploy ACI from beginning, so that is very important to keep in mind that encryption on for backup and restore and have passphrase is in a.52:45Safe environment.53:05And also there's a note here during configuration import, make sure that fail import if a secure field cannot be decrypted, checkbox is enabled, right? And this will verify the passphrase 1st. So keep that in mind about the encryption and passphrase when you are preparing for AC upgrade.53:07So here's the on this slide you can see that the difference right so between versions rate and for graphical user interface, we see here the information about version four.one and five and then ACI version.five.one starting with ACI five.one and onwards. So we can see here the simplified right the schedule, so with NO move the.53:28Need to have that separate steps for scheduler and you know specify that, you know, particular settings for this new and this is something you will will see the through demonstration today from a demo. That's a simplified approach, right? So it's one window with one wizard, right? So we will be able to upload the image, you know the.53:48Validate, download, install et cetera. So we'll take a look through the demonstration today's demonstration as well.54:08So that's, you know, that slide is to show you the difference between, two graphical user interfaces, right? The the the separation between graphical user interfaces and new features introduced here. So monitor device upgrade status, right? So this is something throughout.54:15If you upgrade, you will see a status field on the controller firmware screen. So this is going to tell you whether your upgrade has completely successful, completed successfully or whether it is failed or it stays the same other on the other states, so you will just, you know, be able.54:35To observe and see that, you know, this stage of upgrade, whether it's not scheduled, scheduled, queued, installing, retraining, completed and failed.54:55So that's very, useful money, you know, ability as well. So what to do after the upgrades, right? Firstly, you want to validate that your upgrade was successful, right? The 1st step, in validating the success of your upgrade happens even before.55:05Do you start your upgrade by developing a robust test plan during the planning and designing phases of your project. It is important to do this ahead of time because it needs to be reviewed and approved by all your stakeholders.55:25So, e.g., you have, you know, may have folks who manage your VM environment. You might have application owners and those responsible for data security. So for some organizations, even the network team is split between engineering, implementation teams and the operation man.55:42Management team so you might want to you know make sure that everyone in sync you know for ACI upgrade. So if your organization you know is like that when you have multiple stakeholders, you likely need to have the operational team sign off on your.56:02Test plan.56:22So if you need to help, you know, if you need help developing this plan, Cisco professional services or 1st that particular services, right? And if so, you can reach out to your partner, e.g., or you can reach out to your Cisco representative who can, you know, arrange that, particular.56:23Call a project with Cisco professional services.56:43Alright, so this we have another question coming up, so I'll pass the ball to Don. All right, thank you, Irene. So this next question, what should the following best describe your current ACI software strategy?56:49So we've got strict adherence to long lived releases, aggressive adoption of the latest features. If it's not broke, don't fix it or you do not have a formal software strategy, so let me go ahead and share out the responses.57:06Okay strict adherence to long lived releases, if it's not broke, don't fix it, which are very similar, you know, reasons, right? So it looks like those two are gonna be neck and neck.57:27For this one, which I'm not generally surprised by that. Oh, we got one aggressive adoption of the latest feature releases. Okay, very cool.57:42Eight votes in, we'll give it another nine votes in, we'll give it another 10 s. Anyone wants to sneak one in at the last minute. All right, we'll go ahead and lock it in.57:57The strict adherence to long lived releases followed not that far behind by if it's not broke, don't fix it. Okay, Irene, I'm gonna turn things back to you, I believe.58:14Alright, thank you very much, Tom. Alright, so post upgrade tasks, right? So this couple of slides with dedicated for post upgrade docs. So when the, you know, when the upgrade failure is detected, right? The scheduler pauses in NO.58:32Your nodes begin to upgrade, and the scheduler expect manual intervention to debug any upgrade failures. So as I mentioned at the beginning of this presentation, use this ACR upgrade best practices and troubleshooting article to learn about troubleshooting steps for the most common reasons that the scheduler may.58:51However, I'm right, so if the problem does not have you know if you problem does not have that quick fix contact immediately especially if you have a proactive case logged.59:12So, because if you trying to decommission or wipe in a note, you know, or restart ACI upgrade, so this might make it even worse. So the best practice recommendation, right? If this is something you know you were familiar with the ACI upgrade best practice to troubleshooting, if this is something minor, you can fix, you know, you can fix it, but if not.59:25Not just, you know, not trying to that decommissioning, wiping configuration, all the stuff, just, you know, go ahead and contact immediately.59:45So this is another tool we have, this is, you know, will require a Nexus dashboard, this Nexus dashboard pre upgrade analysis. This is optional, right? It's not mandatory once again, same like a script, but as I said, if you have a Nexus dashboard available or you know running up and running, so you can use that.59:54Tool available pre upgrade checks against appex and switches, you know, direct access to release notes and post upgrade analysis comparing anomalies, configuration changes, operational changes et cetera. So you will run pre upgrade and post upgrade and then compare if this is anything, you know, change.01:00:14You know, we used to do it as a you know Nexus nexus engineer myself, right? We used to do it manually. So before upgrade and after post upgrade, right? So we check the difference between, you know, back and forth. This is now the available with the.01:00:31Nexus dashboard. We can just, you know, select, you know, option in the Nexus dashboard will do it for us. Of course, you know, we'll need to have a Nexus dashboard and ACI sites onboarded.01:00:51Note here that nexus dashboard will not perform the upgrade itself, right? So this still will need to be launched from the apex. So you do have sites on the Nexus dashboard will be able to run this pre upgrade post upgrade checks, but the upgrade itself or you will need to launch from Apex.01:01:04All right. And this will take us back to the our survey. I will pass the ball to Don and then we'll, you know, navigate to our demonstration part of our webinar. All right, thank you, Irene. And so let me go ahead.01:01:22And start this new poll. And this one is how do you currently manage ACI conformance and compliance across your fabric? So our options are manual audits and internal checklists using Nexus dashboard or another automated recording tool.01:01:40Going on TAC.01:02:00Or using, you not currently have a formal confirmation conformance process. So let's share and see how things are looking with this one. So we got two votes so far. One of them is relying on south, another one is don't have a formal conformance.01:02:01This process. We also got one vote for, two votes now for manual audit that looks like relying on TAC during support cases. Okay, six votes in so far, three of them are relying on TAC. Now we've got seven votes in.01:02:21But it looks like the majority is relying on TAC to make those recommendations during some sort of a support case, ok? Seven votes in, see if we can squeeze one or two more in in the last few seconds. If not.01:02:41I will go ahead and stop sharing and I believe as Irene said, it's now time for the demo portion, so I will pass things over to Dave to continue our session. Thanks Don, thanks Irene.01:03:01Okay. Again, Irene's presentation is very helpful if you want to have a successful upgrade, pre planning, positioning of software, remediating anything before the upgrade will, will.01:03:20Greatly increase surchances of success. I'm gonna have a little demo. It's gonna be recording. I had to clip out quite a bit of the dead time in order to fit into the schedule. My lab topology is as follows, it's two spines. It's three leafs, it's not quite a good representative of a production system.01:03:38One of my leafs.01:03:59I needed another leaf in order to get my VPC paired too or not VPC but double linked my single apic. I am going to be doing a upgrade from 532 and 2615. These are both maintenance releases. 532 is, is really.01:04:00Quite old, but it is still available and quite a few people are still on the five.X train. There is a direct path from 522 to 6.5 as Irene mentioned. If you wanted to go past to six two, this would be the intermediate step.01:04:19Again, I'm going to do this as a demo recording because I just can't fit it into the time required. One thing we will do is talk about the validation steps. The minimum you want to do is open up the APEC.01:04:34Go to your system dashboard, look at your controller status, I want to go to single controller and make sure they are all fully fit. That means all the databases are in sync and they're all running properly and make sure all your leafs and spines are in a healthy state. So from here, I'm gonna do some more update.01:04:49And pre update and post update validation checks, they're identical to both the ins pre and posts so I don't want to duplicate that too much. From here I'm just gonna go ahead and go into the recording. I do skip ahead quite a bit in order to save time. I did try to put some timing on each step so you can kind of gauge what's.01:05:07It's needed.01:05:27Oh Possibly a short demo on how we can download the APIC firmware to the Apic itself. This version of code 532F does not allow me to download 615. There's an issue.01:05:31I can work around it by using the SAP commands which are documented. So currently I've got the ISO for 532 and the switch firmware for 532. I've downloaded to this Windows jump server. I've got the ISO for 615 loaded. I'm gonna use SFTP to copy the file over.01:05:47You could use a SCP it doesn't really matter. I just like using this it's called input to the it's gonna push the file too, and this will take several minutes.01:06:11The transfer. Now that's the file's been transferred I need to exit out and log back in under SSH to add it to the repo on the apic.01:06:28So the file is there and I can do a.01:06:48Again I can do a show firmware so you can see what's in there right now and it's all the 532 software and the catalog for that version. So I'm going to go ahead and add it to the repo.01:06:55Yeah, this will take a few minutes to push the repo and then update the repo itself. I'm just running the repo.01:07:13Command itself, if you don't see it yet, it takes a few minutes and there's the 615 and if I refresh my GUI, I have 615. I'm going to stop here so I can actually run a few other commands to show you what's gonna happen in the upgrade.01:07:30And then eventually use the itself do the upgrade. You can't continue in the CLI it's actually just a few more commands, but you'll miss miss a few things that we're gonna demonstrate here. Thanks. So let's go ahead and begin. We're gonna select the controllers.01:07:50Set up an update and or it's a regular update or gonna select 615 and you can see what the ACI 532 validation tool will do. It's similar to what you will see in this script and also in ND, but 532 is quite a bit older s.01:08:11A couple years old at this point. So it's not gonna have all the latest identified issues that tech and engineering have seen. This is pretty similar. We've seen this before, the.01:08:31Singleton leave 313 doesn't have a VPC pair so it's gonna complain that this can cause loss of data if you're connected to it during the upgrade process. A lot of the other things it'll go through and then there's an issue with the simc version, even though we're on a.01:08:42A valid version, it's not the latest version of SIMC. Get to next. Again, it's gonna let me know that I'm able to bypass some critical issues and then begin the install. This installation can take several minutes. I may skip ahead some of the.01:08:59Part of the recording and it.01:09:19Show you what's happening in the meantime. And you can see I've lost connection to the controller and 10 % and again if I go to the KBM, we should start seeing some activity over here that's reloading the system.01:09:21We're on 615E.01:09:45At the controller level, the switches are still at 532.01:09:47Catalog files complete. Fabric upgrade is complete. The switches are the next step. Images. We can start the download of the firmware. So I need to download the 32 and 64 bit images.01:10:05And I've got them on my local jump box and there's a 32 bit I've already downloaded from software.cisco.com. And again, this can take several minutes to download and so again even though it's completed.01:10:22It will take a while to refresh and show on the images. I'm gonna go ahead and start the 64 bit download.01:10:39And again, this could take several minutes, I'll probably skip ahead until both images are loaded and they both completed the download process, but it can take a few minutes for them to.01:10:56Actually get registered and pushed onto all the switches. The 32 bit version is there.01:11:15I'll give it a few more minutes and then the next step will be to upgrade the switches. There are some notes up here about some transactions that are still in process, but again, those are generally they're the upgrade and they will clear out and there's the 64 bit and 32 bit.01:11:21So we're ready for the next step.01:11:41Okay, we're gonna continue on to the switch firmware upgrade. The controller's complete. I've already downloaded the images. What I 1st need to do is create what are called upgrade policy groups that's under the switches menu.01:11:45And I'm gonna create an upgrade group and I'm gonna do an on even discussed in our discussion. You want to pick half of the VPC pair and half of the spine and the easiest way to do that is just say otter even.01:12:05And you want to add switches? Now I can manually select the odd number of switches, which is one way or I can just, as you know, the system is gonna try to push me in this way, I can just select all the odd notes. These all look correct. I can select all of them.01:12:24Go to the next step, and just make sure everything's complete and it's gonna go on to version selection next. And I do want to go to 615.01:12:45I can also there's an advanced settings here. You can make changes. You want to by default use the default selections enforce compatibility, unenforced graceful upgrade, you know, I can deal with issues after the fact, I want to get most of the switches or all the switches upgraded upon this time. I.01:13:01Don't wanna pause during a upgrade failure, otherwise I'm gonna have to go back and revisit this. If there's a failure with the switch, I can deal with that one individually, otherwise, I'd like it to continue.01:13:21Go ahead hit next. It's gonna run its ACI validation. Again, these are scripts based on the time of the release of this code. It's giving me the same issue with 313, it's a singleton. There is NO VPC pair, so it's gonna potentially lose data. So they they do try to warn you.01:13:33There's NO other real issues that I see. I'm gonna go ahead and continue. Again, it's gonna tell me that warned me that there's an issue and I'm gonna accept it and I can begin the system upgrade.01:13:53Again, this is gonna take a bit. I may skip ahead to save some time.01:14:10You probably won't see much activity until the downloading is complete. I can ready to install, I can select this hyperlink, use my switches, I'm gonna go ahead and select actions, begin inst install.01:14:40And you can see it's the process is beginning. Again, this could take take several minutes. I'll skip ahead all three switches are reloading the new image unfortunately.01:14:58They don't have a concept connection, if I did, I could bring it up and you could see the.01:15:18Post an installation going on Okay let's continue with the firmware upgrade. The switches have completed. One thing you might notice in this screen is leaf 313 took over.01:15:22Almost 20 min longer than the other two switches.01:15:42The only real difference between the spine three oh one and the other leaves are these two are running the three two bit firmware and 313 is a newer switch running the 64 bit firmware that may account for the additional time. I did check some of the logs, there was NO, there was NO EPLD upgrade that increased the time, so it was just the additional.01:15:45Firmware that was different from the other systems. So let's go ahead and continue and do the even switches. And so I'm going to create an even policy group and add switches.01:16:06And again, I'll use the shortcut just like the even switches and those are good. Again I'll go to.01:16:25615 echo, and I'll leave the advanced settings alone. It's doing its validation within the apic. And again, nothing really outstanding information I've seen earlier.01:16:41And I can go ahead and hit select next and begin the download. Again, once the download is download is complete, I can go ahead and initiate the installation. This can take a few minutes so I may skip ahead in the recording.01:16:57Okay, the switch firmware has been downloaded, ready to install, I can select the switches and go to actions and begin the installation.01:17:15And again, you'll fire off the installation queue, they'll eventually reboot, and once they log back in, we can they'll come back into the fabric. Again I may skip ahead to eliminate some of the dead time.01:17:35Okay, you may need to manually refresh leaf 02:12 was showing red, but it actually completed several minutes ago.01:18:16I ran a show discovery on the leaf itself and it was fine. At this point, the upgrade of the odd even switches are complete, the APEC is complete, and simpc is running on a supported version although not the latest recommended version. That completes the upgrade of the fabric.01:18:23So I hope that.01:18:43It wouldn't make sense I'll try to put together a summary of the times. The switches took approximately 3030 5 min per grouping and the Apic itself was another 40 min or so. Again, the simc is quite lengthy. It took about an hour and a half to 2 h.01:18:45Again the what we're planning can do scheduling, getting firmware downloaded or ready to download. If you do need to do SIMC upgrades, you may want to schedule those in a separate maintenance window. You need to do them in series, you can't do them in parallel. You have to have a cluster quorum.01:19:05Operational for your system.01:19:25Okay, that concludes it for now. Okay I'm gonna cut back over to live. This is my apec. Again, after the upgrade, you wanna check and make sure your health scoring your leaves, check your operational state, make sure that's all in good shape. A lot of these faults were here before.01:19:28For and after they are not really indicative of an issue with the upgrade. They probably existed beforehand.01:19:48If you want to investigate them before or after again, looking check into the fault, select all faults. And if I kind of look at the descriptions right, we don't apply power to both of our power supplies because of the lab. We also have some hardware issues, NO smart licensing, so nothing that would indicate an issue with the upgrade.01:19:54Secondly, what you can do is in the APIC itself, there's some ACI diag commands you can run, and these are in the notes that Irene provided. So.01:20:16These are all the ACI diag commands you can run. They all start with or end with read. So this is like application vector read is application database read, and it's quite verbose, some things you'll recognize write the top information.01:20:30The fabric domain name information. You may want to run these before and after and just do a comparison, make sure you don't see anything that's obviously in different. You can also look at what's called the fabric node vector read.01:20:49And again, this is just gonna give you the nodes, the spines and leafs. I've got an undiscovered leaf that I'm using for another, another task, so it's not brought into the system at this point. And ACI diag.01:21:13This is the replica vector read or the rec replica vector database read, it's just looking at all the different shards. I've only got a single apex, so ideally everything's in a good state.01:21:31You know, we just make sure you've got multiple systems, you want to check every one of these if you can. And then there's an ACI diag cluster that just looks at the cluster information type, this takes a few minutes.01:21:48Now, and again just checking making sure all your processes are up, make sure the system is fully fit. Check your IP addresses, right? I've got NO IPV six.01:22:07Yeah, it just gives you an idea of what's going on. And then lastly in the ACI diag, verify apic. And this checks your SSH credentials or SSH connectivity. So it gets just making sure your certs are in good shape.01:22:25And SS open SSL is working properly. If you happen to have Nexus Dashboard and you've onboarded into the, your ACI into the fabric, we do have a fabric management tool. Again, as Irene mentioned, we can't do upgrades from Nexus dashboard. Those are still handled by ACI, but we can.01:22:41Dan runs some tools to check things.01:23:01Under manage, we have what's called fabrics fabric software. And in this system I've got both NXOS and ACI, so under ACI, I've got I've loaded 62 for some additional testing. I've run some testing beforehand just because I've done some things.01:23:03So it takes a few minutes to do each one. I can actually look at my history.01:23:23Of this fabric, and that's the nice thing about a controller, it stores information for a long a long time. I've done a bit much testing, you can see I've done one test from five through two to 61E and you see a failure, so let's go, so this is this is the pre and posts that i've done.01:23:27So this is the pre analysis, and I was going to 532615I have a pre analysis report of the controller, if I take a look at it, again, some things we've talked about in the past, I've got an unbalanced VPC pair.01:23:47And I've got a simc incompatibility. Again, it's not a bad SIMC, it's just not the latest released SIMC. And due to an issue between 532 and 615 can't I cannot run up download the 64 bit code onto the controller.01:24:06At this time. And so it's gonna complain about that because there was NO.01:24:25Let me go back to, the switches. So again, it's gonna, it's complaining because it's got nothing to compare. There's NO target firmware at this point. Hey Dave, just want to give you a quick time check.01:24:30Yeah, I'm almost done. And let's just do a quick post analysis update, and so this is after the update, all anomalies are cleared, checks have been passed. You'll see a listing of all the advisories that have been cleared based on going to six two. Most of them are P certs.01:24:47And just go back to the switches, and again now I can do a post update analysis and it can tell me, hey, everything's passed. No, NO issues with the fabric. Again, letting me know it's been cleared based on the upgrade.01:25:05So again, these are just tools you can use to verify your controllers. You may want to check your servers, your your critical systems at this point. So that kind of concludes this demonstration. Okay, back to you. Yeah thank you.01:25:22Both Irene and Dave for your insightful presentation and demo.01:25:43If everyone, let me grab the ball back then, if you're looking for more resources, visits community.cisco.com. The Cisco community is the trusted knowledge sharing destination for customers and part.01:25:47Partners throughout their life cycle by providing guidance, answers, and deep engagement as part of a holistic digital experience. You'll see people from all over Cisco replying to questions that are posted at community.cisco.com. So I definitely encourage you to to check that out if you haven't.01:26:06I'm gonna go ahead and put that link here in the chat. Now we can open the floor for any questions. We had a few come in during the presentation. They were answered in chat or Q and A.01:26:23If you do have a question you'd like to ask, we have a couple of minutes left here, so feel free to throw that into the Q and A or the chat, if not.01:26:41I'll go ahead and open the final slider which is just a survey. So, we'd love to hear if you're interested in a one on one consultation with one of our customer success specialists.01:27:01And if so, put that in one of your answers in the survey if you don't mind. And we also welcome your feedback on how we can improve future sessions. It just takes a couple of seconds and your input is really valuable in helping us enhance our offerings and improve these sessions. I'll keep the survey open for a few minutes after the session closes to allow you time to complete.01:27:17And we do really thank you for taking the time to share your your thoughts and opinions there.01:27:37Again I will go ahead and say that we also encourage you to, to check out the rest of the sessions that we have coming up.01:27:44The expert insight series will cover intersight telemetry for ACI and nexus dashboard, also best practices at ACI and indeed for both security and monitoring the Splunk. And then later in July, we will have a live Q and A with all the panelists from all of these sessions so that could be really interesting, really fun. Be sure and join us.01:27:58For that. If there are NO questions, I will go ahead and conclude today's session. Thank you again for your participation. We really do appreciate your engagement. We love getting the questions, we love getting the feedback.01:28:18We do extend our gratitude to our presenter and demo and our panelists as well. This was the 1st session in our current series of Cisco experts inter insight series, Cisco ACI upgrade conformance and software strategy.01:28:37Thank you all for attending and thank you for your time. Have a great day.01:28:54