Best Practices for ISE Upgrades

Hello everyone and welcome to 2nd session of today's security expert insights series focused on Cisco ICE and unlocking the full power and potential of your.00:10Cisco ICE deployment. Today's afternoon session will focus on upgrade planning and best practices.00:20As many of you are looking at upgrading your current deployments, we'll be focusing on upgrade planning and best practices. Our goal is to ensure you have the roadmap and tools necessary for a smooth transition to the latest versions of ice. To lead us through this deep dive, I'm joined by our presenter from our customer success specialist team Miguel and our demo specialist.00:26Before I hand over to Miguel, we want to make sure this session is as interactive as possible, and we want to hear from you. We'll be using Slido throughout the session to gather your questions, understand your current needs, and address the topics that are top of mind for you. Please put excuse me, please take a moment now to join us over on Slido by using the event code.00:47Good 1777580, that should allow you access without any passwords.01:07Your input definitely helps to drive the flow of these modules, so please don't be shy with any questions and with that I will hand it over to Miguel. Thanks Tim. Good afternoon everyone or good morning, it depends on time zone, but nice to meet you. My name is Neil Martinez. As Tim already mentioned here we are gonna be talking today about.01:15Talk a little bit about upgrade planning and best practices.01:34Eh, of course, here I'm gonna be presenting a lot of theory, a lot of experience I I I want to share with you. Also, you're gonna be helping from his side to explain a little bit more and go on a deep dive with the demo, ok? Important to mention.01:38It's gonna be critical always for you to ask as many questions as you want. Here the main objective is for you to feel more comfortable, so feel free to use a slider for it. Also, another important topic that we need to know is a little bit about what we are gonna be reviewing. I know that earlier today for some of you that probably were on the call, eh.01:55We were talking a little bit about what's new and of course sometimes we need to talk a little bit about things about the upgrade, ok? Remember that it's always important to make different configurations and that's the reason we have this agenda today. Why the upgrade when and prepare perform the upgrade and.02:14Eh at 1st what I want to touch is the reason and also why we need to consider this part is really focused on the what's new. Why? Because at the end, whenever we are looking for upgrade is because we need or we have a requirement, we have a need or a requirement.02:33So it's really important for us to decide and define what's our current our current need. According to that, we are gonna be taking that decision. Sometimes it's because we want to go to a golden release. Sometimes it's because we want to test a new feature.02:51Or we want or we need a new feature also happens or sometimes we only are looking to have a good life cycle maintaining, which is also a justifiable point. In this scenario, that's the reason it's important to consider that the reasons to upgrade maybe really different, but it's always important for you to know.03:07That it's a valid reason, ok? Now, of course, this is gonna move to the plan and preparation. On the preparation, what I want to take a look at or we are gonna be focusing a little bit more is on why or what's the reason behind that preparation. Being honest, I have talked a lot about this.03:27Eh I had also some experience implementing ISO, what I can tell you is that usually whenever we are working with technology, we need to expect and expect, ok? The reason behind it is because at the end, we never know what is gonna happen with technology, of course we can minimize the impact, we can minimize minimize risk.03:46But at the end, we still have a risk. It's almost impossible to reduce those risks to none. So it's always important for you to prepare.04:05And we're gonna be talking about those preparations. What is important for you to consider to get ready and be prepared for that. Now, before we start and we deep dive a little bit more, I would like to understand which version of eyes are you using. And this is because of course according to that.04:14We are gonna need to make some changes. Okay, we are seeing that a lot of you are in three.oh three.one and three.23.32 and three.four is a minimum.x. We still have some minimum, but we still have some.04:32Eh three.five. Okay, here part, of what we already talked about and it's gonna be important for you to consider are the life cycles. What while we are alright in the moment we are seeing this information, it's gonna be critical for us to talk about this.04:49The reason behind it is because, e.g. 3.03.13.2, even three.three now has been announced of life. And of course it's important for you to plan and prepare, ok? Usually what I say to my customers and probably here I can share also with the with the everyone.05:07Sorry, it's important for you to to know and of course understand what it's gonna be important, ok? In this scenario, a lot of you maybe on three.three, of course, a trigger to upgrade from three.three would be an end of life.05:24Why? Because we are gonna have different milestones at some point software maintenance is gonna stop. So of course it's an important trigger. Now, that's the reason life cycle is one of the most important things to follow up. It's always critical? No, it's not it's not always critical to move at at one point of time, but of course we are gonna need.05:41And to consider and be prepared. Otherwise we maybe losing some fixes. Of course, we are gonna stop receiving vulnerability.06:00Fixes that are really important even if we are not always hitting them, but it's always critical to consider this information. Now, here we are seeing that most of you are 3.03.13.2, so it would be important for you to consider this action because at this moment we are already in process on the end of life. So we.06:09You need to make a planning for the upgrade. That's something really, really important.06:29Yeah, three.four at this point of time is the golden release, just for awareness, if anyone has that question, three.five there is NO there's NO date or any specific timeframe for for it to be the golden release, but probably it's gonna happen in the future. So always remember that.06:35It's important to you to consider also the release notes which is critical for you to have more information about version and being able to take that decision. Three.four and three.five, of course three.four is the golden release, but if you are looking for a specific feature and it's on three.five, probably that's the.06:51The reason or it could be the trigger to jump to three.five even if it's not the release, so always make sure to be reviewing and taking a look at the release notes. Now, let's jump very quickly. On the plan and prepare, we need to or we are gonna be reviewing three different four different things.07:07Compatibility checks and upgrade path, pre upgrade activities, upgrade readiness to maintenance window. Now, important to mention, some of these are gonna be changing a little bit in in a future or are changing already, being honest, but it's important for you to always have awareness, ok?07:25We are gonna be discussing about all the different upgrade types we are gonna be talking about them in a, in a few slides, but it's important for you to be aware that some of these process are important whenever we are doing some specific type of upgrade, ok? Now, there are some compatibility checks and upgrade paths for upgrade activities which are.07:43For every type of offer, those are criticals, ok? Now, e.g., let's talk about the supported platforms. Five are supporting the same platform, but it's important to understand that all 03:36 hundreds, 37 hundreds are already in end of life process. This doesn't mean that we.08:02We're not gonna support them anymore now. No, they still have support, but it's important for you to have awareness because at some point of time, new versions may not support these appliances.08:22Now, for the for the the ones that are using VM, this is gonna be critical. Why? Because all Vmiss are aligned to a hardware model, ok? So even if your VM is aligned to one of these appliances as a hardware model, of course you will need to update the specifications on that VM in a future.08:34As I, as I say, not right now, but it's important for you to consider this information. Why? Because e.g. 36 hundreds are gonna in support on 2027 2028 eh approximately, so it's important for you to have like that awareness and start planning. Of course, whenever you are making an upgrade to aversion it.08:53It's always critical to review the specifications.09:13To give an example that happened at some point of time, I heard about a customer that was trying to upgrade, and of course since we didn't they didn't review this information at the point of the upgrade, they suffered an issue. They opened a tech case and with that, they realized that the specifications on their current VM weren't.09:16Not supported anymore. So it's important to always to double check the target version, the information about it. We have different parts that are important for you to take a look at and those are gonna be able to give you a lot of information to get prepared and of course have this ready before Europe.09:36Now let's talk a little bit about which are the current allied platforms, which are the three eight hundreds. These three are the currently supported versions of all the different appliances that we have. Of course, we have three equivalent VMs, the VM small, the VM medium and the VM large, which are equivalent to 3850 3855, and 3895.09:57Remember that.10:16For each one of the 3637 and 3800 families we have different Vmiss So always make sure to review the specifications of your VM to review to which one is aligned, ok? But it's important to double check this information, always make sure and of course we can take a decision about if we are gonna be using appliance.10:17VMs, if we are gonna be using Amazon Web Services Azure. This is gonna change a little bit. What are the options for what are.10:38Are our needs. According to that, we need to make sure that we have the most efficient and of course fulfilling our requirements. E.g., I can give you you know experience a little bit of information. I had a customer in the past that was working with ICE appliances. Because they didn't.10:46One e.g. with the virtualization team for them to move the information about the Vmiss.11:06That happened yes, it maybe a requirement. Sometimes they decide to mix and have a VM with an appliance, it's also feasible. Yes. At the end it's important for you only to review that you have the information and all the specifications needed for that specific version.11:11Now, another topic we need to talk about is the compatibility checks. Why? Because it's always important. These are some examples only, but whenever we are making an upgrade, we need to review also the compatibility information. E.g., with the Microsoft Active Directory, we need to review also with the bean word cloud or Amazon.11:29Services. We need to review catalyst center integrations are really important whenever we're integrating.11:49E.g., catalyst center is one of the most important which we need to know and review that it's compatible with our current version. That's something that we always need to consider. So always make sure to review and make sure that we are or that we have that information.11:55Now, it's important also to review the licensing model. On the licensing model, we need to talk a little bit more about this section. Hold on a second because it's not moving, here my screen. Okay.12:12Why are we talking about licensing model? And here, let's double check it seems something is not oh ok here the topic is that of course we have the two.X and three.x. Probably a lot of you already have sold this part, ok? I saw that most of you are on three.x. Everything is working there correctly.12:29And of course, this is not gonna be changing a lot for you. But of course for the ones that are still on two.x, it's important for you to be aware. The licensing is changing between two.X and three.x. Now everything is smart licensing and of course the features are changing a little bit.12:48E.g., we can see here that the essentials now contains AAA.1X guest and easy connect, ok? And one important thing is that yes, based on essentials are quite much similar, but the plus and Apex has changed. Why? Because as you may remember onto the text, we had a Lego model which.13:06Is that we needed that even per endpoint. If the endpoint was doing that one X with profiling and posturing, we needed to have three license, one of each one for that specific endpoint. Now that has changed. And the reason it has changed is because.13:26At the end, now we have an model, which means that, of course, even if we have a an essential licensing, but we want to use profiling, we don't need an essential licensing. What we need is to have an advantage license. The advantage license is gonna take.13:42That or it's gonna consume only one advantage and it's gonna contain not only the advantage features but also the essential features. That's the reason it's really important.14:03Also, for you to consider it's gonna be always critical eh in this scenario for you to be aware about what's important to consider on this section. E.g., if we had 1000 base licenses and we have 300 plus.14:14Probably now we don't need those licenses. Now the advantage maybe only 300 and also gonna contain the essential part, we can reduce those advantage only to 300 at the essentials to 700. Because at the end we don't need two per endpoint, it's only one license per endpoint.14:31That's a little bit about the importance and the reason I wanted to mention this because it's gonna be critical for you to consider that part. Now also let's review a little bit about how those features have changed a little bit. Now we have the advantage we have on the the.14:51As well as the containment and the endpoint analytics. Poster MDM and it's gonna still remain in the apex or now the premier, which is the equivalent in unlicense. Also we have the user defined network and the location service on the advantage. That's important for you to.15:09Have awareness again this is not critical, it's already a little bit old for some of you, but at the end it's important. Regarding device admin, there is NO changes on licensing or awareness.15:29Now, let's move a little bit here. Migrating from three.X to three.three and plus. Now, it's important for you to have a, this consideration yes remember that always we need to consider the different types of migrations that we have. Of course, if we are migrating from three.X to three.three.15:40We need to review because we need to review also some other topics like e.g. if we need to raise a support case, provide conversion specific because e.g. we still have the VM licensing. Now it's only a VM common. So it's important to always double check the.15:59Licensing information and the specifics for migration for each one of the versions.16:19And again, the upgrade path, this is gonna be critical. I remember some of you were asking here, e.g., it's something that you need to consider and this is gonna be really a good example. We have the information about the simple step upgrade, e.g., we can upgrade to three.three from three.23.one and three.oh.16:25So three.four from three.33.one and three dot.16:45Two. Now, of course, if you are looking to upgrade to three.five, you will have the options of three.43.three, and three.two. So it's always important to double check, make sure that you have the current version, otherwise you need to have a two step upgrade, which means that you need to jump to a middle version.16:48Again, this is gonna be really useful because e.g. if we have, we are on three.O, you need to upgrade 1st to three.13.two eh before moving to what on this scenario to three.two or three.three if you want to move to three.five. So it's gonna be critical for you to consider this information.17:06And of course plan this different upgrade. Now, there are different ways to make the upgrade, you don't need to do it in one step. It's important for you to be aware, we are gonna be learning a little bit more about India upgrades, but it's gonna be important for you to consider this. And just as an advance.17:26E.g., whenever we are talking about a backup and restore, it's much more easy to move in a two step upgrade since we can create a new virtual machine or choose an appliance, make the upgrade to that version, and without moving all the other the all the other nodes, we can, e.g..17:44Installed in that in in that appliance for VM, the new version or the middle version, sorry, the three.three move the backup and then we can get the backup, reinstall the three.five and then restore again the three.three backup. And after that we can start to move everything else. So it's gonna be simple in.18:01If you had a good planning. Again, it's important to to always make a good planning and here this is a one topic that I like always to speak about. The reason why I'm so recurrent and I mentioned always that it's really critical and important to the planning is because again.18:21We're working with technology. We don't know what may happen. It can be an electric allow-it can be some issue with the virtual machines at some point of time. We may face a lot of topics that we may not be prepared for, ok? So in this scenario, what we need to do is minimize risk, ok? And how do we.18:41Do it by preparing, ok? If we prepare, if we follow best practices, if we have these validations, then we are gonna be much more prepared in case anything where it happens, ok? And at least what I have seen in my experience is that it's better to get prepared, even if we are not gonna use it than.19:01Trying to solve something at that point of time that happened suddenly. So it's always important to consider this information. Of course, what we are looking to do is minimize any type of impact, but the best practices we're providing is to reduce that risk, but please always consider to follow them because this is gonna give you.19:21The abilities to recover and also have all the information to get back online as fast as possible, ok? Now, e.g., backups, configuration operational, backups are important. Configuration. Remember that always contains all the information. Operational.19:40It's gonna have all the information and it's good to have it. It's not something that it's not gonna be on on the backup but.20:00It's always good to have it, ok, as a backup, to have all the information there, it's good to also export the endpoint CSV. Load balancers, information about load balancers is gonna be critical, we need to be aware of them. We need to know how we are using those load balancers. Being honest, it's gonna be much more difficult if we don't know what's happening or how.20:09We need to handle them once we are making any changes on the deployment, of course, whatever it's around ice, it's gonna be important, ok?20:29Just to mention something that is in the in the following part, but e.g., also active directory is important. If we're gonna use RSA it's also important, we need to have awareness about what is around because at the end, if we're gonna make any changes, we need to be aware that.20:38Some relationships should be considered just to make sure that everything is working as expected. Now export certificates and private keys, that one is critical, ok? Actually, if there are any customers that I have spoke to in the call, you are probably familiar but.20:55Export certificates and private key is a best practice for upgrading, but I always mention.21:15It's critical always, not only for for operating, ok? Whenever you are importing a new certificate to ice, always make sure to export the certificate and the private key for that, for that one. The reason behind it is because again the during normal operation of the ice, something may happen and if you need to restore something, it's always to.21:20Need to have the certificates.21:40You don't want to miss that part. Export internal CA certificates from CLI, it's also gonna be important for the for you to have them and of course have all the information completely back up from the from this perspective. Now taking notes, taking notes, of course, again, it's important to have the MDM credentials, active directory credential.21:41Don't get credentials like RSA, profiler configuration for HPFN. Remember that here the profiler configuration maybe different, so it's always important for you to consider that this information is really important.22:02And it's always good to have it at hand. And of course, we need to clean up, clean up delete or delete expired certificates as I always mentioned, and also it's a good best practice not only for upgrading but try to clean up your certificates. Try to don't have any nothing new certificates. At the end, we only want to have.22:14The certificate we are using in the eyes, we don't want to have any certificates that we are not using or that they are expired. At the end, those are gonna cause issues or may even.22:34May help us or they may may affect us may affect that the option that we have visibility on ice. E.g., we maybe working with certificates and if a new TLS certificate is gonna expire and we have an alarm of certificates, we may lose it because of that visibility. So always make sure to clean your expired certificates and.22:45Nothing new certificates. Projects operational data inactive endpoints and guest accounts, those are best practices too for the upgrading, but also consider those for the daily operations. It's always good.23:05To clean up the eyes and of course if we have inactive endpoints start the cleanup and the guest accounts, probably those are not in use anymore. We need to keep our review and a constant cleanup. Also, do not forget to disable automatical backups and configure a repository and download the latest URT and upgrade bundle.23:17Of course, it's gonna be affected a little bit on the type of upgrade that we are gonna be taking a decision.23:37Here, e.g., on the upgrade readiness tool, this one is gonna be really useful whenever we are doing a split upgrade, ok? Now, the split upgrade, you are gonna realize that now we have a new split upgrade in a few slides too, and are you gonna help us with the demo for that one. But it's always good that if you are running the the.23:43The basic or the let's say the older split model to download and run the URD. This is gonna help to clone the database and make sure that we everything is ready for the upgrade. Of course, we may have a failure, we can review with that log bundle to get more information about the reasons behind that error or we may have an.24:03Like we can see on the next slide, e.g., we can see some timings on how is our estimated times for the upgrade. Now, of course, this is gonna be critical for us to to understand that.24:24The split upgrade usually it's a little bit longer. It's gonna take more time because we are gonna make or move the the notes a little bit more sequentially instead of doing everything in one philosophy. So just consider that part that part, the URT is gonna give you some estimates, but it's important for you to consider also.24:36How you're gonna be moving those notes. Now, on demand is health check. Another part that I usually talk with my customers and this is really good and it's critical for the upgrade but also from time it's good.24:53To review our health of the eyes, ok? Here we have on the on the ice and the administration system, the health check option. You can run it from there and it's gonna give you or validate information about the deployment, ok? Again some critical errors unless you are not the.25:11I think it's already from two.seven, but if you make you don't find it is because it was introduced eh I think it was introduced also in two.seven in in one patch. I to be honest I don't recall which batch was exactly at this point of time, but it's important if you can run it and you find it in your deployment to run it. The reason is because it's gonna review a lot of this information.25:28And give you if there's any issues yet that you need to take a look at and resolve before making the upgrade.25:48And of course, as I mentioned, in a daily basis, you can also run it not daily, but e.g. in a week or a month to make sure that everything is working correctly. Sometimes we can find issues with the DNS or probably the certificates. Also it could be regarding the disk space or some things regarding the disk specification.25:54The Vmiss.26:14So it's always good to take a look and double check. Now, schedule the maintenance window. Here, this is gonna be important because at the end eh, I think it's critical for you to consider how much time do we need, ok? This is gonna depend a lot on the type of upgrade we are gonna.26:15Need, of course, to consider a few things. Important always communicate the possible downtime. It's always critical because at the end, if we are 15 notes, it's possible to have a downtime or we may have a reduced capacity, so it's simple or important always to consider that information and make the communications as we need it.26:35Minimize downtime, of course, don't upgrade all your psns at once. That's something critical because otherwise we may have a current outage. So if you are planning to do it because we are gonna see that there's an option, you can do it.26:54Sorry, but always remember that if you don't want to lose the services, you need to consider to do not upgrade all your psns at once. Always schedule extra time, you might need it. Again, we don't know what may happen in the middle of the of the maintenance window, maybe we're doing an upgrade.27:10We may find a bug, we may find an issue with the virtual machine, we may find.27:28A lot of topics there. So it's always good to scale that extra time for us to be able to resolve or even run a roll if needed. Now, how we can estimate that information? Well, for the standalone on the administration policy service monitoring, we can consider around 240 min and 60 min for every 15 gb of data. It's an estimate time.27:34And for the distribute here are the times for each one of the notes, e.g., secondary admin, 240 min policy service notes, 180 min and monitoring 240 min plus 60 min for every 15 gigats. At the end, this is gonna change a little bit.27:54And again and always important to mention this also is gonna change according to the type of upgrade. It's not the same whenever we are doing a backup restore, then in this scenario doing also a split upgrade by the graphic interface.28:10So be careful with that part. Now, let's talk a little bit about the perform, the upgrade itself. We're gonna be talking about three different things, the deployment type, upgrade types and process and upgrade options. Now, of course, probably we are familiar with deployment types, don't confuse this with VM.28:26Sizing. Why? Because I have seen a lot of confusion here. We are talking about, oh, I have a small eh, a small VM, a small VM is not equivalent to a small HA deployment. A small HA deployment means that we have two nodes or even three. Now we have.28:44These two notes and we may have an additional PSN and for the medium is when we have two nodes as administration and monitoring and we have up to six pfns, ok? This is gonna be important for you to consider which is the type of deployment you have on the large.29:02Administration monitoring are independent, each one have its own server or its own VM and we have up to 50 psns and up to four PX grid which are independent, ok? Why this is important for you to know because at the end.29:17E.g., according to this to this type of deployment, we can decide regarding the upgrade type. E.g., the split, it's gonna be really useful on the medium and large. On the small, we have only two nodes, so we need to consider that yes, it's gonna be a split, but of course we have only a couple of movements there. So we need to be careful.29:33And of course.29:53Make those considerations really and define the specific steps that and of course the warnings that we need to take. Why? Because of course if we have only small issue at that point of time whenever we are making the upgrade, we may lose that. So be careful always consider and then we are gonna be talking about the upgrade types.29:54And the methods. Now, we have the upgrade types, we have the split and the new split, ok? As you may see here, the split upgrade is a multi step sequential process which is gonna reduce the downtime, and of course it takes longer than the full upgrade, which we are gonna be talking about in a second. Now.30:14The new split is gonna make an upgrade in batches, it's gonna increase the stability and reduce downtime. It's gonna take a lesser time than the older split upgrade, it's not gonna require and it's gonna have some prechecks.30:34So let's say it's an improved split upgrade, ok? So it's gonna be important for you to consider it? Yes. But now we need to consider a few things about the upgrade methods too. Why? Because in this case the new split as I, is gonna show us, it's really important and we are gonna be looking through it.30:47Or the upgrade on the graphic interface While the split sometimes we need to make sure that we have the right split. We are gonna be talking about different split methods that we have, ok? Now, talking about the operate types here a little bit of a comparison between the split and the new split.31:04The total time for upgrade of course is longer on the oldest upgrade, it's less on the new split. Total time for six notes, it's escalated to 24 h with a comparison of 7 h with iterations. Now that you are required, here is not required, rechecks and are not available while in the new split they are available.31:23Available. Recommended maximum notes.31:43By iteration four here on the older split now we have 15 and of course installation can be done on the split upgrade during the upgrade itself. So it's gonna be an improvement. Here, e.g., also it's gonna we are gonna be talking a little bit about in the demo about some upgrade issues that we may find, we may find.31:45In the new split upgrade and how we can detect and help to resolve them, ok? E.g., on a failure after reboot, how we can make a fresh install the node and jump back to the deployment or in the in other notes, e.g., if we are talking about, the failure before the reboot, how we can automatically jump back to the old deployment. So it's gonna be important to.32:05Will check this information. It's gonna be critical for you to take that decision. So.32:25Be really pay pay a lot of attention on the demo because I/O is gonna help us with this part. Now, the next upgrade is the full, this one is the risky one, let's say. The reason behind it is because it's gonna be the quickest. Of course, it's gonna move the deployment almost in two steps, which is the primary admin and then.32:31Everything else? But of course, we are moving all the notes in one in one step. At that point of time of course we have an outage. So it's important for us to consider that if we want to run this full upgrade, it's risky because at the end we are moving everything, we don't have any services at that point of time, but of course it's gonna be faster.32:50Now, let's talk a little bit about the upgrade options on the split upgrade, ok? This is what I mentioned. We have the CLI based interface base and backup and restore. These are three different methods that we have for the upgrade which are gonna be really different. The CLI usually it's more recommended for export users which require to do.33:11Shooting if there's any issue with the graphic interface, of course the CLI is gonna be one important step.33:30Eh, some customers decide to go directly to CLI because they are familiar with the process, they are a little bit more experienced with it. So it's good if you decide to go through that path too. The graphic the graphic interface is usually an single click upgrade, it's gonna be most more simple. Of course, it's gonna maybe be taking a little bit more. It's gonna.33:36Push all the upgrade bundle to all the notes automatically, so usually that's a little bit more friendly let's say. And the backup and restore, let's say that it's a middle step, ok? Usually it helps a lot whenever we are moving regarding virtual machines or appliances.33:56Of course, we can back up the old version and restore on the new version. It's gonna minimize downtime and works best for virtual deployments whenever we can change those VM machines fairly quick, ok? But also if you are moving from e.g. from appliances and you need to build up the the new deployment, maybe you can take advantage of it too.34:11So it's gonna depend a lot on what you are looking to accomplish. Here, let's talk a little bit about the graphic interface very quickly, as I mentioned is a single click. The step two is a customizable option for PSN order operate psns in tandem or groups. And once it's finished, promote the original plan and MNT and install latest plan.34:31Again, patching is always critical. Please don't forget the patching. Usually whenever we have the base versions, of course, those are good, but we always keep improving them with the patches. So whenever you are moving, always double check which is the most important patch.34:51In the scenario, e.g., review the golden the suggested golden release, it would be a good a good practice to double check and make sure that everything it's on the best, on the best way or in the best state as possible, ok? Now.35:07The CL, the CLI as you may see here, it has a little bit more of steps, not a lot. This is recommended for troubleshooting. We it's a manual process. We need to upgrade each node individually, we need to copy the upgrade image to each one of the nodes. We need to prepare and execute the upgrade.35:23And monitor each node individually. Of course, after we finish the upgrade, we need to install the latest patch. But this is really good because we can't see what's happening on each one of the nodes while the upgrade is happening. And also we have the backup and which is mainly backup configuration database, install the new version, restore your backup.35:38And joins the note to the new deployment and install the latest batch.35:58That's pretty much the process to follow up. As you may see, this seems a little bit more simple, but of course since we need to be installing and creating new virtual machines, that maybe a little bit more complex. So what's the best best for you on this oldest split upgrade? Well, it's gonna depend.36:02This is regarding the complexity, as I mentioned, back to the medium one see like the complex, but of course they are gonna have different things or different features in this time, e.g., for the service medium, the graphic interface is the longer.36:19And the CLI itself so medium. But if we are talking about rollback, then restore doesn't have a rollback while the graphic interface is limited and the CLA has the option of the rollback. So it's always good to take a look here. E.g., error resources previous architects, it's gonna be important for you to consider this.36:37So always make sure whenever you are doing a split upgrade regarding these specific scenarios with which is the best for you, ok? Now, very quickly for the full upgrade, we have different prechecks that we need to double check.36:54We're gonna, we need to check if the repository is configured for all those nodes, download the operate bundle and help to prepare the operate database for all the nodes. Of course review that we check if the is enabled. These checks are important for us to be able to be successful with the full upgrade, we are gonna have them and are gonna help us to have a.37:11Let's say I'm more quick and minimize the risk for this full upgrade. Now, being honest, from my experience, I haven't seen a lot of full upgrades because again usually critical on the environments and we are trying to minimize the impact instead of giving more, let's say priority to the speed of the upgrade.37:31But again, if this is your scenario, you may need to consider this part and always make sure that you are complying with the pre checks. Now I would like to share a little bit more about from your side regarding the upgrades and upgrade methods here, which is the preferred one that you have.37:51Graphic interface CLI background restore graphic interface. And yeah, being honest here, the graphic interface now with the with the new split of it's gonna be really, really, really important. So it's something that it's gonna be critical here yep graphic interfaces.38:1070 %. That's amazing. So I don't want to take a lot of time because I want you to take a deeper look here on the graphic interface demo. So we're gonna jump with I/O to review the graphic interface, new split upgrade and the CLI upgrade them.38:28Thank you so much Miguel. And I'm happy to see a lot of people that want to use the graphic interface because, yeah, the new speeds upgrade makes that very, very interesting and that's what we're gonna be walking through in a second. So let me it's trying to.38:46Wait time reopened, give me 1 s.39:04Okay, it wants to end the session to share the screen. I'm not sure why it's trying to do that. Give me 1 min alright while it does this, let me log in here.39:12Okay, so I put in a different setup and it wants me to cancel. So I'm gonna have a second session and I will start talking about this while I come in from there. So give me 1 s. Yeah, but the new graphic interface, there's lots of detail in there.39:28And we'll be working through all of that. Let me put their notes here Okay so what I'll do is I'll have Miguel talk about the.39:44Post upgrade task, which is what should come after this, and I'm just gonna reenter the session because there was a security update and it wants me to update the security settings before I can share. So, I will rejoin in 1 min while Miguel talks about the post upgrade methods.40:04Okay perfect. Thank you. Let me share here again my screen. Let's talk a little bit about before, before the demo. Okay, the post upgrade tasks are critical because at the end, whenever we are finishing the upgrade, it's always important to double check, ok? That's the reason this is gonna be critical.40:19Post upgrade, it's always important. Why, which are some best practices and why it's important because at the end, e.g., we want to run an on demand health check at the end to make sure that at least the sanity checks are run and that everything is working correctly. We don't want to get caught by DNS not failing at that point of time or that something happened.40:39E.g., we missed that in.41:00The password that needs to working, we need, we want to review different things there, ok? And of course the reason we want to test and verify and run an on demand health check is critical for us to make sure that everything is working at least correctly. If we are talking about CLI, e.g., we want to run a cleanup from previous upgrades with application upgrades clean.41:02It's always important because at the end we have that preparation. Of course, remember that one of the best practices was to also disable the backups, ok, that the automatic backups. So in this case, what we need to do is always reconfigure those backups. It's really, really important because remember that backups.41:22Well, I think I don't need to say this, but those are critical whenever we are operating just to have like a a safe in case something happens. And the enable automatical failover is configured and hard bit for the. It's a feature that usually makes the failover of the pen automatically.41:42Of course, not everyone has it enabled, if configure is important for you to enable it and disable it before the upgrade, but in case you you don't have it.42:02This is not gonna impact you. At the end, this is mainly a decision that it's gonna be important. And this is mainly for the post upgrade task, important always to test and verify again. I'm gonna mention it again because that's critical. Not because we have finished try to push everything, always make sure to have time on your maintenance window to step.42:11To test and verify your different use cases, just to make sure that everything is working correctly.42:31Hiya are you ready? Yes, thank you so much. Right. Yeah, the sharing wanted me to update some settings, so now let's go do alright.42:37Let me share this and we are good to go alright now we are there. Alright, so let me set the pace for the demo because it is an interesting one and I've tried to come up with a scenario to make it interactive so that I.42:57And kind of follow and sort of put yourself in my shoes as though you are the one doing this upgrade. So you've been given this task. You have two ICE deployments. One is a six node deployment and the other is an eight node ICE deployment, and they are both on ICE three.two. Now we need to upgrade the six node ICE deployment to ice three.three, which again, we know end of life.43:17Has been announced but it's very lab, so we're gonna go with that.43:37And but our focus is gonna be on the eight node ICE deployments which we are trying to upgrade to ICE three.four. Now, the preference is to use the new split upgrade method, which Miguel just walked us through, so that's gonna be our emphasis. But we are also told that if necessary, you can use the CLI as well. This sounds like.43:40Like a scenario in the CCIE exam, so let's go with it.44:00Okay, so 1st of all, let's get into our three.43.two setup where we are currently running ICE three.two. So let's confirm the version of ICE we have and also the patch. It's always very important to confirm that.44:05As you can see, we have ICE three.two and it is not patched. Now, if we follow the details that we've been sharing, even though we can upgrade with the new splits update from three.two, there's actually a patch requirement and we get to that in a second.44:21Now, before you upgrade, always check your deployments, review the details. As you can see, we have two dedicated admin nodes, two dedicated monitoring, and we have four psns, two with PX grid, two with device admins, so we look good. All right, coming to the upgrade page now, we have the option for full upgrade.44:36And split upgrade. If I decide to upgrade now, if you're familiar with this screen, this is the setup for the old split upgrade method where you only get to upgrade the admin secondary 1st, and then after that you do the monitoring primary, then you can do your psns and after that you have to do the secondary.44:54MNT and finally your primary admin. So this is the old split upgrade method which takes a lot of time and you have at least five iterations to complete it. So we don't want to do this. So what we have to do is to patch our three.two environments to at least.45:16Ice three.two patch three. Now this is the same environment that is now on patch three. And if we go to the upgrade window, you will see the difference. So again, still three.two, but just adding the patch shows us the new split upgrade. Now by default, the full upgrade is highlighted, but that's not what we wanna do.45:32Because that's gonna mean full downtime which we don't want, so we want to go with the splits and it's gonna give you this information letting you know, what is required and if you're doing the splits upgrade services, I'm letting you know that right now any change we are making will not affect our environment. All ice services will keep running, and again we said the same message yesterday.45:52Saying hey.46:12This is a preparation stage. I services will not be suspended so you can proceed. So this is great. Now we're looking at the details and what we need to do to proceed with this new split upgrade method. This checklist, as Miguel was emphasizing is very, very important. And we see that has now been.46:13Embedded into ice. And I think we added some previous versions as well, but you need to go through all these items and we need to check.46:33That box that says I have reviewed this, but we'll come back to those later. Before I even start the upgrade, I want to know the state of my environment. NTP, DNS, general information. So still staying within the GUI, we can go for the health check.46:40Just to review how things are at the moment. So I've triggered the ICL check. Again, this does not impact the environment. It just runs some diagnostics for us and gather the data. So platform check DNS checks, certificates, system sets, our disk space, making sure NTP is working as expected, the load average.46:56MDM validation, our processes. So he said this is.47:16A good thing to run and you can even do this proud to upgrading your ICE environments once in a while, it's good to check how things are running. Now this is very interesting and do pay attention to this. I have three DNS servers in my environment and it's telling me the resolution is working for the 1st DNS server.47:19But it is failing for the other two DNS servers. So again, I saw this, but I said, well, at least one DNS server is working and that is sufficient for me. So I think I'm good to go ahead with the upgrade, but I'm gonna download the reports just in case I want to review the things I checked during my.47:36I'll check, so now that we have this, let's go back to the checklist that we have for the new split upgrade. And I saw this question in the chat. If you need to be on the latest patch of ice before you upgrade, now it is not composory, but as you can see, I have gone from patch three to patch nine. Even though it's not composory, I will.47:55I would say don't be on the patch that is too far from the latest. So patch three to patch nine is a whole lot. So for my lab and the best practice I have chosen to have the latest patch, which is patch nine or three.two. So now we're gonna proceed again with the upgrade.48:15We're gonna go back to our checklist, go through the items on that checklist and walk through them one after the other. Some we have already done, so we'll just have that noted in our notepad, and the ones we haven't done yet, we will take the steps to go through those details and make sure it is done.48:31So 1st of all, we want to make sure we have a backup of our ICE environments, we do have that and we'll see that in a minute. Miguel emphasized the importance of exporting our certificates and private keys and we'll, we'll get to do that as well. To confirm the paths, yeah, we do know that three.two to three.four is a single path. We're gonna review our upgrade bundle in our.48:48Repository and also a patch.49:08We'll review AD credentials as well, and then when necessary, we also need to purge excessive operational data. So let's go ahead, come out of the wizard for now and go review those checklist items. Okay, so 1st of all, let's review and make sure we have the latest backup.49:10As you can see I took a manual backup.49:30Just two days ago and I will review to make sure that is available within ICE. So for this new split upgrade, it checks to make sure you have a backup that is at least seven days old. So anything older than seven days, you'll be required to take a new backup. Now I'm checking my certificate and you don't have to export every.49:33The certificates you have depending on your use cases. In my use case, my key certificate which is composory, the one you're using for your admin, and if you're doing eap authentication, definitely that is very important and most times it won't be a self signed certificate. So in my case, I'm experts in my EAP.49:53Authentication certificates and my admin and guest bottle certificates. So very important don't go for certificates only because without a private key you cannot re import that certificate for use within ISO please we are exporting certificates and the private key.50:11So I have that selected and you have to do this for all the nodes in your deployment. In this scenario, because it's a lab, I'm not gonna do everything during this recording, but the same process you did for your admin, you have to rep repeat that for all the other nodes especially if the certificates.50:26Our generated by individual csrs rather than a wildcast certificate. Alright, that's one other thing of my checklist. Next, I wanna check my repository, make sure it's good, and I'm gonna confirm the files I have in that repository to make sure it has my latest.50:46Backup and also the patch I'll be installing at the end. So this session has expired, let me just refresh this and get back into it. Let's skip that 1 s. All right. We are back in our primary admin node. So let us look at our repository information and confirm the files with.51:01Have in there.51:21So that's our repo or FTP, so let's confirm that we have that. So show repository command will get us into the details. All right, there we go. I do have my patch file and I have my main upgrade bundle as well, as you can see from three.one to three.X.51:22It's trying to go to three.four. We are going from three.two to three.four, so that is our ideal file and we'll also be installing the latest patch of ICE three.four. And as you can see, I also have my backup file in this so.51:42Our repository is working as required. Let's see what is next on our list. Identity certificates, identity sources rather. In this case we are joined to an AD environment and just to see now that I actually have a PSN that is not joined to my.51:57At AD, so I'm gonna fix that.52:17What we need to have ahead of the upgrade are the credentials for our integrations in this case AD. So even if you're not given the credentials, make sure you have someone on your AD team available to jump in if that help is needed. So make sure we have that. So in this case we do have our credentials and as you can see, we've used that.52:19To join our other deployment that was not integrated with AD. Finally, let's check our disk space to see if there is data we need to purch. As you can see, I have a lot of empty space in my disk, so I'm good to go.52:39And but if you need to purge data, now there is NO fixed date of numbers to use, but I usually say pick logs between five to seven days max. If it's a very busy environment, you can even reduce that to three days. So I think after doing the purge now, we're in a good place and please do all these checks.52:56Before the actual day of the upgrade, so you can do this, you know, days listen to the upgrade, do all these checks, make sure everything is good. I've had lots of customers trying to do this during the maintenance window, which is not ideal. All right, so this is the 1st iteration and you'll begin to see the difference between the old and the new splits upgrade. With the old split upgrade made.53:15I thought the only node I can upgrade right now is just my secondary admin node.53:35But now with the new splits upgrade method I'm gonna use two iterations to upgrade eight nodes. Yeah. Just two iterations to upgrade eight nodes. That's one of the things I love about the new space upgrade method. So I'm choosing my secondary admin, my secondary monitoring.53:40And I'm picking one PSN with PX grid and one PSN with device admin. So I've picked four of the nodes. I'm gonna leave the other four for 2nd iteration and we are good to go. This is why I used to be a CLI person for upgrades, but with this new space upgrade method, I've definitely.53:56Change my mind. The other thing is not only can we do the upgrade to the new version of ICE.54:16We can also install a patch along with the upgrade itself, so that's another big plus. So in this case, we're going to four and we're also installing patch three along with that upgrade. Alright, so here is where the preparation is done.54:21Someone asked the question about the upgrade readiness tool and we'll still see that during this demo, but the new speeds upgrade does not require us to do that because that is what is being done here. And as you can see, even though I am upgrading four nodes in the 1st iteration, it's actually pushing the same upgrade bundle to my ideal, my, my.54:38Primary admin nodes. So we see this has been pushed to five nodes. Primary admin, which I'm not upgrading yet.54:58And all the other four nodes that I am upgrading, so the bundle upgrade has been pushed across to all those. Memory looks good. Pan failover has been checked. Again, these are things we will have done manually if we're doing CLI, I'm disabling panfailover, but again, this is addressed by the new splits upgrade. So we are looking very good so far. We give it a few minutes.55:05Is to make sure all this checks out, which is why we're doing a recording because technically all these checks and downloads do take a while.55:25Now, if you remember when we did the L check review, we saw the error about the DNS. As you can see that is still coming up here. Best practice, and we'll come to this later as we go, if you have anything that is not green or blue.55:33Try to address it before you proceed with the upgrade. Anything yellow or anything indicated in red, try to resolve before you upgrade. That's a key best practice, so please owe that and we will see the reason why in a minute.55:49And it's also giving us very good information here, which is something the upgrade readiness tool used to give us as well, giving us an estimate of how long it's gonna take to do this upgrade, and as you can see, everything is looking good, all green, all blue, except my DNS was of.56:05Resolved. So let's start staging. This is another interesting part. Again, I, services will not be impacted. Everything is still working in your environment, NO impact. But this is where we actually do the data upgrade itself, and then we push that across to all the nodes. So you can see the four nodes that I selected for upgrade.56:21They've all, had that file pushed across. So we are finally ready to start the upgrade itself. And this next stage is where there will be impact and we'll see that notification from ICE in a minute. We see the estimate once again, which is great.56:42And it also tells us how we can monitor the upgrade while it's going on, what we can monitor within this primary ice and also what we'll be able to monitor when we have the 1st node upgraded. So it's showing us all the nodes we have in this 1st iteration, we have the time of upgrade.56:57And we are good to go. So this is that final one in letting you know alright now, from this stage, ICE will be impacted. So technically our maintenance window is about to start and there will be impact on the devices we have selected to upgrade.57:15So this is where we cross our fingers and we take our cup of water or coffee and start waiting for the upgrades to proceed. Alright, so we have 5 h 13 min. So you can go take a break and we'll be back, but not for this demo. This demo, we already have all these recorded, so stick with us. Now.57:30Now very important whenever you're upgrading ICE, if it's a VM or physical appliance or even on the cloud, always make sure you have access to the console.57:50And because if you're doing SSH at some point you will lose access when the processes are restarting during the upgrade, so please always always make sure you have access to the console. That way you know what is going on. And there are some commands within ICE that allows you to monitor the upgrade as well and Miguel mentioned some of these as well. So during the show login application.58:00One including upgrade will give you some of those details. So we're not going into all of them today, but as we review, we'll definitely be doing some troubleshooting. So now we have access to the CLI, we can see what is happening when the reload starts. So we have 5 h 10 min.58:20And one thing to also remember that I said at the beginning is we are actually upgrading two deployments. So everything I've been showing you so far is for our eight node deployment that we are upgrading from three.two to three.four.58:38Now this is our three, six node deployments and we've done exactly the same thing so far. The only difference is because this is a six node deployment, I'm upgrading in two iterations of three nodes each. So everything you've seen me do for the eight nodes.58:53I've also done for the six node deployments, the only difference is we're doing three nodes in one iteration. So let's continue and I have the notes in there just let you know we are switched. So our six node deployments and to observe the upgrade. So yeah, we see stopping the high services.59:09We come back after 4 h and 29 min and we can see our upgrade is complete. No problems at all. And this is what we want to see when we upgrade ICE. It also gives you a summary after that 1st iteration of upgrade is completed, and please download these reports. You might not need it.59:25But sometimes we do need it. And after that, you click on next, it lets you know, ok, you've upgraded three nodes successfully. Second iteration, you can choose to upgrade all the other nodes.59:45Or if you want to do more iterations, that is also possible. But for this upgrade, we are choosing to upgrade all the other three nodes in our second iterations to make things go fast. Again, we are doing.59:56The preparation stage for the remaining three notes, this is very similar to what we did for the initial three, so we're just gonna observe the data, make sure everything is completed, and then we'll proceed.01:00:11Okay, so I mean this is good because even for for this iteration if you watch we don't have any DNS errors unlike we had in our eight node deployments and so this is even better, everything is green. And the only thing we have is a blue which is informational, just letting us know of a process that is not running.01:00:31But that process is not enough to affect our upgrade, so we'll ignore that. And now we have started the upgrade for the last three nodes. So this is our second iteration and we are downloading the upgrade data to all those nodes, so again, NO impacts on these ones yet.01:00:48But when this is done, we're gonna start the upgrade. Yeah, and we can see some questions coming in the chat. I know we also have a Slido section to put in the questions so try to use the slider for your questions as much as possible. Alright, this is all green. Upgraded data has been transferred across all these nodes.01:01:04So we are good to proceed to upgrade the last three nodes of our six node ICE deployments. This is gonna take 3 h and if you watch, the time is even shorter. We have 4 h for the initial upgrade, which was also three nodes. Now the final three nodes is taking like above.01:01:2340 min lesser than the 1st one. This is very important information especially if you've been upgrading ice via the CLI. In an ICE deployment, at NO point can you have only the admin personnel running.01:01:40At least there should be admin and MNC personnels running, so it lets us know that during this upgrade, it needs to enable the monitoring personnel and our primary admin node while the old primary, and then it's gonna restart the services. Again, that's the important thing to note here, which is gonna become a useful even as we proceed in this.01:01:56Demo. All right, so we've acknowledged that information. Now we will proceed with the upgrade, and while we are upgrading 2nd iteration, let's actually log into our new primary admin node. So we've upgraded three nodes successfully.01:02:17So let's log into what used to be our secondary admin node, but after the upgrade it is now the new primary admin node and see how things are going. As you can see, that version is now on that node is now on ICE three.three. If we look at the upgrade, so even though we are on the new deployment that has successfully upgraded.01:02:34It is showing us information about the three other nodes that has been upgraded on the older deployments, so again, that's another plus we can review and get those details.01:02:54From the upgraded environments as we proceed. So again, it's looking good and let's see how things go about an hour, about 2 h left for that. And as we wrap this up, we will jump back to our.01:03:04Eight node deployments and see how that is doing as well. So we are almost done with this one. And one thing to also mention, again, let's review the deployments. Okay, that's good. So at this stage you can see all the nodes have been successfully upgraded. So one part of our task is done. Our six node deployment is successful.01:03:20Upgraded. Now switching back to our eight node deployments, let's see how that is going.01:03:40Let's do a quick check. Hopefully things are going well with that upgrade as well. Nope. This is not looking good. We don't want to see red when we're doing ICE upgrades. But as Miguel mentioned, these things do happen when you upgrade and when you use technology. So again, we've done a lot of all that is required of us. We've followed a lot of best practices.01:03:49So why are we running into problems? Well, let's find out. It told me that my secondary node upgraded successfully.01:04:09So let me log into that node and see what is going on. So seventh.eight used to be our old admin, secondary admin, which is now the new primary admin in my ok, so this is good. At least my.01:04:18Secondary admin is now on three.four, which is what I want. So that tells me that notes upgraded successfully. And I'm gonna click on this, which is something Miguel shared with us at the end of the presentation, how you can review the post upgrade tax. So again, the fact that it's showing me all this tells me.01:04:33The, that node upgraded successfully to three.four, and here is a good list of some details to review.01:04:53After you upgraded ICE and make sure all these are done and everything is in a good place. But now I'm still anxious because I have eight nodes in my deployments I was trying to upgrade four of them in an iteration. I am seeing one of them upgraded successfully, which is good.01:05:01But let's find out what happened, why the other three did not upgrade along. So again, not only did it upgrade to three.four, it also installed patch three. So I'm kind of happy, but again, I'm wondering what is going on with the other ones. So as you can see.01:05:17Even though it was just a secondary admin, after it upgraded, it becomes both the admin and the monitoring, because like I mentioned, you will always have admin and monitoring for every ICE deployments you can't have just one of those personnels. So this is where we need to go back to another detail that Miguel shared with us.01:05:32Our upgrade timed out. I have one successful upgrade, I have three upgrades pending. What should I do? Number one, gather as much data as possible. In a production environment, this is where you start thinking about calling TAC. If you did not have a TAC case opened or a proactive tag case and they.01:05:52This is where you have to tell your colleague, hey, I think we need Cisco TAC, let's find out what's going on here. But while you're trying to open the tag case, there are things you can still do on your side and one of them is gathering as much data as possible. So as you can see.01:06:11I am downloading the upgrade reports here just to make sure I have all those data to share with TAC and when they get on. And let's actually review one of these details to see, what's at least went on if we can get more information. So I'm gonna open that and see if it tells me anything new.01:06:24So again, we still we try to upgrade this four nodes. It showed us how much time was taken and still the only thing I see here is that the upgrade was executed, but it did time out. And because it timed out, the other upgrade were canceled. So now I know for sure that the other nodes have not been upgraded, they are still.01:06:43Stuck on three.two.01:07:03And I have only one upgrade on three.four. So I'm gonna try something else here like we always do as engineers or administrators, I'm like, ok, I know ICE zero eight's upgraded successfully. Let me try to upgrade another node in the iteration and it says nope, your secondary admin node must be part of the 1st iteration, so now I know there is a problem.01:07:05And I really need help. So this is where I need to remember information that was also shared with us during the presentation. I'm gonna hold it on here just for a second. So now, which category of this failure am I in? This is, these are scenarios for the new split upgrade.01:07:25The 1st one is for a failed secondary upgrade, which is what I just did. Now I have two options. Failure after reboots or failure before reboots. In my case, there was a reboot, the other node is fully installed, so I know the 1st line is what applies to my scenario, and the suggestion is.01:07:44Do a fresh install for that node and join it back to the deployment. So now I know, ok, with this information, there is NO way I'm gonna get the other three nodes to upgrade in this in this iteration or even the other four. So now I have seven nodes that are healthy.01:08:03And then the other old deployments, only one upgraded successfully. So what I need to do is do a fresh install and get that node back to my deployment. All right, so let's start taking those steps, but one other thing I will encourage to do is.01:08:20You only have one node on three.four, which is not doing anything right now. Let's make sure the seven nodes that are here are healthy and our environment is still up and running, so I've come back here to check. Primary admin looks good. Monitoring two nodes looks good. My four psns, everything is still green, so it looks like we're still in business.01:08:36Now, when there is a field upgrade, you still have upgrade data on these nodes. So we also need to do what is called an upgrade cleanup. So let's go into, the nodes we try to upgrade. Now, this is interesting even though.01:08:56Zero.seven is my primary upgrade node, which I have not attempted to upgrade yet. But if you remember, when the bundle was being pushed, it was pushed to five devices, including my primary admin node. So we run that command application upgrade clean up to start cleaning things up. So second, my primary admin.01:09:14And then I have three other nodes, and I'm gonna do the same. Now this is gonna be repetition and if you can watch, I've increased the speed of my recording just so we can make good with our time. So what I'm doing now is logging into the other three nodes that were involved in the 1st iteration of my upgrade.01:09:34And I'm cleaning up the upgrade data that has been done because if you don't do that and you try to upgrade, you will have some errors especially if you're trying to upgrade your CLI, so now we are doing a cleanup. My colleague opened a case with TAC, but it didn't open it as a SEV one, so we still don't have a TAC engineer on the call yet. So while we are trying to.01:09:51Take this a SEV one case and have TAC join us and let us do what we have learned so far from the best practices.01:10:11Webinars that we've attended. Now, we know we need to reset the node that upgraded successfully, so now put that into the ISO and I'm gonna reinstall ICE three.two on that secondary admin node that upgraded successfully. So again, we're not gonna walk through all the details, but just to let you know when you reboot from the ISO.01:10:18So after ICE is installed, the 1st thing you wanna do is your setup.01:10:38And I'm showing this for your reason because we're still gonna ask something repeated here with respect to our DNS servers that we configured earlier, so I'm config I'm configuring this node exactly the same way I had it before the upgrade, and my goal is once I have it back running ICE three.two patch nine, I'm gonna rejoin it to.01:10:42My deployments and I'm gonna have eight nodes working again. The only thing is those eight nodes are still gonna be on ice three.two. And the goal is to upgrade to ICE three.four.01:11:02So let's see how we achieve that. All right, so I have reinstalled ICE on my secondary admin node. Let me verify that's on patch nine again before we can rejoin, we need to be on patch nine, so yeah, we're back on three.two patch nine.01:11:13So now I can add this node back to my full deployment. Yeah, it's a standalone, which is what we expect. So that's good. All right, so yep, standalone, this looks good. So now I'm gonna go back to my old deployment and I'm gonna add it back.01:11:30As my primary monitoring node, oh secondary admin routers, so we're gonna have this added in a second. And after this is where things are gonna get interesting because now you need to make a decision after I get all my eight nodes restored.01:11:46Do I want to leave the environment on three.two and just have another maintenance window to finish this upgrade or if my maintenance window is running from Friday to Sunday night and I still have two full days, then I might as well try to upgrade via the CLI. And as we mentioned for this demo.01:12:04We are told that if necessary, we can upgrade via the CLI. So I've spoken to my colleague and I've said, hey, you know what? I think we are dangerous enough. We know ICE enough to upgrade via the CLI. So we said, alright, that's what we're gonna do. So let's go for the CLI. Now you see all the nodes are synced up.01:12:22Three.nine patch two, we're looking good. So let's shoot for CLI upgrade. I think we are good enough to do this. So we're gonna make that decision and we're gonna choose to upgrade eyes via the CLI. Now, this is where you begin to see the difference between upgrading via the CLI versus.01:12:40The new space upgrade. The new space upgrade, we just had to go through our checklist, do a few things, check the boxes, and we're told we have 5 h, go grab a coffee and come back. And ICE was doing everything, but via the CLI, you're gonna do the at work. So again, always check the version of ICE. We've confirmed this three.two patch nine.01:12:57Always check to make sure all services are running, we've done that. NTP service looks good. So again, all these prechecks were done for us automatically. One way using the new space upgrade, but for CLI you have to do all this yourself. Check the disk, make sure it looks good, alright?01:13:17So you see we are taking extra precautions because we already had a failure with the upgrade, so now we're checking everything to make sure we don't run into any issues again. So that looks good as well. I think we're doing good with time. Alright, let's check DNS again, we did get a warning about our DNS, but here we've done an NS lookup and it's.01:13:33Telling me, yeah, it's it's working. It's using that 70 seven.five server and the resolution looks good. So DNS looks good. I'm gonna check my repository as well, make sure I have all the files I need.01:13:53Just like we did initially will confirm that as well. The only thing we didn't do via the GUI, which was not required for the new space upgrade, is the upgrade readiness tool, and I saw that question in the chat earlier as well, so.01:14:07Let us go download the upgrade readiness tool that can be found from the download site. So let's log in there. Yeah, I do need to log in. And once we log in, we're gonna navigate to ICE three.four because that's the version you're going to. So for the upward readiness tool.01:14:21And the version you need is a version of ICE upgrading to, not the version of iz you are right now. So we are going to ICE three.four, so I'm gonna go to three.four and I'm gonna search for that file that says upgrade readiness two. We have this downloaded.01:14:38And I'm gonna move this over to my repository via my FTP server. So as you can see, I have put that file and now it should be in my repository, so let's go confirm that with the up arrow, and yes, we have our upgrade readiness tool uploaded. Now remember for the upgrade readiness tool.01:14:54And that is run on the secondary admin node. So all the checks we've been doing so far were on the primary admin node. Again, the upgrade readiness tool does not impact our services, but it should only be run or it can only be run on the, on the secondary admin. If you try to run it on the primary admin, it will actually give you a warning letting you know if you can run.01:15:14On this.01:15:34So again, two things I always do, make sure services are running, make sure you have the right version and patch. Now we run the command for the upgrade readiness tool, which is just application install, put the name of the file and the name of our repository. Now I had this question in the chat as well, like what does the tool actually do? Now.01:15:35Very similar to what we were doing with the new upgrade, where it clones the database, looks at the schema, kind of does a check on your DB because a lot of failed up upgrades are due to database issues. As you can see, it's cloning our database now. I'm grabbing those files.01:15:55Making sure that kind of doing this part of simulation saying, hey, I'm assuming I'm trying to do this upgrade now, i'm cloning the database and I'm gonna attempt to update upgrade it and see that if I run into any issues so that's why it's called the readiness tool. So again, it takes quite some time, about 30 min, but as you can see, it.01:16:14Ran through all these processes, checked our database, check the VM compatibility. I think that question was also asked in the chat earlier. So if you are not running a compatible version of ESXi, that would also be noted here as well when you run this upgrade readiness too. But this is looking good. Everything here is looking successful.01:16:34So we will proceed. Alright, again, just like the GUI gave us the upgrade estimates time, the readiness tool gives us the estimate for each node.01:16:55So it lets us know how long the MNC will take the PSN and the pan. But again, these are estimates, so please always add additional time and also have the habit to remove the tool once you're done. So after we use the readiness tool, I am going to remove the tool.01:17:10So we don't have that staying put on ice. So now I think we are good to start our CLI upgrade. We'll confirm the file and then we'll run the command, so to upgrade via the CLI, there are two commands we, we need. The application upgrade prepare, which does the file extraction.01:17:26And then the application upgrade proceed. So let's run those and I'm gonna upgrade just my secondary admin node for now. Let me do a time check. All right, we have about 15 min, so it might be 5 min over.01:17:43And I think that's because I had to re enter my session. But again, this is a command to upgrade. Application upgrade prepare, put the name of the file, and then your repository name. All right, so we're getting the files across.01:18:02While this is uploading or extracting, I think we have to do the same for all the notes, but for now let's push this across, get this done for the secondary admin node. The, the CLI upgrade is very similar to the old.01:18:17Split upgrade method where you have to do the secondary admin upgrade 1st. So I can't upgrade this and then start doing the PSN upgrade simultaneously. It can cause some failure, even if you want to make it faster, 1st of all, upgrade the secondary admin node.01:18:34After that is upgraded successfully, then you can do multiple nodes along. So again, let's add that in mind. It's, it's best practice to make sure you at least get your secondary node upgraded successfully 1st. All right, let's move things a little faster. We'll run this upgrade, make sure it's.01:18:50Upgraded successfully, and then we can come back to review the details. Again, like I mentioned, always have your console ready. So when it restarts, we can come back and view these details trying to see if I should move it further a little bit.01:19:10But I don't want us to miss any of the key details. So again, we can see the deregistering process in there is getting registered in the new setup, so again this is looking good, so our CLI upgrade is going very well. And now it's gotten to the stage where it says, yeah, it's gonna reboot, and this is where the SSH session ends.01:19:28And it's also advised even within this to make sure you get into the console to see what's going on. And that is what we're gonna do. Alright, so this is our console. We're upgrading, we're looking good and we're gonna see what happens now. After this upgrade is completed, we're gonna rep repeat the same step.01:19:48Yes.01:20:08I see a question there about the UR, I didn't get the other parts of it, but the URLT bundle helps us to validate our database, but after that we delete it and we don't use it for the actual upgrade process itself. All right, so we have our node upgraded successfully. Now we're on three.four. Now, another limitation of the.01:20:11CLI upgrade is.01:20:31You can't install the patch along with the main upgrade, which we could do with the new speeds upgrade. So for now we're gonna leave the patch alone. We want to successfully get all our nodes to three.four. All right, so we're looking good again. We are verifying making sure everything looks good and we.01:20:33We got to this phase when we used the CLI, sorry, the new sheet new speeds upgrade. So let's verify again and make sure we have our dots eight on three.four like we add at the 1st stage of our new speeds upgrade.01:20:53So again, we have that. So again, we are in a good state, NO patch, but we are three.four, which is fine. And we're gonna see this again in our deployment. And I think once this is confirmed, we are confident enough to move.01:21:09All of our psns along with one of the monitoring nodes over. So again, because I had a field L check, I have come in here to run the L check again and everything is looking good. But we can see the DNS is still looking at us, but we have one out of three DNS working, so we're like, hey, there should be NO problem.01:21:29So let's go ahead and do the same for some of the nodes in the other deployments, yeah. Admin monitoring, so we are good to go. All right, so this is where it gets interesting. Now we come in here and we have decided to upgrade all our psns.01:21:49And our monitoring node all at the same time. So we're gonna do that. We're gonna see how it stands for one of the nodes and we're gonna repeat the same across the other nodes. And let me see if I can just push this across a little bit.01:22:07Now 12 min. All right I'm just gonna kind of speed us up a little. So again, the same process we are repeating that pull that across just to get us on time.01:22:26Here we go. All right, there we go. So let's continue from here. Again, as you can see, we've repeated the same process on all the five nodes we are upgrading. Now let's see where things lie here. I think that has saved us a few minutes. Now, we're looking good as you can see.01:22:44All those other nodes we are upgrading have now been joined so that ice or upgraded 1st, so now we are in a good place. So remember we talked about this even during the new space upgrade, you can't have the secondary, you can't have only the admin node at some point. So we're taking a step here to enable.01:23:03Pull the monitoring, we got this notification with the new split upgrade, but because we are doing this manually via the CLI.01:23:23Now we have to do it ourselves, so please this is an important step and don't miss this if you're upgrading eyes via the CLI. So that is good. By the time we come back into this now, we should see our old admin running.01:23:29As both the admin and the MNT. And again, whenever I make a change rather than refreshing the GUI over and over, I like to come and check the status and we can see we are back up and running, so we are good to go. So now as you can observe, we have our deployments, we have the three.two, which is.01:23:46Gradually upgrading and we have only two nodes left in the three.two and now we have six nodes under three.four.01:24:06So before I upgrade the next node here, I want to make sure I have green check marks on the other ICE environment. So let's go back there and see if everything is green. Now there's a problem here because these upgrades have been done for a while, but for some reason I can see that.01:24:13The other nodes are still stuck in upgrading states. Now, this is looking very similar to the problem we had when we were upgrading via the new speeds upgrade. So I'm looking at one of the nodes, I can see it's already on three.4I look at the application status to see if everything is running fine.01:24:30And that's gonna show us. So now I'm worried because I have a new node. We actually have five new nodes that have upgraded successfully to three.four. We have services running on all those five nodes, but for some reason the synchronization is not showing as all green. So there is a problem. So now I'm saying.01:24:49Hey, whatever was the problem with my new space upgrade?01:25:09It's still affecting me even though I have gone and upgraded successfully via CLI. So now I need to go back to my troubleshooting skills and try to see what is going on. Again, I'm verifying services I could do in reality when something is not working, you're checking again like, hey, services are running, right? So yeah, it is running.01:25:12So now let's run some of those commands that we learned with respect to filled.01:25:32Upgrade. Now, ice PSC is one of the very interesting logs you can see within ICE and I've shown that here saying I've run that command and it's showing me something which I do not expect. It says there is a pink failure, and I'm like, wait.01:25:38How can they ping failure? I know all these notes can speak to each other. So let's come to CLI. Let's do what we do as engineers and administrators. Can I ping my admin node from my MNC? Let's see what happens.01:25:54Yeah, it looks good. So why is that a pink failure? Ok, you know what, let's do a pink for the FQDN, so let me try that. I know my DNS was working so this should work as well ok now we do not expect this because DNS was working at the beginning.01:26:10So why can't we ping that and why is it saying the service is not available? Now, at least we know, we kind of understand where the problem is coming from. So even though we are worrying what is going on, we are still happy that we are narrowing it down and that's our corporate. We are resolving with one of the dnss that is not working. Remember we had three dnss and it shows it's in there.01:26:27And we are trying to resolve with one of the DNS servers that is not functioning. So those are the three DNS servers.01:26:48Two of those servers cannot resolve our domain name, only one can resolve it. So now we say, you know what, let's remove those two DNS servers from this deployment and have only the working one. And we need to do this across all the nodes in the deployment. So again, we're saying the pain of us not taking the the action when we noted it.01:26:54During the health check and during the pre staging. If we had fixed the problem at that session when he told us there was a problem with two of the DNS servers, we won't run into this issue, but we didn't fix it. So now let's see what's going on. But again, I've tested with that server and I know that that server is working.01:27:14So again, we're gonna wrap this up in about three to 5 min. Now we know what the problem is. I am gonna update my name server and we're gonna use only the 77.five, which I know is working. So let's make that change.01:27:31And we are hoping that after we make this change across all the notes in our eyes deployment, we will be able to complete this upgrade successfully. Yeah, and this was also interesting when you're making this change, you need to leave the valid one. If you try to, it's gonna prompt you like at least one needs to be in there. So I'm gonna update this command.01:27:46And the command is just a NO, so NO, then IP name server and then remove the servers you don't need. So we do that and it restarts ice services. And I think I saw a question in the chat as well. Someone asking why whenever we make a change, ICE always restarts services. I think it's the way the system works.01:28:06Everything you, the the your DNS, e.g., associated with a lot of services within ICE, it's not like a router or a firewall where the configuration is a single file where you update, you save it, and you can now use that. And in the case of ice, it needs to restart services. And now I'm happy because I have successfully removed the.01:28:26Two name servers that are not working and I have only one name server now which is working, but let's do a quick test. Make sure it is good and fingers crossed. Our ICE upgrade will proceed successfully after these updates we've made. Okay, yeah, now it's looking good. Now I'm smiling again because now I'm I'm.01:28:46Hoping my upgrade will continue successfully. So let's see. Alright, so this is our 77.eight, which is the new primary admin node. Let's come here and fingers crossed let's up. Yeah, it's all green. So all the notes had been upgraded, but because the admin node could not communicate.01:29:04With those notes, we were seeing those things failures in our log, which was due to DNS. Now after we resolve that, all those nodes have been upgraded successfully, so yes, we are almost there. We just need to get these other two nodes across to that deployment and we'll be good to go, but.01:29:24Oh, the one more thing, we have two monitoring nodes here and we also have monitoring nodes over here. That's two. At NO point can an ICE deployment have more than two monitoring nodes? So I need to disable this. If we don't do this and you try to upgrade the monitoring node over there, it's gonna give you an error saying, hey, the.01:29:41That deployment already has two monitoring notes, so you can do that, so.01:30:01Disabled the monitoring to make sure you don't have modern one enabled. So MNC personnel successfully disabled on O eight. So again, to save us time, I I cut that out and now the old PSN has now been upgraded to three.four successfully.01:30:04So now you can see I have my old nine and I have my ten successfully upgraded. So my monitoring nodes fully upgraded, all my psns upgraded. Primary admin is there. So the only node waiting now is my old primary admin node. So our last step is to upgrade that node to ICE three.four.01:30:23And we'll see it in our deployment. And as we wrap things up, now I'm more confident that this is gonna work well and we've seen how to use a new split upgrade, which will have made life easy if only I had attended to the warning it gave me about my DNS, which I didn't resolve, and I've spent.01:30:43Probably 1012 hours trying to fix this issue in in real life, and now we should have the last node successfully added. Let's click that refresh button and we should have our green nodes fingers crossed. There we go, we have successfully.01:31:02Upgraded all our eight nodes from three.two to three.four. Now we need to patch it, which we're not gonna do. We all know how to patch ice. So we're gonna stop the demo here because we have successfully upgraded from three.two to three.four.01:31:20Alright Miguel, I'll pass it back to you now. Let me just hear very quickly some key points to remember. Remember to always choose your upgrade path. It's gonna be important. The upgrade type is also really important because we need to define if we are gonna use a split, new split or a full.01:31:35And always remember to prepare your system for upgrade. Preparations are always critical. Install latest patch of your current release after after the upgrade, that's really important. Never forget to install that patch. Otherwise, we maybe exposed to some risk and also remember to perform the post upgrade task, which are gonna.01:31:54Very important just to make sure that everything is working before we finish our maintenance window.01:32:14Now, before I said, the only topics I have is some live Q and A and additional resources that you can take a look at. There are some ICE videos in YouTube channel, public community compatibility guides, licensing guides, and.01:32:20I also remember that there's a more help in the journey. And I'll get back with team to close this one. Thank you Miguel, and thank you for the great demo there. Thanks everybody for attending today. When you close out the webinar, you'll receive.01:32:35Viva post webinar survey, it would be great if you could provide some feedback there. It definitely helps us to shape our future sessions.01:32:55The recording of today's session will be posted in a post session AMA forum, which you can access using the bottom left QR code here. That will be open for ten days. You can continue to answer questions and we'll be there to answer whatever we can.01:33:02Registration is now open for our upcoming sessions in April covering ICE integrations specifically around Catalyst Center and Entra ID. And then finally over on the right side, we have cyber capsules which can be accessed in both English and Spanish as of today, which are short form videos, basically just covering a.01:33:18Variety of ICE topics. So if anybody's looking forward to, you know, additional short form content that's there, we'll get this recording out to you and would love if you could join the the post session AMA.01:33:37Thanks everybody for attending today and we hope to see you on our future sessions.01:33:50