Deep-Dive on Cisco Intersight - Part 2

I Hello everyone, Welcome to 2nd part of the deep dive to Cisco inter site session. This is the 3rd session in the expert insight series that we are running.00:00Today we have myself Ronak, and Terrence. We are both part of the customer success team in Cisco. We are both customer success specialist. Terence has a vast experience in the domain of data center and he is an expert. He will be delivering.00:20Most of the session today and me Ronak will be there to support him throughout the session today. I am also customer success specialist and I am Cisco certified and CCNA certified as well.00:37And we both will be there today to help you through this session. Without any further ado I will now hand over the ball to Terrence to take the session forward. All right? Thank you Rona.00:52I hope I'm audible. Everybody can hear me? Yes. All right, welcome to our session, expert series deep dive on Cisco Intersight part two. My name is Terrence and I'm a CSS in Cisco Dalian China. 1st of all, I'm not feeling too good these days because my.01:07Sure this is hurting so I can speak loud. My apologies if you find me coughing along the way. We have a lot to cover in this session. It's really a lot and we may not have the time to go through each detail in every slides. Since this is a deep dive I assume that.01:27Most of you would have some basic knowledge on Cisco inter site. Before we start, this is the disclaimer. Basically this is Cisco confidential and the presentation for today's contains a guideline.01:47Right and based on Cisco recommended practices and you may want, we also want to caution that you remain responsible for using this in compliance with laws and regulations and it fits into your.02:07Environmental business. So the session for today is we have start with pools policies, profiles and templates, and then we touch on firmware upgrades and then finally OS installation. So the 1st topic for today, let's look into the four foundation.02:22Constructs.02:43Of inter site configuration objects and here are the areas we'll be covering today. So the building blocks of inter site configurations. So this slide is to give a quick overview of our pools, policies, profiles and templates.02:44Now the question is.03:04The question is, you know, what problems are we solving here? Now we can summarize into three areas and the 1st one is we know as snowflakes. So, a server is known as snowflake when it has been manually tweaked so many times that it caused.03:08Is configuration and settings will become unique, and if you think of snowflakes, you know, they are all unique, you never find two or the same. So when the server becomes unique, it can becomes a black box. So that's the danger and typically it's because due to changes.03:27Not conforming to a standard OSOP. Configuration drift, now even if you have a document or a template that says, you know, all servers should have these settings, but in reality, an engineer logs into the server, make a change, you know, and they walk away just because they want to fix a temporary issue.03:45And ends up, you know, is like differs from a template.04:05And typically that what happens to your testing servers, scalability is that always a challenge if you have to manually configure by yours and then assign MAC address to over 200 over servers.04:10Now hopefully you see the flow from manual change to create to these snowflake issues and then leads to configuration drift and it can becomes a scalability challenge. So inter site try to comes out with a way to cope with this problem and eventually.04:26We have these four basic.04:45Building blocks. And the final point here is that the mental model here is an inter site is a decorative system. You don't configure the device. We define the intent and this intent is actually achieved through pools and policies and the execution of this intent is done through.04:48Profiles and deployment.05:08Right, starting with pools. So, what is pools? Pools are just predefined collections of unique identifiers and we can automatically allocate them to our infrastructures and pools enable.05:12Automations and scaling. So when we deploy our profile values are drawn automatically from the pools, NO manual interventions and even from a perspective from a scaling perspective, the effort is the same even when you have to deal with more servers.05:28Poor location behavior and they are all identifiers are assigned in sequential or ordered allocations. So making it easier to track although we suddenly do that.05:48Types of pools. So there are two types of pools. Identity pools, which is marked in blues, and they provide unique identifiers to servers, and then we have resource pools marked in orange, and these are this is grouped, this actually groups similar resource.06:04A few things to call on the right pertaining to best practices. So 1st thing is the planner OUI prefix that's the organizationally unique identifier. So for Mac we have 0025B50 yeah 0025B5 and four.06:26WNN WWPN, this starts at 2000. So both are Cisco assigned and are recognized universally. So use this prefix to ensure locations are unique and comply with industry standards.06:44The other thing is that when dealing with pools we try to size it two to three times larger than your current footprint because once it's depleted, the expansion is straightforward, but reclaiming one that's really being assigned to a profile will be messy.07:01So make sure we are looking at a sufficient large amount, large blocks of identifiers. Three is that we wanna place pools in the right level in your organizational hierarchy. Intercite actually support a shared organization model so we can have a soft organization.07:19To create the share to to create the resource pool and then we share with other organizations. So this allows multiple teams and subop to reuse the same pool. Finally is that keep separate pools per environment for different environments.07:39Like your development testings, production, they should have their respective isolated pools and we define pools for each environment and then we assign them accordingly in server profiles or policies.07:58For policies, so policies are basically the configurations or settings of your infrastructure, right? They are defined ones and reused anywhere. Now it's part of what we call a decorative state model because meaning we treat the policies as authoritative.08:18Not the hardware and during deployment we basically push.08:36To the hardware. And the policies are attachable and shareable and that's because we can add to multiple profiles and it's shareable because it's being used across by multiple profiles. It can be cloned, it can be cloned and even compare for consistency.08:41So the guideline is a reuse policy aggressively and try not to duplicate where possible.09:01The next part is on this policy layering and grouping and choose the right layer. So different policies actually applies at different layers. So in fact, when you try to create a policy, right? You try to create a policy, you realize that you have to filter by the platform and.09:11Which is what you see here. You have UCS servers domain, UCS servers, then you have UCS domains, chassis or even hyperflex. And at times that you see that they even overlaps because like network connectivity, it can appears in.09:30Both of this. Now DA, if you look at the diagram on the right here, this is the layering and grouping. So it actually becomes logical, you know, when something breaks and you know where to look for it.09:50So e.g., if your VLAN is not working, so likely you check the domain policies because the domain profile will actually controls the network fabric. The other advantage of this grouping is that, you know, you can have different teams to manage different parts of the infrastructure. So e.g..10:06The network team can handle the VLAN or even uplinks and then you have the server or compute team to manage the buyers or the firmware.10:26Now important notes here regarding the composition, you'll find that some policies are not interdependent. So e.g., if you look at the land connectivity policy, it actually requires.10:39The other three policies to be available. So the high level policies can comprise of low level policies. The guideline here is build foundational policy 1st than the high level ones.10:54Shareability, so shareability boundaries, so whether a policy is shareable depends whether it contains unique identities. So a highly shareable ones e.g. with the BIOS, NTP SNMP, it does not contain any values but purely.11:10Configuration settings. Now the less shareable ones, so e.g. like your LAN connectivity policies because it uses IQN. Now, if your IQN is configured to use proofs, right? So it becomes shareable of course to, to some extent, and if you use.11:31It's a static ID then of course these are static IDs is a single single use. So do watch out whether policy is shareable or usable.11:51Changes an update, nothing is gonna happens on the hardware, right? So when you edit the policies, nothing changes on the hardware. The reference policy just simply go out of sync and you must redeploy. So now let's review.12:00And we're gonna bring the constructs together. So the 1st one will be the server pro server profiles. So profiles takes the policies we build bundles them into a single share deployable units.12:19And then we buy them to a physical server. So we can actually represent it this way, right? Having a profile we go to policies plus pools plus assignment. So this line this line basically define what we just mentioned includes the collections of our policies plus pools.12:32Then we have pools encapsulated within policies and then plus assignment. Now talking about assignment, there's a difference between assignment and deployment. Creating the profile alone will not trigger anything. You need to assign to infrastructure and deploy.12:52Now, if a profile is assigned, right, if a profile is assigned to our hardware, this means we are attaching it to the hardware, nothing happens, right? Keep in mind this is a one to one, after assignment we need to deploy, and this means to activate the settings. So we push the con.13:10Configurations to the hardware.13:29Profiles also have life cycle states, so here are some common ones. I'm not gonna go through each of them. I will talk on inconsistent validating and activating. So inconsistent happens when the profile was deployed or previous.13:34Be deployed and now the setting does not match. So the reason can be pending changes or it can be due to out of sync we change the policy or you deliberately go into the CIMC and then you change the bio setting. So this can result inconsistent and for.13:52And this happens immediately after you change something to the profile because once you change.14:12After a while, you know, inter site will have to validate it. So it can also happen during deployment, the initial stage of deployment. Activating this happens in the deployment and it happens right after the validations phase. So the server is going through the process.14:19Test of applying the policies and this state is actually temporary because the duration also depends on the number of policies involved.14:39All right, types of profiles, types of profiles, so here are some examples of different profiles each depending on the platform type. So e.g. in the case of a server profile is basically makeups of policies and pools.14:52Defined for the configuration of a UCS server. Now these servers can also be a standalone FI attached or unified H. At the bottom, this one is more important schematic diagram of a UCS server profile and.15:12This is what I have in terms of relationship, good to know and it helps to understand the hierarchy. So starting from the policy, right, from policy to policy to profile. So this is a many to many because policies are designed to be reusable.15:29And likewise server profile obviously can contain many policies like boot order or storage. Now policies and pools.15:49Policies and pools, this is also many to many because a single land policy can contains IQM pool or even a Mac pool, so it becomes many to many, and likewise for the pool it can be we can have a global Mac pool that is used by different network.16:01Policies. Now the problem here is pools and profile. This is a bit complex, technically is many too many, but I put it in the brackets because this has to be mediated by policies.16:20Right, he has to be mediated by policies because whether it is whether it is IP pool, IQM pool or a WW NM pool you know they are consumed through policies, right? The only exception is this UUID pool because.16:38Because this is consumed directly by the server profile.16:58Now, of course this is not an exhaustive diagram because if you can dive deeper you may find that policies between policies themselves, they can even contain policies like the same connectivity policies. All right, so now let's move on to the, profile.17:02More templates. So a template.17:21It's actually a profile. So the template is actually a profile, but we often call it a master profile or a blueprints or sometimes we refer to as a golden template. So this master profile is special because we can derive multiple profile from it and your inherent change.17:25Just from the master automatically. And in fact, this is how we manage hundreds of servers without hundreds of individual configurations.17:45Some points to take note, synchronized update, this is what I just mentioned. We can derive profiles and we when we make a change to the template, it will be propagated to every derived profile automatically.17:56Override only when needed. So the derived profiles will inherit settings from the templates, but we do have the options to overwrite. The thing here is that, you know, do this only when needed, otherwise, you know, it results in a configuration drift.18:13One template per row. So the row here refers to the service functions or the service tasks. So the best practice is to have one template for each row. So you may have a web tier server.18:33Your database server or analytics server. So the point is don't try to fit everything into one template because it can result in complexity and errors. So at the bottom it's just a simple high level schematic diagram of how templates will look when we propagate because.18:47It can actually create multiple copies of profiles.19:07So putting it all together, let's take a look at some common scenarios. I was hesitating whether to to include these slides because there's a lot of content and to really describe the situations. But anyway, I'll leave it in, but I will just simply.19:14Go through very quickly just to let everybody understand and see how this thing applies. So in the 1st scenario a, we have rapid provisioning, so we have a new chassis arrived with 16 blades and once we have the pools and templates in place, we can just.19:33Completely derived 16 profiles and assigned to the new service and this means all 16 will go from bare metal to OS ready.19:53In under 30 min. So without this model, the same task will probably take days, you know, of manual configurations. Scenario B will be a hardware failure, I think something more closer to.20:04Most of our cases. So a production blade here fails overnight at 02:00 a.m.. And, the 1st thing is that we will disassociate the profile from the failed server since it is attached with the WWPN and Mac identities, and then we reassign a pre stage.20:21Spare profile.20:40So in other words, we are moving the identity or the field server to the new hardware because the profile will carry the original network as storage identities and so from the OS perspective, they will just see this new server as the original. So the new server will boot up with the same identity.20:41E and then there's actually NO need for reconfigurations or your SAN rezoning.21:01No finally is this the last scenario and in this case suppose your IT, your IT team mandates a new BIOS tuning in your environment that is across 200 production server.21:12All right, new BIOS. And what we need to do is to just update the BIOS policy in the template and every derived profile will inherit the change. Then the next thing we need to do is schedule about deploy and the entire fleet.21:31Will get updated.21:51All right, so quickly let's move on to this what should be the build order like, so creating profiles, alright, so we talk about profiles, if you think of it, it's like raw materials. So your pool come 1st because they are the foundational resource for policies and profiles to.21:59Consume two important points, size for growth and scope to the right organization. The next thing is to define the policies, so your policies, if you think a good example is that is the behavior. So it defines the standards or the.22:19Specifications. So one policy per concern. So try not to put everything into one giant policies and keep each policy focused on the single topic. And then the next thing is these templates. So the templates are actually, if you think of it, it will be like your.22:39Manufacturing.22:59Blueprints. So in this case this is your assembling line. We take all the individual policies and then we bundle them into a reusable package. So we mentioned that keep in mind one template per server rules and the reason is very simple. Different servers need different configurations.23:00So DB servers might need higher memory, where else a web server may actually need network throughput.23:20Derived profiles, so we derive profiles with inherit the templates, inherit from the templates, the policies, the pools and all the settings. Now there is three things we need to do, right? 1st is to assign to specific servers, and then you will go through the validation.23:29Just to check the configurations and eventually we deploy the profile.23:48Common peak force alright some common pit force to watch out. The 1st mistake is that very tempting to create a profile, right? We start with the profile, but actually we need to create a pool 1st. So the point here is that, you know, don't try to create a profile before the pools are ready. You may end up.23:57The meaning has NO MAC address assigned, so always set up the pool 1st before moving forward. 2nd mistake is that modi modifying the policies directly on the derived profile. So we should be actually modifying on the templates. And if you do that.24:16The profile and template will go out of sync, and this means that.24:36The link will be broken between the profile and templates. So the next time somebody tries to update template, the change won't apply correctly. And the 3rd one do not.24:41Create a single monolithic template we mentioned try not to create a giant template for all servers and the reason is that when you change your policy in one row, you end up changing for all rows. So, use separate templates.24:57For separate server rules, it might feel more like a lot of work up front, but it saves you from disaster later alright so we come to the summary and and best practice, so I'll just quickly glance through here.25:151st post reduce errors and the main point is that we want to prevent or eliminate duplicates. Second we talk about policies policies, standardized configurations. Always remember policies are reusable, so it reduce variations and avoid your snowflakes.25:35Problem profile encapsulates the configurations, so use profile to include references to.25:55Policies and pools, templates help us scale. So the point here is template managements make easier to make changes make changes easier across multiple systems.26:03Now on the right are some best practice. The 1st one is try to have a standard before you do anything. So agree on the standard to adopt. So this means define what are your standards, what are the approved by your settings, what should be your approved boot configuration?26:21What is your approved firmware versions? Then we create the policies from those standards.26:41Second one is created ones used everywhere. Define your pools and policies in shared organization, make them available to other orgs through resource sharing. The 3rd one is to tag everything systematically, use tag from day one.26:48To label or mark your environment, your roles, the owner, so on your server profile, you can actually create texts for the environment that says, you know, is, is this for productions or for testings? You can even text the owner who owns this, who is responsible.27:07Should it be the infra team or dev team?27:28So why do we say starts from day one? Because if you wait later and then in six months time, you may really have hundreds of profiles and policies, treat configuration as code, right? We can leverage on inter site API and terraform.27:30Provider. Now if you have attended the last session, I think Eric have shown this. So we did not cover this today, but you can actually export your policies and profiles store them in Git repository so that configuration change become auditable, reviewable.27:51One year, even to some extent you can even rule back as a reference.28:105th monitored drift regularly, so inter site has this built in drift detections, if a pool NO longer matches his template, it will display as non compliant and we should catch this drift before it becomes an incident.28:18Alright, so we're gonna move on to today's second topic that is on firmware update and below other areas that we'll be covering. So basic question is why must there be firmware upgrade?28:43So, firmware upgrade basically is to maintain the security posture or improve performance and then ensure compatibility. Now, the more important I want to stress on is firmware and better upgrade in inter sites, you know, is there any difference and what are the.28:59Okay, so whether is it fabric interconnect or servers? Firmware upgrade means bundle upgrade, alright it's called bundle because it's actually a collection of firmware for the related components and most of the firmware.29:19Most of the firmware components are interdependents and they need to be compatible, hence we kind of bundle them and upgrade the components together. So we have two kinds of bundles.29:39Right infrastructure, infrastructure and server HSU bundle. Now the file naming convention for both always end with.bundle.bin. And in both cases, you will also see that they will have within this file, it actually was a.29:55Contains the.30:15Payload.TGZ, which is a compressed archive. Now obviously for these fis, you will contain the NXOS, the input output modules, intelligent fabric modules or your fabric extenders and then power supply.30:17The case of this server HSU.30:37It actually have this what are you called the out of band alright OOB firmware. And then there is also a non OOB component and you'll see that it actually comes in the form of ISO image.30:41So this portion actually involves the HSU agent running on a CMC to mount these ISO. Hence that's why you see this name HSU appearing. Whole server whole software update.30:57Now at the bottom we have this summary table, a more detailed overview difference between infrastructures and the server HSU. So the key summary info is actually on the 1st line under the descriptions. So we can see that for infrastructures is basically a target.31:14That the UCS infrastructure and is accomplished.31:34Through the FI in the case of server HSU, it targets at a server and is a multi stage process and involving HSU agent and also there is also the ISO boot environment. I'm not gonna go through this list because it's straightforward and by.31:38Glancing through, you can see they are.31:58Actually self explanatory. Right, some important concepts during this bundle upgrade. So for the 1st part is on this infrastructure bundle 1st thing is that firmware are stored on the FI cache. So.32:00Meaning the FI has its own cache for storing firmware files, hence the system 1st download store locally, then we do the upgrade.32:20So, since it has cache, we need to monitor this cache, right? Because if the cache is full, the download is gonna fail. So that's why cache management is not optional but necessary. We'll talk more later behind.32:30The 3rd point is on traffic evacuations, so before upgrading FI traffic is moved away to the peer fabric interconnect. So if you look at the diagram, we typically start with FIB, upgrading FIB and we need to evacuate the traffic.32:46So that the flow goes through FIA. I will path connectivity during upgrades. So before the upgrade system checks, I will path are healthy by monitoring the I/O alarms, we know whether the link is down or if there's any failures in the interface. So you.33:02Want to ensure data path are cleared and before allowing user acknowledgement.33:23And the way is through this 60 min alarm and meaning that we have the alarms and if they are not clear and UCS will wait for 60 60 min and if let's say 60 min up and the problem is still not clear.33:27The system will let the user decide whether to upgrade.33:47But typically if you have these alarms, you know, most of our users would click on it to ratify the issues. Finally we have this dual bank, which I will elaborate in just a while. And coming to this HSU bundle, so I mentioned this is a two phase process.33:53The 1st phase will involve the OOB upgrade and we make use of the dual bank, which is this diagram here. So this is basically a storage comes in the form of.34:13Cisco flexflash SD cards and it's make out of two slots, right? One Mark is primary and the other is backup, and at any one time only the primary is inactive. So.34:27It's running the current or existing firmware and during the upgrade new firmware will be written to the back, backup banks and we just simply wait for the reboot and during the reboot process, we will toggle the switch. So in this way the backups becomes the primary and the primary then.34:42And falls back becomes the backup.35:02Now the phase two operations will be involved in non OOB, which make use of the HSU agent. So refer to the diagram on the right here, basically we are upgrading the rates, GPUs and then the drives. So the 1st step is the server reboots and remember we mentioned.35:06But not OOB comes in a form of ISO, so that's where we need to boot up.35:26This ISO to bypass the OS and this this is what is doing, the server will actually put into this HSU ISO bypass the OS and then you would.35:33Performance inventory scans, update the non OOB components. Alright, so in this we're gonna put everything together and quickly walk through to understand. So we start with this, infrastructure firmware upgrade that is the.35:47Fabric interconnect.36:07We download to a cache in FIA, then we sync to FIB, then step two we stage on FIB, we upgrade components, right? Upgrade all these components, and then in the step three we evacuate the FI FIB and.36:09We will activate the FIB and the related components and the reboots and during the reboot we do the switch of the primary and backup banks, and then we wait for the FIB site data path to clear the alarms to clear, and then after that it completes and then we.36:27Completely we proceed with FI FIA. Now a few call outs traffic evacuations I mentioned earlier, right? We need to make sure that all the data paths. And 2nd one is firmware cache management. So we see that cache management is not.36:48Not optional but necessary. And to avoid the cash full, we can use the we can actually manage the cash.37:07What we need to do is simply SSH to the fis and we can issue this list firmware cache to display all the bundles in the cache. Then we identify the old ones and then we just run a clear from firmware cache providing the identities.37:16Have to remove the bundle and just remember to perform on both fis. Viewing safeguards I've more or less explained in earlier slides. So I'm gonna go through and.37:36On this we talked, we could talk on the server upgrade that is hooked up via pair of fi, so it's basically your IMM domain service. So the 1st thing is we trigger the up trigger the upgrade and then we download to FI cache. Now the question.37:53Question here is trigger. How does it trigger? So.38:13If you look at here, I have listed the ways that from firmware can be triggered. One is a manual or direct user actions explicitly we click on upgrade firmware that is we identify the servers and we go through these navigations. The sec.38:17One way is through firmware policies, remember we say firmware defines the desired states. So through firmware policies we can actually enforce and.38:37You'll see this in the demo and then there's two other methods, right? C and D we did not discuss in depth, but this makes use of workflow. So in this case, most likely we are combining the firmware upgrades with other tasks. And finally this.38:47It's only applicable for VMware, so this is the Cisco HSM plus VMware vSphere lifecycle manager. So this probably in the case where the firmware and ESXi life cycle are managed together.39:07Alright, so after we trigger, right, firmware bundle will be downloaded to FIA and then we sync to FIB, then the next stage will be we stage to OOB components, so the OOB upgrades happens 1st because they don't.39:30Depends on the west. And after that we reboot or we activate the OOB, right? Stage firmware becomes active and then that's where we prepare for the non OOB upgrades. So to do that, we mount the HSU ISO via V media and then.39:47The HSU agent running on CIMC.40:07Takes control and he actually configures a one time boot right from this ISO, basically tell the server on the next boot, don't boot up the OS. So we get into a special environment in step five, you get into what you call the.40:10HSU ISO environment and the HSU agent will actually perform inventory scans.40:30And perform the upgrade. So while he's doing the upgrade HSU agent also report status to inter site right via redfish API, and when status completed we return back to the.40:37Option OS.40:57Right, so the last one will be on standalone server, right? Standalone server. So in this case, again, we trigger the upgrade similar to the previous scenario we just discussed in the previous slides. And then the next thing is the CIMC download the bundle, how we.41:00Download depending on your deployment. So we have CVA and PVA.41:20That's where it does the download, and in this case you will start on the utility utility storage. Because this is a standalone, there's NO FI, so there is NO FI cache. So this utility storage is depending on your platform, it can be.41:27Internal SD or Italy partitions. Then we in step four we stage the OV firmware again make use of the dual bank mechanisms and the next step we reboot and that activates that inactive to becomes the primary bank, you know, and by.41:47Vice Versa.42:06And finally we do the non OOB upgrades via ISO and that completes the whole process. So the key takeaway here is firmware means bundle because we sudden do single upgrades or just by.42:07Was alone or anything. So in both bundles they contains firmwares or.42:27Of the hardware or the mix, in fact a mix of related components. Then we know that there's two bundles, right? Infrastructure, and then there is a HSU, and we talk that they are non destructive.42:33Right, they are not disruptive by design because we make use of traffic evacuations, I will path and user acknowledgement finally, and save by default, right? Because we also make use of the dual bank to stage before activating.42:49And finally below this important, right? This one is important to verify your firmware version against your hardware. So these are two references to look up when you need to do firmware upgrades.43:09Okay, before I move on to the OS demo, so I will show you on the video. So I'm going to play a video and in fact, this is fresh from Arvin. It was made yesterday.43:26Or the day before, but I'm gonna play the video. The thing is I'm gonna play the video at 1.5 times speed. So please bear with me and to give a quick background alright to give a quick background in this video, we'll show the firmware policy firmware.43:44Policy can be used to trigger the firmware upgrade on UCS server. So we 1st gonna show you the UCS server, actually it's this one.44:04Right that has the policy attached and it does not contain any firmware policies and in the video we're gonna show to create a new firmware policies and then we add to the masters and then we propagate the change, it goes to the derived profiles and once we made the change.44:13You'll notice there's activations requires reboot. Now, if users will try to do a reboot, you'll find that nothing's happens with the firmware. It remains at the old versions. So basically we need to do a redeploy of the servers.44:31In this video, we'll show how to perform the server firmware under upgrade using policies. Specifically this will be the firmware policy on UCS server. So let's take a look at the server. This is UCSC 220M6 series and complete the under version of firmware version is six.oh we will NO one two.44:4950192.45:09So let's go ahead and create a firmware policy. And I have a policy here. So this is the policy and you can see that it's currently not in use, right? It's not in use. So I will go ahead and.45:11To Edit this policy, so the target platform is a UCS server and update next. So in the server model I will select the UCS 02:26 and the version will be six.O with a minor of two to 6044. So let's keep this and I'm not gonna make any change.45:31And let's check out the server file. So in my server page and this is the server and we can see that it's being attached with the profile, and in factor profile has been deployed having this green check mark icon. So let's click on the profile.45:50And looking by the name, we can tell that it's being profile. So I should be able to find a template, so this is a template and looking into the profile, we can see that under compute section there is NO firmware policies that's being attached. So let's go into the template and make a change, meaning we've attach the firmware policies to this template.46:08And then it should change should be propagated to the profile. So this is the template and and we can see that in the compute section where there is NO firmware, and I'm gonna go ahead and make the change.46:29So in the general page, there's nothing I need to change, and then I'll click next, go to the compute we can see that the firmware is not assigned, so I'm going to select from my policy. This is the one that we have. So I'm going to attach it to the.46:41Right, it's done and quickly I'm gonna go through and make sure that the rest of each of sections does not show any difference. In the summary indeed, I can see the firmware has been attached to this template.46:59Alright I'm not going to derive any profile so I'll just simply close this and let's go to the server profile. We see that this is the profile and it now shows the status of inconsistency, so we will click.47:12On this changes has been propagated from the template to this and we can see from this label that it mentions that the firmware has been added and it requires a reboot and we can quickly take a look and see that there's a change requesting data firmware to be in this version.47:32So let's try to reboot and see the results. It shows inconsistency and here also shows that the reason is the pending changes. So I'm gonna go to the server and locate our server and we try to do a power recycle. So this is the.47:51Server, and let's try to do a power recycle to see how it goes. So in the power, I will click on power cycle, alright and I'll just say yes. So the request has been submitted and we can see that the power cycle best.48:07So let's take a look into the server. It's gonna take a while for the cycle cycle process to start and complete, so I'm gonna wait for a while and then it comes back to this page.48:28Alright, I think the server has been rebooted, so I'm gonna refresh. YESC has been rebooted and we can see that even after the version remains unchanged and the profile status continues to show inconsistent. In fact, this can be real because there has to be a process involving this.48:44Change of the firmware version because the process needs to download the bundle and so.49:04We can see that if you click on the and it will continues to show that activation requires reboot, activation requests reboot inconsistent and so on. So the message activation requests reboot is just so to effectively reboot to sort of reboot the loan will not work because the pro.49:13The process should take up to an hour. Remember is that.49:33Upgrade of the server firmware is actually a two stage process because 1st you need to upgrade the OLD firmware, and then in 2nd stage you need to upgrade the non OOB components, and that means handling the whole non OOB components would require the HSU agent to mount the ISO so that it can bypass the OS and then upgrade. So what I want to show here in this.49:40This is adding a firmware profile will not trigger the firmware upgrade automatically because it actually requires us to redeploy the profile. So we must newly deploy the updated profile to server and this will force it to initiate the firmware upgrade.49:59Process, so we will go ahead, click on the profile, and then we will say deploy so we have initiated the redeployment and we can check at the status.50:14The device of a profile, and this is gonna take a while, so I'm gonna pause momentumly.50:33So going on and currently is at the point of deploying the firmware policies. We can take a look into this stage here by clicking on the view execution flow. So as you can.50:54See that it.51:14Request to initiate the image download to endpoint, so waiting for the image to download. So previously when we see that the simply reboot, it doesn't work because the process, like I said, we need to download the firmware and this is currently processed so that simply rebooting alone is not.51:15Doing to upgrade the server firmware. So this is where we are and it's gonna take time and I'm gonna pause the video for a while.51:35So as mentioned, the download is completed and is being into the utility storage. So we can see clearly that the request here is to upgrade some of this version to a higher version and we can see that now is that's what we have seen updated some inventory for the OV.51:45Components, so we still have the non OOE components to deal with.52:05So let's so this sub phase has been completed and if you look at the overall process so it's rebooting now so the has completed and.52:09Shows success and we can see that the status here is showing ok. So let's take a look and in the profile, so we do not see the inconsistency. So let's check on server.52:29And if I go to my server page, I can see that it now shows as a thing has been deployed successfully and has been upgraded to what we expect. So that.52:40Alright, so, that is the that is the demo for the firmware upgrades. So let's continue with OS installation. Right, so this is the last topic for today OS installation, and again here is.52:55The, topics, areas that we're gonna cover.53:15And hopefully we have time for the demo. Right, so we're gonna talk about the software repository and, image management. So, what is software repository? I assume you have seen it is basically a management front end and UI to keep track.53:19So of the OS image links and files. So basically we are referring to the firmware as you image OS image or even.53:39Sorry about that or even the OS configuration files. So generally the usage and the purpose is used for OS installation, right? Because during the OS installation process, we need the relevant OS or SU image and.53:48This is where software repository is the place to get it. So, apparently to the OS installation, of course, we need to upload this image to the software repository.54:08Now for firmware, as for firmware, it is not needed if you are on because it's opt in from Cisco repository, but for PVA, you probably have to get manually download it and then put it in your local storage.54:19This repository, this repository tools may not physically store the images. It just contains links that points to some host locations and.54:38You'll need, actually need a separate server to physically host this image within your local network and a typical mechanism will be to use the NGINX, right? Use NGINX and another point, last point to take note if you are installing the VMware.54:51Cisco does not provide this.55:13Customized ESXi, ISO images, you have to download from VMware websites. A caveat here do note regarding this web server that holds your image. So if you're using the Python.55:16Built in HTTP server, right, to access your web server to host, it will not work, right? You will run fine, but during the OS installation you will encounter errors like a file server is inaccessible.55:34Now, what are the hosting protocols? So basically for CI FS.55:54So this is suitable when the image file is located on the Windows servers. And in the case of NFS preferred when you, your image are hosted on Linux or Unix file servers. And in the last case, we use use this when the image is hosted on the web.56:00Web server with.56:20HTTP or HTTPS access. One point to take note on the reachability, the software repository needs to be accessible by CIMC when using that site. A reason is that the actual server still relies on CIMC.56:22To access the software repository for downloading and mounting the. Right, another piece of info is this SCU. This is so important. So what is the SCU? This is basically a pre boot.56:40Two provided by Cisco to prepare and configure the server before OS installation. So these two is distributed as an image, right? A bootable software package containing the SU two is and during the OS installation process, the server.57:00Actually boots into this SU image to run the configuration configuration tool. So what this SU does.57:20Or what's the purpose of this SU? It facilitates the installations, prepare the storage, install drivers configures the buyers, boot order. So during the server boot up, the SU is like a container or an environment that holds the SU.57:30So if you look at this diagram, you can see how it fits into the boot sequence and basically is that when it runs, so it prepares the server for OS installations because it contains the necessary.57:50Components to configure, make adjustments and then load drivers and then prepare these and finally he hands off to the OS installer. Now, what happens without the SCU? So imagines if there's NO OS.58:06View the installations are.58:26Will fail or incomplete because when the installations installer starts, it looks for this and if there's NO this configured there's NO usable this in if there's NO, if there's missing drivers where will not be detected.58:29But having said that it is still possible to run without SCU and we'll see this later in OS installation using embedded mode where we can run without SCU. And that means.58:48Some advanced preparation is required, secure boot interactions, so how SEU relates to secure boot during OS installations, some, some tools, right? SU tools or scripts or installer.59:04May not be digitally signed and this means the has to be temporarily disabled. We will discuss more later because we have a slide on UEFI. Limitations I will not go through they are pretty static and.59:21You probably even can find them in the user guide. Okay.59:40Right, so what is the flow looks like? So for OS installations, we checked 1st is to check the firmware and the thing is that we want to ensure that the server firmware meets the minimum requirements. So there is the link and do check up this link and then the.59:47Next step is to add OS and SU image into the repository.01:00:07And then, it starts with the OS installation. So there's a few options here. You can click on the meet mid ball menu or the three horizontal dots or otherwise you can just handpick the servers and then you click actions OS install. So the.01:00:13Depending on the.01:00:33Options we have selected. There is two sub flows, so the one in dotted greens that is for bulk reconfigure. So in this case, the screen will show as site configuration via file, and this method actually allowed deployment of reserves and if you.01:00:35Requires the configurations.01:00:55With the required parameters in a CSV file. So we have to upload a CSV file and that gets into a summary page, we review it and then we confirm and then we proceed with the installation. So let me quickly just show you how is it like. So.01:00:58This is the server page and this is the what I call the three horizontal or meetball menu. So if I click on it, I say install and this is the one I'm saying set configuration via file. So I click on this and you can see that straight away asked to include.01:01:19The CSV file that contains the configurations. So let me browse to it and I will pick.01:01:39This, right, a CSV file, do a validation, and I should be able to click next. Right immediately, I will see that the information has been populated here. Okay, let's take a look at the file. So this is.01:01:46The CSV file and if you look at it, right, the file name is the same and your contains the ESXi, right, which is I specified here, the SU and then.01:02:06The configuration mode Cisco, and then likewise I see the two servers which I have here in my CSV file. Alright, so that's the way to use this option.01:02:23Alright, so the other mode, the other flow would be the orange dotted line that actually triggered the install OS wizard, and this is basically a stick, six step process. So we 1st confirm the server and then.01:02:43We select the OS from the repository, then we go to the configuration sections that's where we need to decide whether to use a Cisco custom embedded.01:03:01And finally we go to the SU, pick the SU, and then we need to select the target days, right? It can be a local days fiber channel or iscsi. And then we get into a summary page and then we do the install.01:03:11All right. So in the next few slides we will dive into these options. So let's start with the Cisco mode. So in this option, we'll be using a Cisco validated template for selected OS and in this mode there's NO hassle with kickstart.01:03:28File or placeholder and.01:03:48It simply ensures deployment follow the verified configurations and best practice. When do we use this mode, best for if you want standard Cisco supported OS deployment and your team does not have the expertise to maddle with the kick start file and you.01:03:52To accept Cisco predefined.01:04:12Installation workflow. So in short, we want simplicity and the flow is quite straightforward, right? So basically we go through this stage of staging the images for these SU and OS.01:04:15In software repository, we launch the wizard, and then we go to the server page, confirm the servers, and then we pick the OS, we get into the configuration page. This is where we pick the Cisco mode and then we select the SU and then it will get us into the.01:04:32Memory and they will actually kick off the installations. So for this part, there are videos available, so I've included here and hopefully Rona can help us to paste them into the chat. All right, likewise.01:04:52For the custom mode. So we have in this option we have to upload the kick start file and this has necessary placeholders for parameters and configurations and inter site will kind of display them.01:05:13In the UI and there's also a link here to You can refer as a lot of sample kick start on this when do we use this mode is best for if you have full control over the OS install you.01:05:28You're likely using a non Cisco ISO and you have, you have the expertise right to manipulate the kick start file and basically you want to customize the installation flow via parameters. So examples of this is this is how.01:05:48Looks so the within this kickstart file and you don't have to hard code, you know, values like.01:06:08Passwords or even a host name. So instead we have this placeholder and during the installation, the wizard will show them as fields and you can input the values and inter site automatically replace them with the real values.01:06:17Process flow is also quite similar except for step four and five. Right, the rest are very similar. So I've highlighted step four and five. So these are the ones that's concerned when it comes to the configurations, we probably select browse and our customs and then we can browse.01:06:35And we need to fill in a placeholder and the steps, the rest are more or less the same. I've also included the links because these videos are available in the YouTube.01:06:53So let me quickly take you through to show you how is it like. So I'm gonna cancel this and on my servers, so I'll go to my server page.01:07:05So suppose I select this server and I click install OS, right? This is a wizard, a six step process, click next, confirm this is my target servers.01:07:22And OS I will select ESXi, and this is the part where where we need to make the change. We need to select custom. So in this case you can select your configuration file from repository or from local. I will select it from.01:07:37And this is the file.01:07:58Right, so over here this is where I need to input the values. So how does it look like if I were to bring up that file, let me try to locate my file. Okay, I should have it here yep.01:08:04Right, so this is my custom file and you can see that on line twelve, right, this is the one that is the change, right? You can see the placeholder for IP address and host name. That's why we have to fill in here manually and simply.01:08:25Because right, I drop this because in my environment probably because net mass and gateway, all these are quite static, so I'm not gonna change and that's why I can provide these as my placeholder for users to input.01:08:45And you can also see the custom values, right? Custom variables that can be included in this custom mode. Alright, so let's quickly move on to the next one and the final one is this.01:09:01Embedded mode. So this one is probably the most difficult because.01:09:21We embate the kick start file into the OS ISO. So it's combined together, hence the final image will contain the necessary reference to the kick start from the bootloader configurations. We can run in two modes with scus and with our scus. So.01:09:26We have to use these enabled answered customizations and driver injections or loading of utilities. Without the scus it means that policies have to be created, put entries get loaded, and also take note that.01:09:46Tracking is unavailable. So when do we use this mode best for convenience? You don't want to upload or manage configuration files, but the downside is that you must know how to modify the boot configurations and embed the kick start file.01:10:05Another scenario is that if you want to install in a hit list mode, so of course this must be supported by the hardware. Hitless mode means that OS will be installed without direct interactions through monitor or keyboard.01:10:25So there will be profile drift, right? There will be profile drifts, just watch out for the message and then should it go out of sync, just redeploy the profile. Process flow wise is quite, it's very different from the rest too, right? It's very different.01:10:41I've highlighted in this the 1st stage, alright, this is where the about all the work is done. We need to prepare the ISO media, edit the ISO config file, and then we need to inject the kick start file package all into the final ISO image.01:11:01And then eventually we upload it to the software repository and the steps remains more or less the same. There is also a link here in YouTube. So let's get into this one on.01:11:21UVFI secure boot. So, what is secure boot is basically a security mechanism built into modern servers allow only trust or site software to run as startup. So in.01:11:41UCS servers enabling the secured boot is associated with UEFI but they are not the same, right? It's not the same. When this is enabled, it all executables such as the bootloaders, EFI drivers, EFI applications.01:11:56All are verified for authenticity before we are allowed to execute.01:12:16And if you look to do this, right, means that the firmware will have to check every boot components, which is what you see here and specifically it verify the bootloader, the OS kernel. So this is what we call the.01:12:21Chain of trust and it checks the component if they are digitally signed against trusted certificates if anything is not trusted or modified, and we'll just refuse to put it.01:12:38Now, what are the OS supported for secure boot? We have, following OS, so basically it's Windows, Ubuntu, and Rehat. So for these three OS, they applies to C series standalone and IMM servers. Now, why only these three servers? Because they.01:12:54Because they provide side bootloaders. So what about ESXi? So ESXi is a bit special, so on the right, we have this section focused on ESXi installations. So if you recall, we discussed earlier.01:13:13On the SCU, we mentioned that ESXi secure boot is not supported. So if you perform OS installation using the workflow, you will notice there's an option, right, that appears in this screenshot, right? There's this option called override secure boot configuration.01:13:30So in this screenshot, this is what you see. And if we click on the options, right? If you were to click on this option, it means inter sites may modify secure boot. So we can think of it this way if our service server if our UCS server settings.01:13:49For secure boot is on, inter site will override by switching it off. Now on the other hand, if our UCS server setting for secure boot is off, right, there will be NO issues, right? The option does not apply because ESXi installed does not support secure boot.01:14:08Now during runtime, we will be prompted again whether to continue or cancel, right? You can see this over here. So continue means ok. All right ok let's ignore the secure boot, continue the installations of on site OS. Cancel will means that.01:14:26Do not.01:14:46Ignore secure boot, right? Keep it on and continue the installation of on site OS. So if we detect any component that's not trusted installation will fail. So how does this fits into the UEFI in.01:14:47Cisco and custom mode alright the UAFI boots get disabled during workflow and you will be re enabled automatically after installations. The reason is that the install process may contains tools, scripts that are not signed.01:15:07So this contradicts with secure boots because secure boots needs them to be signed. So it may end up blocking the process. So that's why we need to.01:15:26Disabled temporary. In the embedded mode, alright, uevfi secure boot get disabled during workflow and we must manually re enable after installation. So the reason is that if you are.01:15:37Is, it actually operates without install tracking because in this mode it may install without tracking and as such it does not know when to turn secure boot back on. That's why we need to manually turn it back on.01:15:54So the most important concern is that, does it mean ESXi installation secure boot will never be turned on. No, it does not mean secure boot is never used, right? It means secure boot is temporary off during installation because it's off during installation.01:16:12It doesn't mean secure boot is never used, right? If you didn't get what this whole slide is just talking, the whole idea is that we just need to turn off or disable secure boot, before installation. And intersight has the option.01:16:32To temporarily turn it off for you. Alright, I'm gonna run out of time, I think. So I will just kick off with the video.01:16:48In this video I'm gonna walk you through on the intersite OS installation for ESXi using workflow. For OS installation.01:17:08Right, so where you have to go through the five steps process of selecting the OS and then the configurations are supplying the SCU and etc. So this method of using the workflow is more straightforward and it does not require user intervention. In this method, there maybe times where you need to manually reboot the server and.01:17:20It boots up you will have to manually acknowledge by pressing F eleven to continue and select the keyboard type something the end user license agreement and selecting the these to install, and eventually when ESXi stops up, it defaults to the HCP, not the static IP provided.01:17:40Alright, so for the server, we will install esxi or the standalone, the UCS 222M6, so it's going to be this, right? So it's a UCS 222 oh M six series and currently the vision is healthy and there is NO server profile.01:17:56Technically speaking as best practice we should have a signed profile, but I'm gonna leave this out in this demo. Also you'll notice that the firmware has been updated at the latest and at this point for in this demo, this is the latest version. And in fact for as best practice is to upgrade to the latest.01:18:16Now in the if we go into the VKVM, right, we can see that server was previously right the server was previously installed having a host name of ESXi one.01:18:35Right esxi one and assign dynamic IP 1 821-668-1126 from the and then in the CIMC for this server, right? So so this is the server, and if we go into the CIMC.01:18:50Under compute by yours, we look at the configure boot order right here on the left. So on the left we see that there is M two rate boot, the media iso, and then there is also this embedded storage. Now we're gonna use the M two rate booth.01:19:08So this will be our target device where we will install the ESXi on. All right, so let's go back to inter site for firmware OS OS image SEU and OS config file so that inter site can reference these files in the upgrade OS installed.01:19:35In this OS image link, so for this demo we have uploaded ESXI seven.0u tree ISO and.01:19:49In case you need to, so basically what you you need to do is just click on this add OS image link and in this case I have used HTTP and provided the right so in this case I have this and then most likely you will just copy the link and then.01:19:57Comes back here and then you just piece it in and then you just go to next and then you fill up the information. So as you can see that we need to have a separate host, right? We need to have this separate machine where we have this install all these images and this whole survey must be running web server like NGINX, so in this case I have this host turned onto phone, 445o.01:20:14Running HTTP server.01:20:34So since I re added I'm gonna skip these steps, so this will be the server that we can use, and we also need the SU link, right? So for this demo we'll be using SU six.two.3C right, so this is the one and we enter the source of SU, which is an ISO image.01:20:37Right, it's an ISO image yes, it's an ISO file and you can see this during the installation when it boots up, you will see the screen showing Cisco SU setting up the SU workplace. So it's a tool that is distributed as an image, a bootable software package that contains the SU tools supplied by Cisco and part of this inter site ecosystem.01:20:54So the same as so similarly, you also need a host, so in this case I also make use of this server and definitely have the SEO right over here. So eventually all get pointed into this software repository. So for more information, I suggest we re.01:21:13Had to.01:21:33There's a, there is a link. So let me open up this link and this is actually the release notes for the SEU. Now the wood of caution, the west installation is very picky and dependent on the version of SU, so my advice is to really read through.01:21:34And seriously confirm the SCU version in the release notes for the OS you're gonna install.01:21:54So I'm gonna head back to inter site and I'm gonna introduce this workflow. So we're gonna look into under so there is this thing called workflow. Now the workflow that we'll be using is called operating operating system.01:22:00Operating system install, right? So as we can see here is the system defined, so we can make any modification and if we click into it and you will go into the designer, right, into the designer tab. Now the way this work is that we break down our entire installation process into specific tasks and conditions. So for instance you know we can have this condition here and.01:22:16We can see that we check the workflow type and in for our case we'll be using a standalone and that will break into this sub workflow.01:22:35Great, so it breaks down into a further support flow task and it's called install operating system on standalone server. So the important note here is that you also need to have the Cisco inter site advantage license to run this and.01:22:44So here is the general page and basically this is the preview of what it looks like. Now to run this workflow so we will simply click on the execute, and we're gonna provide the parameters. So in this case we're gonna call it the workflow name.01:22:59The next one is the server H our ESXi seven one oh you three. And then for next one you come to installation mode. And under the configuration tab, you're gonna select a static IP. All right, for IP address, my gateway.01:23:17Hosting and then it comes to selecting the SU. So in this case it's going to be six.let me six.two.3C And then for the installation target look at this.01:23:36And, so we gotta worker, so we can actually go back to our main screen and on the top there you see a screening circle if you click on it, you can see that the one has been going on.01:23:52And we see that he has proceeded. So one thing is that in using the workflow, we have this option of showing the current input output and logs.01:24:10So let's go over to, right? So it's gonna take a while, right? We probably have to wait for about 05:10 minutes before we start to see that it begins the installation and before they install the UI appears. So on the UCS server deployed.01:24:27Through the install workflow. This stage normally happens after the ISO is mounted and the server books from it. So this process is going to take roughly about 1010 minutes based on this.01:24:47So I'm gonna wait for about 10 min before continuing and this is the SU setting up this temporary environment in the server. This part is gonna take about a minute. At this point, the installer has loaded his core environment into a RAM and is now preparing the temporary old environment and the full graphical installer will now run from, and this is where.01:25:01We see the SCU showing up and this part is gonna take about.01:25:21Ten to 30 min, Cisco server configuration utility and we can see that it's going to be very slow and it appears it's actually progressing in the background. This is not the ESXR installer yet. The server booster SUI so from.01:25:25The 1st one doing an OS installation via intercite. So the Cisco tool that is going to prepare the server hardware checking the storage et cetera, use automate parts of the OS installation and then it hands off to the actual VMware ESXi installer.01:25:44And if we go back to our inter site and check on the request. So this was our workflow request and we can see that the status shows success. So what this means is that if you go into this new execution flow, what basically means that.01:26:00Yeah.01:26:20It actually prepares the OS install configurations and then initiate the OS system installations and map the virtual media and then it track the OS bootstrap process. And this part is complete, completed as concerned, so it's actually handing over to the server itself because the rest are quite automatic. So.01:26:21We end up here where this is actually running the show. So what we can see here is what's happening on the screen here is that we are checking the SU mode, so determining the installation mode and then starting the OS installation process that begins the guided install and then we receive the OS answer file and the config fi.01:26:41Well, so loading any predefined setting if we have provided, you know, like an answer file, otherwise it's using the default and then starting a non interactive OS install, meaning it runs in a non interactive or we can call it automated mode, and then installing the OS ESXi so it starts the.01:27:01ESXi image and eventually clearing the hard disks so that means that the SU is actually wiping or clearing the hardest where ESXi will be installed. So this will actually erase whatever previously on the drive, your previous OS will be overwritten and then mapping the OS ISO to continue this.01:27:18So the SU has kind of finished his reason and we now attempt to unmap the current SCU ISO and then it maps the ESXi ISO, so in this case our and.01:27:38But in the background installation is progressing, it's gonna take roughly about from what I, what I've run this recently it takes about an hour, 20 min. So I'm gonna let this continues and I'll come back in about an hour's time.01:27:50And then following this cleaning up we should see that it will reboot shortly. So this section into boot into the new ESXi.01:28:07However, this is moving extremely slowly on the bottom screen. The system can produce and initialize the local rate as well into the local booth device.01:28:28You know, 90 10 min right so ESXi has been installed. And as you can see that throughout this session.01:28:43There isn't any user intervention at all the study IP and.01:29:00The host name that we assigned earlier alright so that should complete.01:29:15For these last topics and this ended my presentations. So I'm gonna hand over to Ronak. Thank you very much Terence for such a wonderful presentation and the demos.01:29:33I hope all of it was quite useful for all the audience joined us today. This will bring us to the end of the session today. If you have any more questions, please feel free to put them in the chat. You can raise your hand. We will be here for two, 3 min more, but thank you all for joining us today.01:29:50We will have one more session next week, which will be more focused on the Q and A If you have any questions, any queries throughout the series, regardless of which session you are part of, all the three sessions, any questions you have, please join us in the next week webinar so that you can ask your questions live with all the experts who will be joining us.01:30:09In that session as well. The registration link should have reached you, but I will also put it in the chat in case anyone has missed registering for that session. You can go to the link mentioned on the chat and you can register for that session which is happening next week.01:30:29But for now, if you have any more questions, please feel free to put them in the chat or you can raise your hand, you can ask it live to us.01:30:49And we will be more than happy to answer them for you. If not, this brings us to the end of the session. Thank you all for joining us. I Hope to see you in our next week's session as well.01:30:59