Cisco ACI Virtual Edge Migration
Available Languages
Bias-Free Language
The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
- US/Canada 800-553-2447
- Worldwide Support Phone Numbers
- All Tools
Feedback
Feedback
As announced on the Cisco.com Notices page, Cisco ACI Virtual Edge (AVE) will be end-of-sales and end-of-life in Cisco ACI Release 6.0. Before you attempt to upgrade your APIC cluster to Release 6.0 or later, you must migrate VMware AVE VMM domain to another domain such as VMware VDS VMM domain. At the time of writing, the suggested replacement is VMware VDS VMM domain. This document describes the migration procedures of VMware AVE VMM domain with VLAN encapsulation mode to VMware VDS VMM domain.
This document does not cover the following use cases:
· Cisco AVE cloud mode for vPod
· Cisco AVE with VXLAN encapsulation mode
· Cisco AVE Distributed Firewall
Note: The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product.
This document assumes that the reader has a basic knowledge of Cisco ACI technology. For more information, see the Cisco ACI white papers available at Cisco.com: https://www.cisco.com/c/en/us/solutions/data-center-virtualization/application-centric-infrastructure/white-paper-listing.html
This document uses the following terms with which you must be familiar.
● Cisco ACI terminologies:
◦ Endpoint Group (EPG)
◦ uSeg EPG
◦ VMM Domain
● VMware terminologies:
◦ VMware ESXi
◦ VMware vSphere Distributed Switch (VDS)
◦ Physical Network Adapter (pnic)
Cisco ACI Virtual Edge (AVE) with VLAN encapsulation mode overview
● Cisco AVE with VLAN encapsulation mode has the following characteristics:
● One Cisco AVE VM is deployed on each ESXi host.
● Support Local Switching mode. (No Local Switching mode is VXLAN mode only)
● Cisco AVE VM has three interfaces:
◦ Management: Management purpose only.
◦ Inside: trunk internal VLANs facing VMs. Two internal VLANs, one primary VLAN and one secondary VLAN, are allocated for each EPG, which are from the internal VLAN range.
◦ Outside: trunk external VLANs facing ACI leaf nodes. One external VLAN is allocated for each EPG or each uSeg EPG, which is from an external VLAN range.
● Intra-EPG isolation or intra-EPG contract is not supported.
Figures below illustrate an example of VMware AVE VMM domain with VLAN encapsulation mode in Cisco ACI fabric. The port-groups for EPG1 and EPG2 are created and PVLAN is enabled, which creates two VLANs, one primary/promiscuous VLAN and one secondary/isolated VLAN, for each EPG. Thus, communication between virtual machines goes through the Cisco AVE VM via Inside interface and Cisco AVE VM can apply various policies even if it is intra-EPG communication within the same ESXi host.
In the case of inter-EPG or cross ESXi hosts communication, Cisco AVE VM sends the traffic via Outside interface towards the ACI leaf nodes, using the external VLAN for the EPG. Policy enforcement for inter-EPG traffic is done on the leaf node based on the external VLANs for EPGs.
In the case of uSeg EPG, each uSeg EPG is assigned its own VLAN. Even though VM1 and VM2 are connected to the same port-group, Cisco AVE VM sends traffic via Outside interface towards the ACI leaf nodes, using external VLAN for the EPG. This enables the ACI nodes to enforce policy based on the external VLANs for uSeg EPGs.
The following steps are migration step examples of VMware AVE VMM domain with VLAN encapsulation mode to VMware VDS VMM domain. In this document, Example 1 is covered.
Example 1: Move VMs to the port-groups on the new VDS within the same ESXi host
This is an example of migration steps. If each ESXi host has an available uplink that can be used for the new VDS during migration.
1. Create a VMware VDS VMM domain and associate the VMM domain to the EPGs.
2. Add the new VDS to the relevant ESXi hosts.
3. Move one of the uplinks to the new VDS.
4. Move VMs to those new port-groups.
5. Move the rest of uplinks to the new VDS.
6. Repeat #2-4 for other ESXi hosts that have used VMware AVE VMM domain.
7. Delete the Cisco AVE virtual machines.
8. Delete the VMware AVE VMM domain in APIC.
Example 2: Evacuate VMs to different ESXi hosts during migration
If you have enough compute resource in the ESXi cluster, you can move the VMs to another ESXi host during migration, using the following steps:
1. Create a VMware VDS VMM domain and associate the VMM domain to the EPGs.
2. Move the VMs to other ESXi hosts.
3. Associate the new VDS to the appropriate ESXi hosts.
4. Move uplinks of the ESXi host to the new VDS.
5. Move the VMs to the ESXi hosts with the new VDS, selecting the new port-groups on the new VDS.
6. Repeat steps 2 to 4 for other ESXi hosts that have used Cisco AVE VMM Domain.
7. Delete the Cisco AVE virtual machines.
8. Delete the VMware AVE VMM Domain in APIC.
Considerations and Assumptions
The following list includes some key considerations for the migration:
· Does each ESXi host have an available uplink that can be used for the new VDS during migration?
· Is there enough compute resource in the ESXi cluster to evacuate VMs during migration?
· Is Cisco Discovery Protocol (CDP) or Link Layer Discovery Protocol (LLDP) used between ACI leaf nodes and ESXi hosts for host discovery?
If yes, both ends need to be configured accordingly including physical network adapters. For example, Cisco Virtual Interface Card (VIC) enabled LLDP by default, which must be disabled.
If not, Resolution Immediacy option must be set to “pre-provision” at the VMware vDS VMM domain association configuration in the EPGs,
· Is there an intermediate switch between the ACI leaf nodes and ESXi hosts?
If yes, the intermediate switch configuration may need to be configured accordingly. It is generally recommended not to allow unnecessary VLANs on the intermediate switches’ interfaces to avoid unnecessary traffic traversed. In the case of Cisco UCS Fabric Interconnects with vNIC templates, it is recommended to use different vNIC templates for vNICs used for the VDS for AVE and the new VDS.
· Is there a uSeg EPG with VMware AVE VMM domain?
If yes, you must check “Allow Micro-Segmentation” at the VMware vDS VMM domain association configuration in the base EPG to use uSeg EPGs with VMware vDS VMM domain. That configures PVLAN (Private VLAN) on the VMware vDS VMM domain port-group for the base EPG, and it enables proxy-ARP within the base EPG.
If IP based uSeg EPG is used on your existing AVE environment, it’s recommended to simultaneously migrate VMs in the IP based uSeg EPGs to the port-groups on the new VDS.*
· Is there a different LACP policy on the new VDS? For example, enhanced LACP policy.
If yes, ACI leaf interface related configurations such as interface access policy group need to be migrated too. Especially enhanced LACP needs attention because enhanced LAG policy must be specified in VMM domain configuration for each EPG. Please refer Enhanced LACP Policy Support section in Cisco ACI Virtualization Guide for detail of enhanced LACP migration.
· Is there a scale constraint of number of ports * VLANs on the ACI leaf nodes?
If yes, you may need to avoid migrating all of EPGs at once to avoid all of the VLANs, for VMware AVE VMM domain and the new VMware VDS VMM domain, programmed on the ACI leaf nodes at the same time.
Note: (for advanced readers) As IP based uSeg EPG is applicable to routed traffic only, whereas traffic within the same subnet is not routed on the ACI fabric in the case of uSeg EPG on AVE that doesn’t use proxy-ARP. For example, intra-subnet traffic between an endpoint in new vDS VMM domain and an endpoint in AVE VMM domain is bridged instead of routed on the ACI fabric, which means IP based uSeg EPG is not applicable. Instead, other attributes such as MAC and VM attributes are applicable. After the migration, traffic between endpoints in new vDS VMM domain doesn’t have this consideration because proxy-ARP is enabled on the port-group for the base EPG and intra-subnet traffic is routed.
The assumptions in this document are the following:
· The ESXi host has at least 2 uplinks directly connected to ACI leaf nodes and port-channel is not used.
· There is no uSeg EPG.
· There is no scale constraint of number of ports * VLANs on the ACI leaf nodes that have both Cisco AVE VMM domain and VMware VDS VMM domain during migration.
Step 1: Create a VMware VDS VMM domain and associate the VMM domain to the EPGs
This step needs to be performed on APIC. Create a new VMware VDS VMM domain and associate the VMM domain to the EPGs, which creates a new VDS and port-groups on the vCenter. It’s recommended to use the same VDS version and use different VLAN ranges for VMware AVE VMM domain and VMware VDS VMM domain.
Step 2: Associate the new VDS to the appropriate ESXi hosts
This step needs to be performed on the VMware vCenter. Associate the new VDS to the appropriate ESXi hosts (Figure 4). As there is no uplink attached to the new VDS and VMs still use the port-groups on the VDS for AVE, traffic is still forwarded through Cisco AVE VM.
Step 3: Move one of the uplinks to the new VDS
This step needs to be performed on the VMware vCenter. Attach one of the uplinks to the new VDS (Figure 5). As VMs still use the port-groups on the VDS for AVE, traffic is still forwarded through Cisco AVE VM. Note that the available uplink bandwidth and high availability of the VDS for AVE is decreased if you move an uplink from the VDS for AVE. It is recommended to initiate traffic from virtual machines during uplink migration because other leaf nodes might still point to the vPC virtual TEP (VTEP) instead of the physical TEP (PTEP) of the leaf connected to the VDS for AVE for the remote endpoints.
Step 4: Move VMs to those new port-groups
This step needs to be performed on the VMware vCenter. Move VMs to those new port-groups (Figure 6). Because of the change on the VMs’ network adapter configurations, you may experience a small traffic loss. As VMs use the port-groups on the new VDS, traffic is not forwarded through Cisco AVE VM anymore. Ensure traffic is forwarded through the new VDS accordingly before you migrate vmkernel adapters to the new port-groups.
Step 5: Move the rest of uplinks to the new VDS
This step needs to be performed on VMware vCenter. Attach the rest of the uplinks to the new VDS (Figure 7). Ensure traffic is forwarded through the new VDS using multiple uplinks accordingly.
Step 6: Repeat Step 2-4 for other ESXi hosts that have used VMware AVE VMM domain
Repeat Steps 2 to 4 for other ESXi hosts that have used VMware AVE VMM domain Ensure traffic works across different ESXi hosts.
Step 7: Delete the Cisco AVE virtual machines
This step needs to be performed on VMware vCenter. Delete the Cisco AVE virtual machines before deleting the VMware AVE VMM domain in APIC. If you have used the Cisco ACI vCenter Plug-in to install the Cisco AVE, you can perform the removal operation using the Cisco ACI vCenter Plug-in. Refer the Cisco ACI Virtual Edge Uninstallation section, in the Cisco ACI Virtual Edge Installation Guide for the detailed procedure.
Step 8: Delete the VMware AVE VMM domain in APIC
This step needs to be performed on APIC. Delete the VMware AVE VMM domain association from EPGs and then remove the VMWare AVE VMM domain on APIC (Figure 8).
