Configure Nexus Dashboard Orchestrator to Migrate Endpoint from One DC to another DC

Introduction

This document describes the design and configuration changes required to migrate an Endpoint from one Data Center to another Data Center.

Physical Topology

Figure 1. depicts the interconnectivity of two Data Centers.

Figure 1: Physical Topology

DC and DR locations have the Application Centric Infrastructure (ACI). DC and DR locations have the WAN Switches, Border Leaf, Spines, Inter-Site Network Devices (ISN), Server Leaf, and connected Endpoints.

Logical Topology

Figure 2: Logical Topology

Logical objects configured in both sites:

• Tenant Production is configured at the DC and DR sites.
• DC-VRF-WEB and DC-VRF-APP are configured in DC-SITE1. DR-VRF-WEB and DR-VRF-APP are configured in DR-SITE2.
• Each VRF is configured with local L3Outs on Border Leaf towards WAN Switches. Default routes are configured on Border Leaf towards WAN Switches.
• WAN Switches are configured with Static routing for Inter-VRF and Inter-DC communication.
• Both Data Centers are configured with Local BDs and EPGs. DC has DC-BD1-WEB/DC-EPG1-WEB, DC-BD2-WEB/DC-EPG2-WEB and DC-BD-APP/DC-EPG-APP. DR has DR-BD1-WEB/DR-EPG1-WEB, DR-BD2-WEB/DR-EPG2-WEB and and DR-BD-APP/DR-EPG-APP.
• There are endpoints connected in WEB and APP EPG.
• DC-SITE1 and DR-SITE2 are added to the Nexus Dashboard Orchestrator.

Traffic Flow Before Endpoint Migration

There are multiple types of Traffic Flow in Data Centers:

• Intra EPG Traffic flow
• Inter EPG Traffic flow
• Inter VRF Traffic flow
• Inter DC Traffic flow

Intra EPG Traffic Flow

Figure 3: Intra EPG Traffic Flow

Communication between DC-EP-1 and DC-EP-2 is Intra EPG communication, as both Endpoints belong to DC-EPG1-WEB. Communication between DR-EP-1 and DR-EP-2 is Intra EPG communication, as both Endpoints belong to DR-EPG1-WEB.

Inter EPG Traffic Flow

Figure 4: Inter EPG Traffic Flow

DC-EP-1 and DC-EP-3 are part of DC-EPG1-WEB and DC-EPG2-WEB respectively, communication between these two Endpoints is Inter EPG Traffic Flow. DR-EP-1 and DR-EP-3 are part of DR-EPG1-WEB and DR-EPG2-WEB respectively, communication between these two Endpoints is Inter EPG Traffic Flow.

Inter VRF Traffic Flow

Figure 5: Inter VRF Traffic Flow

DC Border Leaf forwards the traffic to DC WAN Switches for any Inter-VRF communication. DC WAN Switches are used for Inter-VRF communication. DC-EP-1/EP-2 (VRF WEB) communicates to DC-EP-4 (VRF APP) through WAN Switches. DR Border Leaf forwards the traffic to DR WAN Switches for any Inter-VRF communication. DR WAN Switches are used for Inter-VRF communication. DR-EP-1/EP-2 (VRF WEB) communicates to DR-P-4 (VRF APP) through WAN Switches.

Inter DC Traffic Flow

Figure 6: Inter DC Traffic Flow

Communication between DC-Endpoints and DR-Endpoints forwarded to Border Leaf. Border Leaf forwards the traffic to WAN Switches. WAN Switches are used for Inter DC communication.

Migration Plan

Nexus Dashboard Orchestrator is used to create the Multisite between both the sites, EPGs/BDs stretched across sites and endpoints must be migrated from DC-SITE1 to DR-SITE2.

Schema-1 Creation

Schema-1 was created through Nexus Dashboard Orchestrator.

Figure 7: Tenant Template - Add Schema

Figure 8: Add Schema Name

Template-VRF-Contract-Stretched Creation

Template-VRF-Contract-Stretched created inside Schema-1. DC-SITE1 and DR-SITE2 must be part of this Template and Tenant-Production in order to be associated with the same Template. This is a stretched template. VRF and Contracts must be part of a separate Template, as these objects are shared across other BD/EPGs. This template must be used in order to stretch the DC-SITE1 VRF and Contract to DR-SITE2.

Figure 9: Add Application Template - Choose ACI Multi-Cloud

Figure 10: Add Template Name Template-WEB-VRF-Contract-Stretched, Choose Tenant Production

Figure 11: Template-WEB-VRF-Contract-Stretched Details

Import VRF-Contract in Template-VRF-Contract-Stretched

Import DC-VRF-WEB and DC-VRF-WEB-Contract from DC-SITE1. Contracts are created for Inter-EPG communication and EPG-to-L3Out communication.

Figure 12: Click Import and Choose DC-SITE1

Figure 13: Choose Contract from DC-SITE1

Figure 14: Choose Filter from DC-SITE1

Figure 15: Choose VRF from DC-SITE1

Figure 16: Template-WEB-VRF-Contract-Stretched with VRF and Contract Information

Deploy Template-VRF-Contract-Stretched

Click Deploy Template-VRF-Contract-Stretched and choose DC-SITE1 and DR-SITE2.

Figure 17:Add Fabrics to Template-VRF-Contract-Stretched

Figure 18: Deploy Out-of-Sync Templates

Figure 19: Deployment Completed

Figure 20: Verify VRF and Contracts Deployed on Both Sites

Template-EPG1-BD1-Stretched Creation

Template-EPG1-BD1-Stretched created inside Schema-1. DC-SITE1 and DR-SITE2 were added to Template and Tenant-Production associated with the same Template. This is a stretched template. This template is used in order to stretch DC-EPG1-WEB and DC-BD1-WEB to DR-SITE2.

Figure 21: Add Application Template - Choose ACI Multi-Cloud

Figure 22: Add Template name Template-EPG1-BD1-Stretched, Choose Tenant Production

Figure 23: Template-EPG1-BD1-Stretched Details

Import EPG1-BD1 in Template-EPG1-BD1-Stretched

Import DC-EPG1-WEB and DC-BD1-WEB from DC-SITE1.

Figure 24: Click Import and Choose DC-SITE1

Figure 25: Choose DC-EPG1-WEB from DC-SITE1

Figure 26: Choose DC-BD1-WEB from DC-SITE1

Change BD Setting in Template-EPG1-BD1-Stretched

Enable L2 Stretch in DC-BD1-WEB settings and add the Gateway IP Address. This template stretches BD across the site and the anycast Gateway configured in DC-SITE1 and DR-SITE2.

Figure 27: Choose L2 Stretch in DC-BD1-WEB

Figure 28: Add Gateway IP/Subnet

Deploy Template-EPG1-BD1-Stretched

Click Deploy Template-EPG1-BD1-Stretched and choose DC-SITE1 and DR-SITE2.

Figure 29: Add Fabrics to Template-EPG1-BD1-Stretched 

Figure 30: Deploy Out-of-Sync Templates

Figure 31: Deployment Completed

Migrate DC-EP-1 from DC-SITE1 to DR-SITE2

Configure static binding in DR-SITE2 in DC-EPG1-WEB and associate DR-SITE2 Physical Domain. Migrate the DC-EP-1 from DC-SITE1 to DR-SITE2.

Figure 32: DC-EP-1 Currently Learned in DC-SITE1

Figure 33: DC-EP-1 removed from DC-SITE1

Figure 34: Adding Physical Domain in DR-SITE2

Figure 35: Adding Static Binding in DR-SITE2

Figure 36: DC-EP-1 Learned in DR-SITE2

Physical Design after DC-EP-1 Migration

DC-EP-1 is connected to DR-SITE2 Server Leaf.

Figure 37: Physical Design after DC-EP-1 Migration

Logical Design after DC-EP-1 Migration

DC-EP-1 is connected to DR-SITE2 Server Leaf. DC-EPG1-WEB, DC-BD1-WEB and DC-VRF-WEB are stretched between DC-SITE1 and DR-SITE2.

Figure 38: Logical Design after DC-EP-1 Migration

Intra EPG Traffic Flow after DC-EP-1 Migration

Figure 39: Intra EPG Traffic Flow after DC-EP-1 Migration

Communication between DC-EP-1 and DC-EP-2 is Intra EPG communication, as both Endpoints belong to DC-EPG1-WEB. This communication happens through DC ISN to DR ISN Multisite/Overlay Links.

Ping Response between DC-EP-1 and DC-EP-2

Figure 40: Ping Response Between DC-EP-1 and DC-EP-2

Routing Table from Spines

DC-EP-1 learned in DC-SP-01/DC-SP-02 from DR-SP-01/DR-SP-02.

DC-EP-1 is learned in DC-SITE1-SP-01 from DR-SITE2-SP-01.

Figure 41: Routing Table from Spines

DR-SITE2-SP-01 Overlay Unicast TEP IP.

Template-EPG2-BD2-Site1 Creation

Inter EPG communication between DC-EP-1 and DC-EP-3 happens, once DC-EPG2-WEB and DC-BD2-WEB  are part of the Nexus Dashboard Orchestrator.

Template-EPG2-BD2-Site1 created inside Schema-1. DC-SITE1 added to Template and Tenant-Production associated with the same Template. This is site site-specific template. This template is used to import the Template-EPG2-BD2-Site1 for the communication between DC-EP-1 and DC-EP-3.

DC-EP-1 and DC-EP-3 communication require DC-EPG2-BD2 to be part of Nexus Dashboard Orchestrator.

Figure 42: DC-EP-1 and DC-EP-3 Not Able to Communicate

Figure 43: Add Application Template - Choose ACI Multi-Cloud

Figure 44: Add Template Name Template-EPG2-BD2-Site1, Choose Tenant Production

Figure 45: Template-EPG2-BD2-Site1 Details

Import EPG2-BD2 in Template-EPG2-BD2-Site1

Import DC-EPG2-WEB and DC-BD2-WEB from DC-SITE1.

Figure 46: Click Import and Choose DC-SITE1

Figure 47: Choose DC-EPG2-WEB from DC-SITE1

Figure 48: Choose DC-BD2-WEB from DC-SITE1

Figure 49: Contracts Associated with DC-EPG2-WEB are Imported

Deploy Template-EPG2-BD2-Site1

Click Deploy Template-EPG2-BD2-Site1 and choose DC-SITE1

Figure 50: Add Fabrics to Template-EPG2-BD2-Site1

Figure 51: Deploy Out-of- Sync Templates

Figure 52: Deployment Completed

Shadow EPG for DC-EPG2-WEB created in DR-SITE2.

Figure 53: DC-EPG2-WEB is Deployed in Both Sites

Inter EPG Traffic Flow after EP-1 Migration

Figure 54: Inter EPG Traffic Flow after EP-1 Migration

Communication between DC-EP-1 and DC-EP-3 is Inter-EPG communication, as both Endpoints belong to DC-EPG1-WEB and DC-EPG2-WEB respectively. This communication happens through DC ISN to DR ISN Multisite/Overlay Links.

Ping Response between DC-EP-1 and DC-EP-3

Figure 55: Ping Response between DC-EP-1 and DC-EP-3

Template-WEB-L3Out-Site1 Creation

Template-Web-L3Out-Site1 created inside Schema-1. DC-SITE1 was added to the template and Tenant-Production was associated with the same Template. This is a site-specific template. This template is used for DC-EP-1 Inter-VRF and Inter-DC communication.

Figure 56: Add Application Template - Choose ACI Multi-Cloud

Figure 57: Add Template Name Template-WEB-L3Out-Site1, Choose Tenant Production

Figure 58: Template-WEB-L3Out-Site1 Details

Import External EPG and L3Out in Template-WEB-L3Out-Site1

Import External EPG and L3Out in Template-WEB-L3Out-Site1

Figure 59: Click Import and Choose DC-SITE1

Figure 60: Choose EXT-APP-EPG from DC-SITE1

Figure 61: Choose DC-APP-L3Out from DC-SITE1

The shadow of EXT-WEB-EPG was created in DR-SITE2 with applied DC contracts.

Figure 62: Contracts Associated with EXT-WEB-EPG are Imported

Deploy Template-WEB-L3Out-Site1

Click Deploy Template-WEB-L3Out-Site1 and choose DC-SITE1.

Figure 63: Add Fabrics to Template-WEB-L3Out-Site1

Figure 64:Deploy Out-of-Sync Templates

Figure 65: Deployment Completed

Verify the Routes in DR Server Leaf for DC-VRF-WEB

Static routes installed in DR Server Leaf for DC-VRF-WEB.

Figure 66: Verify the routes in DR Server Leaf for DC-VRF-WEB

Inter VRF Traffic Flow after DC-EP-1 Migration

Figure 67: Inter VRF Traffic Flow after DC-EP-1 Migration

DC-EP-1 uses DC-WEB-L3Out in order to communicate with DC-EP-4. The traffic flows from DR-ISN to DC-ISN Multisite Links, DC-ISN to DC-SP-01/DC-SP-02, and from DC-SP to DC-BL. DC-BL-01/DC-BL-02 forward the traffic to DC-WAN Switches for Inter-VRF routing.

Ping Response between DC-EP-1 and DC-EP-4

Figure 68: Ping Response Between DC-EP-1 and DC-EP-4

Inter DC Traffic Flow after DC-EP-1 Migration

Figure 69: Inter DC Traffic Flow after DC-EP-1 Migration

DC-EP-1 uses DC-WEB-L3Out in order to communicate with DR Endpoints. The traffic flows from DR-ISN to DC-ISN Multisite Links, DC-ISN to DC-SP-01/DC-SP-02, and from DC-SP to DC-BL. DC-BL-01/DC-BL-02 forward the traffic to DC-WAN Switches for DR Endpoints.

Ping Response between DC-EP-1 and DR-EPs

Figure 70: Ping Response Between DC-EP-1 and DR-EPs

Migrate Remaining Endpoints

Physical Design after Remaining Endpoints Migration

After migrating the remaining Endpoints from DC to DR DC-EPG1-WEB, the physical diagram changed accordingly.

Figure 71: Physical Design after all Endpoints Migration from DC to DR

Logical Design after Remaining Endpoints Migration

DC-EPG1-WEB, DC-BD1-WEB, and DC-VRF-WEB are already stretched between DC and DR Sites. DCs remaining Endpoints migrated from DC to the DR Site.

Figure 72: Logical Design after Remaining Endpoint Migration

Intra EPG Traffic Flow after Remaining Endpoint Migration

Figure 73: Intra EPG Traffic Flow after Remaining Endpoint Migration

Communication between DC-EP-1 and DC-EP-2 is Intra EPG communication, as both Endpoints belong to DC-EPG1-WEB. This communication happens directly within the DR Site.

Inter EPG, Inter VRF, and Inter DC traffic flows remain similar to DC-EP-1 migration.

Undeploy Template-EPG1-BD1-Stretched from the DC Site

All the Endpoints are migrated from DC to the DR site for DC-EPG1-WEB. DC-EPG1-WEB and DC-BD1-WEB are not required in the DC Site. Undeploy the Template-EPG1-BD1-Stretched from DC Site, this deletes the EPG and BD from Site-1.

Figure 74: Click Undeploy Template

Figure 75: Choose DC-SITE1 and Click Undeploy

Dissociate Template-EPG1-BD1-Stretched from DC Site

This step dissociates the Template-EPG1-BD-Stretched from the DC Site.

Figure 76: Click Dissociate Template

Figure 77: Uncheck DC-SITE1

Figure 78: DC-SITE2 Part of Template-EPG1-BD1-Stretched

Logical Design after Undeploying the Template-EPG1-BD1-Stretched from DC

DC-EPG1-WEB and DC-BD1-WEB are not part of the DC Site after Undeploying the Template.

Figure 79: Logical Design after Undeploying the Template

Template-VRF-Contract-Site2 Creation

Template-VRF-Contract-Site2 created inside Schema-1. DR-SITE2 added to Template and Tenant-Production associated with the same Template. This is a site-specific template. This template is used in order to associate VRF and Contract from the DR site for DC-EPG1-WEB and DC-BD1-WEB.

Figure 80: Add Application Template - Choose ACI Multi-Cloud

Figure 81: Add Template Name Template-VRF-Contract-Site2, Choose Tenant Production

Figure 82: Template-VRF-Contract-Site2 Details

Import VRF-Contract in Template-VRF-Contract-Site2

Import DR-VRF-WEB and DR-VRF-WEB-Contract from DR-SITE2.

Figure 83:Click Import and Choose DR-SITE2

Figure 84: Choose Contract from DR-SITE2

Figure 85: Choose Filter from DR-SITE2

Figure 86: Choose VRF from DR-SITE2

Figure 87: Template-WEB-VRF-Contract-Site2 with VRF/Contract Information

Deploy Template-VRF-Contract-Site2

Click Deploy Template-VRF-Contract-Site2  and select DR-SITE2.

Figure 88: Add Fabrics to Template-VRF-Contract-Site2

Figure 89: Deploy Out-of-Sync Templates

Figure 90: Deployment Completed

Associate DR-VRF-WEB to DC-BD1-WEB

Associate DR-VRF-WEB to DC-BD1-WEB from Template-EPG1-BD1-Stretched which was created earlier. DC-BD1-WEB is part of DR-SITE2.

Figure 91: Click Template-EPG1-BD1-Stretched

Figure 92: Associate DR-VRF-WEB to DC-BD1-WEB

Apply DR-Contracts to DC-EPG1-WEB

Apply DR-Contract to DC-EPG1-WEB which uses DR contracts for the communication from DC-EPG1-WEB for Inter-DC, Inter-VRF, and Inter-EPG. DC-EPG1-WEB is part of DR-SITE2.

Figure 93: Delete DC-Contracts from DC-EPG1-WEB

Figure 94: Add DR-Contracts in DC-EPG1-WEB

Figure 95: Template-EPG1-BD1-Stretched Information

Figure 96: Deploy Out-of-Sync Templates

Figure 97: Deployment Completed

DC-Endpoint-1 Traffic Flow

DC-Endpoint-1 starts using DR-L3Out-WEB for communication with DC Endpoints. This communication requires necessary routing changes on WAN Switches.

Figure 98: DC-Endpoint-1 Traffic Flow

Ping Response between DC-EP-1 and DC/DR-EPs

Figure 99: Ping Response between DC-EP-1 and DC-EP-2