Day 0 Operations of Cisco ACI Multi-Site

This chapter contains the following sections:

Day-0 Operations

Cisco ACI Multi-Site Communication Ports

When configuring your Cisco ACI Multi-Site environment, keep in mind that the following ports are used by the Cisco ACI Multi-Site Orchestrator for network communications within the Cisco ACI Multi-Site environment.

Ports required for network communications between the Cisco ACI Multi-Site Orchestrator and Cisco APICs (Sites):

  • TCP Port 80/443 for APIC REST Configuration Deployment

Ports required for network communications between the Cisco ACI Multi-Site Orchestrator nodes:

  • TCP port 2377 for Cluster Management Communications

  • TCP and UDP port 7946 for Inter-Manager Communication

  • UDP port 4789 for Docker Overlay Network Traffic

All control-plane and data-plane traffic between Cisco ACI Multi-Site Orchestrator nodes is encrypted with IPSec's Encapsulating Security Payload (ESP) using IP protocol number 50 to provide security and allow the cluster deployments over a round-trip time distance of up to 150ms. If there is firewall between any Orchestrator nodes, proper rules must be added to allow this traffic.

Defining the Overlay TEP for Cisco APIC Sites Using the Cisco APIC GUI

Before connecting a Cisco APIC cluster (fabric) in a Cisco ACI Multi-Site topology, you must configure the Overlay Tunnel Endpoint (TEP) in the Fabric Ext Connection Policy for each fabric.

The Create Intrasite/Intersite Profile panel in the Cisco APIC GUI is used to add connection details for Cisco APIC multipod, remote leaf switches connecting to the Cisco ACI fabric, and APIC sites managed by Cisco ACI Multi-Site Orchestrator. When the Cisco ACI Multi-Site infrastructure has been configured, the Cisco ACI Multi-Site Orchestrator adds the Intersite Overlay TEP to this Cisco APIC policy.

To configure the Overlay TEP in the Fabric Ext Connection Policy for each Cisco APIC site to be managed by Cisco ACI Multi-Site Orchestrator, perform the following steps:

Procedure

Step 1

On the menu bar, click Tenants > infra.

Step 2

On the navigation pane (prior to Cisco APIC, Release 3.1), expand Networking and Protocol Policies.

Step 3

On the navigation pane (in APIC, Release 3.1 and later), expand Policies and Protocol.

Step 4

Right-click Fabric Ext Connection Policies and choose Create Intrasite/Intersite Profile.

Step 5

Click the + symbol on Pod Connection Profile.

Step 6

Choose the Pod ID from the list.
Step 7

Enter the IP address for overlay traffic to this pod.
Step 8

Click Update and Submit.

Adding Sites Using Multi-Site Orchestrator GUI

This section describes how to add sites using the Cisco ACI Multi-Site Orchestrator GUI.

Procedure

Step 1

Log in to the Multi-Site GUI, in the Main menu, click Sites.

If you are logging in for the first time, the default log in is admin and password is We1come2msc!. Then you are forced to change the password upon initial log in.

The new password requirements are:

  • At least 12 characters

  • At least 1 letter

  • At least 1 number

  • At least 1 special character apart from * and space
Step 2

In the Sites List page, click ADD SITES.

Step 3

In the Sites Details page, perform the following actions:

  1. In the NAME field, enter the site name.

  2. In the LABELS field, choose or create a label.

  3. In the APIC CONTROLLER URL field, enter the Cisco APIC URL. This can be https://<ip_address/dns_registered_hostname> or http://<ip_address/dns_registered_hostname>.

    If you have more than one APIC in a fabric, click the + icon to add additional APICs.

  4. In the USERNAME field, enter the user name.

  5. In the PASSWORD field, enter the password.

  6. You can turn on the SPECIFY DOMAIN FOR SITE switch, if you want to specify a domain name for the site.

    In the DOMAIN NAME field, enter the domain name for the site.

  7. If the Cisco APIC site does not have a site ID, you will receive the following message:

    Cisco APIC does not have an apic-site-id configured. Please provide a unique apic-site-id for this site. Once specified the apic-site-id cannot be changed without factory resetting Cisco APIC.

    • Click Ok.

    • In the SITE ID field, enter the site ID.

      The site ID must be an unique identifier of the Cisco APIC site. The range must be from 1 to 127.

  8. Click SAVE.

Step 4

Repeat these steps to add additional sites.

Configuring Infra Using Cisco ACI Multi-Site Orchestrator GUI

This section describes how to register sites and configure fabric connectivity infra for the sites using the Cisco ACI Multi-Site Orchestrator GUI.

Before you begin

  • Ensure you have at least 2 sites.

    For more information, see Adding Sites Using Multi-Site Orchestrator GUI.

  • In Cisco APIC, you need to have the Multipod Overlay TEP configured on the POD connection profile.

    For more information, see Defining the Overlay TEP for Cisco APIC Sites Using the Cisco APIC GUI.

  • In Cisco APIC, you need to have one POD profile and it must contains a POD policy group. If it does not have a POD policy group you need to create one.

    To check if the POD profile contains a POD policy group:

    • Navigate to the Cisco APIC GUI, Fabric > Fabric Policies > Pod Policies > Profiles > Pod Profile default.

    To create a POD policy group:

    • Navigate to the Cisco APIC GUI, Fabric > Fabric Policies > Pod Policies, right-click Policy Groups and click Create Pod Policy Group. Enter the appropriate information and click Submit.

    To assign the new pod policy group to the POD Profile default:

    • Navigate to the Cisco APIC GUI, Fabric > Fabric Policies > Pod Policies > Profiles > Pod Profile default. Click on the default, choose the new pod policy group and click Update.

  • Any infrastructure changes such as adding, removing spines or spine node ID changes would require a Multi-Site fabric connectivity site refresh.

Procedure

Step 1

Log in to the Cisco ACI Multi-Site Orchestrator GUI, in the Main menu, click Sites.

Step 2

In the Sites List area, click CONFIGURE INFRA.

Step 3

In the Fabric Connectivity Infra page, perform the following actions:

  1. In the Master List, click General Settings.

  2. In the Canvas, in the BGP PEERING TYPE area, from the drop-down list, choose either full-mesh or route-reflector.

    The default is full-mesh.

  3. In the KEEPALIVE INTERVAL (SECONDS) field, enter the keep alive interval seconds.

    The default is 60 seconds.

  4. In the HOLD INTERVAL (SECONDS) field, enter the hold interval seconds.

    The default is 180 seconds.

  5. In the STALE INTERVAL (SECONDS) field, enter stale interval seconds.

    The default is 300 seconds.

  6. In the GRACEFUL HELPER field, choose ON or OFF.

    The default is ON.

  7. In the MAXIMUM AS LIMIT field, enter the maximum as limit.

    The default is 0.

  8. In the BGP TTL BETWEEN PEERS field, enter the BGP TTL between peers.

    The default is 10.

Step 4

In the Property Pane, in the OSPF area, perform the following actions:

  1. You can either modify the msc-ospf-policy-default policy or you can add a new OSPF policy.

    To add a new OSPF, click ADD POLICY.

    • In the POLICY NAME field, enter the policy name.

    • In the NETWORK POINT field, choose either broadcast, point-to-point, or unspecified.

      The default is broadcast.

    • In the PRIORITY field, enter the priority number.

      The default is 1.

    • In the COST OF INTERFACE field, enter the cost of interface.

      The default is 0.

    • In the INTERFACE CONTROLS field, choose advertise-subnet, bfd, mtu-ignore, or passive-participation.

    • In the HELLO INTERVAL (SECONDS) field, enter the hello interval in seconds.

      The default is 10.

    • In the DEAD INTERVAL (SECONDS) field, enter the dead interval in seconds.

      The default is 40.

    • In the RETRANSMIT INTERVAL (SECONDS) field, enter the retransmit interval in seconds.

      The default is 5.

    • In the TRANSMIT DELAY (SECONDS) field, enter the transmit delay in seconds.

      The default is 1.

Step 5

In the Master list, choose a site from the SITE SETTINGS.

  1. In the Property Pane, perform the following actions:

    Note 
    If you add or remove any spines in the Cisco APIC GUI, in the Canvas, click on the site and click refresh. This will discover any new or removed spines and all site-related fabric connectivity to be re-imported from Cisco APIC. Any changes not pushed to Cisco APIC will be lost.

    • In the SITE IS MULTI-SITE ENABLED, turn on the site.

    • In the APIC SITE ID field, only displays the Cisco APIC site ID. You cannot change the site ID.

    • In the OVERLAY MULTICAST TEP field, enter the Overlay multicast TEP IP address.

    • In the BGP AUTONOMOUS SYSTEM NUMBER field, enter the BGP autonomous system number or the IP address.

    • (Optional) In the BGP PASSWORD field, if you have encryption enable then you can set a BGP password.

    • If you are running release 1.1(2) or prior: In the BGP COMMUNITY field, enter the BGP community. The format example is: extended:as2-nn4:4:15. The numbers are variables.

    • In the OSPF AREA ID field, enter the OSPF area ID or the IP address.

      Note 

      When configuring the Multi-Site infra OSPF details, Cisco recommends that you use OSPF Area 0. If you use an Area ID other than 0, in the next step configure it as a regular OSPF area type and not a stub area type.

    • In the OSPF AREA TYPE field, choose either nssa, regular, or stub.

      The default is nssa.

    • In the EXTERNAL ROUTER DOMAIN field, choose a external router domain that you have created in the APIC GUI.

    • In the IP SUBNETS TO IMPORT field, click ADD SUBNET. You can have more than one subnet.

      • In the SUBNET field, enter the subnet. You can either add the IP address or the IP address/netmask.

      • Click SAVE.

  2. In the Cavans, click on the POD and perform the following actions:

    • In the Property Pane, in the OVERLAY UNICAST TEP field, enter the Overlay unicast TEP IP address.

  3. In the Cavans, click on the spine and perform the following actions:

    • In the Property Pane, click ADD PORT and perform the following actions:

      • In the PORT ID field, enter the port ID (1/29).

      • In the IP ADDRESS field, enter the IP address/netmask.

      • In the MTU field, enter the MTU. The range is 576 to 9000 or inherit.

      • In the OSPF POLICY field, choose the OSPF policy.

      • Click SAVE.

        Note 

        • Cisco ACI Multi-Site Orchestrator creates a sub-interface with VLAN 4 with the specified IP ADDRESS under the specified PORT.

        • MTU of the spine port should match MTU on IPN side.

        • OSPF settings under OSPF policy should match on IPN side.

        • Multi-Site does not require to run PIM Bidir inside the IPN.

    • (Optional) In the Property Pane, turn on BGP PEERING.

      • In the BGP-EVPN Router-ID field, enter the BGP-EVPN Router-ID IP address.

    • (Optional) In the SPINE IS ROUTE REFLECTOR field, turn it on if the spine can be route reflected.

    • Repeat step 5c for each spine.

  4. Repeat step 5 for the other sites.
Step 6

(Optional) If you are running release 1.2(1) or later and decide to use the same Overlay Unicast TEP for Cisco ACI Multi-Site.

  1. In the Fabric Connectivity Infra, click on the site.

  2. Click on the POD.

  3. In the Property Pane, you can add the same Overlay Unicast TEP for each POD.

    For more information, in the Overlay Unicast TEP field, click on the i icon.

Step 7

Click APPLY.

Note 
If you receive an error message regarding a value that is incorrect in the a field for a particular site, go to that site and correct the value. Then click APPLY.

Adding Tenants

This section describes how to add tenants using the Multi-Site Orchestrator GUI.

Before you begin

You must have a user with either Power User or Site Manager read-write role to create and manage tenants.

Procedure

Step 1

Log in to the Cisco ACI Multi-Site Orchestrator GUI.

Step 2

From the left navigation pane, select Tenants.

Step 3

In the main pane, click Add Tenant.

Step 4

In the Display Name field, provide the tenant's name.

The tenant's Display Name is used throughout the Orchestrator's GUI whenever the tenant is shown. However, due to object naming requirements on the Cisco APIC, any invalid characters are removed and the resulting Internal Name is used when pushing the tenant to sites. The Internal Name that will be used when creating the tenant is displayed below the Display Name textbox.

You can change the Display Name of the tenant at any time, but the Internal Name cannot be changed after the tenant is created.

Step 5

(Optional) In the Description field, enter a description of the tenant.

Step 6

In the Associated Sites section, add the sites.

  1. Check all sites where you plan to deploy templates that use this tenant.

    Only the selected sites will be available for any templates using this tenant.

  2. From the Security Domains drop-down list, choose the site's security domains.

    Security domains are created using the Cisco APIC GUI and can be assigned to various Cisco APIC policies and user accounts to control their access. For more information, see the Cisco APIC Basic Configuration Guide.

Step 7

In the Associated Users section, add Orchestrator users.

Only the selected users will be able to use this tenant when creating templates.

Step 8

(Optional) Enable consistency checker scheduler.

You can choose to enable regular consistency checks. For more information about the consistency checker feature, see Cisco ACI Multi-Site Troubleshooting Guide.

Step 9

Click SAVE to finish adding the tenant.

Adding Schemas Using Cisco ACI Multi-Site Orchestrator GUI.

This section describes how to add schemas using the Cisco ACI Multi-Site Orchestrator GUI.

Procedure

Step 1

Log in to the Cisco ACI Multi-Site Orchestrator GUI, in the Main menu, click Schemas.

Step 2

In the Schemas List area, click ADD SCHEMA.

Step 3

In the Untitled Schema field, enter the new schema's name.

Step 4

Select a tenant.

In the main window pane, click To build your schema please click here to select a tenant then select a tenant from the SELECT A TENANT drop-down list.

Step 5

(Optional) Import fabric elements.

You can create new objects and push them out to one or more sites or you can import existing site-local objects and manage them using the Multi-Site Orchestrator. To import existing objects:

  1. Click IMPORT button.

  2. Select the site from which you want to import objects

  3. In the Import window that opens, select one or more objects you want to import.

    Note 

    The names of the objects imported into the Multi-Site Orchestrator must be unique across all sites. Importing different objects with duplicate names will cause a schema validation error and the import to fail. If you want to import objects that have the same name, you must first rename them.

Step 6

Add new fabric elements.

  1. Click + Application profile, in the Master List, enter the application profile name.

  2. Click + Add EPG field, in the Master List, perform the following actions:

    1. In the DISPLAY NAME field, enter the EPG name.

    2. Click ADD SUBNET, in the Add Subnet pane, perform the following actions:

      1. In the GATEWAY IP field, enter the gateway IP/netmask.

      2. In the DESCRIPTION field, enter a brief description.

      3. In the SCOPE section, choose Private to VRF or Advertised Externally radio button.

      4. In the SHARED BETWEEN VRFS section, place a check in the check box to share between VRFs.

      5. In the NO DEFAULT SVI GATEWAY section, place a check in the check box to not have a default SVI gateway.

      6. Click SAVE.

      7. Repeat 3d to create another EPG. You should have two EPGs.

  3. In the BRIDGE DOMAIN field, from the drop-down list, choose a bridge domain or enter a bridge domain name to create one.

  4. Click + CONTRACT field, perform the following actions:

    1. In the CONTRACT field, from the drop-down list, choose a contract or enter a contract name to create one.

    2. In the TYPE field, from the drop-down list, choose consumer.

    3. Click SAVE.

  5. Click ADD CONTRACT field to add a second contract, perform the following actions:

    1. In the CONTRACT field, from the drop-down list, choose a contract or enter a contract name to create one.

    2. In the TYPE field, from the drop-down list, choose provider.

    3. Click SAVE.

  6. Click + VRF, in the Master List, perform the following actions:

    1. In the DISPLAY NAME field, enter the VRF name.

  7. Click + Add Bridge Domain, in the Master List, perform the following actions:

    1. In the DISPLAY NAME field, enter the bridge domain name.

    2. In the VIRTUAL ROUTING & FORWARDING field, from the drop-down list, choose a VRF name or enter a VRF name to create one.

    3. In the L2STRETCH section, place a check in the check box to enable Layer 2 stretch.

    4. In the INTERSITEBUMTRAFFICALLOW section, place a check in the check box to allow intersite BUM traffic.

    5. In the L2UNKNOWNUNICAST field, from the drop-down list, choose proxy or flood.

    6. Click [+] Add Subnet, perform the following actions:

      1. In the GATEWAY IP field, enter the gateway IP address/netmask.

      2. In the DESCRIPTION field, enter a brief description of the subnet.

      3. In the SCOPE field, choose Private to VRF or Advertised Externally.

      4. In the SHARED BETWEEN VRFS section, place a check in the check box to share between VRFs.

      5. In the NO DEFAULT SVI GATEWAY section, place a check in the check box to not have a default SVI gateway.

      6. In the QUERIER section, place a check in the check box to querier.

      7. Click OK.

  8. Click Sites +, place a check in the check box for each site.

  9. Click SAVE.

  10. Click Click DEPLOY TO SITES.