Chapter 1. New and Changed Information

This chapter contains the following sections:

- New and Changed Information

New and Changed Information

The following table provides an overview of the significant changes to this guide up to this current release. The table does not provide an exhaustive list of all changes made to the guide or of the new features up to this release.

New Features and Changed Behavior in the Cisco ACI Multi-Site Configuration Guide

Cisco ACI Multi-Site Version

Feature

Description

Where Documented

2.0(2)

Contract Preferred Group

Several EPGs can be added to a Contract Preferred Group to allow full communication between the EPGs in a single VRF

Contract Preferred Groups in Schema Management.

2.0(1)

ACI CloudSec Encryption

CloudSec Encryption provides inter-site traffic encryption.

Cisco ACI CloudSec Encryption in Infrastructure Management.

2.0(1)

Layer-3 Multicast

Starting with Release 2.0(1), you can configure Layer-3 Multicast between sites.

Layer 3 Multicast in Schema Management.

Chapter 2. Overview of the GUI

- Overview of the Cisco ACI Multi-Site Orchestrator GUI

- Dashboard

- Sites Page

- Schemas Page

- Tenants Page

- Users Page

- Admin Page

Overview of the Cisco ACI Multi-Site Orchestrator GUI

The Cisco ACI Multi-Site (Multi-Site) Orchestrator GUI is a browser-based graphical interface for configuring and monitoring your ACI and APIC implementations.

The GUI is arranged according to function. For example, after you log in and are on the Dashboard click Schemas to go to your schemas page. You can view all of your existing schemas or to create a new schema on this page.

The functionality of each Multi-Site Orchestrator GUI page is described in the following sections:

- Dashboard

- Sites Page

- Schemas Page

- Tenants Page

- Users Page

- Admin Page

The top of each page shows the controller status indicating how many controllers are operational.

Click the Welcome link to run a troubleshooting report, reset your password, or to log out.

Dashboard

The Multi-Site dashboard displays the list of all of your site implementations in addition to their current functionality and health.

The following screen shot shows the Multi-Site dashboard display:

Multi-Site Dashboard

The Dashboard has the following functional areas:

- Site Status: The site status table lists your sites according to name and location. The table also indicates the current health status for your implementation according to a descriptive color code.

- The Controller State column indicates the number of controllers available and running. You can have a maximum number of 3 controllers in your Multi-Site implementation. For example, if one out of the 3 controller is down it is represented as 2/3.

- The Connectivity column provides an operational status of the BGP sessions and the dataplane unicast and multicast tunnels that are connected to the peer sites for each site in the dashboard. This functionality is available starting with Cisco ACI Multi-Site, Release 1.0(2).

When one or more BGP sessions or tunnels fail to establish, ACI Multi-Site provides the information about which exact local spines and remote spines failed to establish the BGP session or the tunnel. ACI Multi-Site should be enabled in the site in the infrastructure configuration, for the BGP sessions and the dataplane unicast and multicast tunnels to be established to the peer sites.

BGP Sessions

- When the BGP peering type is full-mesh in Infra-> General Settings, the spine node in a site with the BGP peering enabled will establish the BGP sessions to all the spine nodes with the BGP peering enabled in all the peer sites.

- When the BGP peering type is route-reflector in Infra-> General Settings, the spine node in a site with both BGP peering enabled and route-reflector enabled, will establish the BGP sessions to all the spine nodes with the BGP peering enabled in all the peer sites. In the route-reflector mode, at least the local spine node or the remote spine node or both should have the route-reflector enabled. Otherwise, the BGP session is not established between them.

- If the local and the remote ASNs are different, then it is eBGP. Therefore, the sessions between those sites are always full-mesh, irrespective of the BGP peering type and the route-reflector configuration.

Unicast and Multicast Tunnels: A spine node in a site that is connected to ISN and has infrastructure configuration, will establish a tunnel to all the spine nodes that are connected to ISN in the peer sites.

The color codes indicate the following conditions:

- Critical (red)

- Major (orange)

- Minor (yellow)

- Warning (green)

The numbers in the color indicator columns indicate the number of faults per site.

- + Add Site: enables you to add another site to our implementation. When you click + Add Site, you must provide the following site details information on the Connection Settings page:

- Name: the name of the site

- Labels: the label identifier of the site. Multiple labels can be associated to a site.

- APIC Controller URL: you can add more APIC controllers with a distinguishing URL of a cluster.

- Username and Password: APIC login info with admin level privileges.

- Specify Domain For Site: click the switch to on and provide the domain name if default authentication domain is configured in APIC.

After you have entered your details for your new site, click the Save button.

- Schema Health: provides a listing of your schemas with locales and health.

- Click the magnifying glass icon and enter a schema name to search for a subject schema.

- Click + Add Schema to start the procedure for adding a new schema to your site.

- Click the site locale in the Schema Health table to view the schema details and status for a template.

The Schema Health table provides a heat map type of display; that is, the health of the subject schema is displayed according to color. Schemas that span two columns (i.e, locales) indicate a stretched condition.

- Click the color highlighted table cell to further discover what policies are incorporated into the subject schema. On the schema details page, you can click the arrow to go into the schema builder and update the policy details in the subject schema.

- The color coded slider enables you to select a range for identifying schemas whose health require further review. For example, you can adjust the slider value to between 80 and 100. Then all of your schema implementations that fall within that specific range are displayed on the accompanying Schema Health table.

Sites Page

The Multi-Site Sites page displays all of the sites in your implementation. An example of the Sites page is shown in the following screen shot:

Multi-Site Sites Page

The Sites page consists of the following two panes:

- Site Name or Label: the site status table lists your sites and then indicates the current health status for your implementation according to the following color coded identifiers:

- Critical (red)

- Major (orange)

- Minor (yellow)

- Warning (green)

When you click a specific site, you can view or edit the site's details on the Connection Settings display:

- Name

- Labels

- APIC Controller URL

- Username and Password

- Specify Domain For Site

- APIC Site ID

If you have made changes to the listed fields, click the Save button.

- APIC Controller URLs: the associated APIC URLs for your Multi-Site implementation

- Configure Infra: click this area to configure your Fabric infrastructure connectivity. For more information, refer to the Cisco Application Policy Infrastructure Controller (APIC) page.

- Add Site: click the Add Site button to add a site to your implementation. The following details are required for adding a site:

- Name: the site name.

- Label: select an existing or create a new label.

- APIC Controller URL: the existing URL - click + to add a new APIC Controller URL.

- Username: the site username.

- Password: the unique site password for access.

- Specify Domain for Site: click the selector to On to specify a domain for the site.

- Actions: drop down menu list option to edit, delete, or open a subject site in the APIC user interface.

Audit Logs

Click the Audit Log icon next to the Configure Infra tab to list the log details for the Sites page. The Audit Logs: Sites List page is displayed.

The table on the page displays the following details:

- Date

- Action

- Details

- User

Click the Most Recent tab to select the audit logs during a particular time period. For example, when you select the range from November 14, 2017 to November 17, 2017 and click Apply, the audit log details for this time period are displayed on the Audit Logs page.

Click the Filter icon next to the Most Recent tab to filter the log details using the following criteria:

- User: Select one user name or all users and click Apply to filter the log details using the user name.

- Action: Select the action, for example, created, updated, or deleted, and click Apply to filter the log details according to the action.

Schemas Page

The Multi-Site Schemas page lists all schemas that are associated with your implementation.

The following screen shot shows an example display:

Multi-Site Schemas Page

Use the magnifying glass and associated field to search for a specific schema. Use schemas to configure or import tenant policies, including the VRF, application profile with EPGs, filters and contracts, bridge domains, and external EPGs.

The Schemas table shows the following information in tabular form:

- Name: click the schema name to view or update the settings for the subject schema.

- Templates: displays the name of the template that is used for the schema. Templates are analogous to profiles in the ACI context, which group policies. You can create templates for stretched objects or site-specific objects.

- Tenants: displays the name of the tenant that is used for the subject schema.

- Actions: click the Action field with the associated schema to either edit or delete the subject schema.

Click the Add Schema button to add a new schema to your implementation. Further details on creating a schema are described in Schema Management.

Audit Logs

Click the Audit Log icon next to the Add Schema tab to list the log details for the Schemas page. The Audit Logs: Schemas List page is displayed.

The table on the page displays the following details:

- Date

- Action

- Details

- User

Click the Most Recent tab to select the audit logs during a particular time period. For example, when you select the range from November 10, 2017 to November 14, 2017 and click Apply, the audit log details for this time period are displayed on the Audit Logs page.

Click the Filter icon next to the Most Recent tab to filter the log details using the following criteria:

- User: Select one username or all users and click Apply to filter the log details using the username.

- Action: Select the action, for example, created, updated, or deleted, and click Apply to filter the log details according to the action.

For more information about creating schemas, refer to Schema Management.

Tenants Page

The Multi-Site Tenants page lists all of the tenants that comprise your implementation.

The following screen shot provides an example:

Multi-Site Tenants Page

The table on the Tenants page displays the following:

- Tenant Name

- Assigned to Sites

- Assigned to Users

- Assigned to Schemas

- Actions

The features and functionality on this page include the following:

- Name: click a tenant name to access the Tenant Details settings page. On the Tenant Details page you can edit or update the following sections:

- General Settings: change the Display Name and Description as required.

- Associated Sites: view the sites associated with the subject tenant.

- Associated Users: view the users associated with the subject tenant - you can associate a user with the subject tenant by checking the empty box next to the user name.

- Associated Schemas: click the Associated Schema listing to view the schemas associated with the subject tenant.

- Actions: click the Actions listing to edit the subject tenant's details sites or to create a new network mapping.

Note

You can delete the Tenant object by selecting Delete on the Actions drop down menu.

- Add Tenant: click Add Tenant button to add an existing tenant to your implementation. On the proceeding Tenant Details page, you can add the tenant name, description, security domain, and associated users.

Audit Logs

Click the Audit Log icon next to the Add Tenant tab to list the log details for the Tenants page. The Audit Logs: Tenants List page is displayed.

The table on the page displays the following details:

- Date

- Action

- Details

- User

Click the Most Recent tab to select the audit logs during a particular time period. For example, when you select the range from November 10, 2017 to November 14, 2017 and click Apply, the audit log details for this time period are displayed on the Audit Logs page.

Click the Filter icon next to the Most Recent tab to filter the log details using the following criteria:

- User: Select one user name or all users and click Apply to filter the log details using the user name.

- Action: Select the action, for example, created, updated, or deleted, and click Apply to filter the log details according to the action.

For more information about adding tenants, refer to Tenant Management.

Users Page

The Multi-Site Users page displays all of the identified users in your Multi-Site implementation. An example of the Users page is as follows:

Multi-Site Users Page

The Users page features a table containing all of the identified users by username and associated email and current activity status. If you click a selected Username, you can access the General Setting page attributable to the subject user. On the General Setting page, you can edit the details associated with the subject user such as username, password, email, and switch-on user roles.

Click Add User to add a new user to your Multi-Site implementation. The General Setting page display enables you to assign username, password, email, and switch-on user roles associated with your Multi-Site implementation.

Audit Logs

Click the Audit Log icon next to the Add User tab to list the log details for the Users page. The Audit Logs: Users List page is displayed.

The table on the page displays the following details:

- Date

- Action

- Details

- User

Click the Most Recent tab to select the audit logs during a particular time period. For example, when you select the range from November 10, 2017 to November 14, 2017 and click Apply, the audit log details for this time period are displayed on the Audit Logs page.

Click the Filter icon next to the Most Recent tab to filter the log details using the following criteria:

- User: Select one user name or all users and click Apply to filter the log details using the user name.

- Action: Select the action, for example, created, updated, or deleted, and click Apply to filter the log details according to the action.

Admin Page

When you select the Admin tab from the Cisco ACI Multi-Site Orchestrator navigation bar, it expands the following additional selection of administrative pages:

- Providers

- Login Domains

- Backups

- Audit Logs

- Security

Providers

Cisco ACI Multi-Site Orchestrator Providers Page

The Providers page under the Admin heading displays information about any configured external authentication providers. The following details are shown for each provider:

- Host Name

- Type

- Description

- Port

- Timeout (Sec)

- Retries

Working with external authentication providers is described in Administrative Operations.

Login Domains

Cisco ACI Multi-Site Orchestrator Login Domains Page

The Login Domains page under the Admin heading displays information about the available login domains. The following details are shown for each domain:

- Name

- Description

- Provider

- Status

- Default

Working with login domains is described in Administrative Operations.

Backups

Cisco ACI Multi-Site Orchestrator Backups Page

The Backups page under the Admin heading displays information about any backups that have been created. The following details are shown for each domain:

- Date

- Name

- Size

- Notes

Working with backups is described in Administrative Operations.

Audit Logs

Cisco ACI Multi-Site Orchestrator Audit Logs Page

The Audit Logs page under the Admin heading displays information about the audit logs and records. The following details are shown:

- Date

- Action

- Type

- Details

- User

Working with logs is described in Administrative Operations.

Security

Cisco ACI Multi-Site Orchestrator Security Page

The Security page under the Admin heading displays information about the custom certificates and key rings you have configured for use by the Orchestrator. The following details are shown:

- Certificate Authority

- Name

- Description

- Key Rings

- Name

- Description

- Trustpoint

- State

Working with certificates is described in Administrative Operations.

Chapter 3. Infrastructure Management

- Cisco ACI Multi-Site Communication Ports

- Defining the Overlay TEP for Cisco APIC Sites Using the Cisco APIC GUI

- Configuring Infra Prerequisites and Guidelines

- Configuring Infra Using Cisco ACI Multi-Site Orchestrator GUI

- Adding Sites Using Multi-Site Orchestrator GUI

- Deleting Sites Using Multi-Site Orchestrator GUI

- Cisco ACI CloudSec Encryption

- Multi-Site Cross Launch to Cisco APIC

Cisco ACI Multi-Site Communication Ports

When configuring your Cisco ACI Multi-Site environment, keep in mind that the following ports are used by the Cisco ACI Multi-Site Orchestrator for network communications within the Cisco ACI Multi-Site environment.

Ports required for network communications between the Cisco ACI Multi-Site Orchestrator and Cisco APICs (Sites):

- TCP Port 80/443 for APIC REST Configuration Deployment

Ports required for network communications between the Cisco ACI Multi-Site Orchestrator nodes:

- TCP port 2377 for Cluster Management Communications

- TCP and UDP port 7946 for Inter-Manager Communication

- UDP port 4789 for Docker Overlay Network Traffic

All control-plane and data-plane traffic between Cisco ACI Multi-Site Orchestrator nodes is encrypted with IPSec's Encapsulating Security Payload (ESP) using IP protocol number 50 to provide security and allow the cluster deployments over a round-trip time distance of up to 150ms. If there is firewall between any Orchestrator nodes, proper rules must be added to allow this traffic.

Defining the Overlay TEP for Cisco APIC Sites Using the Cisco APIC GUI

Before connecting a Cisco APIC cluster (fabric) in a Cisco ACI Multi-Site topology, you must configure the Overlay Tunnel Endpoint (TEP) in the Fabric Ext Connection Policy for each fabric.

The Create Intrasite/Intersite Profile panel in the Cisco APIC GUI is used to add connection details for Cisco APIC multipod, remote leaf switches connecting to the Cisco ACI fabric, and APIC sites managed by Cisco ACI Multi-Site Orchestrator. When the Cisco ACI Multi-Site infrastructure has been configured, the Cisco ACI Multi-Site Orchestrator adds the Intersite Overlay TEP to this Cisco APIC policy.

To configure the Overlay TEP in the Fabric Ext Connection Policy for each Cisco APIC site to be managed by Cisco ACI Multi-Site Orchestrator, perform the following steps:

Procedure

Step 1

On the menu bar, click Tenants > infra.

Step 2

On the navigation pane (prior to Cisco APIC, Release 3.1), expand Networking and Protocol Policies.

Step 3

On the navigation pane (in APIC, Release 3.1 and later), expand Policies and Protocol.

Step 4

Right-click Fabric Ext Connection Policies and choose Create Intrasite/Intersite Profile.

Step 5

Click the + symbol on Pod Connection Profile.

Step 6

Choose the Pod ID from the list.

Step 7

Enter the IP address for overlay traffic to this pod.

Step 8

Click Update and Submit.

Configuring Infra Prerequisites and Guidelines

The following sections describe the steps necessary to configure the general as well as site-specific fabric Infra settings.

Before you proceed with Infra configuration, you must have configured and added the sites as described in previous sections, which includes:

- Configuring each site's fabric access policies.

- Configuring direct communication and routable subnets for sites with remote leaf switches.

In addition, keep in mind the following:

- Any infrastructure changes such as adding and removing spine switches or spine node ID changes require a Multi-Site fabric connectivity information refresh.

- The Overlay Unicast TEP, Overlay Multicast TEP, and BGP-EVPN Router-IDs IP addresses assigned on the Orchestrator should not be taken from the address space of the original fabric's Infra TEP pool or from the 0.x.x.x range.

Configuring Infra Using Cisco ACI Multi-Site Orchestrator GUI

This section describes how to register sites and configure fabric connectivity infra for the sites using the Cisco ACI Multi-Site Orchestrator GUI.

Before you begin

- Ensure you have at least 2 sites.

For more information, see Adding Sites Using Multi-Site Orchestrator GUI.

- In Cisco APIC, you need to have the Multipod Overlay TEP configured on the POD connection profile.

For more information, see Defining the Overlay TEP for Cisco APIC Sites Using the Cisco APIC GUI.

- In Cisco APIC, you need to have one POD profile and it must contains a POD policy group. If it does not have a POD policy group you need to create one.

To check if the POD profile contains a POD policy group:

- Navigate to the Cisco APIC GUI, Fabric > Fabric Policies > Pod Policies > Profiles > Pod Profile default.

To create a POD policy group:

- Navigate to the Cisco APIC GUI, Fabric > Fabric Policies > Pod Policies, right-click Policy Groups and click Create Pod Policy Group. Enter the appropriate information and click Submit.

To assign the new pod policy group to the POD Profile default:

- Navigate to the Cisco APIC GUI, Fabric > Fabric Policies > Pod Policies > Profiles > Pod Profile default. Click on the default, choose the new pod policy group and click Update.

- Any infrastructure changes such as adding, removing spines or spine node ID changes would require a Multi-Site fabric connectivity site refresh.

Procedure

Step 1

Log in to the Cisco ACI Multi-Site Orchestrator GUI, in the Main menu, click Sites.

Step 2

In the Sites List area, click CONFIGURE INFRA.

Step 3

In the Fabric Connectivity Infra page, perform the following actions:

1. In the Master List, click General Settings.

2. In the Canvas, in the BGP PEERING TYPE area, from the drop-down list, choose either full-mesh or route-reflector.

The default is full-mesh.

3. In the KEEPALIVE INTERVAL (SECONDS) field, enter the keep alive interval seconds.

The default is 60 seconds.

4. In the HOLD INTERVAL (SECONDS) field, enter the hold interval seconds.

The default is 180 seconds.

5. In the STALE INTERVAL (SECONDS) field, enter stale interval seconds.

The default is 300 seconds.

6. In the GRACEFUL HELPER field, choose ON or OFF.

The default is ON.

7. In the MAXIMUM AS LIMIT field, enter the maximum as limit.

The default is 0.

8. In the BGP TTL BETWEEN PEERS field, enter the BGP TTL between peers.

The default is 10.

Step 4

In the Property Pane, in the OSPF area, perform the following actions:

1. You can either modify the msc-ospf-policy-default policy or you can add a new OSPF policy.

To add a new OSPF, click ADD POLICY.

- In the POLICY NAME field, enter the policy name.

- In the NETWORK POINT field, choose either broadcast, point-to-point, or unspecified.

The default is broadcast.

- In the PRIORITY field, enter the priority number.

The default is 1.

- In the COST OF INTERFACE field, enter the cost of interface.

The default is 0.

- In the INTERFACE CONTROLS field, choose advertise-subnet, bfd, mtu-ignore, or passive-participation.

- In the HELLO INTERVAL (SECONDS) field, enter the hello interval in seconds.

The default is 10.

- In the DEAD INTERVAL (SECONDS) field, enter the dead interval in seconds.

The default is 40.

- In the RETRANSMIT INTERVAL (SECONDS) field, enter the retransmit interval in seconds.

The default is 5.

- In the TRANSMIT DELAY (SECONDS) field, enter the transmit delay in seconds.

The default is 1.

Step 5

In the Master list, choose a site from the SITE SETTINGS.

1. In the Property Pane, perform the following actions:

Note

If you add or remove any spines in the Cisco APIC GUI, in the Canvas, click on the site and click refresh. This will discover any new or removed spines and all site-related fabric connectivity to be re-imported from Cisco APIC. Any changes not pushed to Cisco APIC will be lost.

- In the SITE IS MULTI-SITE ENABLED, turn on the site.

- In the APIC SITE ID field, only displays the Cisco APIC site ID. You cannot change the site ID.

- In the OVERLAY MULTICAST TEP field, enter the Overlay multicast TEP IP address.

- In the BGP AUTONOMOUS SYSTEM NUMBER field, enter the BGP autonomous system number or the IP address.

- (Optional) In the BGP PASSWORD field, if you have encryption enable then you can set a BGP password.

- If you are running release 1.1(2) or prior: In the BGP COMMUNITY field, enter the BGP community. The format example is: extended:as2-nn4:4:15. The numbers are variables.

- In the OSPF AREA ID field, enter the OSPF area ID or the IP address.

Note

When configuring the Multi-Site infra OSPF details, Cisco recommends that you use OSPF Area 0. If you use an Area ID other than 0, in the next step configure it as a regular OSPF area type and not a stub area type.

- In the OSPF AREA TYPE field, choose either nssa, regular, or stub.

The default is nssa.

- In the EXTERNAL ROUTER DOMAIN field, choose a external router domain that you have created in the APIC GUI.

- In the IP SUBNETS TO IMPORT field, click ADD SUBNET. You can have more than one subnet.

- In the SUBNET field, enter the subnet. You can either add the IP address or the IP address/netmask.

- Click SAVE.

2. In the Cavans, click on the POD and perform the following actions:

- In the Property Pane, in the OVERLAY UNICAST TEP field, enter the Overlay unicast TEP IP address.

3. In the Cavans, click on the spine and perform the following actions:

- In the Property Pane, click ADD PORT and perform the following actions:

- In the PORT ID field, enter the port ID (1/29).

- In the IP ADDRESS field, enter the IP address/netmask.

- In the MTU field, enter the MTU. The range is 576 to 9000 or inherit.

- In the OSPF POLICY field, choose the OSPF policy.

- Click SAVE.

Note

- Cisco ACI Multi-Site Orchestrator creates a sub-interface with VLAN 4 with the specified IP ADDRESS under the specified PORT.

- MTU of the spine port should match MTU on IPN side.

- OSPF settings under OSPF policy should match on IPN side.

- Multi-Site does not require to run PIM Bidir inside the IPN.

- (Optional) In the Property Pane, turn on BGP PEERING.

- In the BGP-EVPN Router-ID field, enter the BGP-EVPN Router-ID IP address.

- (Optional) In the SPINE IS ROUTE REFLECTOR field, turn it on if the spine can be route reflected.

- Repeat step 5c for each spine.

4. Repeat step 5 for the other sites.

Step 6

(Optional) If you are running release 1.2(1) or later and decide to use the same Overlay Unicast TEP for Cisco ACI Multi-Site.

1. In the Fabric Connectivity Infra, click on the site.

2. Click on the POD.

3. In the Property Pane, you can add the same Overlay Unicast TEP for each POD.

For more information, in the Overlay Unicast TEP field, click on the i icon.

Step 7

Click APPLY.

Note

If you receive an error message regarding a value that is incorrect in the a field for a particular site, go to that site and correct the value. Then click APPLY.

Adding Sites Using Multi-Site Orchestrator GUI

This section describes how to add sites using the Cisco ACI Multi-Site Orchestrator GUI.

Procedure

Step 1

Log in to the Multi-Site GUI, in the Main menu, click Sites.

If you are logging in for the first time, the default log in is admin and password is We1come2msc!. Then you are forced to change the password upon initial log in.

The new password requirements are:

- At least 12 characters

- At least 1 letter

- At least 1 number

- At least 1 special character apart from * and space

Step 2

In the Sites List page, click ADD SITES.

Step 3

In the Sites Details page, perform the following actions:

1. In the NAME field, enter the site name.

2. In the LABELS field, choose or create a label.

3. In the APIC CONTROLLER URL field, enter the Cisco APIC URL. This can be https://<ip_address/dns_registered_hostname> or http://<ip_address/dns_registered_hostname>.

If you have more than one APIC in a fabric, click the + icon to add additional APICs.

4. In the USERNAME field, enter the user name.

5. In the PASSWORD field, enter the password.

6. You can turn on the SPECIFY DOMAIN FOR SITE switch, if you want to specify a domain name for the site.

In the DOMAIN NAME field, enter the domain name for the site.

7. If the Cisco APIC site does not have a site ID, you will receive the following message:

Cisco APIC does not have an apic-site-id configured. Please provide a unique apic-site-id for this site. Once specified the apic-site-id cannot be changed without factory resetting Cisco APIC.

- Click Ok.

- In the SITE ID field, enter the site ID.

The site ID must be an unique identifier of the Cisco APIC site. The range must be from 1 to 127.

8. Click SAVE.

Step 4

Repeat these steps to add additional sites.

Deleting Sites Using Multi-Site Orchestrator GUI

This section describes how to delete sites using the Multi-Site GUI.

Procedure

Step 1

Log in to the Multi-Site GUI.

Step 2

Ensure you unbind the site from any Schema’s before trying to delete the site.

Step 3

In the Main menu, click Sites.

Step 4

In the Sites List page, hover over the site you want to delete and choose Action > Delete .

Step 5

Click YES.

Cisco ACI CloudSec Encryption

As most Cisco ACI deployments are adopting the Cisco ACI Multi-Site architecture to address disaster recovery and scale, the current security implementation using MACsec encryption within local site is becoming insufficient to guarantee data security and integrity across multiple sites connected by insecure external IP networks interconnecting separate fabrics. Cisco ACI Multi-Site Orchestrator Release 2.0(1) introduces the CloudSec Encryption feature designed to provide inter-site encryption of traffic.

Cisco ACI Multi-Site topology uses three tunnel end-point (TEP) IP addresses to provide connectivity between sites. These TEP addresses are configured by the admin on Cisco ACI Multi-Site Orchestrator and pushed down to each site's Cisco APIC, which in turn configures them on the spine switches. These three addresses are used to determine when traffic is destined for a remote site, in which case an encrypted CloudSec tunnel is created between the two spine switches that provide physical connectivity between the two sites through the Inter-Site Network (ISN).

The following figure illustrates the overall encryption approach that combines MACsec for local site traffic and CloudSec for inter-site traffic encryption.

CloudSec Encryption

Hardware Requirement

The current release supports CloudSec Encryption on the following hardware::

- Cisco Nexus 9364 and 9332 spine switches

- FX line cards in Cisco Nexus 9500 switches

- CloudSec Encryption Terminology

- CloudSec Encryption and Decryption Handling

- CloudSec Encryption Key Allocation and Distribution

- CloudSec Supported Hardware and Port Ranges

- Configuring Cisco APIC for CloudSec Encryption

- Enabling CloudSec Encryption Using Cisco ACI Multi-Site Orchestrator GUI

- Rekey Process During Spine Switch Maintenance

CloudSec Encryption Terminology

CloudSec Encryption feature provides a secure upstream symmetric key allocation and distribution method for initial key and rekey requirements between sites. The following terminology is used in this chapter:

- Upstream device – The device that adds the CloudSec Encryption header and does the encryption of the VxLAN packet payload on transmission to a remote site using a locally generated symmetric cryptography key.

- Downstream device – The device that interprets the CloudSec Encryption header and does the decryption of the VxLAN packet payload on reception using the cryptography key generated by the remote site.

- Upstream site – The datacenter fabric that originates the encrypted VxLAN packets.

- Downstream site – The datacenter fabric that receives the encrypted packets and decrypts them.

- TX Key – The cryptography key used to encrypt the clear VxLAN packet payload. In ACI only one TX key can be active per remote site.

- RX Key – The cryptography key used to decrypt the encrypted VxLAN packet payload. In ACI two RX key can be active per remote site.

- Symmetric Keys – When the same cryptography key is used to encrypt (TX Key) and decrypt (RX Key) a packet stream by the upstream and downstream devices respectively.

- Rekey – The process initiated by the upstream site to replace its old key with a newer key for all downstream sites after the old key expires.

- Secure Channel Identifier (SCI) – A 64-bit identifier that represents a security association between the sites. It is transmitted in encrypted packet in CloudSec header and is used to derive the RX key on the downstream device for packet decryption.

- Association Number (AN) – A 2-bit number (0, 1, 2, 3) that is sent in the CloudSec header of the encrypted packet and is used to derive the key at the downstream device in conjunction with the SCI for decryption. This allows multiple keys to be active at the downstream device to handle out of order packet arrivals with different keys from the same upstream device following a rekey operation.

In ACI only two association number values (0 and 1) are used for the two active RX keys and only one association number value (0 or 1) is used for the TX Key at any point in time.

- Pre-shared key (PSK) – One ore more keys must be configured in the Cisco APIC GUI to be used as a random seed for generating the CloudSec TX and RX keys. If multiple PSK are configured, each rekey process will use the next PSK in order of their indexes; if no higher index PSK is available, a PSK with the lowest index will be used. Each PSK must be a hexadecimal string 64 characters long. Cisco APIC supports up to 256 pre-shared keys.

CloudSec Encryption and Decryption Handling

In order to provide a fully integrated, simple, and cost-effective solution that addresses both, data security and integrity, starting with Release 2.0(1), Cisco ACI Multi-Site provides a CloudSec Encryption feature that allows for complete source-to-destination packet encryption between Multi-Site fabrics.

The following figure shows packet diagram before and after CloudSec encapsulation, followed by descriptions of the encryption and decryption processes:

CloudSec Packet

Packet Encryption

The following is a high level overview of how CloudSec handles outgoing traffic packets:

- The packets are filtered using the outer IP header and Layer-4 destination port information and matching packets are marked for encryption.

- The offset to use for encryption is calculated according to the fields of the packet. For example, the offset may vary based on whether there is a 802.1q VLAN or if the packet is an IPv4 or IPv6 packet.

- The encryption keys are programmed in the hardware tables and are looked up from the table using the packet IP header.

Once the packet is marked for encryption, the encryption key is loaded, and the offset from the beginning of the packet where to start the encryption is known, the following additional steps are taken:

- The UDP destination port number is copied from the UDP header into a CloudSec field for recovery when the packet is decrypted.

- The UDP destination port number is overwritten with a Cisco proprietary Layer-4 port number (Port 9999) indicating that it is a CloudSec packet.

- The UDP length field is updated to reflect the additional bytes that are being added.

- The CloudSec header is inserted directly after the UDP header.

- The Integrity Check Value (ICV) is inserted at the end of the packet, between the payload and the CRC.

- The ICV requires construction of a 128-bit initialization vector. For CloudSec, any use of the source MAC address for ICV purposes is replaced by a programmable value per SCI.

- CRC is updated to reflect the change in the contents of the packet.

Packet Decryption

The way CloudSec handles incoming packets is symmetric to the outgoing packets algorithm described above:

- If the received packet is a CloudSec packet, it is decrypted and the ICV is verified.

If ICV verification passed, the extra fields are removed, the UDP destination port number is moved from the CloudSec header to the UDP header, the CRC is updated, and the packet is forwarded to destination after decryption and CloudSec header removal. Otherwise the packet is dropped.

- If the key store returns two or more possible decryption keys, the Association Number (AN) field of the CloudSec header is used to select which key to use.

- If the packet is not a CloudSec packet, the packet is left unchanged.

CloudSec Encryption Key Allocation and Distribution

Initial Key Configuration

CloudSec Key Distribution

The following is a high level overview of the CloudSec encryption key initial allocation and distribution process illustrated by the figure above:

- The upstream site's Cisco APIC generates a local symmetric key intended to be used for data encryption of VxLAN packets transmitted from its site. The same key that is used by the upstream site for encryption is used for decryption of the packets on the downstream remote receiving sites.

Every site is an upstream site for the traffic it transmits to other sites. If multiple sites exist, each site generates its own site-to-site key and use that key for encryption before transmitting to the remote site.

- The generated symmetric key is pushed to the Cisco ACI Multi-Site Orchestrator (MSO) by the upstream site's Cisco APIC for distribution to downstream remote sites.

- The MSO acts as a message broker and collects the generated symmetric key from the upstream site's Cisco APIC, then distributes it to downstream remote sites' Cisco APICs.

- Each downstream site's Cisco APIC configures the received key as RX key on the local spine switches which are intended to receive the traffic from the upstream site that generated the key.

- Each downstream site's Cisco APIC also collects the deployment status of the RX Key from the local spine switches and then pushes it to the MSO.

- The MSO relays the key deployment status from all downstream remote sites back to the upstream site's Cisco APIC.

- The upstream site's Cisco APIC checks if the key deployment status received from all downstream remote sites is successful.

- If the deployment status received from a downstream device is successful, the upstream site deploys the local symmetric key as its TX key on the spine switches to enable encryption of the VxLAN packets that are sent to the downstream site.

- If the deployment status received from a downstream device is failed, a fault is raised on the Cisco APIC site where it failed and it is handled based on the "secure mode" setting configured on the MSO. In "must secure" mode the packets are dropped and in the "should secure" mode the packets are sent clear (unencrypted) to the destination site.

Note

In current release, the mode is always set to “should secure”.

Rekey Process

Each generated TX/RX key expires after a set amount of time, by default key expiry time is set to 15 minutes. When the initial set of TX/RX keys expires, a rekey process takes place.

The same general key allocation and distribution flow applies for the rekey process. The rekey process follows the "make before break" rule, in other words all the RX keys on the downstream sites are deployed before the new TX key is deployed on the upstream site. To achieve that, the upstream site will wait for the new RX key deployment status from the downstream sites before it configures the new TX key on the local upstream site's devices.

If any downstream site reports a failure status in deploying the new RX key, the rekey process will be terminated and the old key will remain active. The downstream sites will also keep the old and the new RX keys deployed after the new key deployment is finished for some duration to ensure that out of order packet deliveries with either key can be properly decrypted.

Note

Special precautions must be taken in regards to rekey process during spine switch maintenance, see Rekey Process During Spine Switch Maintenance for details.

Rekey Process Failure

In case of any downstream site failing to deploy the new encryption key generated by the rekey process, the new key is discarded and the upstream device will continue to use the previous valid key as TX key. This approach keeps the upstream sites from having to maintain multiple TX keys per set of downstream sites. However, this approach may also result in the rekey process being delayed if the rekey deployment failures continue to occur with any one of the downstream sites. It is expected that the Multi-Site administrator will take action to fix the issue of the key deployment failure for the rekey to succeed.

Cisco APIC's Role in Key Management

The Cisco APIC is responsible for key allocation (both, initial key and rekey distribution), collection of the key deployment status messages from the spine switches, and notification of the Cisco ACI Multi-Site Orchestrator about each key's status for distribution to other sites.

Cisco ACI Multi-Site Orchestrator's Role in Key Management

The Cisco ACI Multi-Site Orchestrator is responsible for collecting the TX keys (both, initial key and subsequent rekeys) from the upstream site and distributing it to all downstream sites for deployment as RX keys. The MSO also collects the RX key deployment status information from the downstream sites and notifies the upstream site in order for it to update the TX key on successful RX key deployment status.

Upstream Model

In contrast to other technologies, such as MPLS, that use downstream key allocation, CloudSec's upstream model provides the following advantages:

- The model is simple and operationally easier to deploy in the networks.

- The model is preferred for Cisco ACI Multi-Site use cases.

- It provides advantages for multicast traffic as it can use the same key and CloudSec header for each copy of the replicated packet transmitted to multiple destination sites. In downstream model each copy would have to use a different security key for each site during encryption.

- It provides easier troubleshooting in case of failures and better traceability of packets from the source to destination consistently for both, unicast and multicast replicated packets.

CloudSec Supported Hardware and Port Ranges

The following table provides the hardware platforms and the port ranges that are capable of CloudSec encryption:

Hardware Platform

Port Range

N9K-C9364C spine switches

Ports 49-64

N9K-C9332C spine switches

Ports 25-32

N9K-X9736C-FX line cards

Ports 29-36

If CloudSec is enabled for a site, but the encryption is not supported by the ports, a fault is raised with unsupported-interface error message.

CloudSec encryption's packet encapsulation is supported if Cisco QSFP-to-SFP Adapters (QSA), such as CVR-QSFP-SFP10G, is used with a supported optic. The full list of supported optics is available from the following link: https://www.cisco.com/c/en/us/support/interfaces-modules/transceiver-modules/products-device-support-tables-list.html.

Configuring Cisco APIC for CloudSec Encryption

You must configure one or more Pre-Shared Keys (PSK) to be used by the Cisco APIC for generating the CloudSec encryption and decryption keys. The PSK are used as a random seed during the re-key process. If multiple PSK are configured, each re-key process will use the next PSK in order of their indexes; if no higher index PSK is available, a PSK with the lowest index will be used.

Because PSK is used as a seed for encryption key generation, configuring multiple PSK provides additional security by lowering the over-time vulnerability of the generated encryption keys.

Note

If no pre-shared key is configured on the Cisco APIC, CloudSec will not be enabled for that site. In that case, turning on CloudSec setting in Cisco ACI Multi-Site will raise a fault.

If at any time you wish to refresh a previously added PSK with a new one, simply repeat the procedure as if you were adding a new key, but specify an existing index.

You can configure one or more pre-shared keys in one of three ways:

- Using the Cisco APIC GUI, as described in Configuring Cisco APIC for CloudSec Encryption Using GUI

- Using the Cisco APIC NX-OS Style CLI, as described in Configuring Cisco APIC for CloudSec Encryption Using NX-OS Style CLI

- Using the Cisco APIC REST API, as described in Configuring Cisco APIC for CloudSec Encryption Using REST API

- Configuring Cisco APIC for CloudSec Encryption Using GUI

- Configuring Cisco APIC for CloudSec Encryption Using NX-OS Style CLI

- Configuring Cisco APIC for CloudSec Encryption Using REST API

Configuring Cisco APIC for CloudSec Encryption Using GUI

This section describes how to configure one or more pre-shared keys (PSK) using the Cisco APIC GUI.

Procedure

Step 1

Log in to APIC.

Step 2

Navigate to Tenants > infra > Policies > CloudSec Encryption

Step 3

Specify the SA Key Expiry Time.

This option specifies how long each key is valid (in minutes). Each generated TX/RX key expires after the specified amount of time triggering a re-key process. The expiration time can be between 5 and 1440 minutes.

Step 4

Click the + icon in the Pre-Shared Keys table.

Step 5

Specify the Index of the pre-shared key you are adding and then the Pre-Shared Key itself.

The Index field specifies the order in which the pre-shared keys are used. After the last (highest index) key is used, the process will continue with the first (lowest index) key. Cisco APIC supports up to 256 pre-shared keys, so the PSK index value must be between 1 and 256.

Each Pre-Shared Key must be a hexadecimal string 64 characters long.

Configuring Cisco APIC for CloudSec Encryption Using NX-OS Style CLI

This section describes how to configure one or more pre-shared keys (PSK) using the Cisco APIC NX-OS Style CLI.

Procedure

Step 1

Log in to the Cisco APIC NX-OS style CLI.

Step 2

Enter configuration mode.

Example:

apic1# configure
apic1 (config)#

Step 3

Enter configuration mode for the default CloudSec profile.

Example:

apic1(config)# template cloudsec default
apic1(config-cloudsec)#

Step 4

Specify the Pre-Shared Keys (PSK) expiration time.

This option specifies how long each key is valid (in minutes). Each generated TX/RX key expires after the specified amount of time triggering a re-key process. The expiration time can be between 5 and 1440 minutes.

Example:

apic1(config-cloudsec)# sakexpirytime <duration>

Step 5

Specify one or more Pre-Shared Keys.

In the following command, specify the index of the PSK you're configuring and the PSK string itself.

Example:

apic1(config-cloudsec)# pskindex <psk-index>
apic1(config-cloudsec)# pskstring <psk-string>

The <psk-index> parameter specifies the order in which the pre-shared keys are used. After the last (highest index) key is used, the process will continue with the first (lowest index) key. Cisco APIC supports up to 256 pre-shared keys, so the PSK index value must be between 1 and 256.

The <psk-string> parameter specifies the actual PSK, which must be a hexadecimal string 64 characters long.

Step 6

(Optional) View the current PSK configuration.

You can view how many PSK are currently configured and their duration using the following command:

Example:

apic1(config-cloudsec)# show cloudsec summary

Configuring Cisco APIC for CloudSec Encryption Using REST API

This section describes how to configure one or more pre-shared keys (PSK) using the Cisco APIC REST API.

Procedure

Configure PSK expiration time, index, and string.

In the following XML POST, replace:

- The value of sakExpiryTime with the expiration time of each PSK.

This sakExpiryTime parameter specifies how long each key is valid (in minutes). Each generated TX/RX key expires after the specified amount of time triggering a re-key process. The expiration time can be between 5 and 1440 minutes.

- The value of index with the index of the PSK you're configuring.

The index parameter specifies the order in which the pre-shared keys are used. After the last (highest index) key is used, the process will continue with the first (lowest index) key. Cisco APIC supports up to 256 pre-shared keys, so the PSK index value must be between 1 and 256.

- The value of pskString with the index of the PSK you're configuring.

The pskString parameter specifies the actual PSK, which must be a hexadecimal string 64 characters long.

Example:

<fvTenant annotation="" descr="" dn="uni/tn-infra" name="infra" nameAlias="" ownerKey="" ownerTag="">
<cloudsecIfPol descr="cloudsecifp" name="default" sakExpiryTime="
10
" stopRekey= "false" status="" >
<cloudsecPreSharedKey index="
1
" pskString="
1234567812345678123456781234567812345678123456781234567812345678
" status=""/>
</cloudsecIfPol>
</fvTenant>

Enabling CloudSec Encryption Using Cisco ACI Multi-Site Orchestrator GUI

The CloudSec encryption can be enabled or disabled for each site individually. However, the communications between two sites will be encrypted only if the feature is enabled on both sites.

Before you begin

Before you enable the CloudSec encryption between two or more sites, you must have completed the following tasks:

- Installed and configured the Cisco APIC clusters in multiple sites, as described in Cisco APIC Installation, Upgrade, and Downgrade Guide

- Installed and configured Cisco ACI Multi-Site Orchestrator, as described in Cisco ACI Multi-Site Orchestrator Installation and Upgrade Guide .

- Added each Cisco APIC site to the Cisco ACI Multi-Site Orchestrator, as described in Cisco ACI Multi-Site Configuration Guide .

Procedure

Step 1

Log in to the Cisco ACI Multi-Site Orchestrator.

Step 2

From the left-hand sidebar, select the Sites view.

Step 3

Click on the Configure Infra button in the top right of the main window.

Step 4

From the left-hand sidebar, select the site for which you want to change the CloudSec configuration.

Step 5

In the right-hand sidebar, toggle the CloudSec Encryption setting to enable or disable the CloudSec Encryption feature for the site.

Rekey Process During Spine Switch Maintenance

The following is a summary of the CloudSec rekey process during typical maintenance scenarios for the spine switches where the feature is enabled:

- Normal Decommissioning – CloudSec rekey process stops automatically whenever a CloudSec-enabled spine switch is decommissioned. Rekey process will not start again until the decommissioned node is commissioned back or the decommissioned node ID is removed from the Cisco APIC

- Spine Switch Software Upgrade – CloudSec rekey process stops automatically if a spine switch is reloaded due to software upgrade. Rekey process will resume after the spine switch comes out of reload.

- Maintenance (GIR mode) – CloudSec rekey process must be manually stopped using the instructions provided in Disabling and Re-Enabling Re-Key Process Using NX-OS Style CLI. Rekey can be enabled back only after the node is ready to forward traffic again.

- Decommissioning and Removal from Cisco APIC – CloudSec rekey process must be manually stopped using the instructions provided in Disabling and Re-Enabling Re-Key Process Using NX-OS Style CLI. Rekey can be enabled back only after the node is removed from Cisco APIC.

- Disabling and Re-Enabling Re-Key Process Using NX-OS Style CLI

- Disabling and Re-Enabling Re-Key Process Using REST API

Disabling and Re-Enabling Re-Key Process Using NX-OS Style CLI

It is possible to manually stop and restart the re-key process. You may be required to manually control the re-key process in certain situations, such as switch decommissioning and maintenance. This section describes how to toggle the setting using Cisco APIC NX-OS Style CLI.

Procedure

Step 1

Log in to the Cisco APIC NX-OS style CLI.

Step 2

Enter configuration mode.

Example:

apic1# configure
apic1(config)#

Step 3

Enter configuration mode for the default CloudSec profile.

Example:

apic1(config)# template cloudsec default
apic1(config-cloudsec)#

Step 4

Stop or restart the re-key process.

To stop the re-key process:

Example:

apic1(config-cloudsec)# stoprekey yes

To restart the re-key process:

Example:

apic1(config-cloudsec)# stoprekey no

Disabling and Re-Enabling Re-Key Process Using REST API

It is possible to manually stop and restart the re-key process. You may be required to manually control the re-key process in certain situations, such as switch decommissioning and maintenance. This section describes how to toggle the setting using Cisco APIC REST API.

Procedure

Step 1

You can disable the rekey process using the following XML message.

Example:

<fvTenant annotation="" descr="" dn="uni/tn-infra" name="infra" nameAlias="" ownerKey="" ownerTag="">
<cloudsecIfPol descr="cloudsecifp" name="default" sakExpiryTime="10" stopRekey= "
true
" status="" />
</fvTenant>

Step 2

You can enable the rekey process using the following XML message.

Example:

<fvTenant annotation="" descr="" dn="uni/tn-infra" name="infra" nameAlias="" ownerKey="" ownerTag="">
<cloudsecIfPol descr="cloudsecifp" name="default" sakExpiryTime="10" stopRekey= "
false
" status="" />
</fvTenant>

Multi-Site Cross Launch to Cisco APIC

Multi-Site currently supports the basic parameters to choose when creating a Tenant and setting up a site. Multi-Site supports most of the Tenant policies, but in addition to that you can configure some advanced parameters.

Use the Multi-Site GUI to manage the basic properties to configure. If you want to configure advanced properties, the capability to cross launch into Cisco APIC GUI directly from the Multi-Site GUI is provided. You can also configure the additional properties directly in Cisco APIC.

There are three different access points in Multi-Site GUI from where you can cross launch into APIC. From these access points in Multi-Site, you can open a new browser tab with access into Cisco APIC. You will log in to Cisco APIC at that point for the first time, and the associated screen is displayed in the Cisco APIC GUI.

- Cross-Launch to Cisco APIC from Sites

- Cross-Launch to Cisco APIC from Schemas

- Cross-Launch to Cisco APIC from the Property Pane

- Viewing Cisco ACI Multi-Site-Managed Objects Using the Cisco APIC GUI

Cross-Launch to Cisco APIC from Sites

Before you begin

- At least one site must be configured in Multi-Site.

- The site must contain at least one tenant with entities such as VRF and bridge domain configured.

Procedure

Step 1

From the left-hand sidebar, open the Sites view.

Step 2

From the Sites list, hover over the name of the appropriate site, click the Actions icon at the end of the row, and choose Open in APIC User Interface to access the Cisco APIC GUI.

The APIC GUI login screen is displayed for logging in with APIC GUI credentials.

Cross-Launch to Cisco APIC from Schemas

Before you begin

- At least one site based on a template must be configured in Multi-Site.

- The site must contain at least one tenant with entities such as VRF and bridge domain configured.

Procedure

Step 1

From the left-hand sidebar, open the Schemas view.

Step 2

From the Schemas list, click the appropriate <schema-name> .

Step 3

From the left-hand sidebar Sites list, hover over the name of the appropriate site, click the Actions icon at the end of the row, and choose Open in APIC User Interface to access the Cisco APIC GUI.

The APIC GUI login screen is displayed for logging in with APIC GUI credentials.

Cross-Launch to Cisco APIC from the Property Pane

Before you begin

- At least one site must be configured in Multi-Site.

- The site must contain at least one tenant with entities such as VRF and bridge domain configured.

Procedure

Step 1

From the left-hand sidebar, open the Schemas view.

Step 2

From the Schemas list, click the appropriate <schema-name> .

Step 3

From the left-hand sidebar Sites list, choose the appropriate site.

Step 4

In the Canvas, choose the name of a specific entity.

For example, choose an available VRF, Contract, Bridge Domain, or another entity as appropriate.

The details for the specific entity are displayed in the Property Pane on the right.

Step 5

In the top right of the Property Pane, click the Open in APIC User Interface icon to access the Cisco APIC GUI.

The APIC GUI login screen is displayed for logging in with APIC GUI credentials.

Viewing Cisco ACI Multi-Site-Managed Objects Using the Cisco APIC GUI

When an APIC cluster is managed by Multi-Site, cloud icons indicate the relationships with other sites.

Viewing Multi-Site-Managed Objects Using the APIC GUI

Before you begin

The APIC cluster/site must be set up to be managed by Cisco ACI Multi-Site.

Procedure

Step 1

To view the relationship of the APIC site with other sites, click the cloud icon at the upper right, next to the settings icons.

In the diagram, hover over the light blue site icon to see the local site details, and hover over the dark blue icon to see the remote site details.

In the image, T1 and its Application Profile, EPG, BD, VRF, and contracts are marked with cloud icons. This indicates that they are managed by Multi-Site. We recommend that you only make changes to these objects in the Multi-Site GUI.

Step 2

To view the localized or stretched usage of a VRF, bridge domain, or other objects, where there is a Show Usage button on the information page, perform the following steps; for example for Bridge Domain and VRF:

1. On the menu bar, click Tenants and double-click on a tenant that is managed by Multi-Site.

2. Click Networking > Bridge Domains > BD-name or Networking > VRFs > vrf-name .

Step 3

Click Show Usage.

Here you can view the nodes or policies using the object.

Note

It is recommended to make changes to managed policies only in the Multi-Site GUI.

Step 4

To set the scope of deployment notification settings for this BD or VRF, click Change Deployment Settings. You can enable warnings to be sent for all deletions and modifications of the object on the Policy tab.

Step 5

To enable or disable Global warnings, check or uncheck the (Global) Show Deployment Warning on Delete/Modify check box.

Step 6

To enable or disable Local warnings, choose Yes or No on the (Local) Show Deployment Warning on Delete/Modify field.

Step 7

To view any past warnings, click the History tab Events or Audit Logs.

Chapter 4. Tenant Management

- Managing Tenants Using the Multi-Site GUI

- Adding Tenants

- Connect Sites Through External EPGs

- Configuring Global Contracts Across Tenants or VRFs

- Configuring Intra-EPG Isolation Using the Multi-Site GUI

- Configuring Microsegmented EPGs Using the Multi-Site GUI

- Associating EPGs with Domains Using the Multi-Site GUI

- Displaying All the Tenants in an Aggregated View

Managing Tenants Using the Multi-Site GUI

Note

To be able to manage tenants in Cisco ACI Multi-Site, the Multi-Site Site and Tenant user account with the Schema Manager role (with complete read/write privileges) must be available.

For the procedures to create a tenant in Multi-Site, see Adding Tenants in the Multi-Site GUI.

The following tenant policies and their associations can be configured in the Multi-Site GUI:

- VRFs

- Bridge Domains with subnets and stretched or localized settings

- Filters and Contracts

- Application Network Profiles with EPGs

- External EPGs to connect sites within a stretched tenant and VRF (but the L3Outs associated with the external EPGs (L3extInstPs) must be configured in APIC)

- Associate EPGs with physical or VMM domains

- Intra-EPG Isolation

- Microsegmented EPGs

- EPGs deployed on a port, PC, or VPC

Other tenant policies, including L3Outs must be configured in the APIC GUI.

After you create a tenant in Multi-Site, there are two ways to add tenant policies:

- Import a fully configured tenant from an APIC site.

- Configure the tenant policies in the Multi-Site GUI.

Adding Tenants

This section describes how to add tenants using the Multi-Site Orchestrator GUI.

Before you begin

You must have a user with either Power User or Site Manager read-write role to create and manage tenants.

Procedure

Step 1

Log in to the Cisco ACI Multi-Site Orchestrator GUI.

Step 2

From the left navigation pane, select Tenants.

Step 3

In the main pane, click Add Tenant.

Step 4

In the Display Name field, provide the tenant's name.

The tenant's Display Name is used throughout the Orchestrator's GUI whenever the tenant is shown. However, due to object naming requirements on the Cisco APIC, any invalid characters are removed and the resulting Internal Name is used when pushing the tenant to sites. The Internal Name that will be used when creating the tenant is displayed below the Display Name textbox.

You can change the Display Name of the tenant at any time, but the Internal Name cannot be changed after the tenant is created.

Step 5

(Optional) In the Description field, enter a description of the tenant.

Step 6

In the Associated Sites section, add the sites.

1. Check all sites where you plan to deploy templates that use this tenant.

Only the selected sites will be available for any templates using this tenant.

2. From the Security Domains drop-down list, choose the site's security domains.

Security domains are created using the Cisco APIC GUI and can be assigned to various Cisco APIC policies and user accounts to control their access. For more information, see the Cisco APIC Basic Configuration Guide.

Step 7

In the Associated Users section, add Orchestrator users.

Only the selected users will be able to use this tenant when creating templates.

Step 8

(Optional) Enable consistency checker scheduler.

You can choose to enable regular consistency checks. For more information about the consistency checker feature, see Cisco ACI Multi-Site Troubleshooting Guide.

Step 9

Click SAVE to finish adding the tenant.

Connect Sites Through External EPGs

In Cisco ACI Multi-Site, Release 1.0(1), site L3Out connections were connected using Network Mappings. In release 1.0(2), use the following process to link the L3Outs for each site.

In this task, you configure an EXTERNAL EPG in a site-specific template for each site, to enable the sites to connect.

Before you begin

- On each site (in Cisco APIC), create an L3Out connection, using the following guidelines:

- The subnet in the L3extInstP must be the same for all inter-related sites (and variable length network masks are not supported).

- The VRF for each L3Out must be the same for all sites. Changing the VRF in APIC, after the external EPGs are deployed, resets the L3Out and requires reconfiguring and redeploying the external EPG for the site.

- When configuring L3Outs for various sites, each L3Out must be dedicated (not shared).

- If you plan to configure GOLF in addition to Multi-Site, they both need separate L3Out policies to the IPN, but they can share a physical interface, as long as the same interface IP address is used.

- In Cisco ACI Multi-Site, create a tenant to be stretched to the sites.

- Create a schema and import or create the VRF, Bridge Domain, Filters and Contracts, and an Application Profile with EPGs.

- In a shared template, deploy the tenant and policies to multiple sites.

Procedure

Step 1

In the same schema you used to deploy a stretched tenant and VRF, start a site-specific template using the following steps, or if the schema already contains site-specific templates, click on a site-specific template and continue at the next step.

1. Click + on the TEMPLATES field to start a new template.

2. Click the edit button and enter the template name.

3. Click + on the TENANT field to start configuring the schema, and choose the stretched tenant from the list.

4. Click + on the SITES field and choose the site for the external EPG.

Step 2

Click + to create an EXTERNAL EPG.

Step 3

Enter the external EPG name.

Step 4

Add the contracts required for the external EPGs to communicate.

Note

If you are associating a contract with the external EPG, as provider, choose contracts only from the tenant associated with the external EPG. Do not choose contracts from other tenants.

If you are associating the contract to the external EPG, as consumer, you can choose any available contract.

Step 5

Click the site-specific template.

Step 6

Click the external EPG.

Step 7

In the external EPG details pane, L3OUT field, choose the L3Out on the site to be used for the external EPG.

Step 8

Optional. Add a subnet for the external EPG, using the following steps:

This is the same as adding a subnet under the L3extInstP in the Cisco APIC GUI.

1. Click + to add a SUBNET.

2. Enter the IP address and subnet mask in the GATEWAY IP field.

3. Click SAVE.

Step 9

Optional. Associate the L3Out with a BD, using the following steps:

1. In the site-specific template, click a BD.

2. In the BD details pane, ckuck + to add an L3Out and choose the L3Out to associate.

3. Click SAVE.

Step 10

Click DEPLOY TO SITES.

Step 11

Confirm the policies to be deployed and click DEPLOY.

What to do next

Repeat these steps for the other sites to be connected (where the tenant and VRF are stretched).

Configuring Global Contracts Across Tenants or VRFs

This use case is for a data center that provides services to EPGs in other tenants or VRFs. It provides contracts that enable all the EPGs to consume the services.

For more information, see the Shared Services with Stretched Provider EPG use case in the Cisco ACI Multi-Site Fundamentals Guide.

Before you begin

Create a schema (for every site that provides and consumes the services) with Tenants, VRFs, bridge domains, application profiles, EPGs, and other contracts.

The tenants, VRFs, BDs, and EPGs do not have to be stretched across the sites.

Procedure

Step 1

Open the provider schema.

Step 2

Create a filter (essentially an Access Control List) with the following steps:

1. Click the + icon to add a filter.

2. Enter the filter name.

3. Click the + icon to add an entry.

4. Enter the entry name.

5. Enter the rest of the data required for the filter and click Save.

Step 3

Create a contract with the following steps:

1. Click the + icon to add a contract.

2. Enter the contract name.

3. Change the contract scope to global.

This enables the contract to be accessible to EPGs in multiple VRFs.

4. Click the + icon to add a filter and choose the filter you created.

5. Click Save.

Step 4

Associate the EPG that provides the services with the contract, with the following actions:

1. Click the EPG.

2. Click the + icon to add a contract.

3. Choose the global contract you previously created.

4. Set the type to provider.

5. Click Save.

6. Click DEPLOY TO SITES.Confirm the sites and click DEPLOY.

Step 5

Associate EPGs with the contract as consumers, with the following actions:

1. Open each consumer schema.

2. Click an EPG.

3. Click the + icon to add a contract.

4. In the Contract field, start typing the contract name. When the contract appears in the list, choose it.

5. Set the type to consumer.

6. Click Save.

7. Associate the contract to any other EPGs in the schema.

8. Click DEPLOY TO SITES.

9. Confirm the sites and click DEPLOY.

Configuring Intra-EPG Isolation Using the Multi-Site GUI

Intra-EPG isolation is allowed between endpoints in an EPG that is operating with isolation enforced. Isolation enforced EPGs reduce the number of EPG encapsulations required when many clients access a common service but are not allowed to communicate with each other. An EPG is isolation enforced for all ACI network domains or none. While the ACI fabric implements isolation directly to connected endpoints, switches connected to the fabric are made aware of isolation rules according to a primary VLAN (PVLAN) tag.

If an EPG is configured with intra-EPG endpoint isolation enforced, these restrictions apply:

- All Layer 2 endpoint communication across an isolation-enforced EPG is dropped within a bridge domain.

- All Layer 3 endpoint communication across an isolation-enforced EPG is dropped within the same subnet.

- Preserving QoS CoS priority settings is not supported when traffic is flowing from an EPG with isolation-enforced to an EPG without isolation enforced.

- In Multi-Site, intra-EPG isolation is not supported in AVS-VLAN mode and DVS-VXLAN mode. Setting Intra-EPG isolation to be enforced may cause the ports to go into a blocked state in these domains.

- Intra-EPG isolation is not supported if the Bridge Domain is configured as "legacy BD mode".

Before you begin

- Create the tenant associated with the EPGs.

- Import the tenant policies or configure a schema containing the tenant's VRF, bridge domain, and the Application Network Profile containing the EPGs that will be subject to intra-EPG isolation.

Procedure

Step 1

Open the schema and template where the EPGs to be isolated are configured.

Step 2

Click an EPG.

Step 3

Choose Enforced, read the warning, and click OK.

Step 4

Optional. Configure other EPGs to be isolation-enforced.

Step 5

Push the template containing the EPGs (configured for intra-EPG isolation) to the site where they will be located.

Step 6

Click the deployed site and template and click an EPG.

Step 7

Click ADD STATIC PORT.

Step 8

Choose the PATH TYPE (Port, Direct Port Channel, or Virtual Port Channel).

Step 9

Choose the LEAF.

Step 10

Choose the PATH.

Step 11

In the PORT ENCAP VLAN field, enter the VLAN number to be used for traffic for the EPG.

Step 12

On the DEPLOYMENT IMMEDIACY field, choose OnDemand or Immediate deployment.

Step 13

On the MODE field, choose Trunk.

Step 14

Optional, repeat the steps for other EPGs that will have isolation enforced.

What to do next

Push the changes to the site where the EPGs are located.

Configuring Microsegmented EPGs Using the Multi-Site GUI

You can use Cisco ACI Multi-Site to configure Microsegmentation to create an attribute-based EPG using a network-based attribute (IP, MAC, DNS) or VM-based attributes (VM ID, VM Name, VMM domain, and so forth). This enables you to isolate VMs or physical endpoints within a single base EPG or VMs or physical endpoints in different EPGs.

Only the basic options for microsegmented (uSeg) EPGs can be configured in Cisco ACI Multi-Site. For procedures for advanced options and for use cases and detailed information about Microsegmented EPGs, see the Microsegmentation with Cisco ACI chapter in Cisco ACI Virtualization Guide, Release 3.0.

Note

When creating an EPG, if you first create an application EPG and want to change it to a uSeg EPG, you must either assign the EPG a different name or remove the application EPG and add the uSeg EPG, with the following process:

1. Delete the application EPG from the schema.

2. Deploy the schema to the sites.

3. Create the uSeg EPG.

4. Redeploy the schema to the sites.

To configure a microsegmented EPG using Cisco ACI Multi-Site, perform the following steps:

Before you begin

- Create the tenant associated with the EPGs that will be microsegmented.

- Import the tenant policies or configure a schema containing the tenant's VRF, bridge domain, and the Application Network Profile containing the EPGs.

- Create at least one application EPG in the tenant.

Procedure

Step 1

Open the schema where the EPGs are configured.

Step 2

Click an EPG.

Step 3

Click USEG EPG.

Step 4

Click ADD USEG ATTRIBUTES.

Step 5

On the DISPLAY NAME field, enter the name for the attribute.

Step 6

Choose the ATTRIBUTE TYPE; it can be one of the following:

- IP

- Mac

- DNS

- VM Name

- VM Data Center

- VM Hypervisor Identifier

- VM Operating System

- VM Tag

- VM Identifier

- VM VMM Domain

- VM VNIC DN (vNIC domain name

Step 7

Save your changes.

What to do next

Associate the USeg EPG with a domain using the Multi-Site GUI.

Associating EPGs with Domains Using the Multi-Site GUI

Before you begin

- Create the tenant associated with the EPGs in Cisco ACI Multi-Site.

- Create the domain profiles (VMM, L2, L3, or Fibre Channel) in APIC.

- Import the tenant policies from Cisco APIC or configure a schema (with template) in Multi-Site, that contains the tenant's VRF, bridge domain, and the Application Network Profile containing the EPGs that will be associated with a domain.

Associate the template with a site.

Procedure

Step 1

In the Sites list, click the site and template for the site where the EPG and domain are configured, and click the EPG.

Step 2

Click ADD DOMAINS.

Step 3

On the DOMAIN ASSOCIATION TYPE field, choose the type, which can be:

- VMM

- Fibre Channel

- L2 External

- L3 External

- Physical

Step 4

On the DOMAIN PROFILE field, choose a previously created profile or phys.

Step 5

On the DEPLOYMENT IMMEDIACY field, choose OnDemand or Immediate.

Step 6

On the RESOLUTION IMMEDIACY field, choose OnDemand, Immediate, or Pre-Provision.

Step 7

Save your changes.

What to do next

Push the template containing the changes to the site.

Displaying All the Tenants in an Aggregated View

Using the Multi-Site GUI Tenants tab, you can view the aggregated list of the tenants.

In the Tenants panel under the Tenants tab, the following fields are displayed in the GUI:

- NAME: Name of the tenant.

- DESCRIPTION: Description of each tenant.

- ASSIGNED TO SITES: The number of the sites that the tenant is assigned to.

- ASSIGNED TO USERS: The number of the users that the tenant is assigned to.

- ASSIGNED TO SCHEMAS: The number of the schemas that the tenant is assigned to.

- ACTIONS: Perform actions for each tenant, for example, Edit, Delete, or configure Network Mappings for the tenant.

Based on the Tenants chart, you can determine the resource utilization of the tenants.

Chapter 5. User Management

- User and Roles

- User Roles and Features

- Guidelines and Limitations

- Creating a User

- Managing Users

User and Roles

The Cisco ACI Multi-Site provides access according to a user’s role through role-based access control (RBAC). Roles are used for both local and external authentication. The following user roles are available in Cisco ACI Multi-Site.

- Power User—A power user can perform all the operations as an admin user.

- Site and Tenant Manager—A site and tenant manager can manage sites, tenants, and associations.

- Schema Manager—A schema manager can manage all schemas regardless of tenant associations.

- Schema Manager - Restricted —A restricted schema manager can manage schemas that contain at least one tenant to which the user is explicitly associated.

- User and Role Manager—A user and role manager can manage all the users, their roles, and passwords.

Admin User

In the initial configuration script, the admin account is configured and the admin is the only user when the system starts. The initial password for the admin user is set by the system. You must change the admin password during the first log in.

- The admin user is assigned the role of a Power User.

- Use the admin user to creating other users and perform all other Day-0 configurations.

- The account status of the admin user cannot be set to Inactive.

User Roles and Features

The following table lists the Cisco ACI Multi-Site features available with a user role.

User Role

Multi-Site Features

Multi-Site av pair

Power User

- Dashboard

- Sites

- Schemas

- Tenants

- Users

- Troubleshooting Reports

shell:msc-roles=powerUser

Site and Tenant Manager

- Dashboard—Sites

- Sites

- Tenants

shell:msc-roles=siteManager

Schema Manager

- Dashboard—Sites and Schema Health

- Schemas

shell:msc-roles=schemaManager

Schema Manager - Restricted

- Dashboard—Sites and Schema Health

- Schemas

shell:msc-roles=schemaEditor

User and Role Manager

- Users

shell:msc-roles=userManager

Guidelines and Limitations

- Users authentication and authorization can be local or external using RADIUS, TACACS+, or LDAP servers. For more information about external authentication, see External Authentication.

- For both local and external authentication, you must associate at least one role with every user. A user may be associated with more than one role. Associating a user to multiple roles offers a combination of features that the user may access.

- Users must be associated with tenants before they can use a tenant on a schema.

Creating a User

Procedure

Step 1

Log in to Cisco ACI Multi-Site.

Step 2

In the Main menu, click Users.

Step 3

Click ADD USER.

Step 4

In the ADD USER page, perform the following actions:

1. In the USERNAME field, enter the user name.

2. In the PASSWORD field, password.

The password must at least be six characters in length, and must contain at least one letter, one number, and a special character. Spaces and * are not allowed.

3. In the CONFIRM PASSWORD field, re-enter the password.

4. In the FIRST NAME field, enter the first name of the user.

5. In the LAST NAME field, enter the last name of the user.

6. In the EMAIL ADDRESS field, enter the email address of the user.

7. In the PHONE NUMBERfield, enter the phone number of the user.

8. In the ACCOUNT STATUS field, choose the account status.

Only Active users are authenticated by Multi-Site .

Step 5

Click the User Role button, to assign a role to a user.

You must associate at least one role with every user. A user may be associated with more than one role. Associating a user to multiple roles offers a combination of features that the user may access. See User and Roles for more information.

Step 6

Click Submit.

Managing Users

Procedure

Step 1

Cisco ACI Multi-Site.

Step 2

In the Main menu, click Users.

Step 3

Select a user and click Actions to perform the following actions.

1. From the Actions menu, choose Delete to delete a user.

You cannot delete an admin user.

2. From the Actions menu, choose Edit to edit a user.

An admin’s user name, account status, and roles cannot be updated.

Step 4

To update the password of a user, click Welcome username.

An admin user or a user associated with the user role Power User or User and Role Manager can update the password of an end user. On initial log in, an end user must update their password.

Chapter 6. Schema Management

- Schema Design Considerations

- Creating a Schema Template

- Contract Preferred Groups

- Layer 3 Multicast

- Shadow EPGs and BDs

Schema Design Considerations

A schema is a collection of templates, which are used for defining policies. This release of Cisco ACI Multi-Site supports up to 60 schemas with 5 templates and 500 objects per schema. It is important to consider schema design approach based on the deployment use-case.

Single Schema Deployment

The simplest schema design approach is a single schema deployment. You can create a single schema and adds all VRFs, Bridge Domains, EPGs, Contracts and other elements to it. You can then create a single application profile or multiple application profiles within the schema and deploy it to one or more sites.

Single Schema

This simple approach to Multi-Site schema creation is illustrated in the figure above and allows for all objects to be readily visible within the same schema. However, the 5 templates and 500 objects per schema limit makes this approach unsuitable for large scale deployments, which could exceed those limits.

Multiple Schemas with Network Separation

Another approach to schema design is to separate the networking objects from the application policy configuration. Networking objects include VRFs, Bridge Domains, and subnets, while the application policy objects include EPGs, Contracts, Filters, External EPGs, and Service Graphs.

You begin by defining a schema that contains the network elements. You can choose to create a single schema that contains all the network elements or you can split them into multiple schemas based on which applications reference them or which sites the network is stretched to.

The following figure shows a single networking template configuration with VRF, BD, and subnets configured and deployed to two sites:

Network Schema

You can then define one or more separate schemas which contain each application's policy objects. This new schema can reference the network elements, such as bridge domains, defined in the previous schema. The following figure shows a policy schema that contains two application templates both of which reference the networking elements in an external schema. One of the applications is local to one site while the other is stretched across two sites:

Policy Schema

After creating and deploying the policy schemas and templates, the networking objects in the networking schema will display the number of external references by the policy schema elements. The object with external references will also be denoted by the ribbon icon as shown in the Network Schema figure above.

Schemas designed this way provide logical separation of networking objects from the policy objects. However, this creates additional complexity when it comes to keeping track of externally referenced objects in each schema.

Multiple Schemas Based On Object Relationships

When configuring multiple schemas with shared object references, it is important to be careful when making changes to those objects. For instance, making changes to or deleting a shared networking object can impact applications in one or more sites. Because of that, you may choose to create a template around each individual site that contains only the objects used by that site and its applications, including the VRFs, BDs, EPGs, Contracts, and Filters. And create different templates containing the shared objects.

One Template per Site

The site1 template in the above figure contains only the objects that are local to Site1 and the template is deployed to only the Miami site. Similarly, the site2 template contains only the object relevant to site2 and is deployed to the San Francisco site. Any change made to any object in either of these templates has no effect on the other one. The shared template contains any objects that are shared between the sites.

You can extend this scenario for an additional site with the following template layout:

- Site 1 template

- Site 2 template

- Site 3 template

- Site 1 and 2 shared template

- Site 1 and 3 shared template

- Site 2 and 3 shared template

- All shared template

Similarly, rather than separating objects based on which site they are deployed to, you can also choose to create schemas and templates based on individual applications instead. This would allow you to easily identify each application profile and map them to schemas and sites as well as easily configure each application as local or stretched across sites.

However, as this would exceed the 5 templates per schema limit, you would have to create additional schemas to accommodate the multiple combinations. While this creates additional complexity with multiple additional schemas and templates, it provides true separation of objects based on site or application.

Creating a Schema Template

Before you begin

- You must have an administrative user account with full read/write privileges.

- You must have a Cisco APIC tenant user account with read/write tenant policy privileges.

For more information, see the User Access, Authentication, and Accounting chapter in the Cisco APIC Basic Configuration Guide.

- You must have at least one available tenant that you want to incorporate into your site.

For more information, refer to Adding Tenants.

Procedure

Step 1

On the Schema page, click the Add Schema button.

Step 2

On the Untitled Schema page, enter a name for the schema you intend to create.

Step 3

Access the Select A Tenant dialog box and select a tenant from the drop-down menu.

Note

Keep in mind, the user account you're using to create a new schema must be associated with the tenant you are trying to add to it, otherwise the tenant will not be available in the drop-down menu. Associating a user account with a tenant is described in Adding Tenants.

- Importing Schema Elements From APIC Sites

- Configuring an Application Profile

- Configuring a Contract

- Configuring a Bridge Domain

- Configuring a VRF for the Tenant

- Configuring a Filter for Contracts

- Configuring an External EPG

- Viewing Schemas

Importing Schema Elements From APIC Sites

You can create new objects and push them out to one or more sites or you can import existing site-local objects and manage them using the Multi-Site Orchestrator. This section describes how to import one or more existing objects, while creating new objects is described later on in this document.

Procedure

Step 1

On the Schema page, click the Import button.

Step 2

Select the site from which you want to import objects.

Step 3

In the Import window that opens, select one or more objects you want to import.

Note

The names of the objects imported into the Multi-Site Orchestrator must be unique across all sites. Importing different objects with duplicate names will cause a schema validation error and the import to fail. If you want to import objects that have the same name, you must first rename them.

Configuring an Application Profile

Describes how to configure an Application Profile with EPGs.

Procedure

Step 1

In the AP field, click + Application Profile.

Step 2

Enter the AP name.

Step 3

Click + Add EPG and enter the EPG name in the Display Name field.

Step 4

Optional. Click + Subnet to add a subnet to your EPG, if appropriate.

You may choose to configure a subnet for the EPG, for example for a VRF route-leaking use-case.

1. On the Add Subnet dialog, enter the Gateway IP address and a description for the subnet you plan to add.

2. In the Scope field select either Private to VRF or Advertised Externally.

3. Click the check box for Shared Between VRFs if appropriate.

4. Click the check box for No Default SVI Gateway if appropriate.

5. Click OK.

Step 5

Optional. Select USEG EPG if appropriate.

1. Enter the Name and Type for the USEG ATTR.

2. Click + USEG ATTRIBUTE to add USeg attributes if appropriate.

3. On the Add uSeg Attributes dialog, enter a Display Name, Description, and Attribute Type.

4. Click SAVE.

Step 6

Select either Enforced or Unenforced for the Intra EPG Isolation field.

Step 7

Choose a bridge domain in the Bridge Domain field.

Step 8

Click + Contract to add a contract if appropriate.

1. On the Add Contract dialog, enter the contract name and type.

2. Click SAVE.

Configuring a Contract

Provides the procedure for configuring contracts to control EPG communications.

Procedure

Step 1

Click + in the box in the Contract pane.

Step 2

Enter a name for the contract in the Contract dialog in the Display Name field under Display Name.

Step 3

Choose a value for Scope using the drop-down menu.

Step 4

Click Apply Both Directions toggle button to apply the filter specified in the contract to either one direction or both directions.

The default setting is ON.

Step 5

Click + Filter.

1. On the Add Filter Chain dialog, click the Name field to choose or find a filter.

2. Optional. Select the available directives in the Directives field.

3. Click SAVE.

What to do next

After you have configured the contract to your specifications, click Deploy to Sites.

Configuring a Bridge Domain

Procedure

Step 1

Click + in the Bridge Domain pane to add a new bridge domain.

Step 2

In the right-hand properties sidebar that opens, provide the following bridge domain details:

- The BD name in the Display Name field.

- The VRF in the Virtual Routing and Forwarding field.

- If appropriate, check the L2 STRETCH checkbox.

- If you enabled L2 STRETCH, you can choose to also enable INTERSITE BUM TRAFFIC ALLOW checkbox.

- If you did not enable L2 STRETCH, you can choose either proxy or flood for the L2 UNKNOWN UNICAST field

Step 3

(Optional) You can choose to add one or more subnets to the bridge domain.

1. Click + Subnet.

An Add Subnet window opens.

2. Enter the subnet's Gateway IP address and a description for the subnet you want to add.

3. In the Scope field, select either Private to VRF or Advertised Externally.

4. If appropriate, check the Shared Between VRFs checkbox.

5. If appropriate, check the No Default SVI Gateway checkbox.

6. If appropriate, check the Querier checkbox.

7. Click SAVE.

Configuring a VRF for the Tenant

Procedure

Step 1

Click the + in the VRF field.

Step 2

Enter a display name for the VRF in the Display Name field.

Configuring a Filter for Contracts

Provides a method to create a filter. A filter is similar to an Access Control List (ACL), used to filter traffic through contracts associated to EPGs.

Procedure

Step 1

Click the + on the Filter object under in the Filter pane.

Step 2

Enter a display name in the Display Name field.

Step 3

Click + Entry to provide information for your schema filter on the Add Entry display:

1. Enter a name for the schema filter entry in the Name field on the Add Entry dialog.

2. Optional. Enter a description for the filter in the Description field.

3. Enter the details as appropriate to filter EPG communication.

For example, to add an entry allowing HTTPS traffic through a filter, choose TYPE: IP, IP PROTOCOL: TCP, and DESTINATION PORT RANGE FROM and DESTINATION PORT RANGE TO: https.

4. Click SAVE.

Configuring an External EPG

In this schema field, you configure an EXTERNAL EPG for each site, to enable the sites to connect. Alternatively, you may want to configure a single External EPG in a template that is in turn associated to multiple sites.

To configure an external EPG for each site, in a site-specific template perform the following steps:

Before you begin

- Create an L3Out in Cisco APIC on all sites where the tenant and VRF are stretched.

- The VRF for each L3Out must be the same for all sites. Changing the VRF in APIC, after the external EPGs are deployed, resets the L3Out and requires reconfiguring and redeploying the external EPG for the site.

Procedure

Step 1

Click + to create an EXTERNAL EPG.

Step 2

Enter the external EPG name.

Step 3

Add the contracts required for the external EPGs to communicate.

Note

If you are associating a contract with the external EPG, as provider, choose contracts only from the tenant associated with the external EPG. Do not choose contracts from other tenants.

If you are associating the contract to the external EPG, as consumer, you can choose any available contract.

Step 4

Click the site-specific template.

Step 5

Click the external EPG.

Step 6

In the external EPG details pane, L3OUT field, choose the L3Out on the site to be used for the external EPG.

What to do next

Optional. You can also add a subnet under the external EPG.

Repeat these steps to create an external EPG for each site.

Viewing Schemas

After you have created one or more schemas, they are displayed both on the Dashboard and the Schemas page.

You can use the functionality available on these two pages to monitor the usage and the health of your schemas when they are deployed. You can also access and edit specific areas of the implemented schema policies using the Multi-Site Orchestrator GUI.

For more information about the functionality of these Multi-Site Orchestrator GUI pages, refer to Overview of the Cisco ACI Multi-Site Orchestrator GUI.

Contract Preferred Groups

Before Release 2.0(2), the Cisco ACI Multi-Site architecture supported communication between EPGs only if a contract was configured between them. If no contract existed, any inter-EPG communication was explicitly disabled by default. Starting with Release 2.0(2), you can include several EPGs in a contract Preferred Group which allows full communication between the EPGs in a single VRF.

Preferred Group Vs Contracts

There are two types of policy enforcements available for EPGs in a VRF which is stretched to multiple sites with a contract preferred group configured:

- Included EPGs – Any EPG that is a member of a preferred group can freely communicate with all other EPGs in the group without any contracts. The communication is based on the source-any-destination-any-permit default rule and appropriate Multi-Site translations.

- Excluded EPGs – EPGs that are not members of preferred groups continue to require contracts to communicate with each other. Otherwise, the default source-any-destination-any-deny rule applies.

The contract preferred group feature allows for greater control and ease of configuration of communication between EPGs across sites in a stretched VRF context. If two or more EPGs in the stretched VRF require open communication while others must have only limited communication, you can configure a combination of a contract preferred group and contracts with filters to control the inter-EPG communication. EPGs that are excluded from the preferred group can only communicate with other EPGs if there is a contract in place to override the source-any-destination-any-deny default rule.

Stretched Vs Shadowed

If EPGs from multiple sites are configured to be part of the same contract preferred group, the Multi-Site Orchestrator creates shadows of each site's EPGs in the other sites in order to correctly translate and program the inter-site connectivity from the EPGs. Contract preferred group policy construct is then applied in each site between a real and shadow EPG for inter-EPG communication.

For example, consider a web-service EPG1 in Site1 and an app-service EPG2 in Site2 added to the contract preferred group. Then if EPG1 wants to access EPG2, it will first be translated to a shadow EPG1 in Site2 and then be able to communicate with EPG2 using the contract preferred group. Appropriate BDs are also stretched or shadowed if the EPG under it is part of a contract preferred group.

Limitations

Inter-site external EPGs are not supported as part of inter-site preferred groups from the Multi-Site Orchestrator. If you want to include a stretched external EPG in a preferred group, you must do so in each site's APIC individually after the external EPG is deployed from the Orchestrator.

- Configuring EPGs for Preferred Group

Configuring EPGs for Preferred Group

Before you begin

You must have one or more EPGs added to a schema template.

Procedure

Step 1

Log in to your Cisco ACI Multi-Site Orchestrator.

Step 2

From the left navigation pane, select the Schemas view.

Step 3

Click the Schema that you want to change.

Step 4

Configure one or more EPGs in the schema to be part of the preferred group.

Note

If you have an existing preferred group in any of the APICs and are planning to import the EPGs from that preferred group into Multi-Site Orchestrator, you must import all EPGs in the group. You must not have a preferred group where some EPGs are managed by the Multi-Site Orchestrator and some are managed by the local APIC.

To add or remove a single EPG:

1. Select an EPG.

2. In the right properties bar, check or uncheck the Include in Preferred Group checkbox.

3. Click SAVE in the top right corner of the main window.

To add or remove multiple EPGs at once:

1. Click SELECT in the top-right corner of the Application Profile tab.

2. Select one or more EPGs by clicking on each one or click Select All to select all EPGs.

3. Click ... in the top-right corner of the Application Profile tab and choose Add EPGs to Preferred Group or Remove EPGs from Preferred Group.

4. Click SAVE in the top right corner of the main window.

What to do next

You can view the full list of EPGs that are configured to be part of the preferred group by selecting a VRF and checking the PREFERRED GROUP EPGS list in the properties sidebar on the right.

Layer 3 Multicast

Note

Layer 3 Multicast across sites is a limited availability feature in Multi-Site 2.0(1). If you plan to enable this feature in your production environment, please consult Cisco for deployment planning and validation.

Cisco Multi-Site Layer 3 multicast is enabled or disabled at three levels, the VRF, the bridge domain (BD), and any EPGs that have multicast sources present.

At the top level, multicast routing must be enabled on the VRF that has any multicast-enabled BDs. On a multicast-enabled VRF, there can be a combination of multicast-enabled BDs and BDs where multicast routing is disabled. Enabling multicast routing on a VRF from the Cisco Multi-Site Orchestrator GUI enables it on the APIC sites where the VRF is stretched.

Once a VRF is enabled for multicast, the individual BDs under that VRF can be enabled for multicast routing. Configuring Layer 3 multicast on a BD enables protocol independent routing (PIM) on that BD. By default, multicast is disabled in all BDs.

When an EPG sends multicast traffic to a remote site where it is not stretched, the MSC creates a shadow EPG on the remote site for each such EPG. This could potentially result in an increased amount of configuration changes, such as subnet routes, being pushed to the remote TORs. To alleviate this, Layer 3 multicast has to also be enabled on the individual EPGs which have multicast sources present, so that the configuration necessary for only those EPGs is pushed to the remote sites. EPGs with multicast receivers do not require enabling Layer 3 multicast.

Multi-Site supports all of the following Layer 3 multicast source and receiver combinations:

- Multicast sources and receivers inside ACI fabric

- Multicast sources and receivers outside ACI fabric

- Multicast sources inside ACI fabric with external receivers

- Multicast receivers inside ACI fabric with external sources

The following is a high level overview of the Layer 3 multicast routing across sites:

- When the multicast source is attached to ACI fabric as End Point (EP) at one site, that site's spine switch will send the multicast traffic to other sites where the source's VRF is instantiated using the Head End Replication (HREP). The multicast traffic will be sent over to other sites where VRF is stretched and multicast traffic will be pruned/forwarded at egress leaf switches based on the group membership.

- The multicast routing solution requires external multicast router to be the Rendezvous Point (RP). Each site must point to the same RP address for a given stretched VRF. The RP must be reachable on each site via the site's local L3Out.

- When the source is outside and the receiver is within a fabric, the receiver will pull traffic via site's local L3Out as PIM joins toward RP and source are always sent via site local L3Out.

- Receivers in each site are expected to draw traffic from source outside the fabric via the site's local L3Out. As such, traffic coming in on L3Out on one site should not be sent to other sites. This is achieved on the spine by pruning multicast traffic from replicating into HREP tunnels.

- All multicast traffic ingressing a TOR's L3out bridge domain from external router is remarked with a special DCSP value in the outer VXLAN header. On the Spine, that DSCP value is matched to prune all multicast traffic from replicating HREP copies into the ISN network

- Traffic sent from one site can be sent out of any site’s L3Out.

- When multicast is enabled on a BD and an EPG from the Multi-Site Orchestrator, all of the BD's subnets are injected into all leaf switches, including the border leaf (BL). This enable receivers attached to the leaf switches to determine the reachability of the multicast source in cases where the source BD is not present on the leaf switches. The subnet is advertised if there is a policy configured on the BL. The /32 host routes are advertised if host-based routing is configured on the BD. The BD's subnets and host routes are advertised if the L3Out policy allows a large subnet range including 0/0 and multicast is enabled on the EPG.

For additional information about multicast routing, see the IP Multicast section of the Cisco APIC Layer 3 Networking Configuration Guide.

- Enabling Layer 3 Multicast

Enabling Layer 3 Multicast

The following procedure describes how to enable Layer 3 multicast on VRF, BD, and EPG using the Cisco ACI Multi-Site Orchestrator GUI.

Before you begin

Cisco ACI Multi-Site Orchestrator cannot create the required local policies on each site, as such you must configure IGMP related policies, PIM related policies, route-maps, RPs, and L3Outs on each APIC site individually for end-to-end solution to work.

For specific information on how to configure those settings on each site, see the Cisco APIC Layer 3 Networking Configuration Guide .

Procedure

Step 1

Log in to your Cisco ACI Multi-Site Orchestrator.

Step 2

From the left-hand sidebar, select the Schemas view.

Step 3

Click on the Schema you want to change.

Step 4

Enable Layer 3 multicast on a VRF.

First, you enable Layer 3 multicast on a VRF that is stretched between sites.

1. Select the VRF for which you want to enable Layer 3 multicast.

2. In the right-hand sidebar, check the L3 Multicast checkbox.

Step 5

Enable Layer 3 multicast on a BD.

Once you have enabled L3 Multicast on a VRF, you can enable L3 multicast on a Bridge Domain (BD) level.

1. Select the BD for which you want to enable Layer 3 multicast.

2. In the right-hand sidebar, check the L3 Multicast checkbox.

Step 6

Enable Layer 3 multicast on an EPG.

Once you have enabled L3 Multicast on the BD, you can select EPGs which have multicast sources. You can only do that if the EPG is part of multicast-enabled BD and VRF.

1. Select the EPG for which you want to enable Layer 3 multicast.

2. In the right-hand sidebar, check the L3 Multicast checkbox.

Shadow EPGs and BDs

When a contract exists between site-local EPGs in stretched VRF or in Shared Services use-cases where provider and consumer are in different VRFs and communicate through Tenant contracts, the EPGs and bridge domains (BDs) are mirrored on the remote sites. These mirrored objects appear as if they are deployed in each of these sites' APICs, while only actually being deployed in one of the sites. These mirrored objects are called "shadow" EPGs or BDs.

For example, if the provider site group tenant and VRF are stretched across Site 1 and Site 2, and the consumer site group tenant and VRF are stretched across Site 3 and Site 4, in the APIC GUI at Site 1, Site 2, Site 3, and Site 4, you can see both tenants and their policies. They appear with the same names as the ones that were deployed directly to each site.

You can distinguish these shadow EPGs and BDs in the APIC GUI as described below:

Note

Shadow objects should not be removed using the APIC GUI.

Procedure

Step 1

To identify a shadow EPG in a pair of EPGs with the same name, in the APIC GUI, navigate to Tenants > tenant-name > Application Profiles > ap-name > Application EPGs > epg-name > Static Ports.

A shadow EPG has no path to the static port.

Step 2

To identify a shadow BD from a pair of BDs with the same name, in the APIC GUI, navigate to Tenants > tenant-name > Networking > Bridge Domains > bd-name > Subnets > subnet-name .

The subnet for a shadow BD has No Default SVI Gateway enabled.

Chapter 7. Administrative Operations

- Viewing Site Status

- Viewing Schema Health

- Viewing Faults for Individual Sites

- Logs

- Configuration Backups

- Configuring Custom SSL Certificates

- External Authentication

- Managing Scope of Schema and Template Deployment

Viewing Site Status

You can use the Multi-Site Orchestrator GUI's Dashboard view to see each site's status, number and types of faults, and schema health.

In the SITE STATUS panel, the following fields are displayed on the dashboard:

- SITE NAME

- CRITICAL Alarms

- MAJOR Alarms

- MINOR Alarms

- WARNING Alarms

Viewing Schema Health

Using the schema health functionality in the Multi-Site Orchestrator GUI dashboard, you can view the health of the individual schemas that are associated with different sites. In the Schema Details window, you can view the policy types that are associated with each site.

You can perform the following tasks using the SCHEMA HEALTH chart in the GUI:

- View the aggregated health score of the entire Multi-Site fabric and all APICs

- View the aggregated fault counts and the fault types for each schema in the Schema Details window

- View the health of the inter-site schemas

- View the health of the multi-sites nodes and their components

- View the health of the connected APICs and ACI clusters

You can view the schema health in the GUI using the following different formats:

- Hovering on an Individual Cell: Each cell in the SCHEMA HEALTH chart represents the health of the schema. If the cell is color coded as Green and if you hover on the cell, it displays the application health score of the schema.

- Clicking in the Cell: If you click the individual cell in the table, it provides the additional schema details for the template and the faults with the associated with each policy type, for example, ANP, EPG, Contract, VRF, and BD.

The faults and warnings are displayed in the columns to the right side of each policy. This functionality is used to collect the details and get more information on the issues causing low health.

- Viewing the Health Score Slider: The health score slider at the top of the page provides capabilities to filter the schemas by the minimum or maximum health score. A range in the slider can be adjusted to view the schemas that match the heath score range. For example, you can adjust the health score to display the schemas matching the health score between 0 to 30 range.

- Using the Search Functionality: The search functionality in the schema health view provides the capabilities to find a schema or a policy based on the keywords that are typed in the search area. When the keywords are typed in the search area, only schemas that contain the keywords are displayed. The results are based on the matching keywords as part of the schema name, template name, or any of the contained policies within that schema.

Viewing Faults for Individual Sites

This section describes how to display the faults for the individual sites using the Multi-Site GUI.

Procedure

Step 1

Log in to Multi-Site Orchestrator GUI.

Step 2

In the Main Menu, click Sites.

Step 3

In the Sites list page, click CONFIGURE INFRA.

Step 4

In the Fabric Connectivity Infra page, click the appropriate site in the Master List. For example, click site1.

The site details with the associated pods and the spines are displayed in the GUI.

The total number of the faults and the fault types, for example, Critical, Major, Minor, and Warning faults are displayed at the top of the panel. Clicking on each fault type displays the fault details with the individual codes and their explanations.

Logs

You can view the Multi-Site Orchestrator logs by selecting Admin > Logs from the main navigation menu.

From the Logs page, you can click the Most Recent field to select a specific time period for which you want to see the logs. For example, when you select the range from November 14, 2017 to November 17, 2017 and click Apply, the audit log details for this time period are displayed on the Audit Logs page.

You can also click the Filter icon to filter the log details using the following criteria:

- User: Select this option to filter the audit logs by the user type, then click Apply to apply the filter.

- Type: Select this option to filter the audit logs by the policy types, for example, site, user, template, application profile, bridge domain, EPG, external EPG, filter, VRF, BGP config, contract, OSPF policy, pod, node, port, domain, provider, RADIUS, TACACS+ and click Apply.

- Action: Select this option to filter the audit logs by an action. The available actions are Created, Updated, Deleted, Added, Removed, Associated, Disassociated, Deployed, Undeployed, Downloaded, Uploaded, Restored, Logged in, Logged Out, Login Failed. Select an action and click Apply to filter the log details according to the action.

- Generating Troubleshooting Report and Logs

- Enabling Log Streaming to an External Log Analyzer

Generating Troubleshooting Report and Logs

This section describes how to generate a troubleshooting report and infrastructure logs file for all the schemas, sites, tenants, and users that are managed by Cisco ACI Multi-Site Orchestrator.

Procedure

Step 1

Log in to your Multi-Site Orchestrator GUI.

Step 2

In the top right corner, click the Options icon and select System Logs.

Step 3

Check the logs you want to download.

Check the Database Backup to download a backup of the Orchestrator database.

Check the Server Logs to download the Orchestrator logs.

Step 4

Click DOWNLOAD.

An archive of the selected items will be downloaded to your system. The report contains the following information:

- All schemas in JSON format

- All sites definitions in JSON format

- All tenants definitions in JSON format

- All users definitions in JSON format

- All logs of the containers in the infra_logs.txt file

Enabling Log Streaming to an External Log Analyzer

Cisco ACI Multi-Site Orchestrator allows you to send the Orchestrator logs to an external log analyzer tool in real time. By streaming any events as they are generated, you can use the additional tools to quickly parse, view, and respond to important events without a delay.

This section describes how to enable Multi-SiteOrchestrator to stream its logs to an external analyzer tool, such as Splunk.

Before you begin

- Set up and configure the log analyzer service provider.

For detailed instructions on how to configure an external log analyzer, consult its documentation.

Note

This release of Cisco ACI Multi-Site Orchestrator only supports Splunk as the service provider.

- Obtain an authentication token for the service provider.

Obtaining an authentication token for Splunk service is detailed in the Splunk documentation, but in short, you can get the authentication token by logging into the Splunk server, selecting Settings > Data Inputs > HTTP Event Collector, and clicking New Token.

Procedure

Step 1

Log in to your Multi-Site Orchestrator GUI.

Step 2

In the top right corner, click the Options icon and select System Logs.

Step 3

In the System Logs window that opens, enable the EXTERNAL STREAMING knob.

Step 4

Select which logs you want to stream.

You can select either all logs or audit logs only.

Step 5

From the SELECT SERVICE dropdown menu, select the log analyzer service.

This release of Cisco ACI Multi-Site Orchestrator supports only Splunk as the service provider.

Step 6

Choose the PROTOCOL for the traffic.

Select UNSECURE for HTTP or SECURE for HTTPS.

Step 7

Provide the service's information.

In the HOST field, enter the host's IP address.

In the PORT field, enter the host's port number.

In the TOKEN field, enter the authentication token you obtain from the service provider.

Step 8

For each Multi-Site Orchestrator node, provide the node's root password.

Note

This is the root user password of each Orchestrator node, not the password you use to log in to the Orchestrator GUI.

Step 9

Click OK to save the changes.

Configuration Backups

You can create backups of your Multi-Site Orchestrator configuration that can facilitate in recovery from Orchestrator failures or cluster restarts. We recommend creating a backup of the configuration before upgrading your Orchestrator to a later version and after every configuration change or deployment. We also recommend exporting the backups to an external storage outside of the Orchestrator nodes' VMs.

Note

Restoring a backup action restores the database on the Multi-Site Orchestrator, but it does not make any changes to the APIC databases on each site. Therefore, after you restore the Orchestrator database, you must also re-deploy any existing schemas to avoid potentially mismatching policies between the Orchestrator and APIC sites.

Using NFS or Local Disks to View the Backup

Cisco ACI Multi-Site is deployed as a 3-node cluster. The backups are stored in the /opt/cisco/msc/backups/ directory.

When you deploy a 3-node cluster out of the box (NFS is not mounted on the nodes), the backup is available on the local disk. The backups can be available on any one node out of the 3 nodes and you can view it using the GUI. You can use NFS or the local disk to view the backup.

- Using NFS to View the Backup: NFS is used to create and store the backups. If you need to use NFS, you should mount it to the destination directory of /opt/cisco/msc/backups/ on all three nodes. After NFS is mounted on all three nodes, run the docker service update msc_backupservice --force command on any one node.

- Using Local Disk to View the Backup: If you have a mounted NFS and you want to use the local disk, you have to unmount NFS on all 3 nodes and run the docker service update msc_backupservice --force command on any one node.

The docker service update msc_backupservice --force command should be run only once after using the mount or unmount command. The mounted or unmounted file system is visible only after running the docker service update msc_backupservice --force command.

- Creating Backups

- Restoring Backups

- Downloading Backups

- Importing Backups

Creating Backups

This section describes how to create a new backup of your Multi-Site Orchestrator configuration.

Before you begin

Procedure

Step 1

Log in to your Multi-Site Orchestrator GUI.

Step 2

From the left navigation menu, select Admin > Backups.

Step 3

In the main window, click New Backup.

Step 4

In the New Backup window that opens, provide the backup details.

In the Name field, enter the name for the backup. The name can contain up to 10 alphanumeric characters and no spaces or underscores.

In the Notes field, enter any additional information to describe the backup.

Step 5

Click SAVE to create the backup.

Restoring Backups

This section describes how to restore a Multi-Site Orchestrator configruation to a previous state.

Before you begin

Procedure

Step 1

Log in to your Multi-Site Orchestrator GUI.

Step 2

From the left navigation menu, select Admin > Backups.

Step 3

In the main window, click the ... actions menu next to the backup you want to restore and select Rollback to this backup.

If the version of the selected backup is different from the running Multi-Site version, the rollback could cause a removal of the features that are not present in the backup version.

Step 4

Click Yes to confirm that you want to restore the backup you selected.

If you click Yes, the system terminates the current session and the user is logged out.

Downloading Backups

This section describes how to download you backup from the Multi-Site Orchestrator.

Before you begin

Procedure

Step 1

Log in to your Multi-Site Orchestrator GUI.

Step 2

From the left navigation menu, select Admin > Backups.

Step 3

In the main window, click the ... actions menu next to the backup you want to download and select Download.

This will download the backup file in .tar.gz format to your system. You can then extract the file to view its contents.

Importing Backups

This section describes how to import an existing backup into your Multi-Site Orchestrator.

Before you begin

Procedure

Step 1

Log in to your Multi-Site Orchestrator GUI.

Step 2

From the left navigation menu, select Admin > Backups.

Step 3

In the main window, click Import.

Step 4

In the Import from file window that opens, click Select File and choose the backup file you want to import.

Importing a backup will add it to the list of the backups displayed the Backups page.

Configuring Custom SSL Certificates

Cisco ACI Multi-Site Orchestrator OVA contains a self-signed SSL certificate that is stored in /opt/cisco/secretdata directory on each node during the Orchestrator installation. By default, the Orchestrator GUI uses this certificate for its HTTPS connections.

While you could previously update these certificates by logging directly into an Orchestrator node server and changing its web server (nginx) configuration, starting with Cisco ACI Multi-Site Orchestrator Release 2.1(1), you can use the GUI to easily add or update custom certificates to be used for the Orchestrator's GUI connection.

When adding custom certificates, you can use one of the following two options:

- Self-Signed Certificate provide you with the ability to create your own public and private keys to be used by the Orchestrator's GUI.

- CA-Issued Certificate allows you to use a certificate provided by an existing Certificated Authority (CA) along with its keys.

You can add multiple CAs and Keyrings containing the public/private key combinations in the GUI, however only a single keyring can be active at any given time and used to secure the communication between the Orchestrator GUI and your browser.

- Adding Custom Certificate Authority

- Adding Custom Keyring

- Activating Custom Keyring

- Custom Certificates Troubleshooting

Adding Custom Certificate Authority

You can add a custom Certificate Authority (CA) to be used for verifying the public key provided by the Orchestrator for HTTPS traffic encryption.

This section describes how to add and configure a custom CA in Multi-SiteOrchestrator GUI. Configuring keyrings and keys is described in the next section.

Procedure

Step 1

Log in to your Multi-Site Orchestrator GUI.

Step 2

From the left navigation menu, select Admin > Security.

Step 3

In the main window, select the Certificate Authority tab and click Add Certificate Authority.

Step 4

In the Add Certificate Authority window that opens, provide the CA details.

In the Name field, enter the CA name.

In the Description field, enter the CA description.

In the Certificate Chain field, enter the CA's certificate chain.

Step 5

Click SAVE to save the changes.

Adding Custom Keyring

You can add a custom keyring containing a public and prviate encryption keys to be used for Orchestrator GUI HTTPS traffic encryption.

This section describes how to add a custom keyring. For instructions on adding a Certificate Authority (CA) that can be used to verify the public key in this keyring, see the previous section.

Procedure

Step 1

Log in to your Multi-Site Orchestrator GUI.

Step 2

From the left-hand navigation menu, select Admin > Security.

Step 3

In the main window, select the Key Rings tab and click ADD KEY RING.

Step 4

In the Create Key Ring window that opens, provide the key ring details.

From the SELECT CERTIFICATE AUTHORITY dropdown menu, select the certificate authority that will contain the key ring.

In the NAME field, enter the key ring name.

In the KEY RING DESCRIPTION field, enter the key ring description.

In the PUBLIC KEY field, enter the ring's public key.

In the PRIVATE KEY field, enter the ring's private key

Step 5

Click SAVE to save the changes.

Activating Custom Keyring

You can add a custom keyring containing a public and private encryption keys to be used for Orchestrator GUI HTTPS traffic encryption.

This section describes how to add a custom keyring. For instructions on adding a Certificate Authority (CA) that can be used to verify the public key in this keyring, see the previous section.

Procedure

Step 1

Log in to your Multi-Site Orchestrator GUI.

Step 2

From the left-hand navigation menu, select Admin > Security.

Step 3

In the main window, select the Key Rings tab.

Step 4

In the main window, click the ... icon next to the keyring you want to activate and choose Make Keyring Active.

Step 5

Click ACTIVATE to activate the keyring.

Activating a key will log you out of the Multi-Site Orchestrator GUI. When the login page is loaded, it will use the new certificate and key.

Custom Certificates Troubleshooting

The following sections describe how to resolve common issues when using custom SSL certificates with Multi-Site Orchestrator.

Unable to Load the Orchestrator GUI

If you are unable to load the Orchestrator GUI page after installing and activating a custom certificate, it is possible that the certificates were not copied correctly to each Orchestrator node. You can resolve this issue by recovering the default certificates and then repeating the new certificate installation procedure again.

To recover the default Orchestrator certificates:

1. Log in to each Orchestrator node directly.

2. Change into the certificates directory:

# cd /data/msc/secrets

3. Replace the msc.key and msc.cert files with msc.key_backup and msc.cert_backup files respectively.

# cp msc.key_backup msc.key
# cp msc.cert_backup msc.cert

4. Restart the Orchestrator GUI service

# docker service update msc_ui --force

5. Re-install and activate the new certificates as described in previous sections.

Adding a New Orchestrator Node to the Cluster

If you add a new node to you Multi-Site Orchestrator cluster:

1. Log in to the Orchestrator GUI.

2. Re-activate the key you are using as described in previous sections.

External Authentication

Starting in Cisco ACI Multi-Site Release 1.1(x), you can configure external authentication and authorization using RADIUS or TACACS+ for users.

As a Multi-Site administrator, you can:

- Add one or more external authentication providers.

It is recommended to set up at least 2 authentication providers for redundancy.

- Create login domains and associate them with providers.

The default domain is the Local domain, for local authentication.

- Assign users to domains.

After you create domains, you can edit, deactivate, or delete them. You cannot delete the Local domain, but you can deactivate it.

Audit logs support external authentication and authorization.

- Guidelines for Configuring External Authentication Servers

- Adding RADIUS or TACACS+ as Authentication Provider

- Adding LDAP as Authentication Provider

- Creating Login Domains

- Editing, Deleting, or Deactivating Login Domains

- Remote User Logon

Guidelines for Configuring External Authentication Servers

When configuring external authentication servers for Multi-Site Orchestrator user authentication:

- You must configure each user on the remote authentication servers.

- For each user, add a custom attribute in the format cisco-av-pair=shell:msc-roles=role1,role2, where each role is one of the Multi-Site Orchestrator roles documented in User Roles and Features.

For example, cisco-av-pair=shell:msc-roles=siteManager,schemaManager.

- For LDAP configurations, we recommend using CiscoAVPair as the attribute string. If, for any reason, you are unable to use an Object ID 1.3.6.1.4.1.9.22.1, an additional Object IDs 1.3.6.1.4.1.9.2742.1-5 can also be used in the LDAP server.

Adding RADIUS or TACACS+ as Authentication Provider

This section describes how to add one or more RADIUS or TACACS+ servers as external authentication servers for authenticating Cisco ACI Multi-Site Orchestrator users.

Procedure

Step 1

Log in to your Cisco ACI Multi-Site Orchestrator as the admin user using the Local domain.

Step 2

From the left-hand navigation pane, select Admin > Providers.

Step 3

In the main window, click ADD PROVIDER.

Step 4

Enter the host name or IP address of the external authentication server.

Step 5

(Optional) Enter a description for the provider you are adding.

Step 6

Select RADIUS or TACACS+ for the provider type you are adding.

Step 7

Enter the KEY and confirm it in the CONFIRM KEY field.

Step 8

(Optional). Configure additional settings.

1. Expand Additional Settings for more settings.

2. You can specify the port used to connect to the authentication server.

The default port is 1812 for RADIUS and 49 for TACACS+.

3. You can specify the protocol used.

You can choose between PAP or CHAP protocols.

4. You can specify the timeout and number of attempts for connecting to the authentication server.

Adding LDAP as Authentication Provider

Starting with Release 2.1(1) you can add one or more LDAP servers as external authentication servers for authenticating Cisco ACI Multi-Site Orchestrator users.

Procedure

Step 1

Log in to your Cisco ACI Multi-Site Orchestrator as the admin user using the Local domain.

Step 2

From the left-hand navigation pane, select Admin > Providers.

Step 3

In the main window, click ADD PROVIDER.

Step 4

Enter the host name or IP address of the external authentication server.

Step 5

(Optional) Enter a description for the provider you are adding.

Step 6

Select LDAP for the provider type you are adding.

Step 7

Enter the BASE DN and BIND DN values for the LDAP server..

Step 8

Enter the KEY and confirm it in the CONFIRM KEY field.

Step 9

(Optional). Configure additional settings.

1. Click Additional Settings to expand.

2. You can specify the port used to connect to the authentication server.

The default port for LDAP is 389.

3. You can specify the timeout and number of attempts for connecting to the authentication server.

4. You can specify the filter used.

The default LDAP filter is (cn=$userid).

For Microsoft, the LDAP filter should be (sAMAccountName=$userid).

5. You can specify the authentication type.

The authentication type can be:

- CISCO-AVPAIR – for authorization based on individual user's attribute.

- LDAP GROUP MAP RULES - for authorization based on the user's group membership.

6. You can specify the attribute value and one or more user groups.

If CISCO-AVPAIR is selected for authentication type, the attribute value is an attribute assigned to individual users, for example ciscoAVPair.

If LDAP GROUP MAP RULES is selected for authentication type, the attribute value is group membership, for example memberOf and a list of groups.

7. If you select LDAP GROUP MAP RULES for authentication type, you must also provide one or more groups to which the LDAP user belongs.

In the LDAP GROUP MAP RULES list, click the + sign.

In the Add New Group Map Rule window that opens, provide the group details, such as Name, Description, Group DN, and Roles. You can add multiple roles for the same group map rule.

Creating Login Domains

A login domain defines the authentication domain for a user. Login domains can be set to the Local, RADIUS, TACACS+, or LDAP authentication mechanisms.

When you are logging in to the Cisco ACI Multi-Site Orchestrator using the GUI, the login screen offers a drop-down list of domains for the user to select from. If no domain is specified, the Local domain is used to look up the username.

When you are logging in to the Cisco ACI Multi-Site Orchestrator using the REST API, the login domain is provided along with the login information in the POST message, for example:

{
"username":"bob",
"password":"we1come!",
"domainId":"59d5b5978d0000d000909f65"
}

To create a login domain using the Cisco ACI Multi-Site Orchestrator GUI:

Before you begin

You must have added one or more authentication providers as described in Adding RADIUS or TACACS+ as Authentication Provider or Adding LDAP as Authentication Provider.

Procedure

Step 1

Log in to your Cisco ACI Multi-Site Orchestrator.

Step 2

From the left-hand navigation pane, select Admin > Login Domains.

Step 3

In the main window, click ADD LOGIN DOMAIN.

Step 4

Enter the domain's name.

Step 5

(Optional) Enter a description for the domain.

Step 6

Select REALM type to specify the authentication provider.

You must have an external authentication provider added before creating login domains.

Step 7

Assign the login domain to one or more providers.

Mark the checkbox next to one or more providers' names to assign the domain.

What to do next

After you create one or more login domains, you can edit, delete, or deactivate them as described in Editing, Deleting, or Deactivating Login Domains.

Editing, Deleting, or Deactivating Login Domains

After you have created one or more login domains, you can use the instruction described in this section to edit, delete, or deactivate them. You cannot delete the Local domain, but you can deactivate it.

Before you begin

You must have created one or more Login domains as described in Creating Login Domains.

Procedure

Step 1

Log in to your Cisco ACI Multi-Site Orchestrator.

Step 2

From the left-hand navigation pane, select Admin > Login Domains.

Step 3

Click the ... menu next to the login domain you want to edit.

You can choose to Edit the domain information, Deactivate the domain so that it cannot be used, or Set as default so it is automatically selected when logging in using GUI.

Remote User Logon

When external authentication is enabled in Cisco ACI Multi-Site, you can log in to the Multi-Site Orchestrator as follows:

Procedure

Step 1

Using a browser, navigate to the Multi-Site URL.

Step 2

Choose your assigned domain from the drop down list.

Step 3

Enter your username and password.

Step 4

Click Submit.

If you are authorized and pass authentication, the Multi-Site Orchestrator GUI is displayed and you have privileges according to the roles that are assigned to you. The first time you log on, you will be prompted to change your password.

Managing Scope of Schema and Template Deployment

Cisco ACI Multi-Site schemas and templates contain the policies and scope of what you intend to deploy to each site.

In order to ensure correct policy deployment and operation, we recommend that you add each schema and template to the sites incrementally and confirm the desired operation and performance.

Preface

This preface includes the following sections:

- Audience

- Document Conventions

- Related Documentation

- Documentation Feedback

- Obtaining Documentation and Submitting a Service Request

Audience

This guide is intended primarily for data center administrators with responsibilities and expertise in one or more of the following:

- Virtual machine installation and administration

- Server administration

- Switch and network administration

Document Conventions

Command descriptions use the following conventions:

Convention

Description

bold

Bold text indicates the commands and keywords that you enter literally as shown.

Italic

Italic text indicates arguments for which the user supplies the values.

[x]

Square brackets enclose an optional element (keyword or argument).

[x | y]

Square brackets enclosing keywords or arguments separated by a vertical bar indicate an optional choice.

{x | y}

Braces enclosing keywords or arguments separated by a vertical bar indicate a required choice.

[x {y | z}]

Nested set of square brackets or braces indicate optional or required choices within optional or required elements. Braces and a vertical bar within square brackets indicate a required choice within an optional element.

variable

Indicates a variable for which you supply values, in context where italics cannot be used.

string

A nonquoted set of characters. Do not use quotation marks around the string or the string will include the quotation marks.

Examples use the following conventions:

Convention

Description

screen font

Terminal sessions and information the switch displays are in screen font.

boldface screen font

Information you must enter is in boldface screen font.

italic screen font

Arguments for which you supply values are in italic screen font.

< >

Nonprinting characters, such as passwords, are in angle brackets.

[ ]

Default responses to system prompts are in square brackets.

!, #

An exclamation point (!) or a pound sign (#) at the beginning of a line of code indicates a comment line.

This document uses the following conventions:

Note

Means reader take note. Notes contain helpful suggestions or references to material not covered in the manual.

Caution

Means reader be careful. In this situation, you might do something that could result in equipment damage or loss of data.

Warning

IMPORTANT SAFETY INSTRUCTIONS

This warning symbol means danger. You are in a situation that could cause bodily injury. Before you work on any equipment, be aware of the hazards involved with electrical circuitry and be familiar with standard practices for preventing accidents. Use the statement number provided at the end of each warning to locate its translation in the translated safety warnings that accompanied this device.

SAVE THESE INSTRUCTIONS

Related Documentation

Application Policy Infrastructure Controller (APIC) Documentation

The following companion guides provide documentation for APIC:

- Cisco APIC Getting Started Guide

- Cisco APIC Basic Configuration Guide

- Cisco ACI Fundamentals

- Cisco APIC Layer 2 Networking Configuration Guide

- Cisco APIC Layer 3 Networking Configuration Guide

- Cisco APIC NX-OS Style Command-Line Interface Configuration Guide

- Cisco APIC REST API Configuration Guide

- Cisco APIC Layer 4 to Layer 7 Services Deployment Guide

- Cisco ACI Virtualization Guide

- Cisco Application Centric Infrastructure Best Practices Guide

All these documents are available at the following URL: http://www.cisco.com/c/en/us/support/cloud-systems-management/application-policy-infrastructure-controller-apic/tsd-products-support-series-home.html

Cisco Application Centric Infrastructure (ACI) Documentation

The broader ACI documentation is available at the following URL: http://www.cisco.com/c/en/us/support/cloud-systems-management/application-policy-infrastructure-controller-apic/tsd-products-support-series-home.html.

Cisco Application Centric Infrastructure (ACI) Simulator Documentation

The Cisco ACI Simulator documentation is available at http://www.cisco.com/c/en/us/support/cloud-systems-management/application-centric-infrastructure-simulator/tsd-products-support-series-home.html.

Cisco Nexus 9000 Series Switches Documentation

The Cisco Nexus 9000 Series Switches documentation is available at http://www.cisco.com/c/en/us/support/switches/nexus-9000-series-switches/tsd-products-support-series-home.html.

Cisco Application Virtual Switch Documentation

The Cisco Application Virtual Switch (AVS) documentation is available at http://www.cisco.com/c/en/us/support/switches/application-virtual-switch/tsd-products-support-series-home.html.

Cisco Application Centric Infrastructure (ACI) Integration with OpenStack Documentation

Cisco ACI integration with OpenStack documentation is available at http://www.cisco.com/c/en/us/support/cloud-systems-management/application-policy-infrastructure-controller-apic/tsd-products-support-series-home.html.

Documentation Feedback

To provide technical feedback on this document, or to report an error or omission, please send your comments to apic-docfeedback@cisco.com. We appreciate your feedback.

Obtaining Documentation and Submitting a Service Request

For information on obtaining documentation, using the Cisco Bug Search Tool (BST), submitting a service request, and gathering additional information, see What's New in Cisco Product Documentation at: http://www.cisco.com/c/en/us/td/docs/general/whatsnew/whatsnew.html

Subscribe to What’s New in Cisco Product Documentation, which lists all new and revised Cisco technical documentation as an RSS feed and delivers content directly to your desktop using a reader application. The RSS feeds are a free service.

Full Cisco Trademarks with Software License

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS" WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.

All printed copies and duplicate soft copies of this document are considered uncontrolled. See the current online version for the latest version.

Cisco has more than 200 offices worldwide. Addresses and phone numbers are listed on the Cisco website at www.cisco.com/go/offices.

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: http://www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1721R)