Multi-Site

The Multi-Site functionality of Cisco Nexus Hyperfabric uses virtual extensible LAN (VXLAN) Ethernet virtual private network (EVPN) to interconnect multiple fabrics, allowing them to function as a unified network. Multi-Site provides these benefits:

  • Centralized management: You can manage all fabrics from a single point.
  • Resiliency: If one fabric encounters issues, other fabrics can remain operational.
  • Optimized resources: You can distribute applications and services among the fabrics for best performance.

Design considerations for a Multi-Site interconnect

These design considerations for apply for Multi-Site:

  • A Multi-Site interconnect can have up to five fabrics.
  • A fabric can be part of only one Multi-Site interconnect.
  • Fabrics in a Multi-Site interconnect must be in the same organization.
  • The fabrics use the full mesh topology. Border gateways will establish Multi-Site interconnect peering with border gateways in other fabrics.
  • Multi-Site interconnect provides Layer 2 VNI extension across fabrics.
  • A border gateway switch must not have any existing loopback interface, host port, or routed port. If your spine switches do not have these interface and ports, you do not need to have a dedicated border gateway switch.
  • Each fabric can have up to two border gateways and each border gateway can have up to two border gateway interfaces.
  • Ensure all border gateways are connected to the same IP network (the underlay) that has enough MTU size to carry VXLAN packets. Every border gateway will automatically establish an EVPN peering with every border gateway in all other fabrics as shown in Design of a Multi-Site interconnect (full mesh).
Design of a Multi-Site interconnect (full mesh)
Illustrates the overall design of a Multi-Site interconnect, which is a full mesh.

Multi-Site use cases

These are some Multi-Site use cases.

Scale and IP mobility

You can use Multi-Site to extend connectivity between active/active fabrics inside a single data center site, which helps with scale and IP mobility.

Scale and IP mobility use case example
Scale and IP mobility use case example

Disaster recovery and IP mobility

You can use Multi-Site to extend connectivity between a combination of active/active and active/standby fabrics within multiple data center sites, which helps with disaster recovery and IP mobility.

Disaster recovery and IP mobility use case example
Disaster recovery and IP mobility use case example

Required IP addresses for Multi-Site

You must assign various IP addresses to use Multi-Site. The addresses can be all IPv4 or all IPv6, but you cannot mix IPv4 and IPv6 addresses. Multi-Site requires these IP addresses:

  • Each fabric requires one virtual IP address for the data plane. This is for terminating the external VXLAN tunnel between border gateways. Nexus Hyperfabric uses a shared virtual IP address that is deployed as the same loopback address. VXLAN tunneled traffic from a border gateway in a different fabric can land on either one of the border gateways in the local fabric, and will be properly switched to an internal VXLAN tunnel to the destination switch.
    One virtual IP address for the data plane per fabric
    Each fabric requires one virtual IP address for the data plane.
  • Each border gateway requires one BGP source IP address for the control plane, which is for Multi-Site Ethernet virtual private network (EVPN) BGP peering between border gateways in different fabrics.
    One BGP source IP address for the control plane per border gateway
    Each border gateway requires one BGP source IP address for the control plane.
  • Underlay BGP peering is between the border gateway interface and external routers to get the required loopback IP addresses above routable between the fabrics. The fabric administrator
    • selects border gateway ports and configures the IP addresses of the ports
    • connects the border gateway ports to the external routers, and
    • configures the underlay BGP peers between the border gateway ports and the external routers.
    Underlay BGP peering
    Underlay BGP peering is based on the border gateway interface.
  • Each border gateway requires one primary IP address to support hosts, port channels or routed ports directly attached to the border gateway. This configuration is reserved for a future feature enhancement.
    One primary IP address per border gateway
    Each border gateway requires one primary IP address to support hosts, port channels or routed ports directly attached to the border gateway.

Create a Multi-Site interconnect

To begin the process of interconnecting multiple fabrics, the network administrator must create a Multi-Site interconnect. This task involves specifying the underlay IP MTU, selecting an authentication method, linking the desired fabrics to the interconnect, and configuring the VNI extensions.

Follow these steps to create a Multi-Site interconnect.

Step 1

Choose Multi-Site interconnects.

Step 2

Click + Add new Multi-Site.

  1. In the Create a new Multi-Site dialog, fill out the fields as necessary.

  2. For Underlay IP MTU, enter the MTU

    The value must be between 1500 and 9100.

  3. For Authentication method, select one of the methods.

    You will see additional fields depending on which method you selected.

  4. If you selected MD5 for Authentication method, then for Create a password and Confirm password, enter a password.

    The password must be between 7 and 126 characters in length.

  5. Click Create.

Step 3

Link fabrics to the interconnect.

  1. Click the Multi-Site interconnect name.

  2. In Fabric interconnects, click + Link Fabric.

  3. For Select fabric, select a fabric.

  4. For Virtual IP for fabric, enter a virtual IP address for the fabric.

    The virtual IP address is deployed as a loopback and the subnet is always /32.

  5. For Fabric external ASN, enter the autonomous system number (ASN) for the fabric.

  6. For BGP source address, enter the source IP addresses for each border gateway that you want to configure.

    The IP address is deployed as a loopback and the subnet is always /32. You can have up to two border gateways per fabric.

  7. Click Link.

  8. Repeat this step for each fabric to link to this Multi-Site interconnect.

Step 4

Choose VNI extensions, then click + VNI extension (L2).

  1. For L2 VNI extension name, enter a name.

  2. For Normalized VNI, enter the VXLAN network identifier (VNI) to use with this VNI extension.

  3. For Select fabric(s), select up to five fabrics that you linked under Fabric interconnects.

  4. Click Add.

  5. Repeat this step for each network.

Step 5

Commit the changes.

See Finish and commit your changes.

What to do next

The fabric administrator must establish the network. See Establish the network for a Multi-Site interconnect.

Establish the network for a Multi-Site interconnect

After the network administrator creates a Multi-Site interconnect, the fabric administrator must establish the network.

Follow these steps to establish the network for a Multi-Site interconnect.

Step 1

Choose Fabrics, then select the fabric for which you want to configure a border gateway.

Step 2

Select Multi-Site interconnects.

Step 3

Select Edit.

Step 4

Configure a border gateway.

  1. In Border gateway configuration, in the area for the first switch, for Switch, select the switch that you want to use for the first border gateway.

  2. For the Port drop-down lists, select the border gateway ports that are connected to the external IP network that interconnects the other fabrics.

    You must select at least one port and can select up to two ports.

  3. For the IP address fields, enter the Classless Inter-Domain Routing (CIDR) IP addresses for the border gateway ports.

  4. If you have a second border gateway IP address allocated from the Multi-Site interconnect configuration, repeat these substeps using the drop-down lists and fields in the area for the second switch.

Step 5

Add one underlay BGP peer per border gateway switch port.

  1. In Underlay peering configuration, select + Add underlay peer.

    Cisco Nexus Hyperfabric supports only TTL=1 for an underlay peer.

  2. For Peer address & interface, enter the IP address of the peer.

  3. For Select a BGP source interface, select the interface to use for the BGP source.

  4. In Session, for Peer AS, enter the autonomous system (AS) number of the peer.

  5. In Security, for Authentication method, select the method.

  6. (Optional) If you selected MD5, for Create a password and Confirm password, enter a password.

  7. Click Save.

  8. Repeat these substeps for each border gateway switch port.

Step 6

Configure the VNI extensions.

  1. In VNI extensions, for Local logical network, select the logical networks to use for this Multi-Site interconnect.

  2. For Extend network, leave the values as Yes.

  3. Select Save.

Step 7

Commit the changes.

See Finish and commit your changes.

Step 8

Repeat Step 4 and the successive steps for each fabric in the Multi-Site interconnect.

Finish and commit your changes

Your changes are not applied to the fabric until you review, commit, and push them.

 Note

For a more detailed description of this procedure, see "Workflow for making changes to the fabric" in Cisco Nexus Hyperfabric—Getting Started.

Follow these steps to finish and commit your changes.

Step 1

Click Review configuration.

Review configuration
Review configuration

Step 2

Verify your changes in the review list.

Step 3

Click Comment and push.

Step 4

In the Comment before pushing configuration dialog box, enter the reason for the change.

Step 5

Click Push configuration.

View the status of a Multi-Site interconnect

Follow these steps to view the status of a Multi-Site interconnect.

Step 1

Choose Fabrics, then select the fabric for which you want to view the status of a Multi-Site interconnect.

Step 2

Select Multi-Site interconnects.

Step 3

In Configuration, select Border gateways.

  1. In main area, expand the information for the switches.

    If the Admin state and Link state are up, then the fabrics have connectivity to one another.

Step 4

In Configuration, select BGP peering.

  1. In main area, view the Status of the peers.

    If a peer's status is Established, then the peering session between that border gateway and its BGP neighbor has been successfully formed and is operational.

Frequently asked questions about Multi-Site

These are some frequently asked questions about multisite.

Q: I created a Multi-Site, but the Multi-Site tab in the fabrics is not populated.

A: You may not have committed some changes in the organization-wide Multi-Site instance, which behaves like a pseudo-fabric with its own commit process. Try to commit changes in the organization-level Multi-Site mesh and check back in the fabric.

Q: Why do I need multiple IP addresses for border gateways?

A: These addresses are used outside the fabric for border gateways (BGWs) to communicate with each other to establish Multi-Site connectivity. For more information, see Required IP addresses for Multi-Site.