Please add breadcrumbs in author

VXLAN EVPN Multi-Site, Release 12.1.3

Tech Article
 
Last updated: April 8, 2024
PDF
Is this helpful? Feedback

New and Changed Information

The following table provides an overview of the significant changes up to this current release. The table does not provide an exhaustive list of all changes or of the new features up to this release.

Release Version Feature Description

NDFC release 12.1.3

Reorganized content

Content within this document was originally provided in the Cisco NDFC-Fabric Controller Configuration Guide or the Cisco NDFC-SAN Controller Configuration Guide. Beginning with release 12.1.3, this content is now provided solely in this document and is no longer provided in those documents.

VXLAN Multi-Site Domain for VXLAN EVPN Fabrics

A VXLAN Multi-Site is a multifabric container that is created to manage multiple member fabrics. A VXLAN Multi-Site is a single point of control for definition of overlay networks and VRFs that are shared across member fabrics. When you move fabrics that are designated to be a part of the multifabric overlay network domain under the VXLAN Multi-Site as member fabrics, the member fabrics share the networks and VRFs created at the VXLAN Multi-Site-level. This way, you can consistently provision network and VRFs for different fabrics, at one go. It significantly reduces the time and complexity involving multiple fabric provisioning.

You can add a Data Center VXLAN EVPN fabric or a VXLAN Multi-Site External Network fabric as member fabrics in a VXLAN Multi-Site. With Release 12.1.3, Cisco Nexus Dashboard Fabric Controller provides support for adding a Campus EVPN VXLAN fabric also as a member fabric in a VXLAN Multi-Site.

As server networks and VRFs are shared across the member fabrics as one stretched network, provisioning new networks and VRFs is provided at the VXLAN Multi-Site fabric level. You can create new networks and VRFs only for the VXLAN Multi-Site. All the member fabrics inherit any new network and VRF created for the VXLAN Multi-Site.

The topology view for the VXLAN Multi-Site fabric displays all member fabrics, and how they are connected to each other, in one view. You can deploy overlay networks and VRFs on member fabrics from a single topology deployment screen, instead of deploying each member fabric separately.

Considerations

  • The VXLAN OAM feature in Cisco NDFC is only supported on a single fabric or site.

  • After you unpair a BGW vPC, perform a Recalculate Config and Deploy Config on the member fabric followed by a Recalculate Config and Deploy Config of the VXLAN Multi-Site fabric.

A few fabric-specific terms:

  • Standalone fabric - A fabric that is not part of a VXLAN Multi-Site is referred as a standalone fabric from the VXLAN Multi-Site perspective. Before the VXLAN Multi-Site concept, all fabrics were considered standalone, though two or more such fabrics can be connected with each other.

  • Member fabrics - Fabrics that are part of a VXLAN Multi-Site are called member fabrics or members. Create a standalone fabric first and then move it within a VXLAN Multi-Site as a member fabric.

When a standalone fabric is added to the VXLAN Multi-Site, the following actions take place:

  • The standalone fabric’s relevant attributes, network and VRF definitions are evaluated with that of the VXLAN Multi-Site. If there are no conflicts, then the standalone fabric becomes a member fabric of the VXLAN Multi-Site. If there is conflict, then adding a standalone fabric to the VXLAN Multi-Site fails and the conflicts are logged in the pending errors log for the VXLAN Multi-Site fabric. You can resolve the conflicts and then add the standalone fabric to the VXLAN Multi-Site again.

  • All the VRFs and networks definitions from the standalone fabric that do not have presence in the VXLAN Multi-Site are copied over to the VXLAN Multi-Site and in turn inherited to each of its other existing member fabrics.

  • The VRFs and networks (and their definitions) from the VXLAN Multi-Site (such as the VRF of the VXLAN Multi-Site, and L2 and L3 VNI parameters that do not have presence in the standalone fabric) are inherited into the standalone fabric that just became a member.

Fabric and Switch Instance Variables

While the VXLAN Multi-Site provisions a global range of network and VRF values, some parameters are fabric-specific and some parameters are switch-specific. The parameters are called fabric instance and switch instance variables.

Fabric instance values can only be edited or updated using the VRFs and Networks configuration page for the fabric. Double-click the appropriate fabric to view Fabric Overview and go to Networks or VRFs tab. Some of the examples of fabric instance variables are BGP ASN, Multicast group per network or VRF, etc. For information about editing multicast group address, see Creating and Deploying Networks and VRFs in a VXLAN Multi-Site Fabric.

You can edit the switch instance values after deploying the network on the switch. For example, VLAN ID.

Performing a Recalculate and Deploy at VXLAN Multi-Site or Member Fabric Level

Certain configuration changes, such as adding or removing a switch in a fabric, might require a recalculate and deploy operation (Actions > Recalculate and Deploy); however, once you have added a member fabric inside a VXLAN Multi-Site Domain (MSD) fabric, the level where you will be performing that recalculate and deploy operation will vary, depending on the configuration change.

The following table provides more information on the appropriate fabric level where you would need to perform a recalculate and deploy operation, depending on the configuration change. In the cases where a recalculate and deploy operation is required at both the member and the MSD fabric level, you would perform the recalculate and deploy at the member level first, then you would perform a second recalculate and deploy at the MSD level afterward.

Configuration Change Perform Recalculate and Deploy At Member Fabric Level Perform Recalculate and Deploy At MSD Fabric Level

Adding or removing a member (child) fabric in an MSD fabric

X

Adding or removing a switch (non-Border Gateway)

X

Adding or removing a Border Gateway switch

X

X

Bringing up an interface, resulting in a new intra-fabric link configuration

X

Bringing up a new link between Border Gateway switches within the same member fabric

X

X

Bringing up a new link between a Border Gateway switch and another switch within the same fabric (for example, adding a leaf switch to a VXLAN fabric, which will result in one or more multiple intra-fabric links between the new leaf switch and the existing Border Gateway spine switches)

X

X

Changing switch role (non-Border Gateway)

X

Changing switch role from or to Border Gateway

X

X

Changing MSD fabric setting

X

Discovering a new DCI link while the Multi-Site underlay IFC Auto Deployment option is turned on

X

Editing a fabric setting of a member fabric, resulting in a "Recalculate and Deploy" pop-up message

X

Pairing or unpairing a ToR or access switch

X

Pairing or unpairing a vPC in a switch (non-Border Gateway)

X

Pairing or unpairing a vPC in a Border Gateway switch

X

X

Upgrading the software image on a non-Border Gateway switch, which might lead to features that are supported in the new image that weren’t supported in the older image

X

Upgrading the software image on a Border Gateway switch, which might lead to features that are supported in the new image that weren’t supported in the older image (for example, Cloudsec is enabled on the MSD fabric, and the switch software image was upgraded from a version that didn’t support Cloudsec to a newer version that does support Cloudsec)

X

X

VXLAN Multi-Site and Member Fabric Process Flow

A VXLAN Multi-Site has multiple sites and hence has multiple member fabrics under a VXLAN Multi-Site. VRFs and networks are created for the VXLAN Multi-Site and get inherited by the member fabrics.

A high-level flow chart of the VXLAN Multi-Site and member fabric creation and VXLAN Multi-Site-to-member fabric inheritance process is depicted in the following figures:

452234.jpg

452235.jpg

The sample flow explains the inheritance from the VXLAN Multi-Site to one member. The following figure illustrates a sample flow from a VXLAN Multi-Site to multiple members:

427503.jpg

In this example, VRF-50000 (and L3 network with ID 50000), and L2 networks with IDs 30000 and 30001 are created in one go. Networks and VRFs are deployed on the member fabric switches, one after another, as depicted in the image.

452236.jpg

You can provision overlay networks through a single VXLAN Multi-Site deployment screen.

If you move a standalone fabric with existing networks and VRFs to a VXLAN Multi-Site, NDFC validates for any conflicts. This is explained in detail in an upcoming section.

Upcoming sections in the document explain the following:

  • Creating a VXLAN Multi-Site fabric.

  • Creating a standalone fabric (as a potential member) and moving the fabric under the VXLAN Multi-Site as a member.

  • Creating networks and VRFs in the VXLAN Multi-Site and their inheritance to the member fabrics.

  • Deploying networks and VRFs from the VXLAN Multi-Site and member fabric topology views.

  • Other scenarios when moving a fabric:

    • Standalone fabric with existing networks and VRFs to a VXLAN Multi-Site fabric.

    • Member fabric from one VXLAN Multi-Site to another.

Creating a VXLAN EVPN Multi-Site and Associating Member Fabrics

The process is explained in two steps:

  1. Create the VXLAN EVPN Multi-Site fabric.

  2. Create a new standalone fabric and move it under the VXLAN EVPN Multi-Site fabric as a member fabric.

Creating the VXLAN EVPN Multi-Site Fabric

  1. In Cisco Nexus Dashboard Fabric Controller, choose LAN > Fabrics.

  2. From the Actions drop-down list, choose Create Fabric.

    The Create Fabric window appears.

  3. Enter a unique name for the fabric and click Choose Fabric.

    A list of all the available fabric templates are listed.

  4. From the available list of fabric templates, choose VXLAN EVPN Multi-Site and click Select.

  5. Enter the necessary field values to create a fabric.

    The tabs and their fields in the screen are explained in the subsequent sections. The overlay and underlay network parameters are included in these tabs.

  6. When you have completed the necessary configurations, click Save.

    A message appears at the bottom right part of the screen, indicating that you have created a new VXLAN EVPN Multi-Site fabric.

    The newly created VXLAN EVPN Multi-Site fabric instance appears in the Fabrics table. It contains the member fabric names as a branch. When no member fabric is created, it is displayed as a standalone fabric.

  7. Click on the fabric to display a summary in the slide-out pane. Click the Launch icon to display the Fabric Overview page.

General Parameters

All mandatory fields in the General Parameters tab are prefilled. Update the relevant fields as needed.

Field Description

Layer 2 VXLAN VNI Range

Specifies the Layer 2 VXLAN segment identifier range.

Layer 3 VXLAN VNI Range

Specifies the Layer 3 VXLAN segment identifier range.

VRF Template

Specifies the default VRF template for leaf devices.

Network Template

Specifies the default network template for leaf devices.

VRF Extension Template

Specifies the default VRF extension template for border devices.

Network Extension Template

Specifies the default network extension template for border devices.

Enable Private VLAN (PVLAN)

Enables private VLAN on VXLAN Multi-Site and its child fabrics.

PVLAN Secondary Network Template

Specifies the default secondary PVLAN network template.

Anycast-Gateway-MAC

Specifies the anycast gateway MAC address.

Multi-Site VTEP VIP Loopback Id

Specifies the multisite routing loopback ID.

Border Gateway IP TAG

Routing tag associated with IP address of loopback and DCI interfaces

ToR Auto-deploy Flag

Enables automatic deployment of the networks and VRFs in VXLAN EVPN fabric connected to ToR switches in an External Fabric. The system enables this configuration on performing Recalculate and Deploy in the VXLAN EVPN Multi-Site fabric.

What’s next: Complete the configurations in another tabs, if required, or click Save when you have completed the necessary configurations for this fabric.

DCI

Field Description

Multi-Site Overlay IFC Deployment Method

Defines how the data centers connect through the border gateways - manually, directly to the border gateways or through a route server.

The Multisite IFCs can be created between the border gateways in the VXLAN EVPN fabrics and router server in external fabric, or back-to-back between border gateways in two VXLAN EVPN fabrics. .

Multi-Site Route Server List

Specifies the IP addresses of the route server. If you specify more than one, separate the IP addresses using a comma.

Multi-Site Route Server BGP ASN List

Specifies the BGP AS Number of the router server. If you specify more than one route server, separate the AS Numbers using a comma.

Enable 'redistribute direct' on Route Servers

Enables auto-creation of multi-site overlay IFCs on route servers. This field is applicable only when you have configured the deployment method for Multi-Site Overlay as Centralized_To_Route_Server.

Route Server IP TAG

Specifies the routing tag associated with route server IP for redistribute direct. This is the IP used in eBGP EVPN peering.

Multi-Site Underlay IFC Auto Deployment Flag

Enables auto configuration. Uncheck the check box for manual configuration.

BGP Send-community on Multi-Site Underlay IFC

Enables the Enable BGP Send-Community both setting in auto-created multi-site underlay IFC, which will generate the send-community both in the eBGP session for multi-site underlay.

BGP log neighbor change on Multi-Site Underlay IFC

Configures the Enable BGP log neighbor change setting in auto created multi-site underlay IFC.

BGP BFD on Multi-Site Underlay IFC

Configures the Enable BGP BFD setting in auto created multi-site underlay IFC.

Delay Restore Time

Specifies the Multi-Site underlay and overlay control planes convergence time. The minimum value is 30 seconds and the maximum value is 1000 seconds.

Multi-Site CloudSec

Enables CloudSec configurations on border gateways. If you enable this field, the remaining three fields for CloudSec are editable. For more information, see Support for CloudSec in Multi-Site Deployment.

CloudSec Key String

Specifies the Cisco Type 7 encrypted octet key string.

CloudSec Cryptographic Algorithm

Choose AES_128_CMAC or AES_256_CMAC for the encryption type.

CloudSec Enforcement

Specifies whether the CloudSec enforcement should be strict or loose.

  • strict - Deploys the CloudSec configuration to all the border gateways in fabrics in VXLAN Multi-Site. If there are any border gateways that don’t support CloudSec, then an error message is generated, and the configuration isn’t pushed to any switch.

    If you select strict, the tunnel-encryption must-secure CLI is pushed to the CloudSec enabled gateways within VXLAN Multi-Site.

  • loose - Deploys the CloudSec configuration to all the border gateways in fabrics in VXLAN Multi-Site. If there are any border gateways that do not support CloudSec, then a warning message is generated. In this case, the CloudSec config is only deployed to the switches that support CloudSec. If you select loose, the tunnel-encryption must-secure CLI is removed, if available.

note.svg

There should be at least two fabrics in VXLAN Multi-Site with border gateways that support CloudSec. If there is only one fabric with a CloudSec capable device, then the following error message is generated: CloudSec needs to have at least 2 sites that can support CloudSec. To remove the error, make sure you have at least two sites that can support CloudSec or disable CloudSec.

CloudSec Status Report Timer

Specifies the CloudSec Operational Status periodic report timer in minutes. This value specifies how often the NDFC polls the CloudSec status data from the switch. The default value is 5 minutes and the range is from 5 to 60 minutes.

Enable Multi-Site eBGP Password

Enables eBGP password for Multi-Site underlay/overlay IFCs.

eBGP Password

Specifies the encrypted eBGP Password Hex String.

eBGP Authentication Key Encryption Type

Specifies the BGP key encryption type. It is 3 for 3DES and 7 for Cisco. . Click the Resources tab.

Resources

Field Description

Multi-Site VTEP VIP Loopback IP Range

Specifies the Multi-Site loopback IP address range used for the EVPN Multi-Site function.

A unique loopback IP address is assigned from this range to each member fabric because each member site must have a Multi-site Routing Loopback IP address assigned for overlay network reachability. The per-fabric loopback IP address is assigned on all the BGWs in a specific member fabric.

DCI Subnet IP Range

Specifies the Data Center Interconnect (DCI) subnet IP address.

Subnet Target Mask

Specifies the DCI subnet mask.

Configuration Backup

Field Description

Scheduled Fabric Backup

Enables daily backup. This backup tracks changes in the running configuration of the fabric devices that are not tracked by configuration compliance.

Scheduled Time

Specifies the scheduled backup time in 24-hour format. This field is enabled if you check the Scheduled Fabric Backup check box.

Select both the check boxes to enable both back up processes.

What’s next: Create a new standalone fabric and move it under the VXLAN EVPN Multi-Site fabric as a member fabric.

Creating and Moving a New Fabric Under the VXLAN EVPN Multi-Site Fabric as a Member

Create a new fabric as a standalone fabric and move it under the VXLAN EVPN Multi-Site as a member. As a best practice, when you create a new fabric that is a potential member fabric of the VXLAN EVPN Multi-Site, do not add networks and VRFs to the fabric. Move the fabric under the VXLAN EVPN Multi-Site and then add networks and VRFs for the VXLAN EVPN Multi-Site. This eliminates the need for validation or conflict resolution between the member and VXLAN EVPN Multi-Site fabric network and VRF parameters.

The following are some points to consider while creating a standalone member fabric:

The parameter values in the Resources tab are automatically generated. The VXLAN VNI ID ranges in the L2 Segment ID Range and L3 Partition ID Range fields allocated for new network and VRF creation are values from the VXLAN EVPN Multi-Site fabric segment ID range. If you want to update the VXLAN VNI ranges or the VRF and Network VLAN ranges, consider the following:

  • Ensure that the new range does not overlap with the other range of values.

  • Update one range of values at a time. If you want to update more than one range of values, do it as separate instances. For example, if you want to update L2 and L3 ranges, you should do the following:

    1. Update the L2 range and click Save.

    2. Click Edit Fabric again, update the L3 range and click Save.

  • Ensure that Anycast Gateway MAC, Network Template and VRF Template field values are the same as the VXLAN EVPN Multi-Site fabric. Else, moving the member fabric to the VXLAN EVPN Multi-Site fails.

Other pointers to consider:

  • The member fabric should have a Site ID configured and the Site ID must be unique among the members.

  • The BGP AS number should be unique for a member fabric.

  • The underlay subnet range for loopback0 should be unique.

  • The underlay subnet range for loopback1 should be unique.

Moving the Member1 Fabric Under VXLAN EVPN Multi-Site-Parent-Fabric

You should go to the VXLAN EVPN Multi-Site fabric Overview to associate a member fabric under it.

  1. Double click on the VXLAN EVPN Multi-Site Fabric to view the Fabric Overview screen.

  2. On the Child Fabrics tab, choose Actions > Move Fabric into VXLAN EVPN Multi-Site.

    You can also click on Fabric Overview > Actions > Add Child Fabrics to add member fabrics to the VXLAN EVPN Multi-Site.

    A list of child fabrics that are not part of any VXLAN EVPN Multi-Site appears. Member fabrics of other VXLAN EVPN Multi-Site container fabrics are not displayed here.

  3. As Member1 fabric is to be associated with the VXLAN EVPN Multi-Site fabric, select the Member1 fabric and click Select.

  4. Select the Fabric and click Select.

    You can see that Member1 is now added to VXLAN EVPN Multi-Site fabric and is displayed in the Child Fabrics in the Fabrics list table.

VXLAN EVPN Multi-Site Fabric Topology View Pointers

The Topology tab displays the configured VXLAN EVPN Multi-Site Fabrics and its child fabrics.

  • VXLAN EVPN Multi-Site fabric topology view - VXLAN EVPN Multi-Site fabric and their member fabrics displayed. A boundary defines each member fabric. All fabric devices of the fabric are confined to the boundary.

    Double click on the member fabric to view further elements.

  • Member fabric topology view - A member fabric and its switches are displayed. In addition, the connected external fabric is displayed.

  • A boundary defines a standalone VXLAN fabric, and each member fabric in the VXLAN EVPN Multi-Site fabric. A fabric’s devices are confined to the fabric boundary. You can move a switch icon by dragging it. For a better user experience, in addition to switches, NDFC allows you to move an entire fabric. To move a fabric, place the cursor within the fabric boundary (but not on a switch icon), and drag it in the desired direction.

Adding and Editing Links

To add a link, choose Actions > More > Add Link. To edit a link, choose Actions > More > Edit Link.

To know how to add links between border switches of different fabrics (inter-fabric links) or between switches in the same fabric (intra-fabric links), refer the Fabric Links topic.

Creating and Deploying Networks and VRFs in a VXLAN Multi-Site Fabric

In standalone fabrics, networks and VRFs are created for each fabric. In a VXLAN Multi-Site fabric, networks and VRFs should be created at the VXLAN Multi-Site fabric level. The networks and VRFs are inherited by all the member networks. You cannot create or delete networks and VRFs for member fabrics. However, you can edit them.

For example, consider a VXLAN Multi-Site fabric with two member fabrics. If you create three networks in the VXLAN Multi-Site fabric, then all three networks will automatically be available for deployment in both the member fabrics.

Though member fabrics inherit the VXLAN Multi-Site fabric’s networks and VRFs, you have to deploy the networks and VRFs distinctly, for each fabric.

A deployment view is introduced for the VXLAN Multi-Site, in addition to the per-fabric deployment view. In this view, you can view and provision overlay networks for all member fabrics within the VXLAN Multi-Site, at once. However, you still have to apply and save network and VRF configurations distinctly, for each fabric.

note.svg

Networks and VRFs are the common identifiers (represented across member fabrics) that servers (or end hosts) are grouped under so that traffic can be sent between the end hosts based on the network and VRF IDs, whether they reside in the same or different fabrics. Since they have common representation across member fabrics, networks and VRFs can be provisioned at one go. As the switches in different fabrics are physically and logically distinct, you have to deploy the same networks and VRFs separately for each fabric.

For example, if you create networks 30000 and 30001 for a VXLAN Multi-Site that contains two member fabrics, the networks are automatically created for the member fabrics and are available for deployment.

You can deploy 30000 and 30001 on the border devices of all member fabrics through a single (VXLAN Multi-Site fabric) deployment screen. Prior to this, you had to access the first member fabric deployment screen, deploy 30000 and 300001 on the fabric’s border devices, and then access the second member fabric deployment screen and deploy again.

Networks and VRFs are created in the VXLAN Multi-Site and deployed in the member fabrics. The steps are explained below:

  1. Create networks and VRFs in the VXLAN Multi-Site fabric.

  2. Deploy the networks and VRFs in the member fabric devices.

Creating Networks in the VXLAN Multi-Site Fabric

Following are the guidelines and limitations when creating networks in a VXLAN Multi-Site fabric:

  • In the VXLAN Multi-Site fabric level, if the Enable L3 Gateway on Border check box is selected and you upgrade the NDFC service, then it is automatically removed from the VXLAN Multi-Site fabric level during upgrade.

  • You can edit the Network Profile part (General and Advanced tabs) of the VXLAN Multi-Site fabric network.

  • A VXLAN Multi-Site can contain multiple fabrics. These fabrics forward BUM traffic via Multicast or Ingress replication. Even if all the fabrics use multicast for BUM traffic, the multicast groups within these fabrics need not be the same.

  • When you create a network in VXLAN Multi-Site, it is inherited by all the member fabrics. However, the multicast group address is a fabric instance variable. To edit the multicast group address, you need to navigate to the member fabric and edit the network. For more information about the Multicast Group Address field, see Creating Networks for the Standalone Fabric.

  • You can only delete networks from the VXLAN Multi-Site fabric, and not member fabrics. You must undeploy the networks on the respective fabric devices before deletion.

  • When you delete networks from the VXLAN Multi-Site fabric, the networks are automatically removed from the member fabrics too.

See the section "Creating Network for Standalone Fabrics" in About Fabric Overview for LAN Operational Mode Setups.

Creating VRFs in the VXLAN Multi-Site Fabric

You cannot delete VRFs at the member fabric level. Delete VRFs in the VXLAN Multi-Site fabric. The deleted VRFs are automatically removed from all member fabrics.

See the section "Creating VRF" in About Fabric Overview for LAN Operational Mode Setups.

Deleting Networks and VRFs in the VXLAN Multi-Site and Member Fabrics

You can only delete networks from the VXLAN Multi-Site fabric, and not member fabrics. To delete networks and corresponding VRFs in the VXLAN Multi-Site fabric, follow this order:

  1. Undeploy the networks on the respective fabric devices before deletion.

  2. Delete the networks from the VXLAN Multi-Site fabric.

  3. Undeploy the VRFs on the respective fabric devices before deletion.

  4. Delete the VRFs from the VXLAN Multi-Site fabric. You can delete multiple VRF instances at once.

    note.svg

    When you delete VRFs from the VXLAN Multi-Site fabric, they are automatically removed from the member fabrics too.

Moving a Standalone Fabric with Existing Networks and VRFs to a VXLAN Multi-Site Fabric

If you move a standalone fabric with existing networks and VRFs to a VXLAN Multi-Site fabric as a member, ensure that common networks (L2 VNI and L3 VNI), anycast gateway MAC, and VRF and network templates are the same across the fabric and the VXLAN Multi-Site. NDFC validates the standalone fabric with the network and VRF information of the VXLAN Multi-Site fabric to avoid conflict entries. An example of conflict entries is two common network names with a different network ID. After validation, the standalone fabric is moved to the VXLAN Multi-Site fabric as a member fabric only if there are no conflicts.

The following are the different points to consider while moving a fabric under a VXLAN Multi-Site:

  • A VXLAN Multi-Site fabric inherits the networks and VRFs of the standalone fabric that do not exist in the VXLAN Multi-Site fabric. These networks and VRFs are in turn inherited by the member fabrics.

  • A newly created member fabric inherits the networks and VRFs of the VXLAN Multi-Site fabric that do not exist in the newly created member fabric.

  • If there are conflicts between the standalone and VXLAN Multi-Site fabrics, validation ensures that an error message is displayed. You can move the standalone fabric to the VXLAN Multi-Site again after updating the fabric configuration. If the move is successful, a message appears at the top of the page indicating that the move is successful.

  • If you move a member fabric from a VXLAN Multi-Site to a standalone fabric, the networks and VRFs remain as they are.

Support for CloudSec in Multi-Site Deployment

CloudSec feature allows secured data center interconnect in a multi-site deployment by supporting source-to-destination packet encryption between border gateway devices in different fabrics.

441314.jpg

CloudSec feature is supported on Cisco Nexus 9000 Series FX2 platform with Cisco NX-OS Release 9.3(5) or later. The border gateways, border gateway spines, and border gateway super spines that are FX2 platforms, and run Cisco NX-OS Release 9.3(5) or later are referred as CloudSec capable switches.

You can enable CloudSec while creating a VXLAN Multi-Site fabric.

note.svg

The CloudSec session is point to point over DCI between border gateways (BGWs) on two different sites. All communication between sites uses Multi-Site PIP instead of VIP. Enabling CloudSec requires a switch to move from VIP to PIP, which could cause traffic disruption for data flowing between sites. Therefore, it is recommended to enable or disable CloudSec during a maintenance window.

Refer to the "Guidelines and Limitations for Secure VXLAN EVPN Multi-Site Using CloudSec" section in the latest Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide for the guidelines and limitations for Secure VXLAN EVPN Multi-Site using CloudSec.

Enabling CloudSec in VXLAN Multi-Site

When you add or remove CloudSec configuration from the switch, the DCI uplinks flaps and triggers multisite BGP session flapping. For multisite with existing cross-site traffic, there will be traffic disruption during this transition. Therefore, it is recommended to make the transition during a maintenance window.

  1. In Cisco Nexus Dashboard Fabric Controller, choose LAN > Fabrics.

  2. Create a new VXLAN Multi-Site fabric by choosing Actions > Create Fabric or edit an existing VXLAN Multi-Site fabric by choosing Actions > Edit Fabric.

  3. In the DCI tab, configure the following CloudSec configuration parameters and click Save.

    Field Description

    Multi-Site CloudSec

    Enables CloudSec configurations on border gateways.

    When Cloudsec is enabled at VXLAN Multi-Site level, NDFC also enables dci-advertise-pip under evpn multisite border-gateway and tunnel-encryption on the uplinks for all Cloudsec capable gateways.

    When you perform Recalculate & Deploy, you can verify theses configs in the Preview Config window for the border gateway switches.

    note.svg

    CloudSec is not supported if the border gateway has TRM enabled on it (that is, if TRM is enabled on the multisite overlay IFC). If CloudSec is enabled in this scenario, appropriate warning or error messages are generated.

    CloudSec Key String

    Specifies the hex key string. Enter a 66 hexadecimal string if you choose AES_128_CMAC or enter a 130 hexadecimal string if you choose AES_256_CMAC.

    CloudSec Cryptographic Algorithm

    Choose AES_128_CMAC or AES_256_CMAC.

    CloudSec Enforcement

    Specifies whether the CloudSec enforcement should be strict or loose.

    • strict - Deploys the CloudSec configuration to all the border gateways in fabrics in VXLAN Multi-Site. If there are any border gateways that don’t support CloudSec, then an error message is generated, and the configuration isn’t pushed to any switch.

      If you select strict, the tunnel-encryption must-secure CLI is pushed to the CloudSec enabled gateways within VXLAN Multi-Site.

    • loose - Deploys the CloudSec configuration to all the border gateways in fabrics in VXLAN Multi-Site. If there are any border gateways that do not support CloudSec, then a warning message is generated. In this case, the CloudSec config is only deployed to the switches that support CloudSec. If you select loose, the tunnel-encryption must-secure CLI is removed, if it exists.

    note.svg

    There should be at least two fabrics in VXLAN Multi-Site with border gateways that support CloudSec. If there is only one fabric with a CloudSec capable device, then the following error message is generated: "CloudSec needs to have at least 2 sites that can support CloudSec." To remove this error, make sure you have at least two sites that can support CloudSec or disable CloudSec.

    CloudSec Status Report Timer

    Specifies the CloudSec Operational Status periodic report timer in minutes. This value specifies how often the NDFC polls the CloudSec status data from the switch. The default value is 5 minutes and the range is from 5 to 60 minutes.

Using the CloudSec feature in NDFC, you can have all the gateways within the VXLAN Multi-Site to use the same keychain (and have only one key string) and policy. You can provide one key chain string for NDFC to form the key chain policy. NDFC forms the encryption-policy by taking all the default values. NDFC pushes the same key chain policy, the same encryption-policy, and encryption-peer policies to each CloudSec capable gateways. On each gateway, there is one encryption-peer policy for each remote gateway that is CloudSec capable, using the same keychain and same key policy.

If you don’t want to use the same key for the whole VXLAN Multi-Site fabric or want to enable CloudSec on a subset of all sites, you can use switch_freeform to manually push the CloudSec config to the switches.

Capture all the CloudSec config in switch_freeform.

For example, the below config is included in the switch_freeform policy:

feature tunnel-encryption
evpn multisite border-gateway 600
  dci-advertise-pip
tunnel-encryption must-secure-policy
tunnel-encryption policy CloudSec_Policy1
tunnel-encryption source-interface loopback20
key chain CloudSec_Key_Chain1 tunnel-encryption
  key 1000
    key-octet-string 7 075e731f1a5c4f524f43595f507f7d73706267714752405459070b0b0701585440
    cryptographic-algorithm AES_128_CMA
tunnel-encryption peer-ip 192.168.0.6
  keychain CloudSec_Key_Chain1 policy CloudSec_Policy1

Add tunnel-encryption in the Freeform Config of the uplink interface policy which will generate config similar to the following:

interface ethernet1/13
  no switchport
  ip address 192.168.1.14/24 tag 54321
  evpn multisite dci-tracking
  tunnel-encryption
  mtu 9216
  no shutdown

For more information, see Enabling Freeform Configurations on Fabric Switches.

If you’re migrating a VXLAN Multi-Site fabric with the CloudSec configuration into NDFC, the Cloudsec related configuration is captured in switch_freeform and interface freeform config. You do not need to turn on Multi-Site Cloudsec in the VXLAN Multi-Site fabric setting. If you want to add more fabrics and establish CloudSec tunnels which share the same CloudSec policy including key as the existing one, then you can enable the CloudSec config in the VXLAN Multi-Site fabric settings. The CloudSec parameters in the VXLAN Multi-Site fabric setting need to match the existing CloudSec configuration on the switch. The CloudSec configuration is already captured in the freeform config, and enabling CloudSec in VXLAN Multi-Site also generates config intents resulting in two intents. For example, if you want to change the CloudSec key in the VXLAN Multi-Site settings, you need to remove the CloudSec freeform config because NDFC does not modify config in the switch_freeform. Otherwise, the key in the VXLAN Multi-Site fabric settings is a conflict with the key in the freeform config.

Viewing CloudSec Operational State

You can use CloudSec Operational View to check the operational status of CloudSec sessions, if CloudSec is enabled on the VXLAN Multi-Site fabric.

  1. Choose a VXLAN Multi-Site fabric.

    The fabric topology window appears.

  2. Select Actions > Detailed View.

  3. On the Link tab, click the CloudSec Operational View tab.

    If CloudSec is disabled, the CloudSec Operational View tab does not appear.

    The following table describes the fields that appear on the Operational View tab.

    Fields Description

    Fabric Name

    Specifies the fabrics that have a CloudSec session.

    Session

    Specifies the fabrics and border gateway switches involved in the CloudSec session.

    Link State

    Specifies the status of the CloudSec session. It can be in one of the following states:

    • Up: Indicates that the CloudSec session is successfully established between the switches.

    • Down: Indicates that the CloudSec session isn’t operational.

    Uptime

    Specifies the duration of uptime for the CloudSec session. Specifically, it’s the uptime since the last Rx and Tx sessions flapped, and the smaller value among the 2 sessions is displayed.

    Oper Reason

    Specifies the reason for the CloudSec session down state.
    note.svg

    After CloudSec is enabled on a fabric, the operational status may not be available until after sessions are created, and the next status poll occurs.

Troubleshooting a CloudSec Session

If a CloudSec session is down, you can find more information about it using Programmable Reports.

  1. Cisco Nexus Dashboard Fabric Controller, choose Operations > Programmable Reports.

  2. Click Create Report.

  3. Specify a unique name for the report in the Report Name field.

  4. From the Select Template drop-down list, select fabric_cloudsec_oper_status and click Select.

  5. Click Next to view the Source & Recurrence tab.

  6. In the Recurrence field, choose the frequency at which the report job should run.

  7. In the Email Report To field, enter an email ID or mailer ID if you want the report in an email.

    You must configure SMTP settings in Settings > Server Settings > SMTP tab. If the Data service IP address is in a private subnet, the static management route for SMTP server must be added in Cisco Nexus Dashboard cluster configuration.

  8. In the Select fabric(s) table, select the VXLAN Multi-Site fabric on which the report job should run.

  9. Click Save.

    The report job will be executed at the configured interval.

Copyright

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS" WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product.

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: http://www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R)

© 2017-2024 Cisco Systems, Inc. All rights reserved.

Need help?