New and Changed Information

The following table provides an overview of the significant changes up to this current release. The table does not provide an exhaustive list of all changes or of the new features up to this release.

Cisco APIC Release Version

Feature

Description

5.0(2)

Support for Azure VNET peering in Cisco Cloud APIC

This release provides support for Azure VNET peering in Cisco Cloud APIC

Data Forwarding Between VNets

Prior to Release 5.0(2), the solution for inter-VNet communication in a Cisco Cloud APIC uses a hub-spoke overlay topology with IPsec tunnels between a pair of Cisco CSR routers in the infra VNet and a Virtual Network Gateways (VNG) in the user VNet. BGP runs over the IPsec tunnels as the routing protocol between the CSR routers and the VNG.

This implementation has several limitations:

  • Azure VPN gateway is an expensive and heavy Azure resource

  • Every user VNet must deploy an Azure VPN gateway in order to forward the traffic to a CSR

  • Each VPN tunnel offers limited bandwidth (1.25 Gbps), which limits the total throughput

  • Creating a virtual network gateway (VPN gateway) can take up to 45 minutes to complete

Beginning in Release 5.0(2), Cisco Cloud APIC supports using Azure VNet peering for inter-VNet connectivity. Azure VNet peering is a service that functions as an internal router to automate connectivity between virtual networks (VNets). The VNets can be in different Azure regions in a cloud site.

VNet peering provides a low-latency, high-bandwidth connection useful in scenarios such as cross-region data replication and database failover. Since traffic is private and remains on the Microsoft backbone, if you have strict data policies, VNet peering becomes preferable because the public internet isn't involved. Since there's no gateway in the path, there are no extra hops, ensuring low-latency connections.

Azure VNet peering, similar to a Cisco CSR, is owned by the infra tenant. However, it’s shared with multiple user accounts.

Cisco APIC Release 5.0(2) is backward-compatible with previous methods of configuring communication between VNets.

About VNet Peering

Virtual network (VNet) peering enables seamless connection between two Azure Virtual Networks and is Microsoft's recommended way of forwarding data between two VNets. The virtual networks appear as one for connectivity purposes. The traffic between virtual machines uses the Microsoft backbone infrastructure. Like traffic between virtual machines in the same network, traffic is routed through Microsoft's private network only. Peerings are bidirectional.

Peering connections are non-transitive. For example, assume three VNets (A, B, and C), where A is bidirectionally peered with B, and B is with C. This does not mean A and C are peered with each other.

Network traffic between peered virtual networks is private. Traffic between the virtual networks is kept on the Microsoft backbone network. No public Internet, gateways, or encryption is required in the communication between the virtual networks.

The benefits of using virtual network peering include:

  • A low-latency, high-bandwidth connection between resources in different virtual networks.

  • The ability for resources in one virtual network to communicate with resources in a different virtual network.

  • No downtime to resources in either virtual network when creating the peering, or after the peering is created.

  • Much higher traffic throughput compared to IPSec tunnels.

For more information on VNet peering, see the article Virtual network peering in the Documentation section in the Azure website.

Cisco Cloud APIC Implementation of VNet Peering

Cisco Cloud APIC uses several components to implement VNet peering.

Hub and Spoke Topology

Cisco Cloud APIC uses a hub and spoke topology for VNet peering rather than a full-mesh topology because a hub-spoke topology is easier to manage. In the hub-spoke topology, all the user VNets peer with a central hub VNet, forming a full hub and spoke topology.

A CSR acts as the network virtual appliance (NVA) in the hub and routes the packets from one VNet to another VNet, and also forms the VXLAN tunnels toward the on-premises site. This kind of connection provides full bandwidth between Azure VNets as supported by your datacenter.

User-Defined Routes

As mentioned previously, peering connections with Azure VNet peering are non-transitive. Since spokes are not peered with each other, Cisco Cloud APIC uses user-defined routes (UDRs) on each user VNet to forward traffic to other VNets.

UDRs are used to override Azure's default system routes, or to add additional routes. UDRs for the spoke are added to a subnet’s route table, with the next-hop pointing to the NLB of the NVA (CSR) in the hub.

Cloud APIC Implementation

The following figure shows how Cloud APIC typically implements VNet peering.

Where:

  1. Infra VNets act as the hub, and tenant VNets act as the spokes. Bi-directional peering is used between each hub-spoke pair.

  2. CSRs in the hub VNet act as the NVAs for spoke-to-spoke routing.

  3. An Azure network load balancer (NLB) is placed in front of the CSRs to perform load-balancing and failover.

  4. In a multi-region cloud site on Azure, each region can have its own hub VNet (infra VNet with a pair of CSRs and a NLB), or can share the hub VNet in other regions. All spoke VNets are peered with all hub VNets, across regions. The types of peerings are:

    • Local peering: Peering with hub in the same Azure region

    • Global peering: Peering with hubs in different Azure regions

  5. The UDR in the spoke VNets redirect traffic to the CSR NLB for traffic destined to other spokes or sites. For example:

    In this example:

    • Spoke-1: VRF1 (42.1.10.0/24)

    • Spoke-2: VRF2 (42.1.20.0/24)

    A UDR is needed in each spoke's route table, with one UDR per CIDR in the destination VRF. The next-hop for the UDR is the IP address of the NLB, where the routes with the local NLB are preferred over those with remote NLBs.

Guidelines and Limitations for Azure VNet Peering

Following are the guidelines and limitations for Azure VNet peering.

  • You can't add, remove, or update a CIDR on the VNet if it has peering connections with other VNets. In this situation, you must remove the VNet peerings, then make the necessary change to the CIDR (add, remove, or update the CIDR), then reestablish the VNet peerings again.

  • Resources in one virtual network can't communicate with the front-end IP address of a Basic Internal Load Balancer (ILB) in a globally-peered virtual network.

  • Some services that use a Basic load balancer don't work over global virtual network peering. For more information, see

    Constraints for peered virtual networks.

  • There is no support for traffic between two user VNets where one of the VNets has only VNet peering configured and the other VNet has only VPN gateway configured.

Prerequisites for Configuring Azure VNet Peering

Complete the following tasks before configuring Azure VNet peering:

Configuring Azure VNet Peering Using the GUI

To configure Azure VNet peering, you set up the cloud site and then configure the necessary information in the cloud context profile.

You use Cisco Cloud APIC to set up the cloud site. You can use Cisco Cloud APIC to configure the cloud context profile; however, if you are using Azure VNet peering in a multisite environment, we recommend that you do so in Cisco Application Centric Infrastructure (ACI) Multi-Site Orchestrator.

Set Up the Cloud Site to Use Azure VNet Peering

Complete this task to set up the Azure cloud site to support VNet peering. This procedure assumes that you have not yet set up the cloud site.

The procedure for setting up the cloud site to use Azure VNet peering is very similar to a typical Cloud APIC first-time setup procedure, with the exception of enabling the Azure VNet peering feature using the VNet Peering option, as described in Step 4.

Before you begin

Procedure


Step 1

Log in to Cisco Cloud APIC.

Step 2

In the Welcome to Cloud APIC dialog box, click Review First Time Setup.

Step 3

In the Let's Configure the Basics dialog box, in the Region Management area, click Begin.

The Setup—Region Management window appears.

Step 4

For Connectivity for Internal Network, choose Virtual Network Peering to enable the Azure VNet peering feature.

This enables VNet peering at the Cloud APIC level, deploying NLBs in all the regions with CSRs in the infra VNet.

Note 

The VPN Connectivity via CSR option is used to enable the traditional VPN connectivity through the overlay IPsec tunnels between CSRs and Azure VPN Gateway routers, instead of using VNet peering.

Step 5

If you want connectivity to the on-premises site or another cloud site—in addition to connectivity within this cloud site—check the Inter-Site Connectivity check box.

Step 6

In the Regions to Manage area, choose one or more regions that you want to manage.

In the current release, a cloud site can consist up to 4 regions, including the home region where the Cloud APIC is deployed.

A Cloud APIC can manage multiple cloud regions as a single cloud site. In the current release, a cloud site can consist up to 4 regions, including the home region where the Cloud APIC is deployed. If a Cloud APIC manages two regions, those two regions are considered a single site.

The following options are available on the row for any region that you select:

  • Cloud Routers: Select this option if you want to deploy CSRs in this region. You must have at least one region with CSRs deployed to have inter-VNET or inter-site communications. However, if you choose multiple regions in this page, you do not have to have CSRs in every region that you choose.

  • Inter-Site Connectivity: Select this option if you want this region to connect to other sites (for example, if you want this region to connect to an on-premises site, or to connect cloud site-to-cloud site, through Cisco ACI Multi-Site). Infra VNETs or VPCs are deployed on all regions selected for inter-site connectivity. Note that when you select inter-site connectivity for a region, the cloud routers option is also selected automatically for this region because you must have two cloud routers deployed for inter-site connectivity hubs.

For example, consider the following configuration:

In this example configuration:

  • The regions Central US and East US have checkmarks in the Cloud Routers column. They will have CSRs and an Azure NLB deployed in their infra VNets that will serve as the hub VNets for VNet peering.

  • The region East US2 will not have a CSR deployed (does not have a checkmark in the Cloud Routers column). This managed region with no CSR will be able to have spoke VNets, but will use the hub VNets in the other regions.

Step 7

Click Next.

Another panel of the Setup—Region Management dialog box appears.

The General area shows the subnets for the cloud routers, which you provided when you installed Cisco Cloud APIC.

Step 8

In the Fabric Autonomous System Number field, enter the BGP autonomous system number (ASN) that is unique to this site.

Note the following Microsoft Azure ASN restrictions:

  • Do not use 64518 as the autonomous system number in this field.

  • Do not use 32-bit ASNs. Azure VPN Gateways support 16-Bit ASNs at this time.

  • The following ASNs are reserved by Azure for both internal and external peerings:

    • Public ASNs: 8074, 8075, 12076

    • Private ASNs: 65515, 65517, 65518, 65519, 65520

    You cannot specify these ASNs for your on-premises VPN devices when connecting to Azure VPN gateways.

  • The following ASNs are reserved by IANA and cannot be configured on your Azure VPN Gateway: 23456, 64496-64511, 65535-65551 and 429496729

Step 9

In the Subnet for Cloud Router field, enter the subnet for the cloud router.

The first subnet pool for the first two regions is automatically populated. If you selected more than two regions, you will need to add another subnet for the cloud routers for the additional regions. Addresses from this subnet pool will be used for inter-region connectivity for any additional regions that are added that need to be managed by the Cloud APIC after the first two regions. This must be a valid IPv4 subnet with mask /24.

Step 10

Click Add Subnet for Cloud Router and enter information for additional subnets, if necessary.

  • For releases prior to Release 5.0(2), enter information on additional subnets based on the number of CSRs that are being deployed.

  • For Release 5.0(2) and later, enter information on additional subnets based on the number of regions that your Cloud APIC is managing.

All CIDRs for the infra tenant will be pre-allocated in peering mode. The system needs three /25 subnets for each managed region with CSRs.

Click the checkbox to accept the values for every subnet that you add.

Step 11

Configure the remaining fields as you normally would.

  1. Under the Cloud Router Template area, in the Number of Routers Per Region field, choose the number of Cisco Cloud Services Routers (CSRs) that will be used in each region.

    The default is two. The current release supports up to four CSRs per region.

  2. In the Username, enter the username for the Cisco Cloud Services Router.

    Note 

    Do not use admin as a username for the Cisco Cloud Services Router when deploying it to an Azure cloud site.

  3. In the Password field, enter the password for the Cisco Cloud Services Router.

  4. In the Throughput of the routers field, choose the throughput of the Cisco Cloud Services Router.

    In some cases, changing the value in this field changes the size of the CSR instance that is deployed. Choosing a higher value for the throughput could result in a larger VM being deployed.

    Note 

    If you wish to change this value at some point in the future, you must delete the CSR, then repeat the processes in this chapter again and select the new value that you would like in the same Throughput of the routers field.

    In addition, the licensing of the CSR is based on this setting. You will need the equivalent or higher license in your Smart account for it to be compliant.

    Note 

    Cloud routers should be undeployed from all regions before changing the router throughput or login credentials.

  5. In the License Token field, enter the license token for the Cisco Cloud Services Router.

    This is the Product Instance Registration token from your Cisco Smart Software Licensing account. To get this license token, go to http://software.cisco.com, then navigate to Smart Software Licensing > Inventory > Virtual Account to find the Product Instance Registration token.

  6. Click the appropriate button, depending on whether you are configuring inter-site connectivity or not.

    • If you are not configuring inter-site connectivity (if you did not select Inter-Site Connectivity when you were selecting regions to manage in the Region Management page), click Save and Continue. You have completed setting up the cloud site in this case.

    • If you are configuring inter-site connectivity (if you selected Inter-Site Connectivity when you were selecting regions to manage in the Region Management page), click Next at the bottom of the page. The Inter-Site Connectivity page appears.

  7. Enter the following information in the Inter-Site Connectivity page:

    • IPSec Tunnels to Inter-Site Routers: This field is necessary only for on-premises connectivity to cloud sites. There is no need to enter information in this field if you don't have an on-premises site.

      In this area, click the + button next to the Add Public IP of IPsec Tunnel Peer field.

      • Enter the peer IP address for the IPsec tunnel termination to the on-premises device.

      • Click the check mark to add this peer IP address.

    • OSPF Area for Inter-Site Connectivity: Enter the underlay OSPF area ID that will be used with on-premises ISN peering (for example, 0.0.0.1)

    • Under the External Subnets for Inter-Site Connectivity heading, click the + button next to the +Add External Subnet field.

      • Enter the subnet tunnel endpoint pool (the cloud TEP) that will be used in Azure. It must be a valid IPv4 subnet with a mask between /16 and /22 (for example, 30.29.0.0/16). This subnet will be used to address the IPsec tunnel interfaces and loopbacks of the Cloud Routers used for on-premises connectivity, and cannot overlap with other on-premises TEP pools.

      • Click the check mark after you have entered in the appropriate subnet pools.

  8. When you have entered all the necessary information on this page, click Save and Continue at the bottom of the page.

    You have completed setting up the cloud site in this case.


What to do next

  1. Perform the tasks in the chapter "Configuring the Cisco Cloud APIC Using the GUI," in the Cisco Cloud APIC for Azure User Guide 5.0(x).

    The tasks include configuring a tenant, an application profile, a VRF, one or more endpoint groups (EPGs), and one or more contracts and filters.

  2. Create a cloud context profile for Azure VNET peering.

    See Create a Cloud Context Profile for those procedures.

Create a Cloud Context Profile

This section explains how to create a cloud context profile using the Cisco Cloud APIC GUI for Azure VNET peering.

The procedure for creating a cloud context profile for Azure VNet peering is very similar to a typical cloud context profile configuration procedure, with the following VNet peering-specific steps:

  • Enable the Azure VNet peering feature in the VNet Peering field.

Before you begin

Set up the cloud site to use Azure VNET peering.

Procedure


Step 1

Log in to the Cloud APIC, if you are not logged in already.

Step 2

In the left navigation bar, navigate to Application Management > Cloud Context Profiles.

The existing cloud context profiles are displayed.

Step 3

Click Actions and choose Create Cloud Context Profile.

The Create Cloud Context Profile dialog box appears.

Step 4

Enter the appropriate values in each field as listed in the following Cloud Context Profile Dialog Box Fields table then continue.

Table 1. Create Cloud Context Profile Dialog Box Fields

Properties

Description

Name

Enter the name of the cloud context profile.

Tenant

To choose a tenant:

  1. Click Select Tenant. The Select Tenant dialog box appears.

  2. From the Select Tenant dialog, click to choose a tenant in the left column then click Select. You return to the Create Cloud Context Profile dialog box.

Description

Enter a description of the cloud context profile.

Settings

Region

To choose a region:

  1. Click Select Region. The Select Region dialog box appears.

  2. From the Select Region dialog, click to choose a region in the left column then click Select. You return to the Create Cloud Context Profile dialog box.

VRF

To choose a VRF:

  1. Click Select VRF. The Select VRF dialog box appears.

  2. From the Select VRF dialog box, click to choose a VRF in the left column then click Select. You return to the Create Cloud Context Profile dialog box.

Add CIDR

Add additional CIDRs, if necessary.

To add a CIDR:

  1. Click Add CIDR. The Add CIDR dialog box appears.

  2. Enter the address in the Address field.

  3. Click Add Subnet and enter the subnet address in the Address field.

  4. Click to check (enabled) or uncheck (disabled) the Primary check box.

  5. When finished, click Add.

VNet Gateway Router

Click to check (enable) or uncheck (disable) in the VNet Gateway Router check box.

VNet Peering

Click to check (enable) or uncheck (disable) the Azure VNet peering feature.

Step 5

Click Save when finished.

Step 6

Configure the Network Contributor role for both the infra and user tenant subscriptions.

For example, assume the following:
  • The infra tenant is using subscription S1 with access credentials/service principal C1

  • The user tenant is using subscription S2 with access credentials/service principal C2

In this situation, you will have to configure the following for peering to work between the user tenant and the infra VNets:

  • You will have to give C1 Network Contributor role permissions to S2 for the hub to spoke peering link

  • You will have to give C2 Network Contributor role permissions to S1 for the spoke to hub peering link

  1. In the yellow window that appears, copy the az command provided.

    • If you have configure the Network Contributor role for the user tenant, copy the text in the area Command to run for User Subscription.

    • If you have configure the Network Contributor role for the infra tenant, copy the text in the area Command to run for Infra Subscription.

  2. Return to the Azure management portal and click Registrations in the left navigation bar.

  3. Open the Cloud Shell.

  4. Select Bash.

  5. Paste the az command that you copied in 6.a.

Step 7

Follow these steps to disable Azure VNet peering, if necessary.

If you decide that you would like to disable Azure VNet peering after you have configured it, follow these steps to disable the VNet peering feature:

  • To disable Azure VNet peering at a global level:

    1. First disable Azure VNet peering at the local level, through the Cloud Context Profile:

      1. Navigate to the Create Cloud Context Profile page:

        Application Management > Cloud Context Profiles, then click Actions and choose Create Cloud Context Profile

      2. Double-click the cloud context profile where you want to disable VNet peeriing.

      3. Uncheck (disable) the Hub Network Peering field.

    2. Then disable Azure VNet peering at the global level:

      1. In the Cloud APIC GUI, click the Intent icon ( ) and select cAPIC Setup. In the Region Management area, click Edit Configuration.

      2. In the Regions to Manage screen, change the Connectivity for Internal Network setting from Virtual Network Peering to VPN Connectivity via CSR.

  • To disable Azure VNet peering at a local level, for a particular Cloud Context Profile:

    1. Navigate to the Create Cloud Context Profile page:

      Application Management > Cloud Context Profiles, then click Actions and choose Create Cloud Context Profile

    2. Uncheck (disable) the VNet Peering field.


Configuring Azure VNet Peering Using the REST API

Complete this task to set up the Azure cloud site to support VNet peering. This procedure assumes that you have not yet set up the cloud site.

Before you begin

Procedure


Step 1

Configure the cloud template for the infra tenant using a post such as the following, where the information used to configure Azure VNet peering is highlighted in bold.


<polUni>
    <fvTenant name="infra">
        <cloudApicSubnetPool subnet="10.10.1.0/24" /> 
        <cloudApicSubnetPool subnet="10.10.2.0/24" /> 
            <cloudtemplateInfraNetwork name="default" numRemoteSiteSubnetPool="2" numRoutersPerRegion="2" status="" vrfName="overlay-1" status="">
                <cloudtemplateProfile name="default" routerPassword="Ins3965!" routerUsername="cisco" routerThroughput="1G"/>
                <cloudtemplateExtSubnetPool status="" subnetpool="11.11.0.0/16"/>
                <cloudtemplateExtNetwork name="default" status="">
                    <cloudRegionName provider="azure" region="eastus" status=""/>
                    <cloudRegionName provider="azure" region="westus" status=""/>
                    <cloudtemplateVpnNetwork name="default">
                        <cloudtemplateIpSecTunnel peeraddr="22.22.219.104"/>
                        <cloudtemplateOspf area="0.0.0.1"/>
                    </cloudtemplateVpnNetwork>
                </cloudtemplateExtNetwork>
                <cloudtemplateIntNetwork name="default">
                    <cloudRegionName provider="azure" region="eastus" status=""/>
                    <cloudRegionName provider="azure" region="westus" status=""/>
                </cloudtemplateIntNetwork>
                <cloudtemplateHubNetwork name="default" status="">
                    <cloudtemplateHubNetworkName name="default" asn="64514"/>
                </cloudtemplateHubNetwork>
            </cloudtemplateInfraNetwork>
    </fvTenant>
</polUni>
Step 2

Configure the cloud context profile for the user tenant using a post such as the following, where the information used to configure Azure VNet peering is highlighted in bold.

Note 

The default value of “type” for cloudRsCtxProfileToGatewayRouterP is “spoke”, so it does not have to be specified when posting a user tenant.


<cloudCtxProfile name="c1" status="">
    <cloudRsCtxProfileToGatewayRouterP tDn="uni/tn-infra/gwrouterp-default" status=""/>
    <cloudRsCtxProfileToRegion status="" tDn="uni/clouddomp/provp-azure/region-westus"/>
    <cloudRsToCtx tnFvCtxName="VRF1"/>
    <cloudCidr name="cidr1" addr="11.0.0.0/16" primary="yes" status="">
        <cloudSubnet ip="11.0.0.0/24" status="">
            <cloudRsZoneAttach status="" tDn="uni/clouddomp/provp-azure/region-westus/zone-default"/>
        </cloudSubnet>
    </cloudCidr>
</cloudCtxProfile>


Verifying the Azure VNet Peering Deployment

After you configure Azure VNet peering, you should verify that it is deployed correctly.

Before you begin

You must have set up the cloud site and configured the cloud context profile.

Procedure


Step 1

Log into Cisco Cloud Application Policy Infrastructure Controller (Cloud APIC).

Step 2

In the left navigation pane, choose Infrastructure > Inter-Region Connectivity.

Step 3

Verify that the information shown in the Virtual Network Peering area is correct.

Step 4

Click on the count (the blue number 2 in the example above) to display more detailed information.

This will display peerings from the hub VNet to the tenant VNets.

Step 5

In the left navigation pane, choose Cloud Resources > Virtual Network Peers, then click the Cloud Resources tab.

The information displayed in this window shows the bi-directional peerings.

Step 6

To check routes, in the same page, click the Overview tab.

UDRs (routes) for peering are driven by contracts defined between EPGs. Check the configuration for contracts if routes are not present.

For each subnet shown in the main window, you can also see the current route table, as shown in the following figure.