Software-Defined Access for Distributed Campus

Solution Adoption Prescriptive Reference — Deployment Guide

July, 2019

NOTE: Works with document’s Advanced Properties “Last Updated” property. Click File | Properties | Advanced Properties | Custom.

                                                                                                                                                   

 

 

 

Table of Contents

Introduction. 5

About Cisco DNA Center 5

About Cisco Digital Network Architecture. 5

About Software-Defined Access. 6

About This Guide. 6

What is Covered in This Guide?. 7

What is Not Covered in This Guide?. 7

Nomenclature Conventions. 7

Define. 8

Fabric Architectural Components. 8

About Cisco Software-Defined Access for Distributed Campus. 9

Design. 10

SD-Access Transit 10

Transit Control Plane Nodes. 10

Transit Control Plane Deployment Location. 10

Key Considerations. 10

Topology Overview.. 11

Identity Services Engine and Shared Services Overview.. 12

Deploy. 13

Process 1: Discovering the Network Infrastructure. 13

Procedure 1: Discover Network Devices – Discovery Tool 13

Procedure 2: Define the Network Device Role – Inventory Tool 19

Process 2: Integrating Cisco DNA Center with the Identity Services Engine. 20

Procedure 1: Configure Authentication and Policy Servers in Cisco DNA Center 20

Process 3: Using the Design Application. 25

Procedure 1: Create the Network Hierarchy. 25

Procedure 2: Define Network Settings and Services. 28

Procedure 3: Create a Global IP Address Pool 33

Procedure 4: Reserve IP Address Pools. 34

Procedure 5: Design Enterprise Wireless SSIDs for SD-Access Wireless. 36

Procedure 6: Design a Guest Wireless SSID for SD-Access Wireless. 39

Process 4: Creating Segmentation with the Cisco DNA Center Policy Application. 42

Procedure 1: Add an Enterprise Overlay Virtual Network – Macro Segmentation. 42

Procedure 2: Add a Guest Overlay Virtual Network – Macro Segmentation. 43

Procedure 3: Create Group-Based Access Control Policies using SGTs – Micro Segmentation. 44

Process 5: Deploying SD-Access with the Provision Application. 47

About Provisioning in SD-Access. 48

Procedure 1: Create Credentials in ISE. 48

Procedure 2: Assign Network Devices to Site and Provision Network Settings. 49

Procedure 3: Assign Wireless LAN Controller to Site and Provision Network Settings. 52

Process 6: Provisioning the Fabric Overlay. 54

Procedure 1: Create IP-Based Transits – Fabric Provisioning Part I 55

Procedure 2: Create SD-Access Transit – Fabric Provisioning Part I (Continued) 56

Procedure 3: Create a Fabric Domain – Fabric Provisioning Part 2. 58

Procedure 4: Add Fabric-Enabled Sites to the Fabric Domain – Fabric Provisioning Part 2 (Continued) 59

Process 7: Assigning Fabric Roles – Fabric Provisioning Part 3. 61

About Fabric Border Nodes. 61

Procedure 1: Provisioning a Site Fabric Overlay. 62

Procedure 2: Provisioning Internal Border Nodes. 69

Procedure 3: Provisioning Redundant Layer-3 Handoffs. 73

Process 8: Configuring Host Onboarding – Fabric Provisioning Part 4. 76

About Authentication Templates. 77

Procedure 1: Assign Authentication Template and Create Wired Host Pool – Host Onboarding Part 1. 78

Procedure 2: Create INFRA_VN Host Pool – Host Onboarding Part 2. 79

Procedure 3: Assign SSID Subnet and Assign Access Point Ports– Host Onboarding Part 3. 80

Operate. 83

Process 1: Providing Access to Shared Services. 83

About Fusion Routers. 83

Procedure 1: Use the Provision Application for BGP and VRF-Lite information – Part 1. 84

Procedure 2: Configure VRF Definitions on the Fusion Routers – Command Runner Tool 85

Procedure 3: Create Layer-3 Connectivity Between Border Nodes and Fusion Routers – Fusion Routers Part I 88

Procedure 4: Establish BGP Adjacencies Between Fusion Routers and Border Nodes – Fusion Routers Part 2. 91

Procedure 5: Use Route-Targets to Leak Shared Services Routes – Fusion Routers Part 3. 94

Process 2: Provisioning and Verifying Access Points. 95

Procedure 1: Access Point Verification. 98

Process 3: Creating IBGP Adjacencies Between Redundant External Border Nodes. 99

Procedure 1: Configure IBGP – Routing Platforms. 100

Process 4: Providing Internet Access for SD-Access for Distributed Campus. 102

Procedure 1: Provision the Connected-to-Internet Border Node. 103

Internet Edge Topology Overview.. 109

Configuring Internet Connectivity. 110

Procedure 2: Use the Provision Application for BGP and VRF-Lite information – Part 2. 110

Procedure 3: Create Layer-3 Connectivity Between Border Nodes and Internet Edge Routers. 111

Procedure 4: Create BGP Adjacencies Between Border Nodes and Internet Edge Routers. 114

About Default Route Advertisement Using BGP. 117

Procedure 5: Advertise the Default Route From the Internet Edge Routers to Fabric VNs. 118

Procedure 6: Verify Default Route Advertisement 118

Procedure 7: Verify Default Route Via LISP – Advanced and Optional 120

Appendix A: Hardware and Software Code Versions. 127

Appendix B: Additional References. 128

Appendix C: Advanced Topology Diagrams. 129

High-Level Overview.. 129

Enterprise Architecture Model Topology. 129

Underlay Connectivity. 130

Overlay Connectivity. 131

BGP Autonomous System Overview.. 132

Fabric Role Overview.. 133

Site-1 Fabric Roles. 133

Site-2 Fabric Roles. 134

Site-3 Fabric Roles. 135

Site-4 Fabric Roles. 136

Site-5 Fabric Roles. 137

Branch Fabric Roles. 138

Layer-2 Overview.. 139

Site-1 Layer-2. 139

Site-2 Layer-2. 140

Site-3 Layer-2. 141

Site-4 Layer-2. 142

Site-5 Layer-2. 143

Branch Layer-2. 144

Layer-3 Overview.. 145

Individual Sites Loopback IP Schema. 145

IP Address Pools. 151

Shared Services and Internet Access. 153

Internet Access. 153

Shared Services. 154

Appendix D: Route Leaking. 155

Procedure 1: Create Layer-3 Connectivity Between Border Nodes and Fusion Routers. 155

Procedure 2: Establish BGP Adjacencies Between Fusion Routers and Border Nodes. 158

Procedure 3: Create IP Prefix lists to Match Fabric Subnets. 160

Procedure 4: Create Route-maps that Match IP Prefix Lists. 160

Procedure 5: Leak routes. 161

Appendix E: IBGP Between Redundant Devices. 164

IBGP for Switching Platforms. 164

About this guide. 168

Feedback & Discussion. 168

 

 

About Cisco DNA Center

Cisco DNA Center is the foundational controller and analytics platform at the heart of Cisco’s Intent-Based Network (IBN) for large and midsize organizations. Intent-based networking embodies the difference between a network that needs continuous attention and one that simply understands what the organization needs and makes it happen. It is the difference between doing thousands of tasks manually and having an automated system that helps focus on business goals. Cisco DNA Center provides a centralized management dashboard for complete control of this new network.  With this platform, IT can simplify network operations, proactively manage the network, provide consistent wired and wireless policy, and correlate insights with contextual cognitive analytics. 

Cisco DNA Center is a dedicated hardware appliance powered through a software collection of applications, processes, services, packages, and tools, and it is the centerpiece for Cisco^® Digital Network Architecture (Cisco DNA™).  This software provides full automation capabilities for provisioning and change management, reducing operations by minimizing the touch time required to maintain the network.  It also provides visibility and network assurance through intelligent analytics that pull telemetry data from everywhere in the network.  This reduces troubleshooting time and addresses network issues through a single pane of management. 

This interconnection of automation and assurance forms a continuous validation-and-verification loop, driving business intent and enabling faster innovation, lower cost and complexity, and enhanced security and compliance.  

About Cisco Digital Network Architecture

Cisco^® Digital Network Architecture (Cisco DNA™) provides a roadmap for digitization and a path to realization of the immediate benefits of network automation, assurance, and security. Cisco DNA is an open, extensible, software-driven architecture that accelerates and simplifies enterprise network operations while enabling business requirements to be captured and translated into network policy.  The result is constant alignment of the network to the business intent.

Cisco DNA begins with the foundation of a digital-ready infrastructure that includes routers, switches, access-points, and wireless LAN controllers (WLC).  The Identity Services Engine (ISE) is the key policy manager for the Cisco DNA solution.  The centerpiece of Cisco DNA is the Cisco DNA Center controller which empowers simplified workflows using the Design, Provision, Policy, and Assurance applications. 

Figure 1       Cisco DNA Center Dashboard

About Software-Defined Access

Cisco Software-Defined Access (SD-Access) is the Cisco DNA evolution from traditional campus LAN designs to networks that directly implement the intent of an organization. It is the intent-based networking solution for the Enterprise built on the principles of Cisco DNA.  The SD-Access solution is enabled by an application package that runs as part of the Cisco DNA Center software and provides automated end-to-end segmentation to separate user, device, and application traffic.  These user access policies are automated so that organizations can ensure that the right policies are established for any user or device with any application anywhere in the network.

SD-Access uses logic blocks called fabrics which leverage virtual network overlays that are driven through programmability and automation to create mobility, segmentation, and visibility.  Network virtualization becomes easy to deploy through software-defined segmentation and policy for wired and wireless campus networks.  Single physical networks are abstracted and can host one or more logical networks which are orchestrated through software.  Error-prone manual operations in these dynamic environments are circumvented altogether, providing consistent policy for users as they move around the wired and wireless network.

About This Guide

This guide provides technical guidance for designing, deploying, and operating Software-Defined Access for Distributed Campus.  It focuses on Cisco DNA Center to deploy the solution after the initial bootstrap of the network and supporting infrastructure is complete.

Figure 2       This Guide’s Four Sections

This guide contains four major sections:

Step 1:          The DEFINE section defines the problem being solved with the SD-Access for Distributed Campus solution and provides information on planning for the deployment and other considerations.

Step 2:          The DESIGN section shows a typical deployment topology and discusses Identity Services Engine and shared services considerations.

Step 3:          The DEPLOY section provides information and steps for the various workflows to deploy the solution along with recommended practices.

Step 4:          The OPERATE section demonstrates the manual configurations necessary for shared services, Internet access, and BGP between redundant device types.

 

What is Covered in This Guide?

This guide provides guidance to SD-Access customers deploying a unified and automated policy across multiple physical locations in a metro-area network.  The process, procedures, and steps listed in this guide are working configurations verified with the Cisco DNA Center, ISE, IOS, IOS-XE, and AireOS code versions listed in Appendix A.

What is Not Covered in This Guide?

Although this deployment guide is about Cisco DNA Center and SD-Access, it does not cover the initial bootstrap and installation of the Appliance or of the ISE distributed deployment.  Shared services installation and deployment, such as DHCP and DNS, along with the network connectivity configuration between various infrastructure components, routers, and switches, are not specifically covered.  SWIM (Software Image Management), LAN automation, multicast, Layer-2 handoff, fabric-in-a-box, and Cisco Catalyst 9800 embedded wireless on Catalyst series switches are also not covered.  

For more information on these items, please see additional references in Appendix B.

Nomenclature Conventions

Known routes, destinations, and prefixes are locations both inside the fabric and in the shared services Domains (DHCP, DNS, WLC, and ISE) that are registered with and known to the fabric control plane nodes. 

Unknown routes, destinations, and prefixes are locations on the Global Internet that are not known or registered with the fabric control plane nodes.

LISP encapsulation or LISP data plane encapsulation in the context of SD-Access refers to the VXLAN-GPO encapsulation.  For brevity, this may be referred to as VXLAN encapsulation.  However, this is not meant to indicate or infer that the encapsulation method is the same VXLAN in RFC 7348 or associated with VXLAN MP-BGP EVPN.

This section provides a high-level overview of the Software-Defined Access solution and components with a focus on elements related to distributed campus.

Fabric Architectural Components

The SD-Access 1.2 solution supports provisioning of the following fabric constructs:

Fabric edge node: Equivalent of an access layer switch in a traditional campus LAN design.  Endpoints, IP phones, and access points are directly connected to edge nodes.

Fabric control plane node: Based on the LISP Map-Server (MS) and Map-Resolver (MR) functionality. The control plane node’s host tracking database tracks all endpoints in a fabric site and associates the endpoints to fabric nodes in what is known as an EID-to-RLOC binding in LISP.

Fabric border node: Serves as the gateway between the SD-Access fabric site and networks external to the fabric.  The border node is the device physically connected to a transit or to a next-hop device connected to the outside world. 

Fabric site: An independent fabric that includes a control plane node and edge node and usually includes an ISE Policy Service Node (PSN) and fabric-mode WLC.  A fabric border node is required to allow traffic to egress and ingress the fabric site.

Virtual Network (VN):  Ostensibly a VRF definition.  VNs are created in the Policy application and provisioned to the fabric nodes as a VRF instance.  VN and VRF are used interchangeably in this document.

Scalable Group Tag (SGT):  A Cisco TrustSec component that operates as a form of metadata to provide logical segmentation based on group membership.

Transit: Connects a fabric site to an external network (IP-Based transit) or to one or more fabric sites (SD-Access transit).  IP-Based transit networks connect the fabric to external networks using VRF-lite.  SD-Access transits carry SGT and VN information inherently carrying policy and segmentation between fabric sites and do not require or use VRF-lite.

Fabric domain: Encompasses one or more fabric sites and any corresponding transit(s) associated with those sites. 

Transit control plane node: The transit control plane node’s database tracks all aggregate routes for the fabric domain and associates these routes to fabric sites. Its functionality is based on LISP Map-Server (MS) and Map-Resolver (MR).

Fabric in a Box: Combines the fabric border node, fabric control plane node, and fabric edge node functionality on the same switch or switch stack.

Host Pool: The binding of a reserved IP address pool to a Virtual Network which associates a segment to a VRF.

 

 

About Cisco Software-Defined Access for Distributed Campus

Cisco Software-Defined Access (SD-Access) for Distributed Campus is a metro-area solution that connects multiple, independent fabric sites together while maintaining the security policy constructs (VRFs and SGTs) across these sites.  While multi-site environments and deployments have been supported with SD-Access for some time, there has not been an automated and simplistic way to maintain policy between sites.  At each site’s fabric border node, fabric packets were de-encapsulated into native IP.  Combined with SXP, policy could be carried between sites using native encapsulation.  However, this policy configuration was manual, mandated use of SXP to extend policy between sites, and involved complex mappings of IP to SGT bindings within the Identity Services Engine. 

With SD-Access for Distributed Campus, SXP is not required, the configurations are automated, and the complex mappings are simplified.  This solution enables inter-site communication using consistent, end-to-end automation and policy across the metro network.

Software-Defined Access for Distributed Campus uses control plane signaling from the LISP protocol and keeps packets in the fabric VXLAN encapsulation between fabric sites.  This maintains the macro- and micro-segmentation policy constructs of VRFs and SGT, respectively, between fabric sites.  The original Ethernet header of the packet is preserved to enable the Layer-2 overlay service of SD-Access Wireless.  The result is a network that is address-agnostic because policy is maintained through group membership. 

This section introduces the SD-Access transit and transit control plane nodes along with key considerations, shows the deployment topology, and discusses the Identity Services Engine and shared services.

SD-Access Transit

The core components enabling the Distributed Campus solution are the SD-Access transit and the transit control plane nodes.  Both are new architectural constructs introduced with this solution.  The SD-Access transit is simply the physical metro-area connection between fabric sites in the same city, metropolitan area, or between buildings in a large enterprise campus. 

The key consideration for the Distributed Campus design using SD-Access transit is that the network between fabric sites and to Cisco DNA Center should be created with campus-like connectivity. The connections should be high-bandwidth (Ethernet full port speed with no sub-rate services), low latency (less than 10ms as a general guideline), and should accommodate the MTU setting used for SD-Access in the campus network (typically 9100 bytes).  The physical connectivity can be direct fiber connections, leased dark fiber, Ethernet over wavelengths on a WDM system, or metro Ethernet systems (VPLS, etc.) supporting similar bandwidth, port rate, delay, and MTU connectivity capabilities.

Transit Control Plane Nodes

The transit control plane nodes track all aggregate routes for the fabric domain and associate these routes to fabric sites.  When traffic from an endpoint in one site needs to send traffic to an endpoint in another site, the transit control plane node is queried to determine to which site’s border node this traffic should be sent.  The role of transit control plane nodes is to learn which prefixes are associated with each fabric site and to direct traffic to these sites across the SD-Access transit using control-plane signaling. 

Transit Control Plane Deployment Location

The transit control plane nodes do not have to be physically deployed in the Transit Area (the metro connection between sites) nor do they need to be dedicated to their own fabric site, although common topology documentation often represents them in this way. For the prescriptive configuration in this guide, a pair of ISR 4451-X routers were used as the transit control plane nodes.  These routers were deployed in their own dedicated site, only accessible through the SD-Access transit Metro-E network, although this is not a requirement. 

While accessible only via the Transit Area, these routers do not act as a physical-transit hop in the data packet forwarding path.  Rather, they function similarly to a DNS server in that they are queried for information, even though data packets do not transit through them.  This is a key consideration. 

Key Considerations

The transit between sites is best represented and most commonly deployed as direct or leased fiber over a Metro Ethernet system.  While Metro-E has several different varieties (VPLS, VPWS, etc.), the edge routers and switches of each site ultimately exchange underlay routes through an Interior Gateway Routing (IGP) protocol.  In SD-Access, this is commonly done using the IS-IS routing protocol, although other IGPs are supported.

IP reachability must exist between fabric sites.  Specifically, there must be a known underlay route between the Loopback 0 interfaces on all fabric nodes.  Existing BGP configurations and peerings on the transit control plane nodes could have complex interactions with the Cisco DNA Center provisioned configuration and should be avoided.  The transit control plane nodes should have IP reachability to the fabric sites through an IGP before being discovered or provisioned.

Traversing the transit control plane nodes in the data forwarding path between sites has not been validated.  Transit control plane nodes should always be deployed as a pair of devices to provide resiliency and high availability.

Topology Overview

Figure 3       Underlay Connectivity – High Level Overview

This deployment guide topology contains twelve fabric sites in a metropolitan area.  It is a large enterprise campus with dispersed buildings in a similar geographic area with each building operating as an independent fabric site.  Each site is connected to a Metro-E circuit at the Enterprise Edge block. A thirteenth site is the Data Center where shared services are located along with the Cisco DNA Center cluster.   The Data Center site has a direct connection to both the Metro-E circuit and to Site-1 in the network.  Site-3 is the main head-end location (represented as Headquarters – HQ) and is connected to both the Metro-E circuit and the upstream Internet edge routers which themselves connect to the Internet edge firewalls.   NAT and ultimately access to the Internet is provided through these firewalls.  The result is that all traffic directed to the Internet from any site must be carried across the Metro-E circuit and then egressed out of Site-3 (HQ). 

Direct Internet access (DIA) is not required at each site, although DIA is deployed depending on the design and deployment needs.  DIA per site is more common in a WAN deployment than a MAN deployment and therefore the configuration has not been validated in this document. 

The Enterprise edge block routers and switches for each site, which ultimately become fabric border nodes, are shown as directly connected to the Metro-E circuit as they are connected to the Service Provider equipment providing access to this circuit. The transit control plane nodes are shown located in the private circuit cloud.  Like every other site, except for HQ, they are accessible only through the Metro-E circuit.  Like the Enterprise edge block routers and switches at each site, the transit control plane nodes are connected to the Service Provider equipment accessing the circuit. 

Within each site, the IS-IS routing protocol is used to advertise underlay routes and to provide IP reachability between Loopback 0 addresses.  Through the Metro-E circuit, the Enterprise Edge routers and switches (fabric border nodes) at each site are IS-IS adjacent with each other so that there is full, end-to-end IP reachability between sites across the private circuit.  

Identity Services Engine and Shared Services Overview

Figure 4       Overlay Connectivity – High Level Overview

The Identity Services Engine environment is configured as a distributed deployment.  There are multiple ISE virtual machines, each running an independent persona.   The deployment consists of two Primary Administration Nodes (PAN), two Monitoring and Troubleshooting Nodes (MnT), and two Platform Exchange Grid (pxGrid) nodes with each pair set up as primary and secondary node.  There is a Policy Services Node (PSN) dedicated to each fabric site in the network.  The ISE nodes are all physically located in the Data Center site.  This is not mandatory, and an optional deployment variation would include deploying the ISE PSNs physically at each site location.

Windows Active Directory (AD) is configured as an external identity store for the ISE deployment.  AD is running on a Windows Server 2016 that is also providing DHCP and DNS services to the entire deployment. 

To satisfy the asymmetric packet path necessary in SD-Access Wireless for Access Point to WLC registration, WLC to control plane node registration, and AP to DHCP server, the Data Center location has underlay reachability through a direct link to the Metro-E circuit along with overlay reachability to the deployment through a pair of fusion routers ultimately connected to a pair of Internal-Only fabric border nodes located in Site-1. 

The Fusion Routers are a pair of Catalyst switches connected upstream to the distribution block at Site-1 and downstream to the data center switches which provide reachability to the UCS servers hosting the management virtual machines. 

This section focuses on the complete deployment workflow and guidelines, from device discovery through to fabric automation. 

Using the Cisco DNA Center GUI, the network is discovered and added to the inventory.  Cisco DNA Center is then integrated with the Identity Services Engine through mutual certificate authentication creating a trusted relationship.  The Design application is used to create site hierarchies, define common network settings, define IP address pools, configure WLAN settings, and set up the ISE guest portal. 

The Policy application is used to create Scalable Group Tags (SGTs) and configure TrustSec related policies.  This workflow is managed in Cisco DNA Center and pushed to ISE using the trust established during integration. 

Finally, the Provision application is used to provision the configuration defined in the Design and Policy workflows to the managed devices discovered and present in the inventory.  The Provision application is also used to create and define the SD-Access overlay network.

Process 1: Discovering the Network Infrastructure

Procedure 1: Discover Network Devices – Discovery Tool

Cisco DNA Center is used to discover and manage the SD-Access underlay network devices.  To discover equipment in the network, the Appliance must have IP reachability to these devices, and CLI and SNMP management credentials must be configured on them. Once discovered, the devices are added to Cisco DNA Center’s inventory, allowing the controller to make configuration changes through provisioning.

The following steps show how to initiate a discovery job by supplying an IP address range or multiple ranges for scanning for network devices.  IP address range discovery provides a small constraint to the discovery job which may save time.  Alternatively, by providing the IP address of an initial device for discovery, Cisco DNA Center can use Cisco Discovery Protocol (CDP) to find neighbors connected to the initial device.

1.     In the Cisco DNA Center dashboard, click Discovery in the Tools section. 

2.     In the New Discovery dialog, supply a Discovery Name (example: Site-1).

3.     Select the Range radio button.

4.     Enter an IP address for the start (From) and end (To) of the IP range (for example, 192.168.10.1, 192.168.10.8).

For a single address, enter the same IP address for both the start and end of the range.

5.     Select Use Loopback for the Preferred Management IP.

6.     Scroll down to the Credentials section. 

7.     Click + Add Credentials on the far right.

The Add Credentials slide-in pane appears.

8.     Select the CLI tab.

9.     Provide a Username and Password and then confirm the Password. 

10.  Provide a password in the Enable Password field and confirm it. 

11.  (Optional) Select the ☑ Save as global settings check box

12.  Click Save.

 

13.  Select the relevant SNMP tab. 

14.  For SNMPv2c, click Read.

15.  Enter a name/description.

16.  Enter a Read Community string and confirm it.

17.   (Optional) Select the ☑ Save as global settings check box.

18.  Click Save.

19.  For SNMPv2c, click Write.

20.  Enter a name/description.

21.  Enter a Write Community string and confirm it.

22.   (Optional) Select the ☑ Save as global settings check box.

23.  Click Save.

 

24.  For SNMPv3, select the SNMPv3 tab.

25.  Select the Mode (example: Authentication and Privacy), the Authentication Type (example: SHA), and the Privacy Type (example: AES128).

26.  After making these selections, fill in the Username, the Auth. Password, and the Privacy Password.

27.  (Optional) Select the ☑ Save as global settings check box.

28.  Click Save.

29.  Repeat the previous steps for any additional credentials required in the network.

30.  Close the Add Credentials dialog.

31.  Verify that the defined Credentials are displayed in the Discovery job. 

32.  Scroll down and click Advanced to expand the section. 

33.  Under Protocol Order, ensure that SSH is ☑ selected and is above Telnet in the protocol order. 

34.  Click Start.

The discovery details are displayed while the Discovery runs.

 

 

Figure 5       Site-1 Discovery Job Details

35.  If there are any discovery failures, inspect the devices list, resolve the problem, and restart the discovery for those devices.

36.  Repeat these steps to discover additional devices in the network and add them to the inventory.

 

Figure 6       Transit Control Plane Node Discovery Job

Procedure 2: Define the Network Device Role – Inventory Tool

The Cisco DNA Center Inventory tool is used to add, update, and/or delete devices that are managed by the Appliance. The Inventory tool can also be used to perform additional functions and actions, such as updating device credentials, resyncing devices, and exporting the device configurations. It also provides the ability to launch the Command Runner tool.  

The Inventory tool has many uses and significant device information is collected and exported by exposing the various columns. However, the tool has a single use in this deployment guide – modifying the device role.  

About Device Roles

The device role is used to position devices in the Cisco DNA Center topology maps under the Fabric tab in the Provision application and in the Topology tool.  The device positions in these applications and tools are shown using the classic three-tiered Core, Distribution, and Access layout.  

Changing a device role does not change the function of the device, nor does it impact the SD-Access configuration that is provisioned.  In SD-Access, it is simply a GUI construct to help with network layout visualization shown in later steps.

Table 1    Device Roles

 

To define the device role:

1.     After the discovery process finishes successfully, navigate to the main Cisco DNA Center dashboard, and click Inventory in the Tools section. The discovered devices are displayed.

2.     Verify that the Last Sync Status of the devices is Managed before making device role changes.

 

 

3.     To update the device role, click the blue pencil icon in the Device Role column, and select the required role.

The following device roles are assigned to the devices discovered in Site-1 and are used to create the Fabric topology layout shown later in this document.

Figure 7       Device Roles - Inventory Tool – Site-1

Process 2: Integrating Cisco DNA Center with the Identity Services Engine

The Identity Services Engine (ISE) is the authentication and policy server required for Software-Defined Access.  Once integrated with Cisco DNA Center using pxGrid, information sharing between the two platforms is enabled, including device information and group information.  This allows Cisco DNA Center to define policies that are pushed to ISE and then rendered into the network infrastructure by the ISE Policy Service Nodes (PSNs).

When integrating the two platforms, a trust is established through mutual certificate authentication.  This authentication is completed seamlessly in the background during integration and requires both platforms to have accurate NTP sync. 

Procedure 1: Configure Authentication and Policy Servers in Cisco DNA Center

The following steps outline how to integrate Cisco DNA Center and the Identity Services Engine. This integration must be done before completing the workflows in the Design and Policy applications so that the results of the workflows can be provisioned to the network equipment to use ISE as the AAA server for users and endpoints via RADIUS and for device administrators via TACACS+.

 

During the integration, all Scalable Group Tags (SGTs) present in ISE are pulled into Cisco DNA Center. Whatever policy is currently configured in the TrustSec matrices of ISE are also pulled into Cisco DNA Center. This is sometimes referred to as Day 0 Brownfield Support: if policies are present in ISE at the point of integration, those policies are populated in the Cisco DNA Center Policy application.

1.     From the main Cisco DNA Center dashboard select the ⚙gear icon in the top-right corner.

2.     Click System Settings.

3.     Navigate to Settings > Authentication and Policy Servers.

4.     Click the + Add button.

 

5.     In the Add AAA/ISE dialog, enter the IP Address of the ISE Primary Administration Node (PAN) in the Server IP Address field (example: 198.51.100.121).

6.     Enter a Shared Secret.

7.     Toggle the Cisco ISE selector to On to reveal additional settings required for integration with ISE.

8.     Enter the ISE Super Admin Username and Password.

9.     Enter the ISE fully qualified domain name for FQDN field.  

10.  Enter a meaningful Subscriber Name (example: dnac) as it will be displayed later in the ISE GUI.

11.  Leave the SSH Key blank. 

12.  Click View Advanced Settings to expand it.

13.  Select the ☑ RADIUS and ☑ TACACS protocols.

14.  Click Apply.

15.  During the establishment of communication, Cisco DNA Center displays Creating AAA server…

Verify that the Status displays as INPROGRESS.

16.  Log in to the ISE GUI.

17.  Navigate to Administration > pxGrid Services.

18.  Please wait at minimum of three minutes.

19.  The client named dnac (the value entered as the Subscriber name) is now showing Pending in the Status column.

20.  If the Subscriber has not appeared, please use the green refresh button.

Integration may take as long as ten minutes although it commonly takes less than five minutes.

21.  Select the checkbox beside the subscriber (dnac).

22.  In the Total Pending Approval drop-down, select Approve All.

23.   Click the second Approve All to double confirm.

24.  A success message displays.

Verify that the subscriber status changes to Online (XMPP).

 

                                                                                                                     

Process 3: Using the Design Application

Cisco DNA Center provides a robust Design application to allow customers of varying sizes and scales to easily define their physical sites and common resources. Using an intuitive hierarchical format, the Design application removes the need to redefine the same resource – such as DHCP, DNS, and AAA servers – in multiple places when provisioning devices. The network hierarchy created in the Design application should reflect the actual physical network hierarchy of the deployment.

The Design application is the building block for every other workflow in both Cisco DNA Center and Software-Defined Access.  The configuration and items defined in this section are used and provisioned in later steps.  The Design application begins with the creation of the network hierarchy of Areas, Buildings, and Floors.  Once the hierarchy has been created, network settings are defined.  These include DHCP and DNS servers, AAA servers and NTP servers, and when applicable, SNMP, Syslog, and Netflow servers.  These servers are defined once in the Design application and provisioned to multiple devices in later steps.  This allows for faster innovation without the repetitive typing of the same server configuration on the network infrastructure.  After network settings are defined, IP address pools are designed, defined, and reserved.  These pools are used for automation features such as border handoff and LAN automation and are used in host onboarding in later steps.  The final step in the Design application in this guide is the creation and configuration of the wireless network settings including a Guest Portal in ISE.

Procedure 1: Create the Network Hierarchy

Using Cisco DNA Center, create a network hierarchy of areas, buildings, and floors that reflect the physical deployment.  In later steps, discovered devices are assigned to buildings so that they are displayed hierarchically in the topology maps using the device role defined earlier in the Inventory tool.

Areas are created first.  Within an Area, sub-areas and buildings are created.  To create a building, the street address must be known to determine the coordinates and thus place the building on the map.  Alternatively, use latitude and longitude coordinates without the street address.  Floors are associated with buildings and support the importation of floor maps.  For SD-Access Wireless, floors are mandatory as this is where Access Points are assigned, as clarified in later steps.  Buildings created in these steps each represent a fabric site in the later Provisioning application procedures.

To create the network hierarchy:

1.     From the main Cisco DNA Center dashboard, click the DESIGN tab.

2.     Select the Network Hierarchy tab.

3.     Click +Add Site to begin the network hierarchy design. 

4.     Select Add Area from the drop-down list.

5.     Supply an appropriate Area Name (example: San Jose).

6.     Click Add.

7.     Click +Add Site.

8.     Select Add Building from the drop-down list.

9.     Provide an appropriate Building Name (example: SJC23).

10.  Select the area created in the previous step as the Parent.

11.  Complete the Address or Latitude and Longitude information to assign a location.

12.   Click Add.

13.  Repeat the previous steps to add sites and buildings, creating a hierarchy that reflects the physical network hierarchy and topology for the organization.

14.  To support SD-Access Wireless, select the gear icon next to a building in the hierarchy, choose Add Floor, and complete the wizard with the applicable details. 

 

Procedure 2: Define Network Settings and Services

In Cisco DNA Center, common network resources and settings are saved in the Design application’s Network Settings tab. Saving allows information pertaining to the enterprise to be stored so it can be reused throughout various Cisco DNA Center workflows.  Items are defined once so that they can be used many times.

Configurable network settings in the Design application include AAA server, DHCP server, DNS server, Syslog server, SNMP server, Netflow collector, NTP server, time zone, and Message of the Day.  Several of these items are applicable to a Cisco DNA Center Assurance deployment.  For SD-Access, AAA and DHCP servers are mandatory and DNS and NTP servers should always be used.

By default, when clicking the Network Settings tab, newly configured settings are assigned as Global network settings. They are applied to the entire hierarchy and inherited by each site, building, and floor.  In Network Settings, the default selection point in the hierarchy is Global. 

It is possible to define specific network settings and resources for specific sites.  In fact, each fabric site in this deployment has its own dedicated ISE Policy Service Node, as shown in the next steps.  For this prescriptive deployment, NTP, DHCP, and DNS have the same configuration for each site and are therefore defined at the Global level. 

1.     Navigate to DESIGN > Network Settings > Network. 

2.     Near the top, next to Network Telemetry, click + Add Servers.

3.     Select the ☑ NTP check box.

4.     Click OK.

 

 

5.     Within the left pane in the site hierarchy, select Global.

6.     Enter the DHCP server IP address (example: 198.51.100.30).

7.     Under DNS Server, enter the domain name (example: dna.local) and the IP address of the Primary DNS server (example: 198.51.100.30).

8.     Under NTP Server, enter the NTP Server IP address (example: 100.119.4.17).

9.     Click the + button under Additional NTP to add an additional NTP field.

10.  Enter a second NTP Server IP address (example: 100.119.4.18).

11.  Click Save.

12.  Navigate to a building (example: Site-1) in the hierarchy using the pane on the left. 

The DHCP, DNS, and NTP servers previously defined under Global are inherited at the site level. 

 

13.  Near the top, next to Network Telemetry, click + Add Servers.

14.  Select the ☑ AAA check box and verify that the NTP check box is already selected.

15.  Click OK. 

 

16.  Verify that the configuration pane is updated with AAA Server. 

17.  Under AAA Server select the ☑ Network and ☑ Client/Endpoint check boxes. 

18.  Under NETWORK > Servers, select the ISE radio button.

19.  In the Network drop-down, select the prepopulated ISE server. 

This is the IP address of the primary PAN. 

20.  Under Protocol, select the TACACS radio button.

21.  In the IP Address (Primary) drop-down, select the IP address of the ISE PSN for the site. 

22.  Under CLIENT/ENDPOINT > Servers, select the ISE radio button.

23.  In the Client/Endpoint drop-down, select the prepopulated ISE server. 

This is the IP address of the Primary PAN. 

24.  Under Protocol, ensure that the RADIUS radio button is selected.

25.  In the IP Address (Primary) drop-down, select the IP address of the ISE PSN for the site. 

26.  Click Save.

The ISE servers for AAA and the servers for DHCP, DNS, and NTP for the selected level in the hierarchy are saved for later provisioning steps.

27.  Repeat the steps above for each site in the Distributed Campus deployment. 

Figure 8       Site-2 Network Settings

Procedure 3: Create a Global IP Address Pool

This section provides information about global IP address pools and shows how to define the global IP address pools that are referenced during the pool reservation process.

About Global IP Address Pools

The Cisco DNA Center IPAM (IP Address Management) tool is used to create and reserve IP address pools for LAN and Border Automation, ENCS (NFV) workflows, and for the binding of the subnet segment to a VN in host onboarding.  IP address pools are defined at the global level and then reserved at the area, building, or floor level.  Reserving an IP address pool tells Cisco DNA Center to set aside that block of addresses for one of the special uses listed above. 

The DHCP and DNS servers are the same for every site in this deployment. Because of the number of pools and sites, not every pool is listed in the chart below. The full IP address pool list can be found in Appendix C.

Table 2    IP Address Pools – Site-1 through Site-4

 

 

IP address pools must be created at the global level first to be reserved later.  Because the deployment is not integrating with a third-party IPAM tool where pools may already be configured, a single, large aggregate block is created at the Global level.

To create a global IP address pool:

 

1.     Navigate to DESIGN > Network Settings > IP Address Pools.

2.     In the site hierarchy on the left, select Global, and click + Add IP Pool.

3.     Enter the IP Pool Name (example: ONE-SEVEN-TWO).

4.     Enter the IP Subnet (example: 172.16.0.0).

5.     Use the drop-down to select the applicable CIDR Prefix (example: /12).

6.     Enter the Gateway IP Address (example 172.16.0.1).

7.     Do not select Overlapping.

8.     Click Save.

Procedure 4: Reserve IP Address Pools

The defined global IP address pool is used to reserve pools at the area, building, or floor in the network hierarchy. For single-site deployments, the entire set of global IP address pools is reserved for that site.  In an SD-Access for Distributed Campus deployment, each site has its own assigned subnets that do not overlap between the sites.

To reserve IP address pools at the site level:

1.     In the Cisco DNA Center dashboard, navigate to DESIGN > Network Settings > IP Address Pools.

2.     In the site hierarchy on the left, select an area, building, or floor to begin the IP address pool reservation

(example: Site-1).

3.     In the top right, click + Reserve IP Pool to display the Reserve IP Pool wizard.  

4.     Enter the IP Pool Name (example: AP-SITE-01).

5.     In the Type drop-down, select LAN.

6.     In the Global IP Pool drop-down, select a global pool (example: ONE-SEVEN-TWO).

7.     Under CIDR Notation / No. of IP Addresses select the portion of the address space to use

(example: 172.16.110.0 and /24 (255.255.255.0)).

8.     Define the Gateway IP Address (example: 172.16.110.1).

9.     Use the drop-downs to select the previously defined DHCP Server(s) and DNS Servers(s).

10.  Click Reserve.

 

11.  Repeat the previous steps for all address blocks required to be reserved in the hierarchy for each site in the network.

 

 

The full reservations for Site-1 and Site-3 are shown below.

Figure 9       IP Address Pool Reservations – Site-1

Figure 10     IP Address Pool Reservations – Site-3

Procedure 5: Design Enterprise Wireless SSIDs for SD-Access Wireless

Cisco DNA Center is used to automate the creation of SSIDs, SSID security, AP profiles and parameters, and wireless Quality of Service (QoS) settings.  Defining these elements in Cisco DNA Center, rather than defining each of these (per SSID) in the GUI of the wireless LAN controller, saves time and reduces potential manual configuration errors.

The wireless workflow in the Design application is a two-step process:

1.     Create the SSID and its parameters.

2.     Create the wireless profile.

 

Creating the SSID involves defining the name along with the type of wireless network (voice, data, or both), wireless band, security type, and advanced options. 

The wireless profile defines whether the wireless network is fabric or non-fabric and defines to which sites and locations within the hierarchy the profile is assigned. Wireless configurations, like IP address pools, are configured at the global level of the hierarchy.

To create the Enterprise wireless network:

1.     In the Cisco DNA Center dashboard, navigate to DESIGN > Network Settings > Wireless.

2.     In the Enterprise Wireless section click + Add.

3.     In the Create an Enterprise Wireless Network wizard, complete the following steps:

 

a.        Enter the Wireless Network Name (SSID) (example: CORP).

b.       Under TYPE OF ENTERPRISE NETWORK, select Voice and Data and Fast Lane.

c.        Ensure BROADCAST SSID: is set to On.

d.       Select the WIRELESS OPTION (example: Dual band operation with band select).

e.       For LEVEL OF SECURITY, select WPA2 Enterprise.

f.         Under ADVANCED SECURITY OPTIONS, select Adaptive.

4.     Click Next to continue in the wizard to the Wireless Profiles section.

5.     Click the + Add next to Profiles.

6.     In the Create Wireless Profile dialog, complete the following steps:

 

a.        Enter a Wireless Profile Name (example: SD-Access Wireless).

b.       Under Fabric, select Yes.

c.        Click Sites to open an additional Sites dialog.

d.       Select the location(s) where the SSID broadcasts (example: San Jose)

e.       Click OK to close the Sites dialog.

f.         Click Add to close the Create Wireless Profile Wizard.

7.     Click Finish.

8.     Repeat this procedure for any additional Enterprise wireless SSIDs.

 

Procedure 6: Design a Guest Wireless SSID for SD-Access Wireless

Designing the guest wireless SSID in Cisco DNA Center is similar to designing the Enterprise wireless SSID.  The primary difference is the Guest Web Authentication section in the workflow.  Cisco DNA Center supports External Web Authentication and Central Web Authentication.

External Web Authentication uses the specified URL to redirect the guest users.  Central Web Authentication uses the Guest Portal sequence of the Identity Services Engine to redirect guest users to the captive portal hosted on ISE.

The Guest wireless SSID creation workflow is a three-step process:

1.     Create the SSID and its parameters.

2.     Create the wireless profile.

3.     Create the portal.

 

When creating the SSID, the SSID name is defined along with the Authentication Type which must be either Web Authentication/Captive Portal or Open.  The profile defines whether the wireless network is fabric or non-fabric and defines to which sites and locations within the hierarchy the profile is assigned.  The Guest Portal of ISE is customized and configured through Cisco DNA Center. 

To create the Guest wireless network:

1.     Navigate to DESIGN > Network Settings > Wireless.

2.     In the Guest Wireless section, click + Add. 

3.     In the Create a Guest Wireless Network wizard, complete the following steps:

a.        Enter the Wireless Network Name (SSID) (example: GUEST).

b.       Under LEVEL OF SECURITY, select Web Auth.

c.        Under AUTHENTICATION SERVER, select ISE Authentication.

d.       Leave the other default selections.

4.     Click Next to continue in the wizard to Wireless Profiles.

5.     In the Wireless Profiles section, select the Profile Name corresponding to the deployment

location (example: SD-Access Wireless). 

 

6.     In the slide-in pane, keep the default Fabric selection of Yes.

7.     Keep the other default selections. 

If a new profile is created, please select the Sites accordingly.

8.     At the bottom of the panel click Save.

9.     Click Next to continue in the wizard to Portal Customization.

10.  In the Portals screen, click + Add.

11.  The Portal Builder wizard appears.

Supply a Guest Portal name (example: GUEST-PORTAL).

12.  Make any desired customizations.

13.  Use the scroll bar to navigate to the bottom of the portal.

14.  Once there, click Save.

15.  A guest web authentication portal is generated for the site(s), and the GUI returns to the previous screen.

 

Click Finish to complete the guest wireless LAN design.

Process 4: Creating Segmentation with the Cisco DNA Center Policy Application

SD-Access supports two levels of segmentation – macro and micro. Macro segmentation uses overlay networks - VRFs.  Micro segmentation uses scalable group tags (SGTs) to apply policy to groups of users or devices.

In a University example, students and faculty machines may both be permitted to access printing resources, but student machines should not communicate directly with faculty machines, and printing devices should not communicate with other printing devices.  This micro-segmentation policy can be accomplished using the Policy application in Cisco DNA Center which leverages APIs to program the ISE TrustSec Matrix.

For a deeper exploration of designing segmentation for SD-Access with additional use cases, see the Software-Defined Access Segmentation Design Guide.

The Policy application supports creating and managing virtual networks, policy administration and contracts, and scalable group tag creation. Unified policy is at the heart of the SD-Access solution, differentiating it from others.  Therefore, deployments should set up their SD-Access policy (virtual networks and contracts) before doing any SD-Access provisioning.

The general order of operation for SD-Access is Design, Policy, and Provision, corresponding with the order of the applications seen on the Cisco DNA Center dashboard.

In this section, the segmentation for the overlay network is defined. (Note that the overlay network will only be fully created until the host onboarding stage).  This process virtualizes the overlay network into multiple self-contained virtual networks (VNs).  After VN creation, the TrustSec policies are created to define which users and groups within a VN are able to communicate.

Use these procedures as prescriptive examples for deploying macro and micro segmentation policies using Cisco DNA Center.

Procedure 1: Add an Enterprise Overlay Virtual Network – Macro Segmentation

Virtual networks are created first, then group-based access control policies are used to enforce policy within the VN.

1.     From the main Cisco DNA Center dashboard, navigate to POLICY > Virtual Network.

2.     Click the + to create a new virtual network.

3.     Enter a Virtual Network Name (example: CAMPUS).

4.     Drag scalable groups from the Available Scalable Groups pane into the Groups in the Virtual Network pane (example: Employees, Contractors, and Network Services).

5.     Click Save.

6.     Verify that the VN with associated groups is defined and appears in the list on the left. 

These virtual network definitions are now available for provisioning the fabric in later steps.

7.     Repeat this procedure for each overlay network.

Procedure 2: Add a Guest Overlay Virtual Network – Macro Segmentation

Creating the guest virtual overlay network is very similar to creating of enterprise overlay virtual network.  However, there must be a method to indicate that this virtual network is for guest access in order to trigger the use of the captive portal and guest SSID(s) created in earlier steps.

1.     From the main Cisco DNA Center dashboard, click the POLICY > Virtual Network.

2.     Click the + to create a new virtual network.  

3.     Enter a Virtual Network Name (example: GUEST).

4.     Drag scalable groups from the Available Scalable Groups pane into the Groups in the Virtual Network pane (example: GUEST).

5.     Select the ☑ Guest Virtual Network check box.

With the check box selected, this virtual network can be used for guest SSID, guest portals, and other guest-related workflows.

6.     Click Save.

About Group-Based Access Control Policies

SD-Access segments the physical network into virtual networks, and security policies are defined to segment traffic inside of the VNs. Cisco DNA Center allows the administrator to explicitly deny or allow traffic between groups (SGTs) within each virtual network. This policy is created in Cisco DNA Center, pushed to ISE, and then finally pushed down to the switches to enforce the policy.

Group-Based Access Control Components

The SD-Access 1.2 solution supports creation and provisioning of the following policy constructs:

Security Group Access Control List (SGACL): Policy enforcement through which the administrator can control the operations performed by the user based on the security group assignments and destination resources.  Cisco DNA Center refers to these as Group-Based Access Control policies to express the intent of these constructs.

Access Contract: Policy enforcement through which the administrator can control the operations performed by the user based on destination port and protocol information.

Procedure 3: Create Group-Based Access Control Policies using SGTs – Micro Segmentation

The following steps demonstrate how use Cisco DNA Center to create micro-segmentation security policies with just a few clicks.  This simple example shows a basic policy that can be used to deny users from the Contractors group from communicating with the Employees group.

Using a single check box, Cisco DNA Center deploys this policy in ISE bidirectionally – also preventing the Employees group from communicating with the Contractors group.  

1.     In Cisco DNA Center, navigate to POLICY > Group-Based Access Control > Group-Based Access Control Policies.

2.     Click + Add Policy. 

3.     From the Available Scalable Groups pane, drag the Contractors group and drop it into the Source pane.

4.     Drag the Employees group into the Destination pane.

5.     Input a Policy Name (example: DENY-CONTRACTORS-TO-EMPLOYEES).

6.     Enter an optional Description.

7.     Select ☑ Enable Policy.

8.     Select ☑ Enable Bi-directional.

9.     Click + Add Contract to show the available Access Contracts. 

10.  Select deny.

11.  Click OK.

12.  Click Save.

 

13.  Verify that the policy is created and listed with a status of DEPLOYED.

14.  Verify that the bidirectionally reverse policy is also created and DEPLOYED.

15.  The policies are now available to be applied to fabrics and are also available in ISE.

To view them and cross-launch ISE using Cisco DNA Center, navigate to POLICY > Group-Based Access Control > Group-Based Access Control Policies > Advanced Options.

16.  Log into ISE in the new browser tab that opens. 

This cross-launch opens directly to the ISE TrustSec Matrix.  

17.  Verify that the policy is programmatically pushed to ISE.

Process 5: Deploying SD-Access with the Provision Application

The Provision application in Cisco DNA Center is used to take the network components defined in the Design application and the segmentation created in the Policy application and deploy them to the devices.  The Provision application has several different workflows that build on one another.  This application can also be used for SWIM and LAN automation although these are beyond the scope of this document.

The process begins by assigning and provisioning devices to a site.  Once devices are provisioned to a site, the fabric overlay workflows can begin.  This starts through the creation of transits, the formation of a fabric domain, and the assignment of sites, buildings, and/or floors to this fabric domain.  Once assigned, Cisco DNA Center lists these locations as Fabric-Enabled.

Once a site is fabric-enabled, devices provisioned to that site can be assigned a fabric role.  This creates the core infrastructure for the fabric overlay.  Host onboarding is completed afterwards to bind VNs and reserved IP address pools together, completing the overlay configuration.

About Provisioning in SD-Access

Provisioning in the SD-Access solution consists of three workflows:

1.       Assign devices to site, building, or floor.

2.       Provision devices added to site, building, or floor.

3.       Provision devices to the fabric roles.

 

Assigning a device to a site causes Cisco DNA Center pushes certain site-level network settings configured in the Design application to the devices, whether used as part of the SD-Access fabric overlay or not.  Specifically, Netflow Exporter, SNMP server and traps, and syslog server information configured in the Design application, are provisioned on the devices. 

After provisioning devices to a site, the remaining network settings from the Design application are pushed down to the devices. These include time zone, NTP server, and AAA configuration.

When devices are defined in fabric roles, the fabric overlay configuration is pushed to the devices. 

The above workflows are described as separate actions as they support different solution workflows.  The first is required for the Assurance solution, and all three workflows are required for the SD-Access solution. 

Procedure 1: Create Credentials in ISE

When devices are provisioned to a site, they receive several configurations, including the centralized AAA server configuration pointing to ISE.  Authentication against ISE is preferred over local login credentials within this configuration. 

Devices provisioned to the site authenticate and authorize their console and VTY lines against the configured ISE server.  To maintain the ability to manage the devices after provisioning, the credentials defined during the discovery job must be available in ISE, either directly or through an external identity source such as Active Directory. 

The following steps show how to create internal user credentials in the Identity Services Engine that match the user credentials defined during the discovery job.

1.     Log into the ISE GUI.

2.     Navigate to Administration > Identity Management > Identities. 

3.     Click +Add.

4.     Enter the Name (example: Operator).

This name must match what was used for Cisco DNA Center discovery job.

5.     Enter the associated Login Password and Re-Enter Password.

6.     Click Submit.

Procedure 2: Assign Network Devices to Site and Provision Network Settings

The first step of the provisioning process begins by selecting devices and assigning them to a site, building, or floor previously created with the Design application.  Both assigning and provisioning to site is completed in a single procedure rather than doing each step independently.

1.     In Cisco DNA Center, navigate to PROVISION > Devices > Inventory.

2.     Select the devices to be assigned and provisioned to a site.

3.     Click Actions.

4.     Click Provision.

5.      

6.     In the first page of the Provision Devices wizard, click Choose a site. 

7.     In the slide-in pane on the right, select the site assignment for the devices.

8.     Click Save. 

9.     Use Next to skip the Configuration and Advanced Configuration screens.

10.  In the Summary screen, review the details for each device.

11.  Finally, click Deploy.

12.  In the Provision Device slide-in pane, leave the default selection of Now.

13.  Click Apply.

14.  Configuration of each device begins, and status messages appear as each device is provisioned successfully. The Device Inventory screen updates with Last Provisioned Time and Provision Status.

Use the Refresh button to view the final status of the devices once the status messages fade.

15.  Repeat the previous steps to assign and provision the remainder of the devices to the applicable sites for the deployment.

Procedure 3: Assign Wireless LAN Controller to Site and Provision Network Settings

The workflow for provisioning site assignment and network settings for the WLC is slightly different from the procedure for the routers and switches.  The WLC must be assigned as the Active or Guest Anchor WLC.  Potentially, APs from multiple locations may be associating with the same WLC, so the WLC must also be set to manage wireless for specific locations.     

1.     In Cisco DNA Center, navigate to PROVISION > Devices > Inventory.

2.     Select a Wireless LAN Controller.

3.     Click Actions > Provision.

 

 

4.     In the first page of the Provision Devices wizard, click Choose a site. 

5.     In the slide-in pane on the right, select the site assignment for the device.

This defines WLC will be present in the fabric topology maps.

6.     Click Save.

7.     Click Next.

8.     Select the appropriate WLC Role (example: Active Main WLC). 

9.     If the WLC is managing APs in locations other than the site to which it is assigned, these locations can be selected by clicking Managing location(s).

10.  Use Next to skip the Advanced Configuration screen.

11.  In the Summary screen, review the details for the device.

12.  Finally, click Deploy.

13.  In the Provision Device slide-in pane, leave the default selection of Now.

14.  Click Apply.

15.  Configuration of the WLC begins, and status messages appear as the device is provisioned successfully. The Device Inventory screen updates with Last Provisioned Time and Provision Status.

Use the Refresh button to view the final status of the device once the status messages fade.

Process 6: Provisioning the Fabric Overlay

Creating the fabric overlay is a multi-step workflow.  Each of fabric overlay steps are managed under the Fabric tab of the Provision application.

Figure 11     Fabric Provisioning Tab

Provisioning the fabric overlay involves the following steps:

1.     Create transits.

2.     Create the fabric domain.

3.     Assign and provision fabric roles.

4.     Set up host onboarding.

 

The first step is to create the transit.  The transit simply represents a connection that joins a fabric site to an external network (IP-Based transit) or to one or more fabric sites (SD-Access transit).  Both IP-based transits and SD-Access transits are created in this guide.  The transits are referenced later during the provisioning of fabric border nodes.

After the transit creation, a fabric domain is created.  A fabric domain is a logical organizational construct in Cisco DNA Center.  It contains fabric-enabled sites and their associated transits.  Sites are added to the fabric domain one at a time.  During fabric role provisioning, a transit is associated with a border node, joining the transit and site together under the domain.

With the domain created and sites added to it, devices in each site are then added to fabric roles.  Border node provisioning is addressed separately in the document due to the different types (Internal, External, and Anywhere) and the separate transit options.  Once the SD-Access overlay infrastructure (fabric devices) is provisioned, host onboarding and port assignment are completed allowing endpoint and Access Point connectivity.

Procedure 1: Create IP-Based Transits – Fabric Provisioning Part I

The IP-based transit represents the remote BGP autonomous system (AS). The local BGP AS is configured as part of the fabric border provisioning in subsequent steps.  The topology in this prescriptive guide includes two IP-based transits and an SD-Access transit. 

The IP-based transits will be used to automate the border connectivity between Site-1 and the Data Center and between Site-3 and the Enterprise edge (Internet edge) respectively.  The SD-Access transit will be used to interconnect the various sites in the distributed deployment thus allowing for unified policy configuration, constructs, and enforcement between these sites. 

1.     In the Cisco DNA Center dashboard, navigate to PROVISION > Fabric.

2.     At the top right, click + Add Fabric Domain or Transit.

3.     Click Add Transit.

4.     In the slide-in pane, select the Transit Type of IP-Based.

5.     For Routing Protocol select BGP – this is the only option currently.

6.     Supply a Transit Name (example: DATA CENTER).

7.     Enter an Autonomous System Number for the remote BGP AS (example: 65333).

8.     Click Add.

A status message appears, and the transit is created.

9.     Repeat the previous steps to create a second IP-Based transit. 

A different Transit Name is required (example: INTERNET EDGE). 

Procedure 2: Create SD-Access Transit – Fabric Provisioning Part I (Continued)

The SD-Access transit is the circuit between sites in an SD-Access for Distributed Campus deployment.  It uses the signaling from transit control plane nodes to direct traffic between fabric sites. 

Unlike creating an IP-based transit, there is no requirement to define a BGP Autonomous System number, as BGP private AS 65540 is reserved for use on the transit control plane nodes and automatically provisioned by Cisco DNA Center.  Using this AS, the SD-Access transit control plane nodes will form EBGP adjacencies with each site border in the Distributed Campus deployment.

When creating an SD-Access transit, the workflow also requires defining the devices used as the transit control plane nodes.  Upon creating the transit and defining the nodes, they are provisioned automatically.  Additional steps are not required to provision these devices. 

1.     In the Cisco DNA Center dashboard, navigate to PROVISION > Fabric.

2.     At the top right, click + Add Fabric Domain or Transit.

3.     Click Add Transit.

4.     In the slide-in pane, select the Transit Type of SD-Access.

5.     Supply a Transit Name (example: METRO-E-SDA).

6.     Using the first Site for the Transit Control Plane drop-down, select the applicable device

as a Transit Control Plane.

7.     Using the second Site for the Transit Control Plane drop-down, select the second applicable device

as a Transit Control Plane.

8.     Click Add.

About Transit Control Plane Node Location

As noted in the Topology Overview section, a pair of ISR 4451-x routers were used as the transit control plane nodes.  These routers were deployed (assigned and provisioned to) their own, dedicated site.  These transit control plane nodes are accessible and have IP reachability via the Metro-Ethernet circuit connecting all the sites.

Figure 12     Transit Control Plane Node Site 

This is just one design option.  Transit control plane nodes could physically be in the same site as other equipment in the deployment.  The primary requirements are to have IP reachability and that the transit control plane nodes cannot also operate as another role in the fabric (example: both transit control plane node and site border).

Procedure 3: Create a Fabric Domain – Fabric Provisioning Part 2

A fabric domain is an administrative construct in Cisco DNA Center.  It is combination of fabric sites along with their associated transits.  Transits are bound to sites later in the workflow when the fabric borders are provisioned.  A fabric domain can include all sites in a deployment or only certain sites depending on the physical and geographic topology of the network and the transits.  The prescriptive deployment includes a single fabric domain that will encompass the buildings (sites) created during previous steps in the Design application. 

1.     In the Cisco DNA Center dashboard, navigate to PROVISION > Fabric.

2.     At the top right, click + Add Fabric or Transit.

3.     Click Add Fabric.

4.     In the slide-in pane, define a Fabric Name (example: SJC).

5.     Select a single location from the Hierarchy (example: Site-1).

6.     Click Add.

Procedure 4: Add Fabric-Enabled Sites to the Fabric Domain – Fabric Provisioning Part 2 (Continued)

Each building created in the Design application is intended to represent a separate fabric site.  In the previous steps, a fabric domain was created, and a single site was added to that domain.  The following steps will add the remainder of the sites to the domain. 

1.     In the Cisco DNA Center dashboard, navigate to PROVISION > Fabric.

2.     Select the newly created fabric domain (example: SJC). 

3.     Select the + button next to Fabric-Enabled Sites. 

4.     In the Add Location slide-in pane, select an additional site in the hierarchy to add to as a fabric-enabled site to this fabric domain.

5.     Click Add.

6.     Verify that the site is added to the domain hierarchy on the left. 

7.     Repeat these steps to add all the sites that are associated with this fabric domain. 

 

The final hierarchy of fabric-enabled sites for the fabric domain is shown below.

Figure 13     Fabric-Enabled Sites

Process 7: Assigning Fabric Roles – Fabric Provisioning Part 3

A fabric overlay consists of three different fabric nodes: control plane node, border node, and edge node.

To function, a fabric must have an edge node and control plane node.  This allows endpoints to traverse their packets across the overlay to communicate with each other (policy dependent).  The border node allows communication from endpoints inside the fabric to destinations outside of the fabric along with the reverse flow from outside to inside.

About Fabric Border Nodes

When provisioning a border node, there are number of different automation options in the Cisco DNA Center GUI. 

A border node can have a Layer-3 handoff, a Layer-2 handoff, or both (platform dependent).  A border node can be connected to an IP transit, to an SDA transit, or both (platform dependent).  A border node can provide connectivity to the Internet, connectivity outside of the fabric site to other non-Internet locations, or both. It can operate strictly in the border node role or can also operate as both a border node and control plane node.  Finally, border nodes can either be routers or switches which creates slight variations in the provisioning configuration to support fabric DHCP and the Layer-3 handoff.  

With the number of different automation options, it is important to understand the use case and intent of each selection to have a successful deployment. 

Border Nodes – Network Types

When provisioning a device as a border node, there are three options to indicate the type of network(s) to which the border node is connected:

·         Rest of Company (Internal)

·         Outside World (External)

·         Anywhere (Internal & External)

 

An Internal border is connected to the known routes in the deployment such as a Data Center.  As an Internal border, it will register these known routes with the site-local control plane node which directly associates these prefixes with the fabric.  

An External border is connected to unknown routes such as the Internet, WAN, or MAN.  It is the gateway of last resort for the local site’s fabric overlay.  A border connected to an SD-Access transit must always use the External border functionality.  It may also use the Anywhere border option as described below.

An Anywhere border is used when the network uses one set of devices to egress the site.  It is directly connected to both known and unknown routes.  A border node connected to an SD-Access transit may use this option if it is also connected to a fusion router to provide access to shared services.

Border Nodes – Fabric Roles

When provisioning a border node, the device can also be provisioned as a control plane node.  Alternatively, the control plane node role and border node role can be independent devices.  While the resulting configuration on the devices is different based on these selections, Cisco DNA Center abstracts the complexity, understands the user’s intent, and provisions the appropriate resulting configuration to create the fabric overlay.

In the GUI, Cisco DNA Center refers to these provisioning options as Add as CP, Add as Border, and Add as CP+Border.

Which option should be deployed is ultimately a design discussion based on the existing technology, processes, and routing that is operating on the intended border node device, along with the scale requirements of the fabric site and the physical layout of the topology itself.  For further discussion, please see the Software-Defined Access Design and Deployment CVDs in Appendix B  along with Cisco Live BRKCRS-2810 and BRKCRS-2811.

 

Procedure 1: Provisioning a Site Fabric Overlay

The following steps will provision the control plane nodes, edge nodes, and the WLC in fabric roles in Site-1.  The (external) border nodes connected to the SD-Access transit will also be provisioned.  In the deployment, two devices in Site-1 are (internal) border nodes connecting to the Data Center through the fusion routers. Their provisioning is addressed in the next procedure.

Site-1 through Site-4 will use independent, dedicated devices in the control plane node roles and the border node roles.  Site-5 will use the control plane node and border node roles operating on the same devices – sometimes referred to as collocated. Please see Appendix C for additional and advanced topology diagrams.

Figure 14     Site-1 Overlay Topology

 

1.     In the Cisco DNA Center dashboard, navigate to PROVISION > Fabric.

2.     Select the Fabric Domain (SJC).

3.     Select the Fabric Site (Site-1) from the Fabric-Enabled Sites on the right.

The topology map loads showing the devices provisioned to that site.

4.     In the fabric topology view, click a node that will be used as a fabric edge node.

5.     Select Add to Fabric from the popup menu.

6.     Repeat this step for additional devices to be used as fabric edge nodes.

7.     Click a Wireless LAN Controller.

8.     Select Add to Fabric from the popup menu.

9.     Click a device that will operate as a control plane node without border node functionality.

10.  Select Add as CP from the popup menu.

11.  Repeat this step for any redundant, dedicated control plane node without border functionality.

12.  Click a device that will operate as the external fabric border node.

This device should be connected to the metro-area private circuit and not to the fusion routers.   

13.  Select Add as Border from the popup menu.

A slide-in pane appears.

14.  As the border is connected to unknown routes or the metro transit,

select Outside World (External) in the slide-in pane.

15.  Supply the Local BGP Autonomous System Number (example: 65001). 

16.  Under Transit, select the previously created SDA: Transit

(example: METRO-E-SDA).

17.  Click Add. 

The transit is added.

18.  Click the blue Add Button at the bottom of the slide-in pane.

19.  A ⚠Warning pop-up provides additional information regarding fabric border nodes.

Click OK.

20.  Repeat these steps for any other External-Only border nodes in the site. 

21.  Verify that the devices are highlighted in blue and have a fabric-role label (example: B, CP, E). 

This indicates the intent to provision these devices to the fabric. 

 

In the bottom right, click Save. 

22.  In the Modify Fabric Domain slide-in pane, leave the default selection of Now.

23.  Click Apply.

Procedure 2: Provisioning Internal Border Nodes

Internal border nodes connect to known routes in the network such as the Data Center.  Internal border nodes are most often used to provide access to shared services networks such as DHCP and DNS.  Most commonly, they are directly connected to another device such as a fusion router (described in the next section).  Internal border nodes vary from external border nodes in that they register prefixes to the host tracking database (control plane node) in a similar way that fabric edge nodes register prefixes and endpoints.  This registration notifies the rest of the fabric site and ultimately the fabric domain that these prefixes are accessible through this internal border node. 

Because these internal border nodes are directly connected to the fusion routers, they will utilize the IP-based transit (DATA CENTER) previously created.  This will instruct Cisco DNA Center to provision a Layer-3 handoff configuration using VRF-lite for use in later steps. 

The following steps will provision internal borders with a Layer-3 handoff which extends the fabric VNs to the next hop fusion router.  This will allow the endpoints in the fabric to access shared services once the fusion router configuration is completed in the Operate section.

1.     In the Cisco DNA Center dashboard, navigate to PROVISION > Fabric.

2.     Select the Fabric Domain (SJC).

3.     Select the Fabric Site (Site-1) from the Fabric-Enabled Sites on the right.

The topology map loads showing the devices provisioned to that site.

4.     Select the device to be used as an internal border.

5.     Select Add as Border from the popup menu.

A slide-in pane appears.

6.     As the border is connected only to known routes, select Rest of Company (Internal).

7.     Enter the Local BGP Autonomous System Number (example: 65001). 

8.     Under Select IP Address Pool, select the previously defined pool to be used for the Layer-3 handoff

(example: BGP-SITE-01). 

9.     Under Transit, select the previously created IP transit (example: DATA CENTER).

10.  Click Add. 

The transit is added.

11.  Click the IP transit name to expand it

12.  Click + Add Interface.

13.  In the new slide-in pane, select the External Interface connected to the first fusion router

(example: TenGigabitEthernet1/11). 

14.  The BGP Remote AS Number is automatically filled in due to the creation of the IP-based transit. 

Click ^ to expand the Virtual Network section. 

15.  Select all Virtual Networks that will require access to shared services.

16.  Click Save.

17.  Verify that the desired External Interfaces and Number of VNs are displayed. 

Click Add at the bottom of the slide-in pane.

 

18.  Click OK in the ⚠Warning pop-up.

19.  Repeat these steps for the redundant internal border node.

20.  Once complete, click Save in the bottom right of the fabric topology screen. 

21.  In the Modify Fabric Domain slide-in pane, leave the default selection of Now.

22.  Click Apply.

Procedure 3: Provisioning Redundant Layer-3 Handoffs

As a redundancy strategy, critical network devices are deployed in pairs.  Fusion routers are no exception.  These too are commonly deployed as a pair of devices to avoid single points of failure in hardware.  To avoid single points of failure in uplinks to the fusion routers, fabric border nodes are connected to each of the fusion routers through separate interfaces. The concept is to build triangles, not squares to produce the highest resiliency and deterministic convergence.

In the previous procedure, a single interface was provisioned for the Layer-3 handoff to a single fusion router on each internal border node.  The following steps show the procedure to provision a second Layer-3 handoff interface on each border node.  This second interface is connected to the redundant upstream fusion router. 

1.     From the fabric topology map for Site-1, click the previously provisioned internal border node.

2.     Select Edit Border in the pop-up menu.   

3.     Under Transits, click > to expand the previously defined IP transit.

4.     The previously defined External Interface and Number of VNs is displayed. 

Click + Add Interface.

5.     In the new slide-in pane, select the External Interface connected to the second fusion router

(example: TenGigabitEthernet1/12). 

6.     The BGP Remote AS Number is automatically filled in due to the creation of the IP-based transit. 

Click ^ to expand the Virtual Network selection. 

7.     Select all Virtual Networks that will require access to shared services.

8.     Click Save.

9.     Verify that the second External Interface and Number of VNs are displayed along with the previously provisioned External Interface.

Click Add.

 

10.  Click OK in the ⚠Warning pop-up.

11.  Repeat these steps for the second internal border node’s interface connected to the second fusion router.

12.  Once complete, click Save in the bottom right of the fabric topology screen. 

13.  In the Modify Fabric Domain slide-in pane, leave the default selection of Now.

14.  Click Apply.

Process 8: Configuring Host Onboarding – Fabric Provisioning Part 4

Host onboarding is the culmination of all the previous steps.  It binds the reserved IP address pools from the Design application with the VN configured in the Policy application, and provisions the remainder of the fabric configuration down to the devices operating in a fabric role.  

Host onboarding is comprised of four distinct steps – all located under the Provision > Fabric > Host Onboarding tab for a fabric site.

1.     Define authentication template.

2.     Create host pools.

3.     Define SSID address pool.

4.     Assign Access Point ports.

 

The first step is to select the authentication template. These templates are predefined in Cisco DNA Center and are pushed down to all devices that are operating as edge nodes within a site. It is mandatory to complete this step first – an authentication template must be defined before host pool creation. 

The second step is to bind the IP address pools to the Virtual Networks (VNs).  Once bound, these components are referred to collectively as host pools.  Multiple IP address pools can be associated with the same VN. However, an IP address pool must not be associated with more than one VN.  Doing so would allow communication between the VNs and break macro-segmentation in SD-Access. 

When deploying SD-Access Wireless, the third step is required.  This step binds an SSID to an IP address pool.  The last step is to selectively modify the authentication template at the port level so that the Access Points can connect to the network.  This step is required if using 802.1x in the deployment.

About Authentication Templates

Authentication templates deploy certain interface-level commands on the downstream (access) ports of all edge nodes. By default, access ports are considered all copper ports operating as switchports (as opposed to routed ports) on the edge nodes.   These interface-level commands are port-based Network Access Control configurations that validate a connected endpoint before it can connect to a network.

There are four supported built-in authentication templates:

1.       Closed Authentication

2.       Open Authentication

3.       Easy Connect

4.       No Authentication

 

These templates are based on the AAA Phased Deployment Implementation Strategy of High Security mode, Low Impact mode, Monitor mode, and No Authentication mode.

Hovering over each option in Cisco DNA Center provides information about the authentication template. 

Figure 15     Cisco DNA Center Authentication Templates

 

Procedure 1: Assign Authentication Template and Create Wired Host Pool – Host Onboarding Part 1

When a host pool is created, a subnet in the form of a reserved IP address pool is bound to a VN.  From the perspective of device configuration, Cisco DNA Center creates the VRF definition on the fabric nodes, creates an SVI or loopback interface (on switches and routers, respectively), defines these interfaces to forward for the VRF, and gives it the IP address defined as the gateway for the reserved pool.

When creating the host pools, there are multiple options including Traffic Type, Layer-2 Extension, Layer-2 Flooding, Groups, Critical Pool, and Auth Policy.  Many are beyond the scope of this document although are listed here for reference.

Figure 16     Host Pool Configuration Options

Traffic Type is defined as either Voice or Data which applies a QoS configuration.  Layer-2 Extension should always be enabled for any post >1.1.8 deployment.  For additional details on the remaining options please see the Software-Defined Access CVDs and additional references in Appendix B.  

The following steps will define the Closed Authentication Template for the Site-1 fabric and create host pools for the wired VNs.

1.     In the Cisco DNA Center dashboard, navigate to PROVISION > Fabric.

2.     Select the Fabric Domain (SJC).

3.     Select the Fabric Site (Site-1) from the Fabric-Enabled Sites on the right.

4.     Click the Host Onboarding tab.

5.     Under Select Authentication Template, select Closed Authentication and click the Save button on the right.

 

 

6.     In the Modify Authentication Template slide-in pane, leave the default selection of Now.

7.     Click Apply to demonstrate the intent to use the desired template. 

The configuration will actually be provisioned at the end of this procedure.

8.     Under Virtual Networks, select a VN to be used for wired clients (example: CAMPUS). 

9.     In the Edit Virtual Network: CAMPUS slide-in pane, select the IP Pool Name(s) to

associate it to the VN (example: CORP-SITE-01). 

10.  Select a Traffic Type of Data.

11.  Verify that Layer 2 Extension is On.

12.  Click Update.

13.  In the Modify Virtual Network slide-in pane, leave the default selection of Now.

14.  Click Apply.

15.  Repeat these steps with the remaining user-defined VNs and IP address pools in the site to create wired host pools.

Procedure 2: Create INFRA_VN Host Pool – Host Onboarding Part 2

Access Points are a special case in the fabric.  They are connected to edge nodes like an endpoint, although they are actually part of the fabric infrastructure.  Because of this, their traffic pattern is unique.  Access Points receive a DHCP address via the overlay network and associate with the WLC via the underlay network.  Once associated with the WLC, they are registered with Cisco DNA Center by the WLC through the overlay network.  To accommodate this traffic flow, the Access Point subnet – which is in the Global Routing Table (GRT) –  is associated with the overlay network.  Cisco DNA Center GUI calls this special overlay network associated with the GRT the INFRA_VN.

INFRA_VN stands for Infrastructure Virtual Network and is intended to represent devices that are part of the network infrastructure, but associate and connect to the network in a similar method to endpoints - directly connected to the downstream ports of an edge node. Both Access Points and Extended Nodes (SD-Access Extension for IoT) are part of the INFRA_VN.

1.     In the Cisco DNA Center dashboard, navigate to PROVISION > Fabric.

2.     Select the Fabric Domain (SJC).

3.     Select the Fabric Site (Site-1) from the Fabric-Enabled Sites on the right.

4.     Click the Host Onboarding tab.

5.     Under Virtual Networks, select INFRA_VN to be used as the Access Point host pool.  

6.     In the Edit Virtual Network: INFRA_VN slide-in pane, select the IP Pool Name(s) to associate it to the VN

(example: AP-SITE-01).

7.     Select AP as the Pool Type.

8.     Verify that Layer-2 Extension is On.

This must be On for the INFRA_VN.

9.     Click Update.

 

10.  In the Modify Virtual Network slide-in pane, leave the default selection of Now.

11.  Click Apply.

Procedure 3: Assign SSID Subnet and Assign Access Point Ports– Host Onboarding Part 3

Each SSID for a fabric site must be assigned an IP address pool so that wireless hosts are associated with the correct subnet when connecting to the wireless network.  This places them in the client overlay network, as the INFRA_VN is the AP overlay network. 

For the Access Points themselves to connect to the network, the authentication template must be modified at the port level.  Cisco DNA Center enables automatic onboarding of APs by provisioning a Cisco Discovery Protocol (CDP-based) macro at the fabric edge nodes when the authentication template is set to No Authentication.  If a different authentication template is used globally, for example Closed Authentication, then the downstream switchport configurations on the edge nodes must be changed in Cisco DNA Center. 

The following steps will assign an IP address pool for the Enterprise and Guest SSIDs and modify the authentication template used at the edge node port level to support Access Port connectivity.

1.     In the Cisco DNA Center dashboard, navigate to PROVISION > Fabric.

2.     Select the Fabric Domain (SJC).

3.     Select the Fabric Site (Site-1) from the Fabric-Enabled Sites on the right.

4.     Click the Host Onboarding tab.

5.     Under Wireless SSID’s, select the IP Address Pool for the Guest SSID

(example: GUEST:172.16.113.0).

6.     Repeat this step for the Enterprise SSID (example: CAMPUS:172.16.112.0).

7.     Click the Save button on the right.

8.     In the Modify SSID Table slide-in pane, leave the default selection of Now.

9.     Click Apply.

10.  Under Select Port Assignment, click an edge node (example: 3850-24P-01).

11.  Select the ☑ check box(es) of the interface(s) to which an Access Point is connected

(example: GigabitEthernet1/0/1). 

12.  Click Assign. 

13.  In the Port Assignments slide-in pane, under Connected Device Type, select Access Point(AP).

14.  Under Auth Template, select No Authentication.

The Address Pool will automatically be filled in based on the INFRA_VN host pool.

15.  Click Update.

 

16.  In the slide-in pane for Modify Interface Segment, leave the default selection of Now.

17.  Click Apply.

 

 

 

The Operate section provides a comprehensive overview on providing shared services and Internet access for the SD-Access fabric domain.  Much of this configuration is performed manually on the CLI of the fusion routers and Internet edge routers.   While shared services are a necessary part of the deployment, they are, strictly speaking, outside of the fabric.  Located outside of the fabric, these devices will not receive automated provisioning from Cisco DNA Center.

The Operate section begins with a discussion on access to shared services using a fusion router along with corresponding information on fusion router requirements and configuration. Once access to shared services is provided to the devices downstream from the edge nodes, Access Points can receive an IP address via DHCP with DHCP Option 43 pointing to their site-local wireless LAN controller.  The Access Points will be registered (brought into the inventory) in Cisco DNA Center by the WLC.  Once in the inventory, they can be provisioned with the specific wireless configuration created during the Design application steps. 

Redundant border nodes have special considerations when it comes to the interaction between BGP and LISP.  To assist with failover scenarios, IBGP sessions are manually configured between these redundant devices.  As the configuration has variations on routers and switches due to subinterface support, both options are discussed.  This section then provides a prescriptive configuration for providing Internet access to the fabric domain and supplies details and considerations regarding connected-to-Internet option in the border provisioning steps.

Process 1: Providing Access to Shared Services

When creating the fabric overlay through the end-to-end workflow, VNs are defined and created.   When these are provisioned to the fabric devices, they are configured as VRF definitions.  In the later host onboarding steps, each VRF is associated with one or more subnets defined during the Design application steps.  The result is that the endpoints in the fabric overlay are associated and route packets through VRF tables.  This is also the first layer of segmentation.

Shared services such as DHCP and DNS are generally accessible through the Global Routing Table (GRT) or could be in their own dedicated VRF, depending on the design.  The challenge is to provide a method to advertise these shared services routes from the GRT/VRF to the VN routing tables on the border nodes so that endpoints in the fabric can access them.  This is accomplished today using a fusion router.

About Fusion Routers

The generic term fusion router comes from MPLS Layer-3 VPN.  The basic concept is that the fusion router is aware of the prefixes available inside each VPN (VRF), either because of static routing configuration or through route peering, and can therefore fuse these routes together.  A generic fusion router’s responsibilities are to route traffic between separate VRFs (VRF leaking) or to route traffic to and from a VRF to a shared pool of resources in such as DHCP and DNS servers in the global routing table (route leaking).  Both responsibilities involve moving routes from one routing table into a separate VRF routing table.

A fusion router in SD-Access has several technological requirements.  It must support:

·         Multiple VRFs

·         802.1q tagging (VLAN tagging)

·         Subinterfaces (when using a router) or switched virtual interfaces (SVI) (when using a Layer-3 switch)

·         BGPv4 and specifically the MP-BGP extensions (RFC 4760 and RFC 7606) for extended communities attributes

 

 

About Route Leaking

In an SD-Access deployment, the fusion router has a single responsibility: to provide access to shared services for the endpoints in the fabric.  There are two primary ways to accomplish this task depending on how the shared services are deployed.

The first option is used when the shared services routes are in the GRT.  IP prefix lists are used to match the shared services routes, route-maps reference the IP prefix lists, and the VRF configurations reference the route-maps to ensure only the specifically matched routes are leaked. This option, along with the reason it was not selected for this prescriptive guide, is discussed in Appendix D. The second option is to place shared services in a dedicated VRF.  With shared services in a VRF and the fabric endpoints in other VRFs, route-targets are used leak between them. 

Each option is viable and valid.  For simplicity, this prescriptive deployment guide places the shared services in a dedicated VRF.  However, production deployments may use one option or the other. 

Access to shared services is a multi-step workflow performed primarily on the command-line interface of the fusion routers.

1.       Create the Layer-3 connectivity between borders nodes and fusion routers.

2.       Use BGP to extend the VRFs from the border nodes and fusion routers.

3.       Use route leaking or VRF leaking to share routes between the routing tables on the fusion router.

4.       Distribute the leaked routes back to the border nodes via BGP.

Figure 17     Route Leaking Workflow

Procedure 1: Use the Provision Application for BGP and VRF-Lite information – Part 1

Cisco DNA Center can be used to determine the Layer-3 handoff configuration that was provisioned in previous steps.  This information is needed to determine the corresponding configuration needed on the fusion routers.

1.     In the Cisco DNA Center dashboard, navigate to PROVISION > Fabric.

2.     Select the Fabric Domain (SJC).

3.     Select the Fabric Site (Site-1) from the Fabric-Enabled Sites on the right.

The topology map loads showing the devices provisioned to that site.

4.     Click the internal border node previously provisioned with the Layer-3 VRF-lite handoff.

5.     In the pop-up menu, select View Device Info. 

6.     In the slide-in pane, scroll to the listed Interfaces and click > to expand the information.

7.     The Layer-3 handoff provisioning information is displayed along with the Local IPs and necessary Remote IPs. 

Note this information as it will be used in the next procedures.

8.     Repeat these steps to collect the information on the redundant border node.

 

Procedure 2: Configure VRF Definitions on the Fusion Routers – Command Runner Tool

The next task is to create IP connectivity between the fusion routers and the border nodes.  Cisco DNA Center has automated the configuration on the border nodes in previous steps.  On each fusion router, IP connectivity must be created for each virtual network that requires connectivity to shared services, and this must be done for each interface between the fusion routers and redundant border nodes. 

Like the chicken and egg dilemma, to configure an interface to be associated with a VRF instance, the VRF must first be created.  As a best practice, the VRF configuration on the border nodes and fusion routers should be the same, including using the same route-target and route-distinguishers on both device sets. 

The VRF configuration on the border nodes can be retrieved using the device’s CLI or through the Command Runner tool in Cisco DNA Center.

To use Command Runner to display the border node VRF configuration:

1.     At the top-right of the Cisco DNA Center dashboard, navigate to TOOLS > Command Runner.

2.     Select one of the borders with the Layer-3 IP handoff (example: 6832-01 (192.168.10.3)). 

3.     The device is added to the device list in the center of the screen.

In the command bar, type the command show running-config | section vrf definition.

4.     Click the Add button.

5.     Verify that the command appears below the device in the Device List. 

 

6.     Click the Run Command(s) button at the bottom of the screen.

7.     After a few moments the command is executed, and the results are returned to Cisco DNA Center.

Click the command (shown with a green border to indicate that it was executed successfully) to view to result.

8.     Copy the displayed VRF configuration to be used for the next steps.

9.     Access the CLI of the fusion routers and enter global configuration mode. 

10.  Paste the VRF configuration so that the RDs, RTs, and VRF names match exactly.

Repeat these copy-and-paste steps for the redundant fusion router.

Procedure 3: Create Layer-3 Connectivity Between Border Nodes and Fusion Routers – Fusion Routers Part I

Cisco DNA Center has provisioned a VRF-lite configuration on the internal border nodes.  The following section will demonstrate how to create the corresponding VRF-lite configuration on the fusion router. 

 

Fusion routers can be either true routers or can be switching platforms.  Both platforms support the role of fabric border as well.  All modern Cisco Layer-3 switches support SVIs and trunks, although not all of these switches support subinterfaces.  Routers must use subinterfaces as they do not support trunk ports.  This results in an inherent difference in the possible configurations for VRF-lite – subinterfaces or SVIs and trunk ports.   

This deployment guide uses Layer-3 switches as the fusion routers.  The internal border nodes in Site-1, as part of the distribution layer, are also Layer-3 switches. The VRF-lite configuration will therefore utilize SVIs and trunk ports to create the connectivity, extend the VRFs to the fusion routers, and exchange routing information.  

The previous steps created the VRFs on the fusion routers.  The next steps will use the IP address and VLAN information from the border node’s View Device Info screen from Cisco DNA Center.

When the fusion router is a switching platform, the VLANs must be created first.  Once created, the corresponding SVIs are created, configured to forward for a VRF, and given an IP address.  Finally, a trunk port is created that forwards for the appropriate VLANs creating connectivity between the border node and fusion router.

 

Using the information gleaned from the View Device Info screen, the following connectivity will be created on the fusion routers in the next steps.

Figure 18     Fusion Router and Internal Border – VRF-lite

1.     In global configuration on the first fusion router, create the VLANs.

2.     Configure each SVI to forward for a VRF and to have the applicable IP address based on the information from the View Device Info screen.

 

Remember to use no shutdown to bring the SVI to the up/up state.

3.     Configure a trunk interface that allows the defined VLANs.

4.     Repeat these steps for the second interface on the first fusion router.

5.     Verify IP connectivity between the first fusion router and the border nodes using ping commands.

6.     Repeat these steps on the second fusion router.

7.     Verify IP connectivity between the second fusion router and the border nodes using ping commands.

Procedure 4: Establish BGP Adjacencies Between Fusion Routers and Border Nodes – Fusion Routers Part 2

Now that IP connectivity has been established and verified, BGP peering can be created between the fusion routers and the border nodes. 

On the border nodes, Cisco DNA Center has created SVIs forwarding for a VRF and a BGP configuration listening for peering adjacencies using those SVIs as the update source.  A corresponding configuration will be applied on the fusion routers with one deviation.  Because the shared services are in a VRF on the fusion routers, they will advertise the SHARED_SERVICES prefixes using address-family ipv4 vrf in BGP configuration and not the default address-family ipv4 which is associated with the GRT.

1.     Create the BGP process on the first fusion router. 

2.     Use the corresponding Autonomous-System defined in the IP-based transit from early steps.

As a recommended practice, the Loopback 0 interface is used as the BGP router ID.

3.     Enter IPv4 Address-Family for vrf SHARED_SERVICES and complete the following steps:

a.        Define the first border node neighbor and its corresponding AS Number.

(Remember that this corresponds with the GRT on the border nodes).

b.       Define the update source to use the applicable SVI.

c.        Activate the exchange of NLRI with the first border node.

4.     Repeat these steps for the adjacency with the redundant border node.

5.     Configure the adjacencies for the remaining VRFs using the applicable SVIs as the update source for the peering.

6.     Once this configuration is finished on the first fusion router, repeat on the second fusion router.

 

Procedure 5: Use Route-Targets to Leak Shared Services Routes – Fusion Routers Part 3

With the shared services routes in a VRF on the fusion routers and BGP adjacencies formed, route-targets can be used to leak routes between the VRFs.  This leaking will occur internally in the routing tables of the fusion routers.  Once leaked, these routes will be advertised to the fabric border nodes and be accessible to endpoints in the fabric needing these services. 

The VRF configuration has already been copied from the border nodes and configured on the fusion routers.  Each VRF is already importing and exporting its own route-targets.    This simply means the VRF is exporting its own routes using the defined route-targets and importing routes from any other VRF with the defined route-targets. 

This procedure will complete the following steps and achieve these results:

·         Each VRF on the fusion routers will import the route-target associated with the SHARED_SERVICES VRF (1:4097).

·         By importing this route-target (1:4097), each VRF will import the shared services subnets into the routing table.

·         The SHARED_SERVICES VRF will import the route-targets associated with the other VRFs (1:4099, 1:4100, 1:4101).

·         These actions fuse the routes between the various VRF tables.

 

The resulting configuration will create the following outcome:

Figure 19     Route-Target and Prefix Import and Export

1.     On both fusion routers, import the SHARED_SERVICES VRF route-target into the CAMPUS VRF.

2.     Repeat this SHARED_SERVICES VRF route-target import step for the remaining VRFs.

3.     Under the SHARED_SERVICES VRF, import the route-targets for the fabric VRFs.

Process 2: Provisioning and Verifying Access Points

Now that the edge nodes ports connecting to Access Points have been provisioned, the internal border nodes and fusion routers have been provisioned and configured, and route leaking of the shared services prefixes is complete, Access Points are able to access both the WLC and the DHCP server in order to receive an IP address, associate with the correct WLC through DHCP Option 43, and be registered to the Cisco DNA Center inventory by the WLC. 

Once in the inventory, APs must be provisioned in order to receive the correct RF profile configuration and to join the overlay network in the SD-Access Wireless Access Point role.  The following steps will provision the Access Points to a floor in a building (site) and provision them with an RF profile, allowing them to operate in the fabric role.

1.     In the Cisco DNA Center dashboard, navigate to PROVISION > Devices > Inventory.

2.     Select the Access Points.

3.     Click Actions > Provision.

A Provision Devices wizard screen appears.

 

 

4.     Within the first wizard screen, Assign Site, click Choose a floor. 

5.     From the slide-in pane on the right, select the Floor Assignment (Choose a floor) for the devices.

6.     Click Save. 

7.     If the APs are all in the same site, use the ☑ Apply to All check box. 

Different site locations can be selected for each device where applicable.

8.     On the Configuration page of the Wizard, select the desired RF Profile (example: Typical). 

 

☑ Apply to All can be used to apply the same RF profile to multiple Access Points. 

9.     Click Next.

 

10.  Verify that the Summary screen displays the RF profile information which includes the Radio Type, Channel Width, and Data Rates. 

11.  Click Deploy. 

12.  In the slide-in pane for Provision Device, leave the default selection of Now.

13.  Click Apply.

14.  A ⚠Warning message appears indicating that Access Points will be rebooted. 

Click OK.

15.  Configuration of the Access Points begins, and status messages appear as each device is provisioned successfully. The Device Inventory screen updates with Last Provisioned Time and Provision Status.

 

Use the Refresh button to see the final status of device provisioning.

 

Procedure 1: Access Point Verification

Traditionally, Access Points take the 802.11 traffic from the endpoint, convert it to 802.3, and then either put in on the local wired network or use CAPWAP to tunnel the traffic to the WLC which then places the traffic on the wired network.  When an SSID is enabled for the fabric, it tells the Access Points to treat the packets differently.  In the fabric, an AP converts 802.11 traffic to 802.3, but also encapsulates it into VXLAN to encode the VN and SGT information of the client.  This VXLAN tunnel is terminated at the first-hop switch (the edge node). 

The Command Runner tool can be used to validate that the Access Points have joined the fabric overlay and are communicating with their connected edge node by verifying this VXLAN Tunnel. 

1.     At the top-right of the Cisco DNA Center dashboard, navigate to TOOLS > Command Runner.

2.     Select an edge node with attached Access Points

(example: 3850-24p-01 (192.168.10.5)).

3.     Verify that the device is added to the device list in the center of the screen.

4.     In the command bar, type the command show access-tunnel summary.

5.     Click the Add button to add the command.

6.     Verify that the command appears below the device in the device list. 

7.     Click the Run Command(s) button at the bottom of the screen.

8.     After a few moments, the command is executed, and the results are returned to Cisco DNA Center.

Click the command (shown with a green text to indicate that it was executed successfully) to view to result.

Figure 20     Successful VXLAN Tunnel Between Access Points and Edge Node

Process 3: Creating IBGP Adjacencies Between Redundant External Border Nodes

The fabric overlay works by tunneling traffic between routing locators (RLOC).  The RLOCs in SD-Access are always the Loopback 0 interface of the device operating in the fabric role. 

When border nodes are configured with the Outside World (External) option, there is no redistribution of external network into SD-Access fabric – this is a function performed by internal border nodes. 

Any traffic with a destination outside of the fabric site must traverse the border nodes.  If an uplink interface or upstream device fails for a border node, the Loopback 0 interface (the RLOC) of that border node is still reachable by other devices in the fabric.  This leads to a potential black-holing of packets. 

There is no built-in method in the Cisco DNA Center provisioning or in LISP to mitigate this issue. The preferred method to address this potential problem is to form IBGP adjacencies between redundant border nodes.  This is accomplished using a cross-link between redundant devices. From a design and recommended practice perspective, and to allow uninterrupted traffic, IBGP adjacencies should be formed between all redundant border types and redundant fusion routers.  These adjacencies need to be formed for the global routing table and all VRFs.

Once the IBGP peering sessions are established, a redirection path is provided in the event of a failed interface or upstream device.  It is an additional network traffic path that helps avoid traffic black-holing in basic failure situations. 

Border nodes and fusion routers may be either Layer-3 switches or a true routing platform. Among Layer-3 switches, the device may support subinterfaces – such as with the Catalyst 6800 Series – or may only support Switched Virtual Interfaces (SVIs).  This results in many deployment-specific variations for the IBGP configuration.  This prescriptive deployment guide uses routing platforms as the fabric border nodes in most of the fabric-enabled sites. Therefore, the IBGP configuration will be shown using routers.

Layer-3 switches are also used for both the internal border nodes and the fusion routers.  For an example IBGP configuration using a switching platform as a border node or fusion router in a redundant pair, please see Appendix E.

Procedure 1: Configure IBGP – Routing Platforms

Routing platforms such as the Integrated Services Routers (ISR) and Aggregation Services Routers (ASR) support subinterfaces (dot1q VLAN subinterface).  Subinterfaces on routers should be used to create the IBGP adjacency between redundant devices. 

This procedure will first create the subinterfaces, assign them to an existing VRF and give them an IP address.  Once IP connectivity is established and verified, the BGP adjacencies will be created using the subinterfaces as the update source. 

1.     In the CLI of the first external border node router, bring up the physical interface.

2.     Define a subinterface that will forward for the Global Routing Table.

3.     Define the VLAN number and the IP address.

4.     Define subinterfaces, VLANs, and IP addresses for the remaining VRFs. 

5.     In the CLI of the second external border node router, bring up the physical interface.

6.     Define a subinterface that will forward for the Global Routing Table.

7.     Define the corresponding VLAN number and IP address.

8.     Define the corresponding subinterfaces, VLANs, and IP addresses for the remaining VRFs. 

9.     Verify connectivity between the subinterfaces on the border nodes.

10.  On the first border node, enable IBGP the redundant peer for the Global Routing Table. 

Use the subinterfaces as the update source.

11.  Enable IBGP between the redundant peer for the remainder of the VRFs.

Use the subinterfaces as the update source.

12.  Complete the corresponding configuration on the redundant border node.

Process 4: Providing Internet Access for SD-Access for Distributed Campus

Internet access for SD-Access for Distributed Campus operates differently from Internet access in a single-site SD-Access deployment, particularly with border node provisioning.  In a single site, the egress point for the network to the Internet is the external border nodes defined in network provisioning.  In an SD-Access for Distributed Campus deployment, which by its definition has multiple fabric sites, many sites in the deployment may have direct Internet access (DIA) or all Internet traffic may traverse the SD-Access transit to a single-site location in order to egress the fabric domain.  Both designs are valid for a deployment, although the latter is the most common in a metro-area with traffic tunneled across the MAN to an HQ location.    

Within an individual site or a single-site deployment, the External or Anywhere border node is considered the “fabric site gateway of last resort.”  From a traffic flow perspective, the site-local control plane node is queried by the fabric infrastructure for where to send packets.  If the site-local control plane node does not have an entry in its database for the destination, it instructs the requesting device to send the traffic to the external (or anywhere) border node.    

In SD-Access for Distributed Campus, the site border nodes are the fabric site gateway of last resort, although the control plane traffic differs from a single-site deployment.  Within the site, the same control plane requests and replies occur as described above.  However, the site border nodes may receive packets for which they have no forwarding path.  In this case, they query the transit control plane nodes to determine if the traffic should be sent to another fabric site or if the traffic should be sent towards the Internet. If traffic is bound for the Internet – if there is no entry in the database –  the transit control plane node instructs the site borders to send traffic to specific devices in the deployment.

Figure 21     Unknown or Internet Destination Packet Flow

In the Cisco DNA Center GUI, these specific devices are border nodes flagged with the ☐ Is this site connected to Internet? option when provisioning.  Selecting ☑ this option enables this border node as a “fabric domain gateway of last resort.”

When a site border queries the transit control plane node to reach an unknown destination and there is no entry in the control plane database, the site border is instructed to send the traffic to the border configured with the connected-to-Internet option.  Traffic is sent to each border node with this flag in a round-robin fashion.  This is an important consideration if more than one site in the deployment has Internet access. 

Procedure 1: Provision the Connected-to-Internet Border Node

In the deployment topology, the border node for Site-3 is connected to both the SD-Access transit and to the upstream Internet edge router.  The workflow will demonstrate provisioning both the SD-Access transit and IP-based transit with a Layer-3 handoff using VRF-lite on a connected-to-Internet border node.

1.     In the Cisco DNA Center dashboard, navigate to PROVISION > Fabric.

2.     Select the Fabric Domain (SJC).

3.     Select the Fabric Site (Site-3) from the Fabric-Enabled Sites on the right.

The topology map loads showing the devices provisioned to that site.

4.     Click a device to perform the fabric border node role.

5.     For provisioning the IP-based transit, this device should also be connected to the upstream Internet edge router.

6.     Select Add as Border (or Add as CP+Border where applicable) from the popup menu.

A slide-in pane appears.

7.     As the border is connected to an upstream provider and the metro-E transit,

select Outside World (External) in the slide-in pane.

8.     Supply the BGP Local AS Number (example: 65003). 

9.     Under Select IP Address Pool, chose the IP address pool to be used for the

IP-based handoff (example: BGP-SITE-03 (172.16.131.0/24)).

10.  Because this device is the egress point to the Internet and has a default route pointing to an upstream device, select the option ☑ Is this site connected to Internet?

11.  Under Transits, select the previously-created SDA transit (example: SDA: METRO-E-SDA).

12.  Click the gray Add Button. 

The SDA transit is added.

13.  Under Transits again, select the previously-created IP-based transit (example: IP: INTERNET EDGE).

14.  Click the gray Add Button. 

The IP-based transit is added.

15.  Click the IP transit name to expand it.

16.  Click + Add Interface.

17.  In the new slide-in pane, select the External Interface connected to the first Internet edge router

(example: GigabitEthernet0/1/2). 

18.  The BGP Remote AS Number is automatically filled in due to previous steps in the workflow.  

Click ^ to expand the Virtual Network selection. 

19.  Select all VNs that will require access to the Internet.

Click Save.

20.  Verify that the defined External Interface and Number of VNs are displayed.

21.   Click + Add Interface.

 

22.  In the new slide-in pane, select the External Interface connected to the second Internet edge router

(example: GigabitEthernet0/1/3). 

This should be a different interface than the interface used in the previous steps. 

23.  The BGP Remote AS Number is automatically filled in due to previous steps in the workflow. 

Click ^ to expand the Virtual Network selection. 

24.  Select all VNs that will require access to the Internet. 

These should be the same VNs as selected for the other interface.

25.  Click Save.  

26.  Click the blue Add button at the bottom of the border node slide-in pane.

27.  A ⚠Warning pop-up provides additional information regarding fabric borders.

Click OK. 

28.  Repeat these steps for the second External-Only border node connected to the Internet edge routers. 

29.  Once complete, click Save in the bottom right of the fabric topology screen. 

30.  In the Provision Device slide-in pane, leave the default selection of Now.

31.  Click Apply.

Figure 22     Site-3 Topology – Fabric Role Intent

 

 

 

Internet Edge Topology Overview

External (Internet) connectivity for the fabric domain has a significant number of possible variations, and these variations are based on underlying network design. The common similarity among these variations is that the connected-to-Internet border node will be connected to a next-hop device that will be traversed to access the Internet due to its routing information.  This device could be an actual Internet edge router, a fusion router, the ISP router, or some other next-hop device. 

In this prescriptive deployment, the Site-3 border nodes are connected to a pair of Internet edge routers as their external next-hop.  These Internet edge routers are connected to upstream Enterprise edge firewalls that are configured to use NAT and are ultimately connected to the ISP.  The Site-3 border nodes have a static default route to Internet edge routers via the Global Routing Table.  The Internet edge routers have a static default route to the Enterprise edge firewalls via their Global Routing Table. 

Figure 23     Default Route Overview – Site-3

 

Configuring Internet Connectivity

Regardless of how the rest of the network itself is designed or deployed outside of the fabric, a few things are going to be in common in deployments due to the configuration provisioned by Cisco DNA Center.  A connected-to-Internet border node will have the SD-Access prefixes in its VRF routing tables.  As a prerequisite for being connected-to-Internet, it will also have a default route to its next hop in its Global Routing Table.  This default route must somehow be advertised from the GRT to the VRFs.  This allows packets to egress the fabric domain towards the Internet.  In addition, the SD-Access prefixes in the VRF tables on the border nodes must be advertised to the external domain – outside of the fabric domain – to draw (attract) packets back in.

Providing Internet access to the SDA fabric (leaking between the global and VRF routing tables) is a multi-step process performed in the CLI of the Internet edge routers.

1.       Create Layer-3 connectivity between the border nodes and Internet edge routers.

2.       Use BGP to create adjacencies and receive VRF routes.

3.       Advertise the default route back to the VRF tables of the Site-3 border.

 

Figure 24     Default Route Advertisement Workflow

On the Site-3 border nodes, the VRF and BGP configurations have been provisioned by Cisco DNA Center along with the Layer-3 handoff.   For Internet access in this prescriptive guide, their upstream peers will not become VRF-aware.  All fabric domain prefixes will be learned in the Global Routing Table of the Internet edge routers, and the default route from these devices will be advertised back to Site-3 border nodes via BGP. 

Procedure 2: Use the Provision Application for BGP and VRF-Lite information – Part 2

Cisco DNA Center is used to determine the Layer-3 handoff configuration that was provisioned in previous steps.  This is needed for the corresponding configuration on the Internet edge routers.

1.     In the Cisco DNA Center dashboard, navigate to PROVISION > Fabric.

2.     Select the Fabric Domain (SJC).

3.     Select the Fabric Site (Site-3) from the Fabric-Enabled Sites on the right.

The topology map loads showing the devices provisioned to that site.

4.     Click the Connected-to-Internet Border provisioned in the previous steps.

5.     In the pop-up menu, select View Device Info. 

A new slide-in pane appears.

6.     Scroll to the interfaces listed in blue, and click > to expand the section.

The Layer-3 handoff provisioning information is displayed. 

Note the information as it will be used in the next steps.

7.     Repeat these steps to collect the information on the second border node.

Procedure 3: Create Layer-3 Connectivity Between Border Nodes and Internet Edge Routers

The next task is to create IP connectivity from the border nodes to Internet edge routers.  This must be done for each virtual network that requires connectivity to the Internet, on each Internet edge router, and for each interface between the routers and redundant borders. 

Cisco DNA Center has configured the border nodes in previous steps.  This configuration will remain unchanged.  The Internet edge routers will not become VRF-aware.  All SD-Access prefixes (the EID-space) will be learned in their Global Routing Table, and the default route will be advertised back to border nodes via BGP.  Using the method in this guide, the BGP configuration on Internet edge routers is not required to use multi-protocol BGP (address-family vrf or address-family vpnv4) or to form adjacencies using VRFs.

Why Will This Work?

VRF’s are locally significant.  The automated BGP configuration on the Site-3 border nodes is expecting BGP adjacencies via VRF routing tables.  This means that routes learned from the neighbor will be installed into the VRF tables instead of the Global Routing Table.  On the other side of the physical link – the Internet edge routers – the BGP neighbor relationship does not need to be formed using VRFs. When a neighbor is defined under BGP configuration, it simply means that the router is going to exchange routes from the VRF or the address family with the defined neighbor.

Figure 25     Internet Edge Router to Border Node – VRF to GRT

The following steps will create subinterfaces on the Internet edge routers.  These are used for connectivity and BGP peering, but will have no VRF-awareness.

1.     On the first Internet edge router, create a subinterface on the link connected to the first border node.

2.     Use the corresponding VLAN and IP address for the INFRA_VN from the View Device Info from earlier steps.

3.     Create the remaining subinterfaces defining the appropriate VLAN and IP address.

Each subinterface will be in the GRT but will peer with a VRF on the border node.

4.     Repeat these steps for the second interface on the Internet edge router which is connected to the second border node.

5.     Test the IP reachability between the first Internet edge router and the two border nodes.

6.     Complete a similar corresponding configuration on the first interface on second Internet edge router.

7.     Complete a similar corresponding configuration on the second interface on second Internet edge router.

8.     Test the IP reachability between the second Internet edge router and the two border nodes.

Procedure 4: Create BGP Adjacencies Between Border Nodes and Internet Edge Routers

Since IP connectivity has been established and verified, BGP adjacencies can be created between the Internet edge routers and the border nodes. 

On the border nodes, a subinterface was provisioned that forwards for a VRF along with a BGP configuration listening for adjacencies using that subinterface as the update source.  A corresponding configuration will be applied on the Internet edge routers although the adjacency is through the GRT.

1.     Create the BGP process on the Internet edge router. 

2.     Use the corresponding Autonomous-System defined in the IP-based transit in early steps.

As a recommended practice, the Loopback 0 interface is used as the BGP router ID.

3.     Define the neighbor and its corresponding AS Number.

4.     Define the update source to use the appropriate subinterface.

5.     Activate the exchange of NLRI with the first border node.

6.     Configure the remaining adjacencies using the applicable subinterfaces as the update-source for the peering.

7.     Continue these steps to build the adjacencies and activate the exchange of NLRI with the second border node.

 

8.     Verify that the adjacencies have formed with each border node.

9.     Complete the corresponding configuration on the second Internet edge router.

10.  Verify that the adjacencies have formed with each border node.

 

 

About Default Route Advertisement Using BGP

There are four different methods to advertise a default route in BGP – each with its own caveats.

The first three methods are very similar and result in the same effect: a default route is injected into the BGP RIB resulting in a general advertisement to all BGP neighbors.  The fourth method selectively advertises the default route to a specific neighbor.

1.  network 0.0.0.0

·         This will inject the default route into BGP if there is a default route present in the Global Routing Table.

·         The route is then advertised to all configured neighbors.

 

2.  redistribute <IGP>

·         The default route must currently be in the Global Routing table and be learned from that routing protocol that is being redistributed (example: redistributing IS-IS into BPG and the default route learned via IS-IS).

·         This will inject the default route into BGP and then advertise it to all configured neighbors.

 

3.  default-information originate

·         This will cause the default route to be artificially generated and injected into the BGP RIB regardless of whether or not it is present in the Global Routing Table. 

·         The default-information originate command alone is not enough to trigger the default route advertisement to all configured neighbors. In current Cisco software versions, an explicit redistribution of 0.0.0.0/0 is required to trigger the default route to be advertised via BGP when using this method. 

 

4.       neighbor X.X.X.X default-originate

·         This command is different in that it will only advertise the default route to a specified neighbor while each of the previous three methods will advertise the default route to all neighbors. 

·         The default route is artificially generated and injected into BGP similarly to default-information originate.

·         The default route will not be present in the BGP RIB of the router where this command is configured. 

This prevents its advertisement to all neighbors.

Each approach has its own benefits and considerations.  The neighbor X.X.X.X default-originate option is particularly useful in cases with stub-autonomous systems – an AS where a full BGP table is not required.  It also provides the most granularity and control over the default route advertisement, as any other method will advertise the default route to all neighbors.  With the number of possible variations in upstream router configuration and peering, fine-grained granularity is needed, and thus it is the method used in this prescriptive deployment guide. 

 

Procedure 5: Advertise the Default Route From the Internet Edge Routers to Fabric VNs

The following steps will use the fourth method discussed above to advertise the default route from the Internet edge routers to the fabric border nodes.

1.     On the first Internet edge router, advertise the default route to the first border node using BGP.

2.     Advertise the default route to the second border node.

3.     Repeat these steps on the second Internet edge router.

Procedure 6: Verify Default Route Advertisement

Once the default route is advertised from the upstream routers to the connected-to-Internet border nodes, it will be present in the VRF tables of those border nodes. 

1.     On each connected-to-Internet border node, verify the presence of the default route in each VRF.

 

 

 

2.     Verify the presence of the default route in the Global Routing Table.

 

Procedure 7: Verify Default Route Via LISP – Advanced and Optional

Troubleshooting and verification of LISP requires a strong understanding of the protocol, its interaction with the RIB and CEF, and a solid understanding and familiarity with the command line interface.  Exhaustive study of LISP is beyond the scope of this guide.  However, some relatively straightforward show commands can be used to verify that the remaining fabric sites know about the Site-3 connected-to-Internet border node and will send packets destined for the Internet to this site.

In SD-Access for Distributed Campus, the ☑ Is this site Connected to Internet? option will advertise that border node to all other sites as a gateway of last resort for the fabric domain.  It will also configure the other sites’ border nodes to query the transit control plane node for unknown destinations.  

Packet Walk

Figure 26     Packet Walk Traffic Flow

The following steps will demonstrate how to validate the packet flow in the diagram above hop-by-hop.  For the purposes of this demonstration, LIG will be used to manually trigger the control plane lookup.  Each step in the packet walk will be shown along with commentary on what information to glean from the resulting CLI output.

1.     On the edge node, display the current LISP map cache.

 

There is no map-cache entry for the desired prefix 208.67.222.222, and the only entry covering that prefix is the LISP default entry 0.0.0.0/0.

2.     Use lig to trigger a control plane lookup.

 

LISP computes this aggregate (208.0.0.0/4) as the largest possible block that does not contain known EID prefixes but does cover the queried address. This is due to LISP’s history as a method to solve the DFZ and TCAM space concerns. 

3.     Redisplay the LISP map-cache.

 

The map-cache entry has a state of forward-native but an action of Encapsulating to proxy ETR.  What is the result when the packet is forwarded?  To understand the forwarding decision, the LISP configuration and CEF tables must observed.

4.     Verify a proxy-ETR is configured on the edge node for the instance ID.

 

The Site-1 border nodes (192.168.10.7 and 192.168.10.8) are configured as proxy-ETR entries.  This configuration is done at the default service ipv4 level under router lisp and therefore inherited by all instance IDs.  With the presence of this configuration, the edge node should forward packets destined for 208.67.222.222 to the Site-1 borders based on the LISP map cache entry.    

5.     Verify the logical forwarding path for the LISP map cache entry.

 

From the perspective of CEF, the Site-1 border node will check to see if the packet meets the LISP eligibility checks.  If met, the packet will be forwarded using the LISP virtual interface 4099.  What is the physical forwarding path for this prefix?

6.     Verify the physical forwarding path for the LISP map cache entry.

 

From the perspective of CEF, the edge node will encapsulate the packet as it is LISP eligible and send it from the interface LISP0.4099 with a destination of either 192.168.10.7 or 192.168.10.8. To reach either of these IP addresses, the GigabitEthernet 1/1/2 interface is used with a next-hop router of 10.10.45.4.

7.     On the border node, display the current LISP map cache.

 

There is no map-cache entry for the desired prefix 208.67.222.222, and the only entry covering that prefix is the LISP default entry 0.0.0.0/0.

8.     Use lig to trigger a control plane lookup.

 

LISP computes this aggregate (208.0.0.0/4) as the largest possible block that does not contain known EID prefixes but does cover the queried address.  This is due to LISP’s history as a method to solve the DFZ and TCAM space concerns. 

9.     Redisplay the LISP map-cache.

 

The mapping-source of this map-cache entry can be displayed to confirm that it was sourced from the transit control plane nodes.

10.  Verify the source of the map-cache entry.

 

The map-cache entry has a state of forward-native but an action of Encapsulating to proxy ETR.  What is the result when the packet is forwarded?  To understand the forwarding decision, the LISP configuration and CEF tables must observed.

11.  Verify the proxy-ETR and use-petr configuration.

 

The router itself is configured as a proxy ETR.  This configuration is done at the default service ipv4 level under router lisp and would otherwise be inherited by all instance IDs if a more specific entry is not added. With the presence of this configuration, the border is a site-local gateway of last resort. 

 

The Site-3 border nodes (192.168.30.7 and 192.168.30.8) are configured as proxy-ETR entries under the instance IDs with the use-petr command.  If the site-local border node receives a negative map-reply from the transit control plane node (a reply indicating there is no entry in the database), traffic will be forwarded to the connected-to-Internet border nodes.

12.  Verify the logical forwarding path for the LISP map cache entry.

 

From the perspective of CEF, the Site-1 border node will check to see if the packet meets the LISP eligibility checks.  If met, the packet will be forwarded using the LISP virtual interface 4099.  What is the physical forwarding path for this prefix?

13.  Verify the physical forwarding path for the LISP map cache entry.

                       

From the perspective of CEF, the Site-1 border node will encapsulate the packet as it is LISP eligible and send it from the interface LISP0.4099 with a destination of either 192.168.30.7 or 192.168.30.8.  To reach either of these IP addresses, the GigabitEthernet 0/0/0 interface is used with a next-hop router of 10.10.71.254 which traverses the Metro-E circuit and SD-Access transit.  Once reaching the connected-to-Internet border nodes in Site-3, the packet will use the default-route learned from BGP to reach the Internet.

 

 

Table 3    Device Platform, Model, and Software Version

Table 4    Cisco DNA Center Package Versions

 

 

CVD – Software-Defined Access Design Guide –  Solution 1.2 (December 2018)

CVD – Software-Defined Access Deployment Guide – Solution 1.2 (October 2018)

CVD – SD-Access Segmentation Design Guide – May 2018

Cisco Design Zone – Design Guides

Cisco Validated Design (CVDs) – Cisco Community

SD-Access Resources – Cisco Community

SD-Access Wireless Design and Deployment Guide

Cisco Identity Services Engine Installation Guide, Release 2.4 – Distributed ISE Deployments

Designing ISE for Scale & High Availability - BRKSEC-3699 Cisco Live 2018 (Orlando)

ISE Security Ecosystem Integration Guides

 

High-Level Overview

Enterprise Architecture Model Topology

Figure 27     Enterprise Architecture Model Topology

 

 

Underlay Connectivity

Figure 28     Underlay Topology

 

Overlay Connectivity

Figure 29     Overlay Topology

 

BGP Autonomous System Overview

Figure 30     BGP Autonomous System Topology

 

Fabric Role Overview

Site-1 Fabric Roles

Figure 31     Fabric Roles – Site-1

 

Site-2 Fabric Roles

Figure 32     Fabric Roles – Site-2

 

Site-3 Fabric Roles

Figure 33     Fabric Roles – Site-3

 

Site-4 Fabric Roles

Figure 34     Fabric Roles – Site-4

 

Site-5 Fabric Roles

Figure 35     Fabric Roles – Site-5

 

Branch Fabric Roles

Figure 36     Fabric Role – Branch Sites

 

Layer-2 Overview

Site-1 Layer-2

Figure 37     Site-1 Layer-2 Topology

 

 

 

Site-2 Layer-2

Figure 38     Site-2 Layer-2 Topology

 

Site-3 Layer-2

Figure 39     Site-3 Layer-2 Topology

Site-4 Layer-2

Figure 40     Site-4 Layer-2 Topology

 

Site-5 Layer-2

Figure 41     Site-5 Layer-2 Topology

Branch Layer-2

The remaining branch sites have the same general matching topology of a pair of branch edge routers connected to a switch stack downstream.

Figure 42     Branch Sites Layer-2 General Topology