Guest

Cisco Intelligent WAN

Cisco Intelligent WAN (IWAN) Design Guide

  • Viewing Options

  • PDF (2.3 MB)
  • Feedback

Contents

Introduction. 3

IWAN Solution Overview.. 3

Design Overview.. 5

Transport-Independent Design. 5

Internet as WAN Transport 6

WAN-Aggregation Designs. 6

IWAN Hybrid Design Model 7

WAN Remote-Site Designs. 8

IP Multicast 9

Deploying Transport-Independent Topology. 9

Design Overview.. 9

DMVPN Hub Routers. 9

Remote Sites - DMVPN Spoke Router Selection. 10

VRFs and Front Door VRF. 11

Design Details. 12

EIGRP.. 13

Encryption and MTU.. 14

DMVPN.. 15

Deployment Details. 17

DMVPN Hub Router Configuration. 18

Example. 33

Example. 44

Deploying Intelligent Path Control 51

Design Overview.. 51

PfR Components. 51

PfR Classes of Service. 52

Design Details. 53

Deployment Details. 53


Introduction

The Cisco® Intelligent WAN (IWAN) solution provides design and implementation guidance for organizations looking to deploy WAN transport with intelligent path control, application optimization, and secure encrypted communications between branch locations while reducing the operating cost of the WAN. IWAN takes full advantage of cost-effective Internet services to increase bandwidth capacity without compromising performance, reliability, or security of collaboration or cloud-based applications.

IWAN Solution Overview

With the advent of globalization, wide area networks (WAN) have become a major artery for communication between remote offices and customers in any corner of the world. Additionally, with data center consolidation, applications are moving to centralized data centers and clouds. WAN now plays an even more critical role as business survival is dependent on the availability and performance of the network.

Until now, the only way to get reliable connectivity with predictable performance was to take advantage of a private WAN using MPLS or leased line service. However, carrier-based MPLS and leased line service can be expensive and are not always cost-effective for an organization to use for WAN transport to support growing bandwidth requirements for remote-site connectivity. Organizations are looking for ways to lower operating budget while adequately providing the network transport for a remote site.

Nearly half of businesses (46 percent) are migrating, or are planning to migrate their WAN to the Internet (Nemertes Research, Benchmark 2012-13 Emerging WAN Trends). Why? As bandwidth demands have increased, the Internet has become a much more stable platform, and the price-to-performance gains are very attractive. However, businesses are primarily deploying “Internet as WAN” in their smaller sites or as a backup path because of the risks. Now this cost-effective, performance-enhancing opportunity can be realized at all your branch offices with Cisco IWAN.

Cisco IWAN can enable organizations to deliver an uncompromised experience over any connection. With Cisco IWAN IT organizations can provide more bandwidth to their branch office connections using less expensive WAN transport options without affecting performance, security, or reliability. With the IWAN solution, traffic is dynamically routed based on application service-level agreement (SLA), endpoint type, and network conditions to deliver the best quality experience. The realized savings from IWAN not only pays for the infrastructure upgrades, but also frees resources for business innovation.

Figure 1 outlines the components of the IWAN solution. Each component is explained in more detail below.

Figure 1. Cisco IWAN Components

Secure and flexible transport-independent design: Using Dynamic Multipoint VPN (DMVPN) IWAN provides capabilities for easy multi-homing over any carrier service offering, including Multiprotocol Label Switching (MPLS), broadband, and cellular 3G/4G/LTE. More importantly, the design simplifies the routing design with a single routing control plane and minimal peering to providers, making it easy for organizations to mix and match and change providers and transport options. Two or more WAN transport providers are recommended to increase network availability up to 99.999%. Additionally, the Cisco DMVPN solution provides an industry-proven and U.S. government FIPS 140-2 certified IPsec solution for data privacy and integrity protection, and automatic site-to-site IP Security (IPsec) tunnels.

Intelligent path control: By using Cisco Performance Routing (PfR), this component improves application delivery and WAN efficiency. PfR dynamically controls data packet forwarding decisions by looking at application type, performance, policies, and path status. PfR protects business applications from fluctuating WAN performance while intelligently load-balancing traffic over the best performing path based on the application policy. PfR monitors the network performance - jitter, packet loss, delay - and makes decisions to forward critical applications over the best performing path based on the application policy. Cisco PfR consists of border routers that connect to the broadband service, and a master controller application supported by Cisco IOS® Software on a router. The border routers collect traffic and path information and send it to the master controller, which detects and enforces the service policies to match the application requirement. Cisco PfR can select an egress WAN path to intelligently load-balance traffic based on circuit costs, to reduce a company's overall communications expenses. IWAN intelligent path control is the key to providing a business-class WAN over Internet transport.

Application optimization: Cisco Application Visibility and Control (AVC) and Cisco Wide Area Application Services (WAAS) provide application performance visibility and optimization over the WAN. With applications becoming increasingly opaque due to increase reuse of well-known ports such as HTTP (port 80), static port classification of application is no longer sufficient. Cisco AVC provides application awareness with deep packet inspection of traffic to identify and monitor applications' performance. Visibility and control at the application level (layer 7) is provided through AVC technologies such as Network-Based Application Recognition 2 (NBAR2), NetFlow, quality of service (QoS), Performance Monitoring, Medianet, and more. Cisco AVC allows IT to determine what traffic is running across the network, tune the network for business-critical services, and resolve network problems. With increased visibility into the applications on the network, better QoS and PfR policies can be enabled to help ensure that critical applications are properly prioritized across the network. Cisco WAAS provides application-specific acceleration capabilities that improve response times while reducing WAN bandwidth requirements.

Secure connectivity protects the WAN and offloads user traffic directly to the Internet. Strong IPsec encryption, zone-based firewalls, and strict access lists are used to protect the WAN over the public Internet. Routing branch users directly to the Internet improves public cloud application performance while reducing traffic over the WAN. Cisco Cloud Web Security (CWS) service provides a cloud-based web proxy to centrally manage and secure user traffic accessing the Internet.

Design Overview

The Intelligent WAN Design Guide provides a design that supports highly available, secure, and optimized connectivity for multiple remote branches.

The primary focus of the design is to allow active/active usage of WAN transports in the following deployment scenarios:

Hybrid WAN design

Dual Internet WAN design

Transport-Independent Design

Dynamic Multipoint VPN (DMVPN) is the building block for the transport-independent design. It provides easy multi-homing over any carrier service offering while providing a single routing control plane with minimal peering to providers. It is a solution for building scalable, automatic, site-to-site IPsec VPNs that support a variety of applications. DMVPN is widely used for encrypted site-to-site connectivity over public or private IP networks and can be implemented on all WAN routers used in this design guide.

DMVPN is used as the solution for data privacy and integrity protection over Internet transport because it supports dynamic, on-demand, full meshed connectivity with a simple hub-and-spoke configuration and a zero-touch hub deployment model for adding remote sites. Additionally, DMVPN supports spoke routers that have dynamically assigned Internet IP addresses.

DMVPN makes use of multipoint generic routing encapsulation (mGRE) tunnels to interconnect the hub to all of the spoke routers. These mGRE tunnels are also sometimes referred to as DMVPN clouds in this context. This technology combination supports unicast, multicast, and broadcast IP, including the ability to run routing protocols within the tunnels.

Internet as WAN Transport

The Internet is essentially a large-scale public WAN composed of multiple interconnected service providers. The Internet can provide reliable, high-performance connectivity between various locations. However, it lacks any explicit guarantees for these connections. Despite its “best effort” nature, the Internet is a sensible choice for a primary transport when it is not feasible to connect with another transport option. Additional resiliency is provided by using the Internet as an alternate transport option.

Internet connections are typically included in discussions relevant to the Internet edge, specifically for the primary site. Remote-site routers also commonly have Internet connections, but do not provide the same breadth of services using the Internet.

The WAN uses the Internet for VPN site-to-site connections and MPLS transport, both as primary WAN transport. For security and other reasons, Internet access at remote sites is traditionally routed through the primary site for centralized security policy control.

WAN-Aggregation Designs

There are two WAN-aggregation design models that are documented in this design guide. The first one is the IWAN Hybrid design model, which uses Internet VPN as transport in conjunction with traditional WAN to provide more bandwidth for key applications while balancing SLA guarantees for the applications. The second is the IWAN Dual Internet design model, which uses dual Internet service providers to further reduce cost while maintaining five nines of reliability for the network transport.

This guide documents the Hybrid deployment module as organizations adopt the IWAN solution to complement their existing WAN connection. From the WAN-aggregation perspective, there are no functional differences between these two methods.

Following figure shows the deployment models for IWAN today.

Figure 2. IWAN Deployment Model

In all of the WAN-aggregation designs, tasks such as IP route summarization are performed at the distribution layer. There are other various devices supporting WAN edge services, and these devices should also connect into the distribution layer.

IWAN Hybrid Design Model

The IWAN Hybrid design model:

Has a single MPLS VPN carrier or a single layer 2 WAN

Uses a single Internet link

Uses Front Door VRF (FVRF) with static routes into MPLS VPN carrier or Internet carrier for IPSec VPN setup

Figure 3 shows the Hybrid design.

Figure 3. IWAN Hybrid Design Model for MPLS WAN

In the IWAN Hybrid design model, the DMVPN hub router connects to the Internet indirectly through a firewall demilitarized zone (DMZ) interface contained within the Internet edge. For details about the connection to the Internet, see the Firewall and IPS Design Guide. The VPN hub routers are connected into the firewall DMZ interface, rather than connected directly with Internet service provider routers.

WAN Remote-Site Designs

This guide documents multiple WAN remote-site designs (Figure 4). They are based on various combinations of WAN transports mapped to the site specific requirements for service levels and redundancy.

Figure 4. WAN Remote-Site Designs

The remote-site designs include single or dual WAN edge routers. These can be a customer edge router (for MPLS or Layer 2 WAN) and a VPN spoke router. In some cases, a single WAN edge router can perform the role of both a customer edge router and VPN spoke router.

Most remote sites are designed with a single router WAN edge; however, certain remote-site types require a dual router WAN edge. Dual router candidate sites include regional office or remote campus locations with large user populations, or sites with business-critical needs that justify additional redundancy to remove single points of failure.

The overall WAN design methodology is based on a primary WAN-aggregation site design that can accommodate all of the remote-site types that map to the various link combinations.

The modular nature of the IWAN network design helps you to create design elements that can be replicated throughout the network.

The WAN-aggregation designs and all of the WAN remote-site designs are standard building blocks in the overall design. Replication of the individual building blocks provides an easy way to scale the network and allows for a consistent deployment method.

IP Multicast

IP Multicast allows a single IP data stream to be replicated by the infrastructure (routers and switches) and sent from a single source to multiple receivers. IP Multicast is much more efficient than multiple individual unicast streams or a broadcast stream that would propagate everywhere. IP telephony music on hold (MOH) and IP video broadcast streaming are two examples of IP Multicast applications.

To receive a particular IP Multicast data stream, end hosts must join a multicast group by sending an Internet Group Management Protocol (IGMP) message to their local multicast router. In a traditional IP Multicast design, the local router consults another router in the network that is acting as a rendezvous point to map the receivers to active sources so that they can join their streams.

The rendezvous point is a control-plane operation that should be placed in the core of the network or close to the IP Multicast sources on a pair of layer 3 switches or routers. IP Multicast routing begins at the distribution layer if the access layer is layer 2 and provides connectivity to the IP Multicast rendezvous point. In designs without a core layer, the distribution layer performs the rendezvous point function.

This design is fully enabled for a single global scope deployment of IP Multicast. The design uses an Anycast rendezvous point implementation strategy. This strategy provides load sharing and redundancy in Protocol Independent Multicast sparse mode (PIM SM) networks. Two rendezvous points share the load for source registration and the ability to act as hot backup routers for each other.

The benefit of this strategy from the WAN perspective is that all IP routing devices within the WAN use an identical configuration referencing the Anycast rendezvous points. IP PIM SM is enabled on all interfaces, including loopbacks, VLANs and subinterfaces.

Deploying Transport-Independent Topology

Design Overview

DMVPN Hub Routers

The most critical devices are the WAN routers that are responsible for reliable IP forwarding and QoS. The amount of bandwidth required at the WAN-aggregation site determines which model of router to use. The choice of whether to implement a single router or dual router is determined by the number of DMVPN clouds that are required in order to provide connections to all of the remote sites.

Cisco ASR 1000 Series Aggregation Services Routers represent the next-generation, modular, services-integrated Cisco routing platform. They are specifically designed for WAN aggregation, with the flexibility to support a wide range of 3- to 130-Mpps (millions of packets per second) packet-forwarding capabilities, 2.5- to 200-Gbps system bandwidth performance, and scaling.

The Cisco ASR 1000 Series is fully modular from both hardware and software perspectives, and the routers have all the elements of a true carrier-class routing product that serves both enterprise and service provider networks.

This design uses the following routers as DMVPN hub routers:

Cisco ASR 1002-X Router configured with an Embedded Services Processor (ESP) default bandwidth of 5Gbps upgradable with software licensing options to 10 Gbps, 20 Gbps, and 36 Gbps

Cisco ASR 1002 Router configured with an Embedded Services Processor 5 (ESP5)

Cisco ASR 1001 Router fixed configuration with a 2.5 Gbps ESP

All of the design models can be constructed using any of the DMVPN hub routers listed in Table 1. You should consider the following: the forwarding performance of the router using an Ethernet WAN deployment with broad services enabled, the router's alignment with the suggested design model, and the number of remote sites.

Table 1. DMVPN Hub Router Options

Option

ISR 4451-X

ASR 1001

ASR 1002

ASR 1002-X

Ethernet WAN with IWAN services

430 Mbps

250 Mbps

500 Mbps

500 Mbps - 1.5 Gbps

Software redundancy option

No

Yes

Yes

Yes

Redundant power supply

Optional

Default

Default

Default

Supported design models

All

All

All

All

Suggested number of remote sites

100

100

250

250+

Remote Sites - DMVPN Spoke Router Selection

There are many factors to consider in the selection of the WAN remote-site routers. Among those, and key to the initial deployment, is the ability to process the expected amount and type of traffic. You also need to make sure that you have enough interfaces, enough module slots, and a properly licensed Cisco IOS Software image that supports the set of features that is required by the topology. Cisco tested multiple integrated service router models as DMVPN spoke routers. Expected performance is shown in Table 2

Table 2. WAN Remote-Site Cisco Integrated Service Router Options

Option

892fsp

2911+ISM

2951+ISM

3945+ISM

4451-X

Ethernet WAN with IWAN services1

18 Mbps

36 Mbps

85 Mbps

136 Mbps

430Mbps

On-board GE ports2

10

3

3

3

4

Service module slots

0

1

2

4

2

Redundant power supply option

No

No

No

Yes

Yes

Notes:

1. The performance numbers are conservative numbers obtained when the router is passing IMIX traffic with IWAN services (DMVPN, EIGRP, NBAR2/DSCP Marking, PfRv2, H-QOS) enabled.

2. A single-router, dual-link remote site requires four router interfaces when using a port channel to connect to an access or distribution layer. Add the EHWIC-1GE-SFP-CU to the Cisco 2900 and 3900 Series Integrated Services Routers (ISRs) in order to provide the additional WAN-facing interface.

The DMVPN spoke routers at the WAN remote sites connect to the Internet directly through a router interface. More details about the security configuration of the remote-site routers connected to the Internet are discussed later in this guide. The single link DMVPN remote site is the basic of building blocks for any remote location.

The IP routing is straightforward and can be handled entirely by static routes at the WAN-aggregation site and static default routes at the remote site in conjunction with FVRF configuration on the WAN interface. This ensures separation of the routing plane used for IPSec tunnel setup in the FVRF and the data plane for carrying user traffic inside the DMVPN tunnel. The dual-router, dual-link design continues to improve upon the level of high availability for the site. This design can tolerate the loss of the primary router and traffic can be rerouted using the secondary router (through the alternate path). Figure 5 outlines the IWAN Hybrid spoke design.

Figure 5. IWAN Hybrid Spoke Design

For IWAN Hybrid Transport Independent design, DMVPN is deployed across MPLS and Internet Transport. This greatly simplifies the routing by using a single routing protocol, creating a succine routing domain. The design provides active-active WAN paths that take full advantage of DMVPN for consistent IPsec overlay. The MPLS and Internet connections can be terminated on a single router, or terminated on two separate routers for additional resiliency. The same design can be used over MPLS, Internet, or 3G/4G transports, making the design tranport-independent. The single-router and dual-router options are shown respectively in Figure 5.

VRFs and Front Door VRF

Virtual Routing and Forwarding (VRF) is a technology used in computer networks that allows multiple instances of a routing table to co-exist within the same router at the same time. Because the routing instances are independent, you can use the same or overlapping IP addresses without conflicting with each other.

You can implement VRF in a network device by having distinct routing tables, also known as Forwarding Information Bases (FIBs), one per VRF. Alternatively, a network device may have the ability to configure different virtual routers, where each one has its own FIB that is not accessible to any other virtual router instance on the same device.

The simplest form of VRF implementation is VRF Lite. In this implementation, each router within the network participates in the virtual routing environment on a peer-by-peer basis. VRF Lite configurations are only locally significant.

The IP routing policy used in this design for the WAN remote sites does not allow direct Internet access for web browsing or other uses; any remote-site hosts that access the Internet must do so through the Internet edge at the primary site. The end hosts require a default route for all Internet destinations; however, this route must force traffic across the primary or secondary WAN transport DMVPN tunnels. This requirement conflicts with the more general VPN spoke router requirement for an Internet-facing default route to bring up the VPN tunnel.

The multiple default routes conundrum is solved through the use of VRFs on the router. A router can have multiple routing tables that are kept logically separate on the device. This separation is similar to a virtual router from the forwarding plane perspective. The global VRF corresponds to the traditional routing table, and additional VRFs are given names and route descriptors. Certain features on the router are VRF-aware, including static routing and routing protocols, interface forwarding, and IPsec tunneling. This set of features is used in conjunction with DMVPN to permit the use of multiple default routes for both the DMVPN hub routers and DMVPN spoke routers. This combination of features is referred to as front-door VRF (FVRF); because the VRF faces the Internet and the router internal interfaces and the mGRE tunnel all remain in the global VRF (see Figure 6). More technical details regarding FVRF can be found in the Technical Feature Supplement appendix.

Design Details

The DMVPN hub routers connect to a resilient switching device in the distribution layer and in the DMZ. The DMVPN routers use EtherChannel connections consisting of two port bundles. This design provides both resiliency and additional forwarding performance. Additional forwarding performance can be accomplished by increasing the number of physical links within an EtherChannel.

The DMVPN hub routers must have sufficient IP-routing information to provide end-to-end reachability. Maintaining this routing information typically requires a routing protocol. Cisco Enhanced Interior Gateway Routing Protocol (EIGRP) is used for this purpose. Multiple EIGRP processes are used - one for internal routing on the LAN (EIGRP-100) and a separate one for the DMVPN networks (EIGRP-IWAN). The primary reason for the separate EIGRP processes is to ensure compatibility with the route selection process at the WAN-aggregation site when deploying other Cisco Validated Deign WAN designs. This method ensures DMVPN learned routes appear as EIGRP external routes after they are redistributed into the EIGRP-100 process used on the campus LAN.

An example of the routing details for the IWAN Hybrid design model is shown in figure 7.

Figure 7. IWAN Hybrid Design - Routing Details

At the WAN-aggregation site, the Internet DMVPN router is connected to the DMZ-VPN that provides Internet connectivity. The Internet DMVPN hub router uses FVRF and has a static default route with the INET-PUBLIC VRF pointing to the firewall DMZ interface. On the MPLS DMVPN router, it also uses FVRF and has a static default route with the INET-MPLS pointing to the provider edge router interface.

EIGRP

IWAN solution uses EIGRP as the primary routing protocol because it is easy to configure, and does not require a large amount of planning. EIGRP has flexible summarization and filtering capability, and can scale to large networks. As networks grow, the number of IP prefixes or routes in the routing tables grows as well. By performing IP summarization, you can reduce the amount of bandwidth, processor, and memory necessary to carry large route tables, and reduce convergence time associated with a link failure.

In this design, EIGRP process IWAN is the primary EIGRP process used for the DMVPN tunnels and is referred to as EIGRP-IWAN. Unlike the classic configuration, where an autonomous system number is used for EIGRP, in IWAN design the EIGRP name mode is used for creating an EIGRP routing instance.

Router eigrp < virtual-instance-name >

There are several benefits for running EIGRP name mode:

Name mode allows you to create a single instance of EIGRP that can be used for all address family types, such as IPv4, IPv6, VRFs, SAF, etc.

It supports multiple VRFs and is limited only by available system resources.

It is a single place for all commands needed to completely define an instance. You no longer have to split configuration between the router EIGRP and the interface configuration in the classic mode.

Tech Tip

You can use the eigrp upgrade-cli command to convert an existing EIGRP configuration from classic mode to named mode. If multiple classic mode configurations exist, you must use this command per EIGRP autonomous system number in classic mode.

Encryption and MTU

The encryption process creates an additional header and trailer on to the original IP packet. The original packet, payload, and header are encrypted and then encapsulated with a new header (or multiple headers) and transmitted across the network. The additional headers introduce a certain amount of overhead to the overall packet length. Table 3 highlights the packet overhead associated with encryption based on the additional headers required for various combinations of IPsec and GRE.

Table 3. Overhead Associated with IPsec and GRE

Encapsulation

Overhead

GRE only

24 bytes

IPsec (transport mode)

36 bytes

IPsec (tunnel mode)

52 bytes

IPsec (transport mode) plus GRE

60 bytes

IPsec (tunnel mode) plus GRE

76 bytes

There is a maximum transmission unit (MTU) parameter for every link in an IP network and typically the MTU is 1500 bytes. IP packets larger than 1500 bytes must be fragmented when transmitted across these links. Fragmentation is not desirable and can impact network performance. To avoid fragmentation, the original packet size plus overhead must be 1500 bytes or less, which means that the sender must reduce the original packet size. To account for other potential overhead, Cisco recommends that you configure tunnel interfaces with a 1400 byte MTU.

There are dynamic methods for network clients to discover the path MTU, which allow the clients to reduce the size of packets they transmit. However, in many cases, these dynamic methods are unsuccessful, typically because security devices filter the necessary discovery traffic. This failure to discover the path MTU creates the need for a method that can reliably inform network clients of the appropriate packet size. The solution is to implement the iptcp adjust-mss [size] command on the WAN routers, which influences the TCP maximum segment size (MSS) value reported by end hosts.

The MSS defines the maximum amount of data that a host is willing to accept in a single TCP/IP datagram. The MSS value is sent as a TCP header option only in TCP SYN segments. Each side of a TCP connection reports its MSS value to the other side. The sending host is required to limit the size of data in a single TCP segment to a value less than or equal to the MSS reported by the receiving host.

The IP and TCP headers combine for 40 bytes of overhead, so the typical MSS value reported by network clients will be 1460. This design includes encrypted tunnels with a 1400-byte MTU, so the MSS used by endpoints should be configured to be 1360 to minimize any impact of fragmentation. In this solution, you implement the ip tcp adjust-mss 1360 command on all tunnel interfaces.

DMVPN

This solution uses the Internet for WAN transport. For data security and privacy concerns any site-to-site traffic that traverses the Internet must be encrypted. Multiple technologies can provide encryption, but the method that provides the best combination of performance, scale, application support, and ease of deployment is DMVPN.

The IWAN independent transport solution requires a DMVPN dual-cloud design, each with a single hub router. The DMVPN routers use tunnel interfaces that support IP unicast as well as IP multicast and broadcast traffic, including the use of dynamic routing protocols. After the initial spoke-to-hub tunnel is active, it is possible to create dynamic spoke-to-spoke tunnels when site-to-site IP traffic flows require it.

The information required by a spoke to set up dynamic spoke-to-spoke tunnel (Figure 8) is provided by the Next Hop Resolution Protocol (NHRP). Spoke-to-spoke tunnels allow for the optimal routing of traffic between locations without traffic hair-pinning at the hub. Idle spoke-to-spoke tunnels gracefully time out after a period of inactivity.

It is common for a firewall to be placed between the DMVPN hub routers and the Internet. In many cases, the firewall may provide Network Address Translation (NAT) from an internal RFC-1918 IP address (such as 10.4.128.33) to an Internet-routable IP address. The DMVPN solution works well with NAT but requires the use of IPsec transport mode to support a DMVPN hub behind static NAT.

DMVPN requires the use of Internet Security Association and Key Management Protocol (ISAKMP) keep alive for Dead Peer Detection (DPD), which is essential to facilitate fast reconvergence and for spoke registration to function properly in case a DMVPN hub is reloaded. This design enables a spoke to detect that an encryption peer has failed and that the ISAKMP session with that peer is stale, which then allows a new one to be created. Without DPD, the IPsec security association must time out (the default is 60 minutes) and when the router cannot renegotiate a new security association, a new ISAKMP session is initiated. The maximum wait time is approximately 60 minutes.

Figure 8. DMVPN Dynamic Spoke-Spoke Tunnel

One of the key benefits of the DMVPN solution is that the spoke routers can use dynamically assigned addresses, often using DHCP from an Internet provider. The spoke routers can take advantage of an Internet default route for reachability to the hub routers and also other spoke addresses.

The DMVPN hub routers have static IP addresses assigned to their public-facing interfaces. This configuration is essential for proper operation as each of the spoke routers have these IP addresses embedded in their configurations. Figure 9 displays a hub technology and associated addresses.

Deployment Details

Figure 9. Hub Topology and Addresses

The procedures in this section provide examples for some settings (Table 4). The actual settings and values that you use are determined by your current network configuration. This process is used for the Dual DMVPN design (repeat for each DMVPN hub router).

Table 4. Examples of Parameters Used in the Deployment

Host Name

Loopback IP Address

Port Channel IP Address

VPN-ASR1002-1

10.4.32.241/32

10.4.32.2/28

VPN-ASR1002-2

10.4.32.243/32

10.4.32.3/28

DMVPN Hub Router Configuration

Procedure 1: Configure the WAN Aggregation Routers

This process applies to both routers connecting to the MPLS network customer edge router and the router connecting to the Internet. The example shown includes the configuration example for Internet DMVPN router, but the same procedure is reused for MPLS DMVPN router.

Reader Tip

This process assumes that the distribution switch has already been configured following the guidance in the Campus Wired LAN Design Guide. Only the procedures required to support the integration of the WAN aggregation router into the deployment are included.

The LAN distribution switch is the path to the organization’s main campus and data center. A layer 3 port-channel interface connects to the distribution switch to the WAN aggregation router and the internal routing protocol peers across this interface.

Tech Tip

As a best practice, use the same channel numbering on both sides of the link where possible.

Within this design, there are features and services that are common across all WAN aggregation routers. These are system settings that simplify and secure the management of the solution.

Step 1. Configure the common features and services for the routers, including:

Device host name for ease of identifying the device

The local login account and password, which provides basic access

Network Time Protocol (NTP) to synchronize system clock network devices

service timestamps debug datetime msec localtime
service timestamps log datetime msec localtime
service password-encryption
!
hostname VPN-ASR1002-1
!
username admin password c1sco123
enable secret c1sco123
aaa new-model
!
clock timezone PST -8
clock summer-time PDT recurring
ntp server 10.4.48.17

Step 2. Configure device management protocols

Use Secure Shell (SSH) for secure management of the network device

Specify the transport preferred none on vty lines to prevent errant connection attempts from the mistyped command-line interface (CLI) commands

Enable Simple Network Management Protocol version 2c (SNMPv2c) for the read-only and read-write community string

ip domain-name cisco.local
!
ip ssh version 2
no ip http server
ip http secure-server
snmp-server community cisco RO
snmp-server community cisco123 RW
!
line vty 0 15
transport input ssh
transport preferred none
line con 0
logging synchronous

(Optional) In networks where network operational support is centralized you can increase network security by using an access list to limit the networks that can access your device. In this example, only devices on the 10.4.48.0/24 network will be able to access the device through SSH or SNMP.

access-list 55 permit 10.4.48.0 0.0.0.255
line vty 0 15
access-class 55 in
!
snmp-server community cisco RO 55
snmp-server community cisco123 RW 55

Tech Tip

If you configure an access list on the vty interface you may lose the ability to use SSH to log in from one router to the next for hop-by-hop troubleshooting.

Step 3. (Optional) Configure centralized user authentication.

As networks scale in the number of devices, Cisco recommends the best practice to use a centralized Authentication, Authorization, and Accounting (AAA) service to reduce operational tasks per device and provides an audit log of user access for security compliance and root-cause analysis. When AAA is enabled for access control, all management access to the network infrastructure devices (SSH and HTTPS) is controlled by AAA.

TACACS+ is the primary protocol used to authenticate management logins on the infrastructure devices to the AAA server. A local AAA user database is also defined on each network infrastructure device to provide a fallback authentication source in case the centralized TACACS+ server is unavailable.

tacacs server TACACS-SERVER-1
address ipv4 10.4.48.15
key SecretKey
!
aaa group server tacacs+ TACACS-SERVERS
server name TACACS-SERVER-1
!
aaa authentication login default group TACACS-SERVERS local
aaa authorization exec default group TACACS-SERVERS local
aaa authorization console
ip http authentication aaa

Step 4. Configure an in-band management interface.

The loopback interface is a logical interface that is always reachable as long as the device is powered on and any IP interface is reachable to the network. Because of this capability, the loopback address is the best way to manage the switch in band. Layer 3 processes and features are also bound to the loopback interface to ensure process resiliency.

The loopback address is commonly a host address with a 32-bit address mask. Allocate the loopback address from the IP address block that the distribution switch summarizes to the rest of the network.

interface Loopback 0
ip address 10.4.32.243 255.255.255.255
ip pim sparse-mode

The ip pim sparse-mode command will be explained further in the process.

Bind the device processes for SNMP, SSH, Protocol Independent Multicast (PIM), TACACS+ and Network Time Protocol (NTP) to the loopback interface address for optimal resiliency:

snmp-server trap-source Loopback0
ip ssh source-interface Loopback0
ip pim register-source Loopback0
ip tacacs source-interface Loopback0
ntp source Loopback0

Step 5. Enable IP Multicast routing.

IP Multicast allows a single IP data stream to be replicated by the infrastructure (routers and switches) and sent from a single source to multiple receivers. Using IP Multicast is much more efficient than using multiple individual unicast streams or a Broadcast stream that would propagate everywhere. IP telephony MOH and IP video broadcast streaming are two examples of IP Multicast applications.

To receive a particular IP Multicast data stream, end hosts must join a multicast group by sending an Internet Group Management Protocol (IGMP) message to their local multicast router. In a traditional IP Multicast design, the local router consults another router in the network that is acting as a rendevous point to map the receivers to active sources so they can join their streams.

In this design, which is based on sparse mode multicast operation, auto-rendevous point is used to provide a simple, yet scalable way to provide a highly resilient rendevous point environment.

Enable IP Multicast routing on the platforms in the global configuration mode.

ip multicast-routing

The Cisco ASR 1000 Series router requires the distributed keyword.

ip multicast-routing distributed

Every layer 3 switch and router must be configured to discover the IP Multicast rendevous point with autorp. Use the ip pim autorp listener command to allow for discovery across sparse mode links. This configuration provides for future scaling and control of the IP Multicast environment and can change based on network needs and design.

ip pim autorp listener

All layer 3 interfaces in the network must be enabled for sparse mode multicast operation.

ip pim sparse-mode

Procedure 2: Configure VRF Lite

Table 5. VRF and Crypto Parameters Example for DMVPN Hub

Parameter

DMVPN-INTERNET Cloud

DMVPN-MPLS Cloud

vrf

INET-PUBLIC

INET-MPLS

rd

65512:1

65512:2

crypto keyring

DMVPN-KEYRING1

DMVPN-KEYRING2

crypto isakmp profile

FVRF-ISAKMP-INET-PUBLIC

FVRF-ISAKMP-INET-MPLS

crypto ipsec profile

DMVPN-INTERNET

DMVPN-MPLS

Tunnel interface

Tunnel 10

Tunnel 20

A dedicated Internet-facing and MPLS-facing VRF is created to support FVRF for DMVPN-Internet cloud and DMVPN-MPLS cloud respectively. The VRF name is arbitrary, but it is useful to select a name that describes the VRF. This design uses VRF Lite so that the route distinguisher value can be chosen arbitrarily. It is a best practice to use the same VRF and route distinguisher combination across multiple devices when using VRFs in a similar manner. However, this convention is not strictly required.

ip vrf INET-PUBLIC
rd 65512:1

Tech Tip

Command Reference:

A route distinguisher is either access service network (ASN)-related (composed of an ASN and an arbitrary number) or IP-address-related (composed of an IP address and an arbitrary number)

You can enter a route distinguisher in either of these formats:

16-bit autonomous-system-number: your 32-bit number

For example, 65512:1

32-bit IP address: your 16-bit number

For example, 192.168.122.15:1

Procedure 3: Connect WAN Aggregation Router to the Service Provider

Option 1: Connect to Internet DMZ

The DMVPN hub requires a connection to the Internet, and in this design the DMVPN hub is connected through a Cisco AS A5500 Adaptive Security Appliance using a DMZ interface specifically created and configured for a VPN termination router.

Step 1. Enable the interface, select the VRF, and assign the IP address.

The IP address that you use for the Internet-facing interface of the DMVPN hub router must be an Internet routable address. There are two possible methods to accomplish this task:

Assign a routable IP address directly to the router

Assign a non-routable RFC-1918 address directly to the router and use a static NAT on the Cisco ASA 5500 to translate the router IP address to a routable IP address.

This design assumes that the Cisco ASA 5500 is configured for static NAT for the DMVPN hub router.

The DMVPN design is using FVRF so this interface must be placed into the VRF configured in the previous procedure.

interface GigabitEthernet0/0/3
ip vrf forwarding INET-PUBLIC
ip address 192.168.18.10 255.255.255.0
no shutdown

Step 2. Configure the VRF specific default routing

The VRF created for FVRF must have its own default route to the Internet. This default route points to the ASA 5500 DMZ interface IP address. Figure 10 offers different views for DMZ connection.

ip route vrf INET-PUBLIC 0.0.0.0 0.0.0.0 192.168.18.1

Figure 10. Physical and Logical Views for DMZ Connection

Option 2: Connect to MPLS Provider Edge Router

Step 1. Assign the interface bandwidth.

The bandwidth value should correspond to the actual interface speed. Or, if you are using a subrate service, use the policed rate from the carrier.

The example shows a Gigabit interface (1000 Mbps) with a subrate of 300 Mbps.

interface GigabitEthernet0/0/3
bandwidth 300000

Tech Tip

Command reference:
bandwidth kbps
(300 Mbps = 300,000 kbps)

Step 2. Assign the IP address and netmask of the WAN interface.

The IP addressing used between customer edge and provider edge routers must be negotiated with your MPLS carrier. Typically, a point-to-point netmask of 255.255.255.252 is used.

interface GigabitEthernet0/0/3
ip vrf forwarding INET-MPLS
ip address 192.168.3.1 255.255.255.252
no shutdown

Step 3. Administratively enable the interface and disable Cisco Discovery Protocol (CDP).

We do not recommend the use of CDP on external interfaces.

interface GigabitEthernet0/0/3
no cdp enable

Step 4. Configure the VRF-Specific Default Routing

The VRF created for FVRF must have its own default route to the MPLS network. This default route points to the MPLS PE interface IP address.

ip route vrf INET-MPLS 0.0.0.0 0.0.0.0 192.168.3.2

Procedure 4: Configure ISAKMP and IPsec

Step 1. Configure the crypto keyring.

The crypto keyring defines a pre-shared key (or password) valid for IP sources that are reachable within a particular VRF. This key is a wildcard pre-shared key if it applies to any IP source. A wildcard key is configured using the 0.0.0.0 0.0.0.0 network/mask combination.

crypto keyring DMVPN-KEYRING vrf INET-PUBLIC
pre-shared-key address 0.0.0.0 0.0.0.0 key cisco123

Step 2. Configure the ISAKMP policy.

The ISAKMP policy for DMVPN uses:

Advanced Encryption Standard (AES) with a 256-bit key

Secure Hash Standard (SHA)

Authentication by Pre-Shared Key (PSK)

Diffie-Hellman group: 2

crypto isakmp policy 10
encr aes 256
hash sha
authentication pre-share
group 2

Step 3. Create the ISAKMP profile.

The ISAKMP profile creates an association between an identity address, a VRF, and a crypto keyring. A wildcard address within a VRF is referenced with 0.0.0.0.

crypto isakmp profile FVRF-ISAKMP-INET-PUBLIC
keyring DMVPN-KEYRING
match identity address 0.0.0.0 INET-PUBLIC

Step 4. Define the IPsec transform set.

A transform set is an acceptable combination of security protocols, algorithms, and other settings to apply to IPsec-protected traffic. Peers agree to use a particular transform set when protecting a particular data flow.

The IPsec transform set for DMVPN uses the following:

Encapsulating Security Payload (ESP) with the 256-bit AES encryption algorithm

ESP with the SHA (Hashed Message Authentication Code [HMAC] variant) authentication algorithm

Because the DMVPN hub router is behind a NAT device, the IPsec transform must be configured for transport mode.

crypto ipsec transform-set AES256/SHA/TRANSPORT esp-aes 256 esp-sha-hmac
mode transport

Step 5. Create the IPSec profile.

The IPsec profile creates an association between an ISAKMP profile and an IPsec transform-set.

crypto ipsec profile DMVPN-INTERNET
set transform-set AES256/SHA/TRANSPORT
set isakmp-profile FVRF-ISAKMP-INET-PUBLIC

Procedure 5: Configure the mGRE Tunnel

Table 6. DMVPN Tunnel Parameters

DMVPN Cloud

Tunnel Interface

Tunnel IP Address

EIGRP AS

NHRP Network ID

Internet DMVPN

Tunnel 10

10.4.34.1/23

IWAN

101

MPLS DMVPN

Tunnel 20

10.4.36.1/23

IWAN

102

Step 1. Configure basic DMVPN tunnel interface settings.

Tunnel interfaces are created as they are configured. The tunnel number is arbitrary, but it is best to begin tunnel numbering at 10 or above, because other features deployed in this design may also require tunnels and they may select lower numbers by default.

The bandwidth setting should be set to match the Internet bandwidth of the respective primary or secondary carrier.

Configure the IP MTU to 1400 and the ip tcp adjust-mss to 1360. There is a 40-byte difference, which corresponds to the combined IP and TCP header length.

interface Tunnel 10
bandwidth 10000
ip address 10.4.34.1 255.255.254.0
no ip redirects
ip mtu 1400
ip tcp adjust-mss 1360

DMVPN uses multipoint GRE (mGRE) tunnels. This type of tunnel requires a source interface only. Use the same source interface that you use to connect to the Internet. Set the tunnel vrf command to the VRF defined previously for FVRF.

Enabling encryption on this interface requires the application of the IPsec profile configured in the previous procedure.

interface Tunnel 10
tunnel source GigabitEthernet0/0/3
tunnel mode gre multipoint
tunnel vrf INET-PUBLIC
tunnel protection ipsec profile DMVPN-INTERNET

Step 2. Configure NHRP.

NHRP is used by remote routers for setting up dynamic spoke-to-spoke tunnels.

NHRP requires all devices within a DMVPN cloud to use the same network ID and authentication key. The NHRP cache holdtime should be configured to 600 seconds.

EIGRP (configured in the following procedure) relies on a multicast transport and requires NHRP to automatically add routers to the multicast NHRP mappings.

interface Tunnel 10
ip nhrp authentication cisco123
ip nhrp map multicast dynamic
ip nhrp network-id 101
ip nhrp holdtime 600

Step 3. Enable PIM non-broadcast multiple access (NBMA) mode for the DMVPN tunnel.

Spoke-to-spoke DMVPN networks present a unique challenge because the spokes cannot directly exchange information with one another, even though they are on the same logical network. This inability to directly exchange information can also cause problems when running IP Multicast.

To resolve this issue requires a method where each remote PIM neighbor has its join messages tracked separately. A router in PIM nonbroadcast multaccess (NBMA) mode treats each remote PIM neighbor as if it were connected to the router through a point-to-point link.

Tech Tip

Do not enable PIM on the Internet DMZ interface, as no multicast traffic should be requested from this interface.

interface Tunnel 10
ip pim sparse-mode
ip pim nbma-mode

Procedure 6: Configure the EIGRP Routing Process

A separate EIGRP process is used over the DMVPN network. The primary reason for the additional process is to ensure that routes learned from the WAN remotes appear as EIGRP external routes on the WAN distribution switch. This helps in limiting the EIGRP query domain and increases the network scalability and stability.

Step 1. Enable an EIGRP process for DMVPN.

Configure EIGRP for the DMVPN mGRE interface. Routes from the other EIGRP process are redistributed. Because the routing protocol is the same, no default metric is required.

The tunnel interface is the only EIGRP interface, and you need to explicitly list its network range.

router eigrp IWAN
!
address-family ipv4 unicast autonomous-system 200
network 10.4.34.0 0.0.1.255
eigrp router-id 10.4.32.243
exit-address-family
!

Step 2. Configure EIGRP properties for the tunnel interface.

Spoke-to-spoke in DMVPN phase 2 networks present a unique challenge because the spokes cannot directly exchange information with one another, even though they are on the same logical network. This limitation requires that the DMVPN hub router advertise routes from other spokes on the same network. This advertisement of these routes would normally be prevented by split horizon, and can be overridden by the no ip split-horizon command under the EIGRP name mode. In addition, the no next-hop-self command preserves the next hop routing information for the dynamic spoke-to-spoke tunnel setup.

The EIGRP hello interval is increased to 20 seconds and the EIGRP hold time is increased to 60 seconds to accommodate up to 500 remote sites on a single DMVPN cloud.

router eigrp IWAN
!
address-family ipv4 unicast autonomous-system 200
af-interface Tunnel 10
hello-interval 20
hold-time 60
no next-hop-self
no split-horizon
exit-af-interface
exit-address-family
!

Step 3. Tag and redistribute the routes.

This design uses route tag to prevent routing loop between the two DMVPN networks. Routes learned over the MPLS DMVPN network are tagged with 200 while routes learned over the Internet DMVPN are tagged with 100. The routes are shared between the MPLS DMVPN and Internet DMVPN hub routers to ensure reachability. However, the tagging scheme ensures the routes learned over MPLS DMVPN path are not readvertised out to Internet DMVPN spokes and vice versa to prevent routing loops.

It is important to tightly control how routing information is shared between different routing domains; otherwise, it is possible to experience route flapping, where certain routes are repeatedly installed and withdrawn from the device routing tables. Proper route control ensures the stability of the routing table.

An inbound distribute list is used on the WAN aggregation routers to tag the routes so they can be differentiated later on. An outbound distribute list is used to select which routes are allowed to be advertised out. The specific route tags in use are shown in Table 7. In the route map, a prefix list is used to define what routes are originated from the data center. In this case, we are allowing default route, the summary route for data center, and the loopback addresses with 32 network masks.

Table 7. Route Tag Information for DMVPN Hub Router

Tag

Route Source

Route Map Name

Tag Method

Action

100

DMVPN over Internet

INTERNET-DC1-IN
INTERNET-DC1-OUT

Explicit

Tag

200

DMVPN over MPLS

MPLS-DC1-IN
MPLS-DC1-OUT

Explicit

Tag

This example includes all WAN route sources in the reference design. Depending on the actual design of your network, you might need to use more tags.

router eigrp IWAN
!
address-family ipv4 unicast autonomous-system 200
!
topology base
distribute-list route-map INTERNET-DC1-IN in Tunnel10
distribute-list route-map INTERNET-DC1-OUT out Tunnel10
redistribute static
exit-af-topology
exit-address-family
!
ip prefix-list DC1-LOCAL-ROUTES seq 5 permit 0.0.0.0/0
ip prefix-list DC1-LOCAL-ROUTES seq 10 permit 10.9.0.0/16
ip prefix-list DC1-LOCAL-ROUTES seq 15 permit 10.0.0.99/32
!
route-map INTERNET-DC1-IN deny 10
match ip address prefix-list DC1- LOCAL-ROUTES
!
route-map INTERNET-DC1-IN permit 20
set tag 100
!
route-map INTERNET-DC1-OUT permit 10
match ip address prefix-list DC1 - LOCAL-ROUTES
set tag 100
!
route-map INTERNET-DC1-OUT permit 20
description Advertise routes learned from MPLS DMVPN cloud
match tag 100
!

Procedure 7: Configure the MPLS DMVPN Hub

Repeat Procedures 1 - 6 in the DMVPN Hub Router Configuration section to provision the MPLS DMVPN hub router. Refer to Table 6 and replace the attributes used for Internet with attributes for MPLS as listed in the table.

Remote-Site Dual DMVPN Spokes Configuration

Use this set of procedures for remote site with dual router, dual link design.

This set of procedures is for the initial configuration of a remote site DMVPN spoke router. You can use these procedures when you configure each router of the dual-router, dual-link design.

Procedure 1: Configure the WAN Remote Router

Within this design, there are features and services that are common across all WAN remote site routers. These are system settings that simplify and secure the management of the solution.

Step 1. Configure the common features and services for the routers, including:

Device host name for ease of identifying the device

The local login account and password to provide basic access

Network Time Protocol (NTP) to synchronize system clock network devices

service timestamps debug datetime msec localtime
service timestamps log datetime msec localtime
service password-encryption
!
hostname [hostname]
!
username admin password c1sco123
enable secret c1sco123
aaa new-model
!
clock timezone PST -8
clock summer-time PDT recurring
ntp server 10.4.48.17
ntp update-calendar
!

Step 2. Configure device management protocols.

Use Secure Shell (SSH) for secure management of the network device

Specify the transport preferred none on vty lines to prevent errant connection attempts from the mistyped CLI commands

Enable SNMPv2c for read-only and read-write community strings

ip domain-name cisco.local
!
ip ssh version 2
no ip http server
ip http secure-server
snmp-server community cisco RO
snmp-server community cisco123 RW
!
line vty 0 15
transport input ssh
transport preferred none
line con 0
logging synchronous

(Optional) In networks where network operational support is centralized you can increase network security by using an access list to limit the networks that can access your device. In this example, only devices on the 10.4.48.0/24 network will be able to access the device through SSH or SNMP.

access-list 55 permit 10.4.48.0 0.0.0.255
line vty 0 15
access-class 55 in
!
snmp-server community cisco RO 55
snmp-server community cisco123 RW 55

Tech Tip

If you configure an access list on the vty interface you may lose the ability to use SSH to log in from one router to the next for hop-by-hop troubleshooting.

Step 3. (Optional) Configure centralized user authentication.

As networks scale in the number of devices to maintain, it poses an operational burden to maintain local user accounts on every device. A centralized AAA service reduces operational tasks per device and provides an audit log of user access for security compliance and root-cause analysis. When AAA is enabled for access control, all management access to the network infrastructure devices (SSH and HTTPS) is controlled by AAA.

TACACS+ is the primary protocol used to authenticate management logins on the infrastructure devices to the AAA server. A local AAA user database is also defined on each network infrastructure device to provide a fallback authentication source in case the centralized TACACS+ server is unavailable.

tacacs server TACACS-SERVER-1
address ipv4 10.4.48.15
key SecretKey
!
aaa group server tacacs+ TACACS-SERVERS
server name TACACS-SERVER-1
!
aaa authentication login default group TACACS-SERVERS local
aaa authorization exec default group TACACS-SERVERS local
aaa authorization console
ip http authentication aaa

Step 4. Configure an in-band management interface.

The loopback interface is a logical interface that is always reachable as long as the device is powered on and any IP interface is reachable to the network. Because of this capability, the loopback address is the best way to manage the switch in-band. Layer 3 processes and features are also bound to the loopback interface to ensure process resiliency.

The loopback address is commonly a host address with a 32-bit address mask. Allocate the loopback address from a unique network range that is not part of any other internal network summary range.

interface Loopback 0
ip address [ip address] 255.255.255.255
ip pim sparse-mode

The ip pim sparse-mode command will be explained in the next step.

Bind the device processes for SNMP, SSH, PIM, TACACS+, and NTP to the loopback interface address for optimal resiliency.

snmp-server trap-source Loopback0
ip ssh source-interface Loopback0
ip pim register-source Loopback0
ip tacacs source-interface Loopback0
ntp source Loopback0

Step 5. Configure IP Multicast routing.

IP Multicast allows a single IP data stream to be replicated by the infrastructure (routers and switches) and sent from a single source to multiple receivers. Using IP Multicast is much more efficient than using multiple individual unicast streams or a broadcast stream that would propagate everywhere. IP telephony MOH and IP video broadcast streaming are two examples of IP Multicast applications.

To receive a particular IP Multicast data stream, end hosts must join a multicast group by sending an IGMP message to their local multicast router. In a traditional IP Multicast design, the local router consults another router in the network that is acting as a rendevous point to map the receivers to active sources so they can join their streams.

In this design, which is based on sparse mode multicast operation, auto-rendevous point is used to provide a simple yet scalable way to provide a highly resilient rendevous point environment.

Enable IP Multicast routing on the platforms in the global configuration mode.

ip multicast-routing

Every layer 3 switch and router must be configured to discover the IP Multicast rendevous point with autorp. Use the ip pim autorp listener command to allow for discovery across sparse mode links. This configuration provides for future scaling and control of the IP Multicast environment and can change based on network needs and design.

ip pim autorp listener

All layer 3 interfaces in the network must be enabled for sparse mode multicast operation.

ip pim sparse-mode

Procedure 2: Configure VRF Lite

Table 8. VRF and Crypto Parameters Example for Dual Routers Branch

Parameter

DMVPN-INTERNET Cloud

DMVPN-MPLS Cloud

vrf

INET-PUBLIC

INET-MPLS

rd

65512:1

65512:2

crypto keyring

DMVPN-KEYRING1

DMVPN-KEYRING2

crypto isakmp profile

FVRF-ISAKMP-INET-PUBLIC

FVRF-ISAKMP-INET-MPLS

crypto ipsec profile

DMVPN-INTERNET

DMVPN-MPLS

Tunnel interface

Tunnel 10

Tunnel 20

A dedicated Internet-facing and MPLS-facing VRF is created to support FVRF for DMVPN-Internet cloud and DMVPN-MPLS cloud respectively. The VRF name is arbitrary, but it is useful to select a name that describes the VRF. This design uses VRF Lite so that the route distinguisher value can be chosen arbitrarily. It is a best practice to use the same VRF/ route distinguisher combination across multiple devices when using VRFs in a similar manner. However, this convention is not strictly required.

ip vrf INET-PUBLIC
rd 65512:1

Tech Tip

Command Reference:

A route distinguisher is either ASN-related (composed of an ASN and an arbitrary number) or IP-address-related (composed of an IP address and an arbitrary number).

You can enter a route distinguisher in either of these formats:

16-bit autonomous-system-number: your 32-bit number

For example, 65512:1

32-bit IP address: your 16-bit number

For example, 192.168.122.15:1

Procedure 3: Connecting to the Service Provider

Option 1: Receive a DHCP address from the Internet Service Provider

The remote sites that are using DMVPN can use either static or dynamically assigned IP addresses. Cisco tested the design with a DHCP assigned external address, which also provides a dynamically configured default route.

The DMVPN spoke router connects directly to the Internet without a separate firewall. This connection is secured in two ways. Because the Internet interface is in a separate VRF, no traffic can access the global VRF except traffic sourced through the DMVPN tunnel. This design provides implicit security. Additionally, an IP access list permits only the traffic required for an encrypted tunnel, as well as DHCP and various Internet Control Message Protocols (ICMPs) for troubleshooting.

Step 1. Enable the interface, select VRF, and enable DHCP.

The DMVPN design uses FVRF so this interface must be placed into the VRF configured in the previous procedure.

interface GigabitEthernet0/0
ip vrf forwarding INET-PUBLIC
ip address dhcp
no cdp enable
no shutdown

Do not enable PIM on this interface because no multicast traffic should be requested from this interface.

Step 2. Configure and apply the access list.

To secure the connectivity and prevent unauthorized access to the router, an access list must be applied to the Internet-facing interface. The IP access list must permit the protocols specified in Table 9. The access list is applied inbound on the WAN interface so filtering is done on traffic destined to the router.

Table 9. Required DMVPN Protocols

Name

Protocol

Usage

non500-isakmp

UDP 4500

IPsec via NAT-T

isakmp

UDP 500

ISAKMP

esp

IP 50

IPsec

bootpc

UDP 68

DHCP

Example

interface GigabitEthernet 0/0
ip access-group ACL-INET-PUBLIC in
ip access-list extended ACL-INET-PUBLIC
permit udp any any eq non500-isakmp
permit udp any any eq isakmp
permit esp any any
permit udp any any eq bootpc

The additional protocols listed in Table 10 may assist in troubleshooting, but are not explicitly required to allow DMVPN to function properly.

Table 10. Optional Protocols – DMVPN Spoke Router

Name

Protocol

Usage

icmp echo

ICMP Type 0, Code 0

Allow remote pings

icmp echo-reply

ICMP Type 8, Code 0

Allow ping replies (from our requests)

icmp ttl-exceeded

ICMP Type 11, Code 0

Allow traceroute replies (from our requests)

icmp port-unreachable

ICMP Type 3, Code 3

Allow traceroute replies (from our requests)

UDP high ports

UDP > 1023, TTL=1

Allow remote traceroute

The additional optional entries for an access list to support ping are:

permit icmp any any echo
permit icmp any any echo-reply

The additional optional entries for an access list to support traceroute are:

permit icmp any any ttl-exceeded ! for traceroute (sourced)
permit icmp any any port-unreachable ! for traceroute (sourced)
permit udp any any gt 1023 ttl eq 1 ! for traceroute (destination)

Option 2: Static Address and Static Routing with Service Provider

A static IP address is assigned to the WAN interface connecting to the service provider. The interface is associated with a VRF. This simplifies the routing and allows the use of a default route for the FVRF for the WAN interface.

Step 1. Enable the interface, select VRF, and assign an IP address.

interface GigabitEthernet 0/1
ip vrf forwarding INET-MPLS
ip address 192.168.3.129 255.255.25.252
no cdp enable
no shutdown

Do not enable PIM on this interface because no multicast traffic should be requested from this interface.

Step 2. Configure static default route in the FVRF to the service provider.

ip route vrf INET-MPLS 0.0.0.0 0.0.0.0 192.168.3.130

Procedure 4: Configure ISAKMP and IPsec

Step 1. Configure the crypto keyring.

The crypto keyring defines a PSK (or password) valid for IP sources reachable within a particular VRF. This key is a wildcard pre-shared key if it applies to any IP source. A wildcard key is configured using the 0.0.0.0 0.0.0.0 network/mask combination.

crypto keyring DMVPN-KEYRING1 vrf INET-PUBLIC
pre-shared-key address 0.0.0.0 0.0.0.0 key cisco123

Step 2. Configure the ISAKMP policy and dead peer detection.

The ISAKMP policy for DMVPN uses:

AES with a 256-bit key

Secure Hash Standard (SHA)

Authentication by PSK

Diffie-Hellman group: 2

DPD is enabled with keepalive intervals sent at 40-second intervals with a five-second retry interval, which is considered to be a reasonable setting to detect a failed hub.

crypto isakmp policy 10
encr aes 256
hash sha
authentication pre-share
group 2
!
crypto isakmp keepalive 40 5

Step 3. Create the ISAKMP profile.

The ISAKMP profile creates an association between an identity address, a VRF, and a crypto keyring. A wildcard address within a VRF is referenced with 0.0.0.0.

crypto isakmp profile FVRF-ISAKMP-INET-PUBLIC
keyring DMVPN-KEYRING1
match identity address 0.0.0.0 INET-PUBLIC

Step 4. Define the IPsec transform set.

A transform set is an acceptable combination of security protocols, algorithms, and other settings to apply to IPsec-protected traffic. Peers agree to use a particular transform set when protecting a particular data flow.

The IPsec transform set for DMVPN uses:

ESP with the 256-bit AES encryption algorithm

ESP with the SHA (HMAC variant) authentication algorithm

Since the DMVPN hub router is behind a NAT device, the IPsec transform must be configured for transport mode.

crypto ipsec transform-set AES256/SHA/TRANSPORT esp-aes 256 esp-sha-hmac
mode transport

Step 5. Create the IPsec profile.

The IPsec profile creates an association between an ISAKMP profile and an IPsec transform-set.

crypto ipsec profile DMVPN-INTERNET
set transform-set AES256/SHA/TRANSPORT
set isakmp-profile FVRF-ISAKMP-INET-PUBLIC

Procedure 5: Configure the mGRE Tunnel

Step 1. Configure basic DMVPN tunnel interface settings.

Tunnel interfaces are created as they are configured. The tunnel number is arbitrary, but it is best to begin tunnel numbering at 10 or above because other features deployed in this design may also require tunnels and they may select lower numbers by default.

The bandwidth setting should be set to match the Internet bandwidth.

Configure the IP MTU to 1400 and the ip tcp adjust-mss to 1360. There is a 40-byte difference, which corresponds to the combined IP and TCP header length.

interface Tunnel10
bandwidth [bandwidth (kbps)]
ip address [IP address] [netmask]
no ip redirects
ip mtu 1400
ip tcp adjust-mss 1360

DMVPN uses multipoint GRE (mGRE) tunnels. This type of tunnel requires a source interface only. The source interface should be the same interface used to connect to the Internet. The tunnel vrf command should be set to the VRF defined previously for FVRF.

Enabling encryption on this interface requires the application of the IPsec profile configured in the previous procedure.

interface Tunnel 10
tunnel source GigabitEthernet 0/0
tunnel mode gre multipoint
tunnel vrf INET-PUBLIC
tunnel protection ipsec profile DMVPN-INTERNET

Step 2. Configure NHRP.

NHRP is used by remote routers to determine the tunnel destinations for peers attached to the mGRE tunnel.

The spoke router requires several additional configuration statements to define the NHRP server (NHS) and NHRP map statements for the DMVPN hub router mGRE tunnel IP address. EIGRP (configured in Procedure 6) relies on a multicast transport. Spoke routers require the NHRP static multicast mapping.

The value used for the NHS is the mGRE tunnel address for the DMVPN hub router. The map entries must be set to the outside NAT value of the DMVPN hub, as configured on the Cisco ASA 5500. This design uses the values shown in Table 11.

Table 11. DMVPN Tunnel Parameters

Parameter Value

Parameter Value

DMVPN cloud

INTERNET

MPLS

VRF

INET-PUBLIC

INET-MPLS

DMVPN hub public address (actual)

192.168.18.10

192.168.3.1

DMVPN hub public address (externally routable after NAT)

172.16.130.1

N/A

Tunnel IP address (NHS)

10.4.34.1

10.4.36.1

Tunnel number

10

20

NHRP network ID

101

102

NHRP requires all devices within a DMVPN cloud to use the same network ID and authentication key. The NHRP cache holdtime should be configured to 600 seconds.

This design supports DMVPN spoke routers that receive their external IP addresses through DHCP. It is possible for these routers to acquire different IP addresses after a reload. When the router attempts to register with the NHRP server, it may appear as a duplicate to an entry already in the cache and be rejected. The registration no-unique option allows you to overwrite existing cache entries. This feature is only required on NHRP clients (DMVPN spoke routers).

interface Tunnel 10
ip nhrp authentication cisco123
ip nhrp map 10.4.34.1 172.16.130.1
ip nhrp map multicast 172.16.130.1
ip nhrp network-id 101
ip nhrp holdtime 600
ip nhrp nhs 10.4.34.1
ip nhrp registration no-unique

Step 3. (Optional) Configure the backup NHS feature for a redundant hub.

When there is more than one DMVPN hub that a DMVPN spoke is peering to, a backup NHS feature must be used. The backup NHS feature allows the spoke router to limit the number of active DMVPN hub connections to one while providing failover capability to an alternate hub in case of a connection failure to the primary hub. This is needed for PfR to provide intelligent path control later on in the procedure.

interface Tunnel 10
ip nhrp nhs 10.4.34.1 priority 1
ip nhrp nhs [Secondary Hub address] priority 2
ip nhrp nhs cluster 0 max-connections 1

Procedure 6: Configure IP Multicast Routing

This procedure includes additional steps for configuring IP Multicast for a DMVPN tunnel on a router with IP Multicast already enabled.

Step 1. Enable PIM non-broadcast multiple access (NBMA) mode for the DMVPN tunnel.

Spoke-to-spoke DMVPN networks present a unique challenge because the spokes cannot directly exchange information with one another, even though they are on the same logical network. This inability to directly exchange information can also cause problems when running IP Multicast.

To resolve the NBMA issue, you need to implement a method where each remote PIM neighbor has its join messages tracked separately. A router in PIM NBMA mode treats each remote PIM neighbor as if it were connected to the router through a point-to-point link.

interface Tunnel 10
ip pim sparse-mode
ip pim nbma-mode

Step 2. Configure the designated router priority for the DMVPN spoke router.

Proper multicast operation across a DMVPN cloud requires that the hub router assumes the role of a PIM-designated router. Spoke routers should never become the designated router. You can prevent that by setting the designated router priority to 0 for the spokes.

interface Tunnel10
ip pim dr-priority 0

Procedure 7: Configure the EIGRP Routing Process

Step 1. Enable an EIGRP process on the spoke router.

A single EIGRP-IWAN process runs on the DMVPN spoke router. All interfaces on the router are EIGRP interfaces, but only the DMVPN tunnel interface is non-passive. The network range must include all interface IP addresses, either in a single network statement or in multiple network statements. This design uses a best practice of assigning the router ID to a loopback address.

router eigrp IWAN
!
address-family ipv4 unicast autonomous-system 200
!
af-interface default
passive-interface
exit-af-interface
!
af-interface Tunnel 10
no passive-interface
exit-af-interface
!
af-interface GigabitEthernet0/0/0
no passive-interface
exit-af-interface
!
network 10.4.34.0 0.0.1.255
network 10.5.0.0 0.0.255.255
network 10.255.0.0 0.0.255.255
eigrp router-id [IP address of Loopback0]
exit-address-family
!

Step 2. Configure EIGRP properties for the tunnel interface.

The EIGRP hello interval is increased to 20 seconds and the EIGRP hold time is increased to 60 seconds to accommodate up to 500 remote sites on a single DMVPN cloud.

router eigrp IWAN
!
address-family ipv4 unicast autonomous-system 200
!
af-interface Tunnel 10
hello-interval 20
hold-time 60
exit-af-interface
exit-address-family
!

Step 3. Configure the summary address for the remote-site LAN networks must be advertised.

The IP assignment for the remote sites was designed so that all of the networks in use can be summarized within a single aggregate route. The summary address as configured below suppresses the more specific routes. If any network within the summary is present in the route table, the summary is advertised to the DMVPN hub, which offers a measure of resiliency. If the various LAN networks cannot be summarized, then EIGRP continues to advertise the specific routes.

router eigrp IWAN
!
address-family ipv4 unicast autonomous-system 200
!
af-interface Tunnel10
summary-address [summary network] [summary mask]
exit-af-interface
exit-address-family
!

Step 4. Configure and apply the outbound distribute list.

This design uses route tag to prevent routing loop between the two DMVPN networks. Routes learned over the MPLS DMVPN network are tagged with 200 while routes learned over the Internet DMVPN are tagged with 100. The routes are shared between the MPLS DMVPN and Internet DMVPN spoke routers to ensure reachability.

An outbound distribute list is used on the branch spoke routers to select which routes are allowed to be advertised out. The goal is that routes coming from the hub will have a tag of either 100 or 200 and the route-map BRANCH-OUT will prevent these routes from being advertised back out to the hubs. This ensures the routes learned over the MPLS DMVPN path are not re-advertised out to Internet DMVPN spokes and vice versa to prevent routing loops.

In the route map, a prefix list is used to define the address block assigned to the tunnels. This ensures the tunnel addresses are not advertised across the two networks, thus the second deny statement. The third and last permit statement ensures that everything else that is not matching the explicitly denied statements will be advertised out to the hubs.

router eigrp IWAN
!
address-family ipv4 unicast autonomous-system 200
!
topology base
distribute-list route-map BRANCH-OUT out Tunnel 10
exit-af-topology
exit-address-family
!
ip prefix-list TUNNEL-ROUTES seq 5 permit 10.0.224.0/19 le 32
!
route-map BRANCH-OUT deny 10
match tag 100 200
route-map BRANCH-OUT deny 20
match ip address prefix-list TUNNEL-ROUTES
route-map BRANCH-OUT permit 1000
!

This example includes all WAN route sources in the reference design. Depending on the actual design of your network, you might need to use more tags.

Step 5. Enable EIGRP stub configuration and configure a leak map.

All DMVPN spoke routers should run EIGRP stub routing to improve network stability and reduce resource utilization. To allow for routing exchange between two stub routers, leak map is needed. The example below allows a spoke router to share all routes learned from a hub router with its peer.

router eigrp IWAN
!
address-family ipv4 unicast autonomous-system 200
eigrp stub connected summary redistributed leak-map LEAK-EIGRP
exit-address-family
!
route-map LEAK-EIGRP permit 1000

Procedure 8: Configure the MPLS DMVPN Spoke

Repeat Procedures 1 – 8 in the Remote-Site Dual DMVPN Spokes Configuration section to provision the second DMVPN spoke router. Refer to Table 8 and replace the attributes used for Internet with attributes for MPLS as listed in the table.

Remote-Site Single DMVPN Spoke Dual Links Configuration

Use this set of procedures for remote site with single router and dual link design.

Procedure 1: Configure the WAN Remote Router

Within this design, there are features and services that are common across all WAN remote site routers. These are system settings that simplify and secure the management of the solution.

Step 1. Configure the common features and services for the routers, including:

Device host name, for ease of identifying the device

The local login account and password, to provide basic access

Network Time Protocol (NTP) to synchronize system clock network devices

service timestamps debug datetime msec localtime
service timestamps log datetime msec localtime
service password-encryption
!
hostname [hostname]
!
username admin password c1sco123
enable secret c1sco123
aaa new-model
!
clock timezone PST -8
clock summer-time PDT recurring
ntp server 10.4.48.17
ntp update-calendar
!

Step 2. Configure device management protocols.

Configure Secure Shell (SSH) for secure management of the network device

Specify the transport preferred none on vty lines to prevent errant connection attempts from the mistyped CLI commands

Enable SNMPv2c for read-only and read-write community strings

ip domain-name cisco.local
!
ip ssh version 2
no ip http server
ip http secure-server
snmp-server community cisco RO
snmp-server community cisco123 RW
!
line vty 0 15
transport input ssh
transport preferred none
line con 0
logging synchronous

(Optional) In networks where network operational support is centralized you can increase network security by using an access list to limit the networks that can access your device. In this example, only devices on the 10.4.48.0/24 network will be able to access the device through SSH or SNMP.

access-list 55 permit 10.4.48.0 0.0.0.255
line vty 0 15
access-class 55 in
!
snmp-server community cisco RO 55
snmp-server community cisco123 RW 55

Tech Tip

If you configure an access list on the vty interface you may lose the ability to use SSH to log in from one router to the next for hop-by-hop troubleshooting.

Step 3. (Optional) Configure centralized user authentication.

As networks scale in the number of devices to maintain, it poses an operational burden to maintain local user accounts on every device. A centralized AAA service reduces operational tasks per device and provides an audit log of user access for security compliance and root-cause analysis. When AAA is enabled for access control, all management access to the network infrastructure devices (SSH and HTTPS) is controlled by AAA.

TACACS+ is the primary protocol used to authenticate management logins on the infrastructure devices to the AAA server. A local AAA user database is also defined on each network infrastructure device to provide a fallback authentication source in case the centralized TACACS+ server is unavailable.

tacacs server TACACS-SERVER -1
address ipv4 10.4.48.15
key SecretKey
!
aaa group server tacacs+ TACACS-SERVERS
server name TACACS-SERVER -1
!
aaa authentication login default group TACACS-SERVERS local
aaa authorization exec default group TACACS-SERVERS local
aaa authorization console
ip http authentication aaa

Step 4. Configure an in-band management interface.

The loopback interface is a logical interface that is always reachable as long as the device is powered on and any IP interface is reachable to the network. Because of this capability, the loopback address is the best way to manage the switch in-band. Layer 3 processes and features are also bound to the loopback interface to ensure process resiliency.

The loopback address is commonly a host address with a 32-bit address mask. Allocate the loopback address from a unique network range that is not part of any other internal network summary range.

interface Loopback 0
ip address [ip address] 255.255.255.255
ip pim sparse-mode

The ip pim sparse-mode command will be explained in the next step.

Bind the device processes for SNMP, SSH, PIM, TACACS+, and NTP to the loopback interface address for optimal resiliency.

snmp-server trap-source Loopback0
ip ssh source-interface Loopback0
ip pim register-source Loopback0
ip tacacs source-interface Loopback0
ntp source Loopback0

Step 5. Configure IP Multicast routing.

IP Multicast allows a single IP data stream to be replicated by the infrastructure (routers and switches) and sent from a single source to multiple receivers. Using IP Multicast is much more efficient than multiple individual unicast streams or a broadcast stream that would propagate everywhere. IP telephony MOH and IP video broadcast streaming are two examples of IP Multicast applications.

To receive a particular IP Multicast data stream, end hosts must join a multicast group by sending an IGMP message to their local multicast router. In a traditional IP Multicast design, the local router consults another router in the network that is acting as a rendevous point to map the receivers to active sources so they can join their streams.

In this design, which is based on sparse mode multicast operation, auto- rendevous point is used to provide a simple, yet scalable, way to provide a highly resilient rendevous point environment.

Enable IP Multicast routing on the platforms in the global configuration mode.

ip multicast-routing

Every layer 3 switch and router must be configured to discover the IP Multicast rendevous point with autorp. Use the ip pim autorp listener command to allow for discovery across sparse mode links. This configuration provides for future scaling and control of the IP Multicast environment and can change based on network needs and design.

ip pim autorp listener

All layer 3 interfaces in the network must be enabled for sparse mode multicast operation.

ip pim sparse-mode

Procedure 2: Configure VRF Lite

Table 12. VRF and Crypto Parameters Example for Single Router Dual Link Branch

Parameter

DMVPN-INTERNET Cloud

DMVPN-MPLS Cloud

vrf

INET-PUBLIC

INET-MPLS

rd

65512:1

65512:2

crypto keyring

DMVPN-KEYRING1

DMVPN-KEYRING2

crypto isakmp profile

FVRF-ISAKMP-INET-PUBLIC

FVRF-ISAKMP-INET-MPLS

crypto ipsec profile

DMVPN-INTERNET

DMVPN-MPLS

Tunnel interface

Tunnel 10

Tunnel 20

A dedicated Internet-facing and MPLS-facing VRF is created to support FVRF for DMVPN-Internet cloud and DMVPN-MPLS cloud, respectively. The VRF name is arbitrary, but it is useful to select a name that describes the VRF. This design uses VRF Lite so that the route distinguisher value can be chosen arbitrarily. It is a best practice to use the same VRF/route distinguisher combination across multiple devices when using VRFs in a similar manner. However, this convention is not strictly required.

ip vrf INET-PUBLIC
rd 65512:1

Tech Tip

Command Reference:

A route distinguisher is either ASN-related (composed of an ASN and an arbitrary number) or IP-address-related (composed of an IP address and an arbitrary number).

You can enter a route distinguisher in either of these formats:

16-bit autonomous-system-number: your 32-bit number

For example, 65512:1.

32-bit IP address: your 16-bit number

For example, 192.168.122.15:1

Procedure 3: Connecting to the Service Provider

The remote sites that are using DMVPN can use either static or dynamically assigned IP addresses. Cisco tested the design with a DHCP-assigned external address, which also provides a dynamically configured default route.

Option 1: Receive a DHCP Address from the Internet Service Provider

The DMVPN spoke router connects directly to the Internet without a separate firewall. This connection is secured in two ways. Because the Internet interface is in a separate VRF, no traffic can access the global VRF, except traffic sourced through the DMVPN tunnel. This design provides implicit security. Additionally, an IP access list permits only the traffic required for an encrypted tunnel, as well as DHCP and various ICMP protocols for troubleshooting.

Step 1. Enable the interface, select VRF, and enable DHCP.

The DMVPN design uses FVRF so this interface must be placed into the VRF configured in the previous procedure.

interface GigabitEthernet 0/1
ip vrf forwarding INET-PUBLIC
ip address dhcp
no cdp enable
no shutdown

Step 2. Configure and apply the access list.

The IP access list must permit the protocols specified in Table 13. The access list is applied inbound on the WAN interface so filtering is done on traffic destined to the router.

Table 13. Required DMVPN Protocols

Name

Protocol

Usage

non500-isakmp

UDP 4500

IPsec via NAT-T

Isakmp

UDP 500

ISAKMP

Esp

IP 50

IPsec

bootpc

UDP 68

DHCP

Example

interface GigabitEthernet 0/1
ip access-group ACL-INET-PUBLIC in
ip access-list extended ACL-INET-PUBLIC
permit udp any any eq non500-isakmp
permit udp any any eq isakmp
permit esp any any
permit udp any any eq bootpc

The additional protocols listed in Table 14 may assist in troubleshooting, but are not explicitly required to allow DMVPN to function properly.

Table 14. Optional Protocols – DMVPN Spoke Router

Name

Protocol

Usage

icmp echo

ICMP Type 0, Code 0

Allow remote pings

icmp echo-reply

ICMP Type 8, Code 0

Allow ping replies (from our requests)

icmp ttl-exceeded

ICMP Type 11, Code 0

Allow traceroute replies (from our requests)

icmp port-unreachable

ICMP Type 3, Code 3

Allow traceroute replies (from our requests)

UDP high ports

UDP > 1023, TTL=1

Allow remote traceroute

The additional optional entries for an access list to support ping are:

permit icmp any any echo
permit icmp any any echo-reply

The additional optional entries for an access list to support traceroute are:

permit icmp any any ttl-exceeded ! for traceroute (sourced)
permit icmp any any port-unreachable ! for traceroute (sourced)
permit udp any any gt 1023 ttl eq 1 ! for traceroute (destination)

Option 2: Static Address and Static Routing with Service Provider

A static IP address is assigned to the WAN interface connecting to the service provider. The interface is associated with a VRF. This simplifies the routing and allows the use of a default route for the FVRF for the WAN interface.

Step 1. Enable the interface, select VRF, and assign an IP address.

interface GigabitEthernet 0/0
ip vrf forwarding INET-MPLS
ip address [IP address] [netmask]
no cdp enable
no shutdown

Do not enable PIM on this interface because no multicast traffic should be requested from this interface.

Step 2. Configure a static default route in the FVRF to the service provider.

ip route vrf INET-MPLS 0.0.0.0 0.0.0.0 [Gateway Address]

Procedure 4: Configure ISAKMP and IPsec

Step 1. Configure the crypto keyring.

The crypto keyring defines a PSK (or password) valid for IP sources reachable within a particular VRF. This key is a wildcard PSK if it applies to any IP source. A wildcard key is configured using the 0.0.0.0 0.0.0.0 network/mask combination.

crypto keyring DMVPN-KEYRING1 vrf INET-PUBLIC
pre-shared-key address 0.0.0.0 0.0.0.0 key cisco123

Step 2. Configure the ISAKMP policy and dead peer detection.

The ISAKMP policy for DMVPN uses:

AES with a 256-bit key

Secure Hash Standard (SHA)

Authentication by PSK

Diffie-Hellman group: 2

DPD is enabled with keepalive intervals sent at 40-second intervals with a five-second retry interval, which is considered to be a reasonable setting to detect a failed hub.

crypto isakmp policy 10
encr aes 256
hash sha
authentication pre-share
group 2
!
crypto isakmp keepalive 40 5

Step 3. Create the ISAKMP profile.

The ISAKMP profile creates an association between an identity address, a VRF, and a crypto keyring. A wildcard address within a VRF is referenced with 0.0.0.0.

crypto isakmp profile FVRF-ISAKMP-INET-PUBLIC
keyring DMVPN-KEYRING1
match identity address 0.0.0.0 INET-PUBLIC

Step 4. Define the IPsec transform set.

A transform set is an acceptable combination of security protocols, algorithms, and other settings to apply to IPsec-protected traffic. Peers agree to use a particular transform set when protecting a particular data flow.

The IPsec transform set for DMVPN uses:

ESP with the 256-bit AES encryption algorithm

ESP with the SHA (HMAC variant) authentication algorithm

Since the DMVPN hub router is behind a NAT device, the IPsec transform must be configured for transport mode.

crypto ipsec transform-set AES256/SHA/TRANSPORT esp-aes 256 esp-sha-hmac
mode transport

Step 5. Create the IPsec profile.

The IPsec profile creates an association between an ISAKMP profile and an IPsec transform-set.

crypto ipsec profile DMVPN-INTERNET
set transform-set AES256/SHA/TRANSPORT
set isakmp-profile FVRF-ISAKMP-INET-PUBLIC

Procedure 5: Configure the mGRE Tunnel

Step 1. Configure basic interface settings.

Tunnel interfaces are created as they are configured. The tunnel number is arbitrary, but it is best to begin tunnel numbering at 10 or above because other features deployed in this design may also require tunnels and they may select lower numbers by default.

The bandwidth setting should be set to match the Internet bandwidth.

Configure the IP MTU to 1400 and the ip tcp adjust-mss to 1360. There is a 40-byte difference, which corresponds to the combined IP and TCP header length.

interface Tunnel 10
bandwidth [bandwidth (kbps)]
ip address [IP address] [netmask]
no ip redirects
ip mtu 1400
ip tcp adjust-mss 1360

Step 2. Configure the tunnel.

DMVPN uses multipoint GRE (mGRE) tunnels. This type of tunnel requires a source interface only. Use the same source interface that you use to connect to the Internet. Set the tunnel vrf command to the VRF defined previously for FVRF.

Enabling encryption on this interface requires the application of the IPsec profile configured in the previous procedure.

interface Tunnel 10
tunnel source GigabitEthernet 0/1
tunnel mode gre multipoint
tunnel vrf INET-PUBLIC
tunnel protection ipsec profile DMVPN-INTERNET

Step 3. Configure NHRP.

The DMVPN hub router is the NHRP server for all of the spokes. NHRP is used by remote routers to determine the tunnel destinations for peers attached to the mGRE tunnel.

The spoke router requires several additional configuration statements to define the NHRP server (NHS) and NHRP map statements for the DMVPN hub router mGRE tunnel IP address. EIGRP relies on a multicast transport. Spoke routers require the NHRP static multicast mapping.

The value used for the NHS is the mGRE tunnel address for the DMVPN hub router. The map entries must be set to the outside NAT value of the DMVPN hub, as configured on the Cisco ASA 5500. This design uses the values shown in Table 15.

Table 15. DMVPN Tunnel Parameters

Parameter Values

DMVPN cloud

INTERNET

MPLS

VRF

INET-PUBLIC

INET-MPLS

DMVPN hub public address (actual)

192.168.18.10

192.168.3.1

DMVPN hub public address (externally routable after NAT)

172.16.130.1

N/A

Tunnel IP address (NHS)

10.4.34.1

10.4.36.1

Tunnel number

10

20

NHRP network ID

101

102

NHRP requires all devices within a DMVPN cloud to use the same network ID and authentication key. The NHRP cache hold time should be configured to 600 seconds.

This design supports DMVPN spoke routers that receive their external IP addresses through DHCP. It is possible for these routers to acquire different IP addresses after a reload. When the router attempts to register with the NHRP server, it may appear as a duplicate to an entry already in the cache and be rejected. The registration no-unique option allows you to overwrite existing cache entries. This feature is only required on NHRP clients (DMVPN spoke routers).

interface Tunnel 10
ip nhrp authentication cisco123
ip nhrp map 10.4.34.1 172.16.130.1
ip nhrp map multicast 172.16.130.1
ip nhrp network-id 101
ip nhrp holdtime 600
ip nhrp nhs 10.4.34.1
ip nhrp registration no-unique

Step 4. (Optional) Configure the backup NHS feature for a redundant hub.

When there are more than one DMVPN hubs that a DMVPN spoke is peering to, the backup NHS feature must be used. The backup NHS feature allows the spoke router to limit the number of active DMVPN hub connections to one, while providing failover capability to an alternate hub in case of a connection failure to the primary hub. This is needed for PfR to provide intelligent path control later on in the procedure.

interface Tunnel 10
ip nhrp nhs 10.4.34.1 priority 1
ip nhrp nhs [Secondary Hub address] priority 2
ip nhrp nhs cluster 0 max-connections 1

Procedure 6: Configure IP Multicast Routing

This procedure includes additional steps for configuring IP Multicast for a DMVPN tunnel on a router with IP Multicast already enabled.

Step 1. Configure PIM on the DMVPN tunnel interface.

Enable IP PIM sparse mode on the DMVPN tunnel interface.

interface Tunnel 10
ip pim sparse-mode

Step 2. Enable PIM NBMA mode for the DMVPN tunnel.

Spoke-to-spoke DMVPN networks present a unique challenge because the spokes cannot directly exchange information with one another, even though they are on the same logical network. This inability to directly exchange information can also cause problems when running IP Multicast.

To resolve the nonbroadcast multiaccess (NBMA) issue, you need to implement a method where each remote PIM neighbor has its join messages tracked separately. A router in PIM NBMA mode treats each remote PIM neighbor as if it were connected to the router through a point-to-point link.

interface Tunnel 10
ip pim nbma-mode

Step 3. Configure the designated router priority for the DMVPN spoke router.

Proper multicast operation across a DMVPN cloud requires that the hub router assumes the role of PIM designated router. Spoke routers should never become the designated router. You can prevent that by setting the designated router priority to 0 for the spokes.

interface Tunnel 10
ip pim dr-priority 0

Procedure 7: Configure EIGRP

Step 1. Enable the EIGRP process on the spoke router.

A single EIGRP-IWAN process runs on the DMVPN spoke router. All interfaces on the router are EIGRP interfaces, but only the DMVPN tunnel interface is non-passive. The network range must include all interface IP addresses, either in a single network statement or in multiple network statements. This design uses a best practice of assigning the router ID to a loopback address.

router eigrp IWAN
!
address-family ipv4 unicast autonomous-system 200
!
af-interface default
passive-interface
exit-af-interface
!
af-interface Tunnel 10
no passive-interface
exit-af-interface
!
af-interface Tunnel 20
no passive-interface
exit-af-interface
!
af-interface GigabitEthernet0/0/0
no passive-interface
exit-af-interface
!
network 10.4.34.0 0.0.1.255
network 10.5.0.0 0.0.255.255
network 10.255.0.0 0.0.255.255
eigrp router-id [IP address of Loopback0]
no auto-summary

Step 2. Configure EIGRP properties for the tunnel interface.

The EIGRP hello interval is increased to 20 seconds and the EIGRP hold time is increased to 60 seconds to accommodate up to 500 remote sites on a single DMVPN cloud.

router eigrp IWAN
!
address-family ipv4 unicast autonomous-system 200
!
af-interface Tunnel 10
hello-interval 20
hold-time 60
exit-af-interface
!
af-interface Tunnel 20
hello-interval 20
hold-time 60
exit-af-interface
exit-address-family

Step 3. Configure the summary address for the remote-site LAN networks must be advertised

The IP assignment for the remote sites was designed so that all of the networks in use can be summarized within a single aggregate route. The summary address as configured below suppresses the more specific routes. If any network within the summary is present in the route table, the summary is advertised to the DMVPN hub, which offers a measure of resiliency. If the various LAN networks cannot be summarized, then EIGRP continues to advertise the specific routes.

router eigrp IWAN
!
address-family ipv4 unicast autonomous-system 200
!
af-interface Tunnel 10
summary-address [summary network] [summary mask]
exit-af-interface
exit-address-family
!

Step 4. Configure and apply the outbound distribute list.

This design uses route tags to prevent routing loops between the two DMVPN networks. Routes learned over the MPLS DMVPN network are tagged with 200 while routes learned over the Internet DMVPN are tagged with 100. The routes are shared between the MPLS DMVPN and Internet DMVPN spoke routers to ensure reachability.

An outbound distribute list is used on the branch spoke routers to select which routes are allowed to be advertised out. The goal is for routes coming from the hub to have a tag of either 100 or 200 and the route-map BRANCH-OUT will prevent these routes from being advertised back out to the hubs. This ensures the routes learned over MPLS DMVPN path are not readvertised out to Internet DMVPN spokes and vice versa to prevent routing loops.

In the route map, a prefix list is used to define the address block assigned to the tunnels. This ensures the tunnel addresses are not advertised across the two networks, thus the second deny statement. The thrid and last permit statement ensures everything else that is not matching the explicitly denied statements will be advertised out to the hubs.

router eigrp IWAN
!
address-family ipv4 unicast autonomous-system 200
!
topology base
distribute-list route-map BRANCH-OUT out Tunnel 10
distribute-list route-map BRANCH-OUT out Tunnel 20
exit-af-topology
exit-address-family
!
ip prefix-list TUNNEL-ROUTES seq 5 permit 10.0.224.0/19 le 32
!
route-map BRANCH-OUT deny 10
match tag 100 200
route-map BRANCH-OUT deny 20
match ip address prefix-list TUNNEL-ROUTES
route-map BRANCH-OUT permit 1000
!

This example includes all WAN route sources in the reference design. Depending on the actual design of your network, you might need to use more tags.

Step 5. Enable an EIGRP stub configuration and configure leak map.

All DMVPN spoke routers should run EIGRP stub routing to improve network stability and reduce resource utilization. To allow routing exchange between two stub routers, leak map is needed. The example below allows a spoke router to share all routes learned from a hub router with its peer.

router eigrp IWAN
!
address-family ipv4 unicast autonomous-system 200
eigrp stub connected summary redistributed leak-map LEAK-EIGRP
exit-address-family
!
route-map LEAK-EIGRP permit 1000

Deploying Intelligent Path Control

Design Overview

Intelligent Path Control using Cisco Performance Routing (PfR) improves application delivery and WAN efficiency. PfR enables intelligence of Cisco IOS routers to improve application performance and availability. PfR allows customers to protect critical applications from fluctuating WAN performance while intelligently load balancing traffic over all WAN paths.

PfR monitors network performance and selects the best path for each application based upon advanced criteria such as reachability, delay, jitter, loss, and mean opinion scores (MOS). PfR can evenly distribute traffic to maintain equivalent link utilization levels using an advanced load balancing technique - even over links with differing bandwidth capacities. IWAN Intelligent Path Control is key to providing a business-class WAN over Internet transports.

PfR Components

PfR is comprised of two major components, a master controller and a border router. The master controller is a policy decision point at which policies are defined and applied to various traffic classes that traverse the border router systems. The master controller can be configured to learn and control traffic classes on the network.

Border routers are in the data forwarding path. Border routers collect data from their Netflow cache and from the IP SLA probe results, provide a degree of aggregation of this information, and influence the packet switching path to manage user traffic.

Master controller is the policy decision maker. At a large site, such as data center or campus, the master controller is a standalone chassis. For smaller branch locations, the master controller is typically collocated (configured) on the same platform as the border router. As a general rule, the large locations manage more network prefixes and applications than a branch deployment, thus consuming more CPU and memory resources for the master controller function. Therefore, it makes a good design practice to dedicate a chassis for the master controller at large sites.

The branch typically manages fewer active network prefixes and applications. And due to the costs associated with dedicating a chassis at each branch, the network manager can co-locate the master controller and border router on the same platform. CPU and memory utilization should be monitored on platforms operating as master controllers, and if utilization is high the network manager should consider a master controller platform with a higher capacity CPU and memory.

The master controller communicates with the border routers over an authenticated TCP socket, but has no requirement for populating its own IP routing table with anything more than a route to reach the border routers.

Because PfR is an intelligent path selection technology, there must be at least two external interfaces under the control of PfR and at least one internal interface. There must be at least one border router configured. If only one border router is configured, then both external interfaces are attached to the single border router. If more than one border router is configured, then the two or more external interfaces are configured across these border routers. External links, or exit points, are therefore owned by the border router; they may be logical (tunnel interfaces) or physical links (serial, Ethernet, etc.).

Tech Tip

Cisco IOS Software Release 15.2(3)T and IOS-XE 3.6S and later support significant simplification enhancements, including target discovery and better default behavior. This guide will only focus on the latest implementation.

PfR Classes of Service

The enterprise traffic is divided into three main application service classes that have their own policies.
Voice and Video traffic – VOICE_VIDEO Service Class

Voice and video traffic is running between the central site and the branches. This traffic is already marked with Differentiated Services Code Point Expedited Forwarding (DSCP EF) and the transport is Rapid Transport Protocol (RTP). To capture video we also match DSCP AF41 and CS4 which is used by Cisco TelePresence® endpoints.

We would like to track delay, jitter, and loss.

We want to have deterministic path selection, and therefore use the primary service provider called MPLS.

If the primary WAN experiences brownouts or a blackout, we want PfR to failover this traffic to the secondary path.

Critical Application - CRITICAL Service Class

There is a highly important application running between branches and the central site network, and running over TCP.

This traffic is marked with DSCP AF31.

This application is interactive so it is sensitive to delay. In addition, it does not respond well to packet loss. The servers being used for this important application are also responsible for other applications that are not only less important, but that have different requirements of the network.

We want to have deterministic path selection, and therefore use the primary SP MPLS.

If the primary WAN experiences brownouts or a blackout, we want PfR to failover this traffic to the secondary path.

Best Effort Applications – BEST_EFFORT Service Class

The rest of the traffic is just HTTP or email and should be load balanced over all exit interfaces, MPLS, and Internet.

QoS is already in place with classification and marking procedures. Therefore, packets entering on the border router are already classified and marked directly on the access switch connecting the station, IP phone, or multimedia terminal. That means PfR can use the DSCP field as an efficient way for the traffic profiling phase. This guide will use the DSCP values as the classification criteria.

You could also use a combination of destination prefixes and DSCP, or even NBAR. If the traffic is not marked when entering the border router, then a good option is to classify on ingress, mark the DSCP, and then use the DSCP as the classification criteria within PfR. Classification can be done with the usual use of access lists.

Design Details

In this guide we want to protect voice, video, and critical applications against soft errors, brownouts, and blackouts. We want to be able to track jitter, delay, and loss, and we also want to have a fast failover. Therefore, performance measurement will be in active mode and will use jitter probes. Jitter probe configuration for all the remote sites is a painful process. Cisco PfR is now using a new feature called Target Discovery to help simplify the configuration and deployment in such cases.

A peering session is configured between the master controller on the central site and the master controllers on the remote sites. The master controllers which participate in this peering will exchange the list of inside prefixes and IP SLA probe responder addresses that belong to the sites, along with the site name (configurable).

Deployment Details

This section covers:

Deployment details for master controllers

Deployment details for border routers

Implementing Dedicated Master Controller

Procedure 1: Configure the Master Controller Router

This section describes configuring a dedicated master controller for a large site such as a campus or data center. Only the core relevant features are described. Figure 11 illustrates a typical hub technology, along with the IP addresses. Table 16 outlines the parameters used in our deployment examples.

Figure 11. Hub Topology and Addresses

Table 16. Parameters Used in the Deployment Examples

Host Name

Loopback IP Address

Port Channel IP Address

VPN-ASR1002-1

10.4.32.241/32

10.4.32.2/28

VPN-ASR1002-2

10.4.32.243/32

10.4.32.3/28

MC-ASR1002

10.4.32.251/32

10.4.32.38/30

Within this design, there are features and services that are common across all WAN aggregation routers. These are system settings that simplify and secure the management of the solution.

Step 1. Configure the common features and services for the routers, including:

Device host name for ease of identifying the device

The local login account and password to provide basic access

Network Time Protocol (NTP) to synchronize system clock network devices

service timestamps debug datetime msec localtime
service timestamps log datetime msec localtime
service password-encryption
!
hostname [hostname]
!
username admin password c1sco123
enable secret c1sco123
aaa new-model
!
clock timezone PST -8
clock summer-time PDT recurring
ntp server 10.4.48.17

Step 2. Configure device management protocols

Use Secure Shell (SSH) for secure management of the network device

Specify the transport preferred none on vty lines to prevent errant connection attempts from the mistyped CLI commands

Enable SNMPv2c for read-only and read-write community strings

ip domain-name cisco.local
!
ip ssh version 2
no ip http server
ip http secure-server
snmp-server community cisco RO
snmp-server community cisco123 RW
!
line vty 0 15
transport input ssh
transport preferred none
line con 0
logging synchronous

(Optional) In networks where network operational support is centralized you can increase network security by using an access list to limit the networks that can access your device. In this example, only devices on the 10.4.48.0/24 network will be able to access the device through SSH or SNMP.

access-list 55 permit 10.4.48.0 0.0.0.255
line vty 0 15
access-class 55 in
!
snmp-server community cisco RO 55
snmp-server community cisco123 RW 55

Tech Tip

If you configure an access list on the vty interface you may lose the ability to use SSH to log in from one router to the next for hop-by-hop troubleshooting.

Step 3. (Optional) Configure centralized user authentication.

As networks scale in the number of devices, Cisco recommends the best practice to use a centralized AAA service to reduce operational tasks per device and provides an audit log of user access for security compliance and root-cause analysis. When AAA is enabled for access control, all management access to the network infrastructure devices (SSH and HTTPS) is controlled by AAA.

TACACS+ is the primary protocol used to authenticate management logins on the infrastructure devices to the AAA server. A local AAA user database is also defined on each network infrastructure device to provide a fallback authentication source in case the centralized TACACS+ server is unavailable.

tacacs server TACACS-SERVER-1
address ipv4 10.4.48.15
key SecretKey
!
aaa group server tacacs+ TACACS-SERVERS
server name TACACS-SERVER-1
!
aaa authentication login default group TACACS-SERVERS local
aaa authorization exec default group TACACS-SERVERS local
aaa authorization console
ip http authentication aaa

Step 4. Configure an in-band management interface.

The loopback interface is a logical interface that is always reachable, as long as the device is powered on and any IP interface is reachable to the network. Because of this capability, the loopback address is the best way to manage the switch in-band. Layer 3 processes and features are also bound to the loopback interface to ensure process resiliency.

The loopback address is commonly a host address with a 32-bit address mask. Allocate the loopback address from the IP address block that the distribution switch summarizes to the rest of the network.

interface Loopback 0
ip address [ip address] 255.255.255.255

Bind the device processes for SNMP, SSH, TACACS+, and NTP to the loopback interface address for optimal resiliency.

snmp-server trap-source Loopback0
ip ssh source-interface Loopback0
ip tacacs source-interface Loopback0
ntp source Loopback0

Procedure 2: Configure Connectivity to the LAN

The master controller is not in line with the data flow and should only need one connection to the distribution switches. In this guide two interfaces are used to configure EtherChannel connection for link redundancy.

Step 1. Configure a layer 3 interface.

interface Port-channel21
ip address 10.4.32.151 255.255.255.192
no shutdown

Step 2. Configure EtherChannel member interfaces.

Configure the physical interfaces to tie to the logical port channel by using the channel-group command. The number for the port channel and channel group must match. Not all router platforms can support Link Aggregation Control Protocol (LACP) to negotiate with the switch, so EtherChannel is configured statically.

interface GigabitEthernet 0/0
description WAN-D3750X Gig1/0/9
!
interface GigabitEthernet 0/1
description WAN-D3750X Gig2/0/9
!
interface range GigabitEthernet 0/0 , GigabitEthernet0/1
no ip address
channel-group 21
no shutdown

Step 3. Configure a default route. This provides reachability information for the Key Server to reach border routers by using a default route.

ip route 0.0.0.0 0.0.0.0 10.4.32.129

Procedure 3: Configure Master Controller Policies

This section provides the foundation configuration required on the master controller. The configuration enables automatic learning of traffic flowing through border routers using Netflow. In addition, PfR route control is enabled to allow load balancing across all configured external interfaces.

Step 1. Configure the key chain.

The key chain is used to authenticate the connection with the border router.

key chain pfr
key 0
key-string cisco
!

Step 2. Configure the master controller.

Enable PfR master controller and designate the border routers' IP addresses and internal/external interfaces on the border routers. Link group is used to specify the carrier and color the exit interface. The key chain is used to authenticate the connection with border routers.

pfr master
!
border 10.4.32.241 key-chain pfr
interface Tunnel 20 external
link-group MPLS
interface Port-channel1 internal
!
border 10.4.32.243 key-chain pfr
interface Tunnel 10 external
link-group Internet
interface Port-channe l3 internal
!

Step 3. Configure PfR policies.

Set the load-balancing range to 30 percent. This setting is conservative and helps reduce the churn result from movement of traffic classes across the links.

Disable auto tunnels as border routers are layer 2-adjacent.

Enable the use of policy-based routing (PBR) for finer application control

Maximum transmit utilization is 90 percent

pfr master
max-range-utilization percent 30
!
no mode auto-tunnels
periodic 90
probe packets 20
!
!

Tech Tip

Note the following will be enabled by default:

Automatic Learning
Mode route control
Maximum transmit utilization at 90%
Load balancing is used as the last resolver and cannot be disabled

Step 4. Enable NetFlow version 9 export (optional).

The following configuration defines the flow exporter. The data that is collected by NetFlow version 9 is sent to a NetFlow collector for further analysis and storage. Flow exporter uses User Datagram Protocol (UDP) as transport protocol.

flow exporter MYEXPORTER
destination 10.151.1.131
source Loopback0
transport udp 9991
option interface-table timeout 300
option sampler-table timeout 300
option application-table timeout 300
!
! Add NetFlow export to PfR
!
pfr master
exporter MYEXPORTER
!

The following configuration enables logging to syslog. This will log PfR-related messages on the console.

pfr master
logging
!

Implementing Dedicated Border Routers

This section describes adding dedicated border router functionality to an already configured WAN router with a DMVPN tunnel at a large site such as a campus or data center. It includes only the additional steps required to enable the border router capabilities.

Procedure 1: Configure Branch Router Functionality

The PfR configuration is centralized on the master controller. The border router configuration only includes:

The source address used to peer with the master controller

The master controller IP address

The key chain used to authenticate with the master controller

Step 1. Configure the key chain.

The key chain is used to authenticate the peering connection with the master controller.

key chain pfr
key 0
key-string cisco
!

Step 2. Configure the border router.

pfr border
logging
local Loopback0
master 10.4.32.251 key-chain pfr

Implementing Combined Master Controller and Border Router for a Branch

The following section shows configuration steps needed for enabling a branch router with combined master controller and branch router functionality.

Procedure 1: Configure Combined Master Controller and Branch Router on a Router

Step 1. Configure the key chain

The key chain is used to authenticate the connection with the border router.

key chain pfr
key 0
key-string cisco
!

Step 2. Configure the master controller.

Enable the PfR master controller and designate the border router's local loopback IP addresses and internal/external interfaces on the border routers. Link group is used to specify the carrier and color the exit interface. The key chain is used to authenticate the connection with border routers.

pfr master
logging
!
border 10.2.10.10 key-chain pfr
interface Ethernet0/0 internal
interface Tunnel 10 external
link-group MPLS
interface Tunnel 20 external
link-group Internet
!

Step 3. Configure border router functionality.

pfr border
logging
local Loopback0
master 10.2.10.10 key-chain pfr
!

Implementing Target Discovery on Master Controller

This section applies to the master controller in either dedicated or shared mode. Target Discovery is applied for the hub-and-spoke traffic model.

The PfR Target Discovery feature introduces a scalable solution for managing the performance of video, voice, and critical applications across large, enterprise branch networks by automating the identification and configuration of IP SLA responders. To optimize media applications using voice and video traffic, PfR uses jitter, loss, and delay measurements. The IP SLA udp-jitter probe provides these measurements but requires an IP SLA responder. The initial PfR solution was to manually configure the probes for all remote sites. PfR Target Discovery uses the PfR Domain Peering.

Tech Tip

Target Discovery feature requires the router run IOS Software Release 15.2(3)T and later releases.

Procedure 1: Configure Target Discovery on the Hub Master Controller

Step 1. Define the prefix of interest.

We want to define the IP address of the shadow router used as a responder on the hub. We also want to control the list of prefixes advertised to the spokes. Therefore, we use the responder list and inside prefix options.

ip prefix-list HQ_PREFIX seq 5 permit 10.10.1.0/24
ip prefix-list HQ_PREFIX seq 10 permit 10.10.2.0/24
ip prefix-list HQ_PREFIX seq 15 permit 10.10.3.0/24
ip prefix-list HQ_PREFIX seq 20 permit 10.10.4.0/24
ip prefix-list RESPONDER_PREFIX seq 5 permit 10.10.10.2/32
!

Step 2. Enable Target Discovery on the hub master controller.

pfr master
mc-peer domain 200 eigrp Loopback0
target-discovery responder-list RESPONDER_PREFIX inside-prefixes HQ_PREFIX
!

Procedure 2: Enable Target Discover on the Branch Master Controller

Step 1. Enable Target Discovery on the spoke master controller.

On the entire spoke master controllers, configure the IP address of the hub master controller. To simplify the configuration on the branch devices, we do not want to manually define the target and prefix list. PfR will automatically generate what is needed.

pfr master
mc-peer domain 200 eigrp Loopback0
target-discovery
!

Procedure 3: Tuning Service Advertisement Framework (SAF) Parameters

Step 1. Configure the hub master controller.

The PfR Target Discovery introduces master controller peering and uses service routing through the EIGRP Service Advertisement Framework (SAF) to advertise, discover, and auto-configure IP SLA responders and associated destination IP prefixes. The following procedure tunes the SAF parameters to increase PfR stability and scalability.

The master controller peering is enabled and PfR Target Discovery is configured in dynamic mode. The SAF configuration is shown here under the service-family command section, and this configuration is assumed to exist before the PfR master controller peering and Target Discovery overlay configuration is added.

Increase SAF Hello timer to 200 seconds.

Increase SAF Hold timer to 600 seconds

Disable split horizon on the loopback interface

Enable dynamic peering with unicast listen

router eigrp IWAN
!
service-family ipv4 autonomous-system 200
!
sf-interface default
shutdown
exit-sf-interface
!
sf-interface Loopback0
no shutdown
hello-interval 200
hold-time 600
no split-horizon
exit-sf-interface
!
topology base
exit-sf-topology
remote-neighbors source Loopback0 unicast-listen
exit-service-family
!

Step 2. Configure the branch location master controller.

At the branch location the master controller peering is enabled and PfR target discovery is configured in dynamic mode.

Increase SAF Hello timer to 200 seconds.

Increase SAF Hold timer to 600 seconds

Define Hub MC address through unicast neighbor statement

router eigrp IWAN
!
service-family ipv4 autonomous-system 200
eigrp stub connected
!
sf-interface default
shutdown
exit-sf-interface
!
sf-interface Loopback0
no shutdown
hello-interval 200
hold-time 600
exit-sf-interface
!
neighbor 10.4.32.251 Loopback0
exit-service-family
!

Tuning Automatic Learning on PfR

This process tunes the automatic learning capability for the PfR and optimizes the traffic going to the branches. PfR will optimize mission-critical traffic based on policy parameters while sending the rest of the traffic based on a different set of policy parameters.

Procedure 1: Tuning Learning Configuration

Step 1. Define the access list for deny global learning.

ip access-list extended DENY_GLOBAL_LEARN_LIST
deny ip any any
!

Step 2. Define the access list for voice and video traffic.

The classification of the traffic is based on DSCP marking. A common DSCP value for voice is EF, and AF41 and CS4 are for video.

ip access-list extended VOICE_VIDEO
permit ip any any dscp ef
permit ip any any dscp af41
permit ip any any dscp cs4
!

Step 3. Define the access list for critical business applications.

This shows the classification of business applications based on DSCP AF31. The DSCP value can be modified to match the specific business app you want to guarantee service.

ip access-list extended CRITICAL
permit ip any any dscp af31
!

Step 4. Define the access list for best-effort traffic. All other traffic will be classified as best effort.

ip access-list extended BEST_EFFORT
permit ip any any dscp default
!

Step 5. Define the IP prefix list to track traffic going to the branches only.

ip prefix-list BRANCH_PREFIX seq 10 permit 10.1.0.0/16 ge 24
!

Step 6. Applying the learning parameter to the PfR master controller.

Assume the traffic is already marked when it enters the border routers on all sites. This step defines learning based on DSCP values while filtering based on the branch prefix. The filtering, based on the prefix, will make sure PfR only learns traffic destined to the branch locations while the DSCP based access list will help categorize the interesting traffic into different classes of service such as voice and video, critical, and best effort. The classes of service are then associated with different optimization policies.

pfr master
!
learn
throughput
traffic-class filter access-list DENY_GLOBAL_LEARN_LIST
!
list seq 10 refname LEARN_VOICE_VIDEO
traffic-class access-list VOICE_VIDEO filter BRANCH_PREFIX
aggregation-type prefix-length 24
count 2000 max 10000
throughput
!
list seq 20 refname LEARN_CRITICAL
traffic-class access-list CRITICAL filter BRANCH_PREFIX
aggregation-type prefix-length 24
count 2000 max 10000
throughput
!
list seq 30 refname LEARN_BEST_EFFORT
traffic-class access-list BEST_EFFORT filter BRANCH_PREFIX
aggregation-type prefix-length 24
count 2000 max 10000
throughput
!
!

Procedure 2: Define Advanced Policies Per Group

Step 1. Define service level policies for voice and video.

For voice and video traffic, network characteristics such as latency, jitter, and loss are important to the voice and video quality and user experience. Mode monitor is set to fast to allow quick decision-making of alternate paths when an out-of-policy condition is detected.

Match command is to specify that this policy should be applied to all the traffic classes learned under list LEARN_VOICE_VIDEO

Delay threshold is configured as 200 msec. The delay measured by PfR is round-trip time

Loss threshold is configured as five percent

Jitter threshold is configured as 30 ms

Probe frequency is set to eight seconds

Probes are automatically configured and generated by Target Discovery

pfr-map PFRMAP 10
match PfR learn list LEARN_VOICE_VIDEO
set periodic 90
set delay threshold 200
set loss threshold 50000
set jitter threshold 30
set mode monitor fast
set resolve delay priority 1 variance 5
set resolve loss priority 2 variance 5
set resolve jitter priority 3 variance 5
set probe frequency 8
set link-group MPLS fallback Internet
!

Step 2. Define service policies for critical data.

Match command is to specify that this policy should be applied to all the traffic classes learned under list LEARN_CRITICAL

Re-evaluate the alternate path every 90 seconds (periodic 90)

Delay threshold is configured as 250 msec. The delay measured by PfR is round-trip time

Loss threshold is configured as five percent

Probe frequency is set to eight seconds and can be changed to a lesser value if the application being controlled is critical

Probes are automatically configured and generated by Target Discovery

pfr-map PFRMAP 20
match pfr learn list LEARN_CRITICAL
set periodic 90
set delay threshold 250
set mode monitor fast
set resolve delay priority 1 variance 20
set resolve loss priority 5 variance 10
set probe frequency 8
set link-group MPLS fallback Internet
!

Step 3. Define policy for best-effort traffic.

Match command is to specify that this policy should be applied to all the traffic classes learned under list LEARN_BEST_EFFORT

Monitor mode is set to both. This is the default mode. Monitoring is passive and echo probes are used to help find a new exit when a traffic class is out of policy

Default policies used: link utilization first, then load balancing

pfr-map PFRMAP 30
match pfr learn list LEARN_BEST_EFFORT
set periodic 90
set mode monitor both
!

Step 4. Apply the policies on the PfR master controller for them to take effect.

pfr master
policy-rules PFRMAP
!