High availability details

The goal of any high availability solution is to ensure that all network services are resilient to failure. Such a solution aims to provide continuous access to network resources by addressing the potential causes of downtime through functionality, design, and best practices. The core of the Cisco Catalyst SD-WAN high availability solution is achieved through a combination of three factors:

• Functional hardware device redundancy. The basic strategy consists of installing and provisioning redundant hardware devices and redundant components on the hardware. These devices are connected by a secure control plane mesh of Datagram Transport Layer Security (DTLS) connections among themselves, which allows for rapid failover should a device fail or otherwise become unavailable. A key feature of the Cisco Catalyst SD-WAN control plane is that it is established and maintained automatically, by the Cisco IOS XE Catalyst SD-WAN devices and software themselves.
• Robust network design.
• Software mechanisms ensure rapid recovery from a failure. To provide a resilient control plane, the Cisco Catalyst SD-WAN Overlay Management Protocol (OMP) regularly monitors the status of all Cisco IOS XE Catalyst SD-WAN devices in the network and automatically adjusts to changes in the topology as devices join and leave the network. For data plane resiliency, the Cisco Catalyst SD-WAN software implements standard protocol mechanisms, specifically Bidirectional Forwarding Detection (BFD), which runs on the secure IPsec tunnels between routers.

​​Recovery from a failure is a function of the time it takes to detect the failure and then repair or recover from it. The Cisco Catalyst SD-WAN solution provides the ability to control the amount of time to detect a failure in the network. In most cases, repair of the failure is fairly instantaneous.

A standard best practice in any network setup is to install redundant hardware at all levels, including duplicate parallel routers and other systems, redundant fans, power supplies and other hardware components within these devices, and backup network connections. Providing high availability in the Cisco Catalyst SD-WAN solution is no different. A network design that is resilient in the face of hardware failure should include redundant Cisco SD-WAN Validators, Cisco SD-WAN Controllers, and routers and any available redundant hardware components.

Recovery from the total failure of a hardware component in the Cisco Catalyst SD-WAN overlay network happens in basically the same way as in any other network. A backup component has been preconfigured, and it is able to perform all necessary functions by itself.

In addition to simple duplication of hardware components, the high availability of a Cisco Catalyst SD-WAN network can be enhanced by following best practices to design a network that is robust in the face of failure. In one such network design, redundant components are spread around the network as much as possible. Design practices include situating redundant Cisco SD-WAN Validators and Cisco SD-WAN Controllers at dispersed geographical locations and connecting them to different transport networks. Similarly, the routers at a local site can connect to different transport networks and can reach these networks through different NATs and DMZs.

The Cisco Catalyst SD-WAN software support for high availability and resiliency in the face of failure is provided both in the control plane, using the standard DTLS protocol and the proprietary Cisco Catalyst SD-WAN Overlay Management Protocol (OMP), and in the data plane, using the industry-standard protocols BFD, BGP, OSPF, and VRRP.

The Cisco Catalyst SD-WAN control plane operates in conjunction with redundant components to ensure that the overlay network remains resilient if one of the components fails. The control plane is built on top of DTLS or TLS connections between the Cisco devices, and it is monitored by the Cisco Catalyst SD-WAN OMP protocol, which establishes peering sessions (similar to BGP peering sessions) between pairs of Cisco SD-WAN Controllers and routers, and between pairs of Cisco SD-WAN Controllers. These peering sessions allow OMP to monitor the status of the Cisco devices and to share the information among them so that each device in the network has a consistent view of the overlay network. The exchange of control plane information over OMP peering sessions is a key piece in the Cisco Catalyst SD-WAN high availability solution:

• Cisco SD-WAN Controllers quickly and automatically learn when a Cisco SD-WAN Validator or a router joins or leaves the network. They can then rapidly make the necessary modifications in the route information that they send to the routers.
• Cisco SD-WAN Validator quickly and automatically learn when a device joins the network and when a Cisco SD-WAN Controller controller leaves the network. They can then rapidly make the necessary changes to the list of Cisco SD-WAN Controller IP addresses that they send to routers joining the network.
• Cisco SD-WAN Validators learn when a domain has multiple Cisco SD-WAN Controllers and can then provide multiple Cisco SD-WAN Controller addresses to routers joining the network.
• Cisco SD-WAN Controllers learn about the presence of other Cisco SD-WAN Controllers, and they all automatically synchronize their route tables. If one Cisco SD-WAN Controller fails, the remaining systems take over management of the control plane, simply and automatically, and all routers in the network continue to receive current, consistent routing and TLOC updates from the remaining Cisco SD-WAN Controllers.

A highly available Cisco Catalyst SD-WAN network contains two or more Cisco SD-WAN Controllers in each domain. A Cisco Catalyst SD-WAN domain can have up to eight Cisco SD-WAN Controllers, and each Cisco IOS XE Catalyst SD-WAN device, by default, connects to two of them.

The redundancy provided by each of the Cisco Catalyst SD-WAN hardware devices supports network high availability.

The combination of hardware component redundancy with the architecture of the Cisco Catalyst SD-WAN control plane results in a highly available network, one that continues to operate normally and without interruption when a failure occurs in one of the redundant control plane components. Recovery from the total failure of a Cisco SD-WAN Controller, Cisco SD-WAN Validator, or router in the Cisco Catalyst SD-WAN overlay network happens in basically the same way as the recovery from the failure of a regular router or server on the network: A preconfigured backup component is able to perform all necessary functions by itself.

In the Cisco Catalyst SD-WAN solution, when a network device fails and a redundant device is present, network operation continues without interruption. This is true for all Cisco devices, Cisco SD-WAN Validators, Cisco SD-WAN Controllers, and routers. No user configuration is required to implement this behavior; it happens automatically. The OMP peering sessions running between Cisco devices ensure that all the devices have a current and accurate view of the network topology.

Failure recovery occurs on a device-by-device basis.

For data plane resiliency, the Cisco Catalyst SD-WAN software implements the standard BFD protocol, which runs automatically on the secure IPsec connections between routers. These IPsec connections are used for the data plane, and for data traffic, and are independent of the DTLS or TLS tunnels used by the control plane. BFD is used to detect connection failures between the routers. It measures data loss and latency on the data tunnel to determine the status of the devices at either end of the connection.

BFD is enabled, by default, on connections between Cisco IOS XE Catalyst SD-WAN devices and Cisco vEdge devices. BFD sends Hello packets periodically (by default, every 1 second) to determine whether the session is still operational. If a certain number of the Hello packets are not received, BFD considers that the link has failed and brings the BFD session down (the default dead time is 3 seconds). When BFD sessions go down, any route that points to a next hop over that IPsec tunnel is removed from the forwarding table (FIB), but it is still present in the route table (RIB).

In the Cisco Catalyst SD-WAN software, you can adjust the Hello packet and dead time intervals. If the timers on the two ends of a BFD link are different, BFD negotiates to use the lower value.

In the Cisco Catalyst SD-WAN overlay network, all Cisco IOS XE Catalyst SD-WAN devices and Cisco vEdge devices establish control connections to Cisco SD-WAN Controllers, to ensure that the routers are always able to properly route data traffic across the network. As networks increase in size, with routers at thousands of sites and with Cisco SD-WAN Controllers in multiple data centers managing the flow of control and data traffic among routers, network operation can be improved by limiting the number of Cisco SD-WAN Controllers that a router can connect to. When data centers are distributed across a broad geography, network operation can also be better managed by having routers establish control connections only with the Cisco SD-WAN Controllers collocated in the same geographic region.

Establishing affinity between Cisco SD-WAN Controllers and Cisco IOS XE Catalyst SD-WAN devices allows you to control the scaling of the overlay network, by limiting the number of Cisco SD-WAN Controllers that a Cisco IOS XE Catalyst SD-WAN device can establish control connections. When you have redundant routers in a single data center, affinity allows you to distribute the Cisco IOS XE Catalyst SD-WAN device control connections across the Cisco SD-WAN Controllers.

Similarly, when you have multiple data centers in the overlay network, affinity allows you to distribute the control connections between edge devices and the Cisco SD-WAN Controller, across all data centers. This way, a Cisco IOS XE Catalyst SD-WAN device has controller redundancy and data center redundancy. If the link between a Cisco IOS XE Catalyst SD-WAN device and any one of the data centers goes down, the Cisco SD-WAN Controllers in the other data center are available to continue servicing the overlay network.

This figure illustrates this scenario, showing three Cisco SD-WAN Controllers in each of the two data centers. Each of the three Cisco IOS XE Catalyst SD-WAN devices establishes a control connection to one controller in the West data center and one in the East data center.

You might think of this scenario as one where there are redundant data centers in the same region of the world, such as in the same city, province, or country.

For an overlay network that spans a larger geography, such as across continents, you can use affinity so the Cisco IOS XE Catalyst SD-WAN devices establish control connections with data centers in their geographic region and only connect to more distant regions should their closer data centers become unavailable.