Cisco IOS®-XE SD-WAN Installs OSPF External Route with DN-Bit

Introduction

This document describes the expected behavior of IOS®-XE SDWAN software when Open Shortest Path First (OSPF) external routes are installed into the routing table.

Cisco IOS®-XE SD-WAN Installs OSPF External Route with DN-Bit

The router that runs Cisco IOS®-XE IOS®-XE SDWAN software installs OSPF external routes (E1 or E2) into the routing table. For the purpose of the demonstration, consider this simple topology diagram:

Here is a pair of routers R1 and R2 that run Cisco IOS®-XE IOS®-XE SDWAN software establish OSPF peering over service-side vpn (vrf 2 in this example). Routers have system-ip 10.10.10.204 and 10.10.10.205 correspondingly. System-ip is equal to OSPF router-id. Some other router advertises prefix 192.168.1.0/24 via Overlay Management Protocol (OMP) to this site. 

Both routers are configured in a similar manner. Relevant configuration is provided here (the main point is that mutual redistribution between OSPF and OMP is done):

route-map omp2ospf permit 10
 set metric 1000
 set metric-type type-1
!
router ospf 2 vrf 2
 compatible rfc1583
 distance ospf external 110
 distance ospf inter-area 110
 distance ospf intra-area 110
 redistribute omp route-map omp2ospf
!
 omp
  no shutdown
  send-path-limit  4
  ecmp-limit       4
  graceful-restart
  no as-dot-notation
  timers
   holdtime               60
   advertisement-interval 1
   graceful-restart-timer 43200
   eor-timer              300
  exit
  address-family ipv4 vrf 2
   advertise ospf external
   advertise connected
   advertise static
  !
  address-family ipv4
   advertise connected
   advertise static
  !
  address-family ipv6
   advertise connected
   advertise static
  !

When normal condition routing table entry is done, 192.168.1.0/24 is installed into a RIB from OMP and redistributed to OSPF. This entry looks like:

R1#sh ip route vrf 2 192.168.1.0 255.255.255.0

Routing Table: 2
Routing entry for 192.168.1.0/24
  Known via "omp", distance 251, metric 0, type omp
  Redistributing via ospf 2
  Advertised by ospf 2 subnets route-map omp2ospf
  Last update from 10.10.10.201 00:03:00 ago
  Routing Descriptor Blocks:
  * 10.10.10.201 (default), from 10.10.10.201, 00:03:00 ago
      Route metric is 0, traffic share count is 1

R1#show ip ospf database external 192.168.1.0

            OSPF Router with ID (172.16.1.204) (Process ID 2)

		Type-5 AS External Link States

  LS age: 354
  Options: (No TOS-capability, DC, Downward)
  LS Type: AS External Link
  Link State ID: 192.168.1.0 (External Network Number )
  Advertising Router: 172.16.1.204
  LS Seq Number: 80000001
  Checksum: 0x25AE
  Length: 36
  Network Mask: /24
	Metric Type: 1 (Comparable directly to link state metric)
	MTID: 0
	Metric: 1000
	Forward Address: 0.0.0.0
	External Route Tag: 0

  LS age: 355
  Options: (No TOS-capability, DC, Downward)
  LS Type: AS External Link
  Link State ID: 192.168.1.0 (External Network Number )
  Advertising Router: 172.16.1.205
  LS Seq Number: 80000001
  Checksum: 0x1FB3
  Length: 36
  Network Mask: /24
	Metric Type: 1 (Comparable directly to link state metric)
	MTID: 0
	Metric: 1000
	Forward Address: 0.0.0.0
	External Route Tag: 0

R2#sh ip route vrf 2 192.168.1.0 255.255.255.0

Routing Table: 2
Routing entry for 192.168.1.0/24
  Known via "omp", distance 251, metric 0, type omp
  Redistributing via ospf 2
  Advertised by ospf 2 subnets route-map omp2ospf
  Last update from 10.10.10.201 00:04:13 ago
  Routing Descriptor Blocks:
  * 10.10.10.201 (default), from 10.10.10.201, 00:04:13 ago
      Route metric is 0, traffic share count is 1

R2#show ip ospf database external 192.168.1.0

            OSPF Router with ID (172.16.1.205) (Process ID 2)

		Type-5 AS External Link States

  LS age: 317
  Options: (No TOS-capability, DC, Downward)
  LS Type: AS External Link
  Link State ID: 192.168.1.0 (External Network Number )
  Advertising Router: 172.16.1.204
  LS Seq Number: 80000001
  Checksum: 0x25AE
  Length: 36
  Network Mask: /24
	Metric Type: 1 (Comparable directly to link state metric)
	MTID: 0
	Metric: 1000
	Forward Address: 0.0.0.0
	External Route Tag: 0

  LS age: 316
  Options: (No TOS-capability, DC, Downward)
  LS Type: AS External Link
  Link State ID: 192.168.1.0 (External Network Number )
  Advertising Router: 172.16.1.205
  LS Seq Number: 80000001
  Checksum: 0x1FB3
  Length: 36
  Network Mask: /24
	Metric Type: 1 (Comparable directly to link state metric)
	MTID: 0
	Metric: 1000
	Forward Address: 0.0.0.0
	External Route Tag: 0

As you can see, both routers installed route into the RIB and then redistributed it into the OSPF. Both routers set DN-bit to external LSA type 5 and that should prevent these routes from being installed into the RIB as OSPF routes and hence redistributed back to the OMP, essentially preventing the loop. This is the same mechanism described in RFC 4576 and RFC 4577. 

All routers have OMP peering established with vSmart controllers:

R1#show sdwan omp peers
R -> routes received
I -> routes installed
S -> routes sent

                         DOMAIN    OVERLAY   SITE
PEER             TYPE    ID        ID        ID        STATE    UPTIME           R/I/S
------------------------------------------------------------------------------------------
10.10.10.229     vsmart  1         1         1         up       1:19:35:34       30/12/5
10.10.10.230     vsmart  1         1         3         up       1:19:35:33       26/1/5

R2#show sdwan omp peers
R -> routes received
I -> routes installed
S -> routes sent

                         DOMAIN    OVERLAY   SITE
PEER             TYPE    ID        ID        ID        STATE    UPTIME           R/I/S
------------------------------------------------------------------------------------------
10.10.10.229     vsmart  1         1         1         up       0:01:38:48       30/10/6
10.10.10.230     vsmart  1         1         3         up       1:19:35:36       25/1/6

Now, R1 loses connectivity with both OMP peers:

Oct 11 12:53:57.777: %Cisco-SDWAN-Router-OMPD-3-ERRO-400002: R0/0: OMPD: vSmart peer 10.10.10.229 state changed to Init
Oct 11 12:53:57.777: %Cisco-SDWAN-Router-OMPD-6-INFO-400005: R0/0: OMPD: Number of vSmarts connected : 1
Oct 11 12:53:58.777: %Cisco-SDWAN-Router-OMPD-3-ERRO-400002: R0/0: OMPD: vSmart peer 10.10.10.230 state changed to Init
Oct 11 12:53:58.777: %Cisco-SDWAN-Router-OMPD-6-INFO-400005: R0/0: OMPD: Number of vSmarts connected : 0

R1#show sdwan omp peers
R -> routes received
I -> routes installed
S -> routes sent

                         DOMAIN    OVERLAY   SITE
PEER             TYPE    ID        ID        ID        STATE    UPTIME           R/I/S
------------------------------------------------------------------------------------------
10.10.10.229     vsmart  1         1         1         init-in-gr                 30/12/0
10.10.10.230     vsmart  1         1         3         init-in-gr                 26/1/0

R1 will mark the OMP route as stale (see OMP route state S), but continues keeping the route in the RIB installed by OMP protocol until graceful-restart-timer expired:

R1#show sdwan omp routes 192.168.1.0/24 | exclude not set
---------------------------------------------------
omp route entries for vpn 2 route 192.168.1.0/24
---------------------------------------------------
            RECEIVED FROM:
peer            10.10.10.229
path-id         1076
label           1002
status          C,I,R,S
    Attributes:
     originator       10.10.10.201
     type             installed
     tloc             10.10.10.201, biz-internet, ipsec
     overlay-id        1
     site-id          201207
     origin-proto     connected
     origin-metric    0
            RECEIVED FROM:
peer            10.10.10.230
path-id         775
label           1002
status          C,R,S
    Attributes:
     originator       10.10.10.201
     type             installed
     tloc             10.10.10.201, biz-internet, ipsec
     overlay-id        1
     site-id          201207
     origin-proto     connected
     origin-metric    0

R1#sh ip route vrf 2 192.168.1.0 255.255.255.0

Routing Table: 2
Routing entry for 192.168.1.0/24
  Known via "omp", distance 251, metric 0, type omp
  Redistributing via ospf 2
  Advertised by ospf 2 subnets route-map omp2ospf
  Last update from 10.10.10.201 00:23:35 ago
  Routing Descriptor Blocks:
  * 10.10.10.201 (default), from 10.10.10.201, 00:23:35 ago
      Route metric is 0, traffic share count is 1

The default graceful-restart-timer timer is 43,200 seconds (12 hours). Once it is expired, route to 192.168.1.0/24 will be still there.

R1#sh ip route vrf 2 192.168.1.0 255.255.255.0

Routing Table: 2
Routing entry for 192.168.1.0/24
  Known via "ospf 2", distance 252, metric 1100, type extern 1
  Redistributing via omp
  Last update from 10.28.7.205 on Vlan2807, 00:04:11 ago
  Routing Descriptor Blocks:
  * 10.28.7.205, from 172.16.1.205, 00:04:11 ago, via Vlan2807
      SDWAN Down
      Route metric is 1100, traffic share count is 1

R1#show ip ospf database external 192.168.1.0

            OSPF Router with ID (172.16.1.204) (Process ID 2)

		Type-5 AS External Link States

  LS age: 339
  Options: (No TOS-capability, DC, Downward)
  LS Type: AS External Link
  Link State ID: 192.168.1.0 (External Network Number )
  Advertising Router: 172.16.1.205
  LS Seq Number: 80000004
  Checksum: 0x19B6
  Length: 36
  Network Mask: /24
	Metric Type: 1 (Comparable directly to link state metric)
	MTID: 0
	Metric: 1000
	Forward Address: 0.0.0.0
	External Route Tag: 0

It is installed as OSPF External Type 1 route now despite the fact that corresponding OSPF LSA has a DN-bit set. 

Also, note that administrative distance is always 1 unit more than administrative distance (AD) of OMP (251 is default for OMP, hence 252 in this case).

It is important to explain why router installs this route with Active Directory (AD) greater than the AD of OMP route. This is due to the fact that you try to prevent loop scenarios when OMP peering is reestablished again and reachability to the fabric is restored. 

Process of route installation with AD=252 is also clearly seen if debug ip routing and debug ip ospf rib redistribution commands are enabled:

Oct 11 14:13:28.302: RT(2): del 192.168.1.0 via 10.10.10.201, omp metric [251/0]
Oct 11 14:13:28.303: RT(2): delete network route to 192.168.1.0/24
Oct 11 14:13:28.307: OSPF-2 REDIS: Notification to redistribute 192.168.1.0/24
Oct 11 14:13:28.307: RT(2): updating ospf 192.168.1.0/24 (0x2) [local lbl/ctx:1048577/0x0] omp-tag:0  :
    via 10.28.7.205 Vl2807  0 1048578 0x100001
Oct 11 14:13:28.307: RT(2): add 192.168.1.0/24 via 10.28.7.205, ospf metric [252/1100]

This is expected behavior that was specifically introduced in IOS®-XE SDWAN software in order to avoid traffic blackhole scenarios when one of the routers is partitioned from the SD-WAN overlay. Blackhole might happen because a service side traffic is still load-balanced via both routers e.g. because two static routes pointing to both routers or some routes pointing to only one router that is partitioned.

In case if ECMP (when R1 is partitioned from fabric) traffic follows two paths:

LAN -> R1 -> R2 -> remote router -> 192.168.1.0/24

LAN -> R2 -> remote router -> 192.168.1.0/24

Here, you can also see examples of outputs from R1 when R1 is partitioned from the fabric. As you can see, connectivity to LAN subnet 192.168.1.0/24 is still preserved via R2 (10.27.7.205 next-hop):

R1#ping vrf 2 192.168.1.1
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 192.168.1.1, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 4/33/44 ms
R1# traceroute vrf 2 192.168.1.1 numeric
Type escape sequence to abort.
Tracing the route to 192.168.1.1
VRF info: (vrf in name/id, vrf out name/id)
  1 10.28.7.205 4 msec 0 msec 0 msec
  2 192.168.1.1 4 msec *  0 msec