Contents

Introduction

Prerequistes

Components Used

Background Information

Working Scenario

Topology Used

Triger On-demand Tunnel Activation

Common Issue Scenarios

Topology Used

Scenario 1: Backup path through the hub considered invalid and unresolved by spokes

Symptom

Troubleshoot

Solution

Scenario 2: BFD sessions between spokes remain up

Symptom

Troubleshoot

Solution

Scenario 3: No backup routes from hub are learened or installed in spokes

Symptom

Troubleshoot

Solution

Useful commands

Introduction

This document describes troubleshoot commands that can be used when configuring or checking an issue related to SD-WAN dynamic on-demand tunnels.

Prerequistes

Components Used

This document is based on these configuration reference, software and hardware versions:

- vManage version 20.9.3

- Edge Router ISR4K version 17.9.3

- All devices were configured for establish dynamic on-demand tunnels based in official documentation

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Note: Refer to this document for Dynamic On-Demand tunnels configuration.

Background Information

Cisco SD-WAN supports dynamic on-demand tunnels between any two Cisco SD-WAN spoke devices. These tunnels are triggered to be set up only when there is traffic between the two devices optimizing bandwidth usage and device performance.

Working Scenario

Topology Used

In a normal operation scenario the On-Demand tunnels trigger conditions are:

- BFD sessions between spokes cannot be established or even appear as down in the show sdwan bfd sessions

- BFD sessions can be triggered when interest traffic is sent between the endpoints

- Basic Dynamic On-Demand tunnels  configurations must be set and confirmed

Triger On-demand Tunnel Activation

- Initially BFD sessions between spokes are not up, only sessions from Spokes to Hub are up and on-demand system status can be seen as inactive in both Spokes and in OMP table, backup route from Hub is set as C,I,R while route from Spoke 2 is set as I,U,IA

Spoke 1#show sdwan bfd sessions

SOURCE TLOC REMOTE TLOC             DST PUBLIC    DST PUBLIC         DETECT TX

SYSTEM IP     SITE ID STATE COLOR       COLOR       SOURCE IP   IP            PORT         ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

10.10.10.100  100     up    blue        blue        10.10.10.1  10.100.100.1  12366        ipsec     7       1000          2:13:14:35 6

Spoke 1#show sdwan system on-demand remote-system

SITE-ID SYSTEM-IP   ON-DEMAND STATUS   IDLE-TIMEOUT-EXPIRY(sec)

---------------------------------------------------------------------------

2       10.10.10.2  yes       inactive  -

Spoke 1#show sdwan omp routes vpn 10 10.2.2.2/32

Generating output, this might take time, please wait ...

Code:

C -> chosen

I -> installed

Red -> redistributed

Rej -> rejected

L -> looped

R -> resolved

S -> stale

Ext -> extranet

Inv -> invalid

Stg -> staged

IA -> On-demand inactive

U -> TLOC unresolved

BR-R -> border-router reoriginated

TGW-R -> transport-gateway reoriginated

AFFINITY

PATH                 ATTRIBUTE                                        GROUP

TENANT    VPN       PREFIX      FROM PEER  ID LABEL STATUS    TYPE     TLOC IP      COLOR     ENCAP PREFERENCE NUMBER REGION ID REGION PATH

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

0         10       10.2.2.2/32 192.168.0.1 61 1005   C,I,R   installed 10.10.10.100 blue      ipsec  -     None    None -

192.168.0.1 62 1003   I,U,IA  installed 10.10.10.2   default   ipsec  -     None    None -

192.168.0.1 64 1005   C,R     installed 10.10.10.100 blue      ipsec  -     None    None -

192.168.0.1 65 1003   I,U,IA  installed 10.10.10.2   private1  ipsec  -     None    None -

192.168.0.1 67 1005   Inv,U   installed 10.10.10.100 blue      ipsec  -     None    None -

192.168.0.1 68 1003   I,U,IA  installed 10.10.10.2   private2  ipsec  -     None    None -

192.168.0.2 71 1005   C,R     installed 10.10.10.100 blue      ipsec  -     None    None -

192.168.0.2 72 1003   U,IA    installed 10.10.10.2   default   ipsec  -     None    None -

192.168.0.2 74 1005   C,R     installed 10.10.10.100 blue      ipsec  -     None    None -

192.168.0.2 75 1003   U,IA    installed 10.10.10.2   private1  ipsec  -     None    None -

192.168.0.2 77 1005   Inv,U   installed 10.10.10.100 blue      ipsec  -     None    None -

192.168.0.2 78 1003   U,IA    installed 10.10.10.2   private2  ipsec  -     None    None -

Spoke 2#show sdwan bfd sessions

SOURCE TLOC REMOTE TLOC              DST PUBLIC    DST PUBLIC          DETECT TX

SYSTEM IP     SITE ID STATE COLOR       COLOR        SOURCE IP   IP            PORT         ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

10.10.10.100   100    up     blue        blue        10.10.10.2 10.100.100.1    12366       ipsec     7        1000        0:11:10:01 1

Spoke 2#show sdwan system on-demand remote-system

SITE-ID SYSTEM-IP  ON-DEMAND STATUS IDLE-TIMEOUT-EXPIRY(sec)

---------------------------------------------------------------------------

1       10.10.10.1 yes       inactive -

- To trigger on-demand tunnel activation interest traffic is needed. In this example ICMP traffic is used, after sending traffic the status of on-demand remote-system changes from status inactive to status active in both ends and destination prefix changes in OMP table from a C,I,R status from Hub to a C,I,R status from Spoke 2

Spoke 1#ping vrf 10 10.2.2.2 re 20

Type escape sequence to abort.

Sending 20, 100-byte ICMP Echos to 10.2.2.2, timeout is 2 seconds:

!!!!!!!!!!!!!!!!!!!!

Success rate is 100 percent (20/20), round-trip min/avg/max = 1/3/31 ms

Spoke 1#show sdwan system on-demand remote-system

SITE-ID SYSTEM-IP   ON-DEMAND   STATUS IDLE-TIMEOUT-EXPIRY(sec)

---------------------------------------------------------------------------

2       10.10.10.2  yes         active 56

Spoke 1#show sdwan bfd sessions

SOURCE TLOC REMOTE TLOC            DST PUBLIC   DST PUBLIC       DETECT TX

SYSTEM IP     SITE ID STATE COLOR       COLOR     SOURCE IP    IP           PORT       ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

10.10.10.100   100     up blue          blue      10.10.10.1   10.100.100.1 12366      ipsec    7        1000 0:11:14:51 1

10.10.10.2     2       up default       default   10.10.10.1   10.12.12.2   12366      ipsec    7        1000 0:00:00:53 1----> BFD session established due of interest traffic and on-demand configuration

10.10.10.2     2       up blue          blue      10.10.10.1   10.12.12.2   12366      ipsec    7        1000 0:00:00:52 1

Spoke 1#show sdwan omp routes vpn 10 10.2.2.2/32

Generating output, this might take time, please wait ...

Code:

C -> chosen

I -> installed

Red -> redistributed

Rej -> rejected

L -> looped

R -> resolved

S -> stale

Ext -> extranet

Inv -> invalid

Stg -> staged

IA -> On-demand inactive

U -> TLOC unresolved

BR-R -> border-router reoriginated

TGW-R -> transport-gateway reoriginated

AFFINITY

PATH                    ATTRIBUTE                                             GROUP

TENANT   VPN PREFIX      FROM PEER  ID LABEL STATUS         TYPE      TLOC IP     COLOR         ENCAP PREFERENCE NUMBER REGION ID REGION PATH

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

0        10 10.2.2.2/32 192.168.0.1 61 1005     R          installed 10.10.10.100  blue         ipsec   -     None      None -

192.168.0.1 62 1003     C,I,R      installed 10.10.10.2    default      ipsec   -     None      None -

192.168.0.1 64 1005     R          installed 10.10.10.100  blue         ipsec   -     None      None -

192.168.0.1 65 1003     C,I,R      installed 10.10.10.2    private1     ipsec   -     None      None -

192.168.0.1 67 1005     Inv,U      installed 10.10.10.100  blue         ipsec   -     None      None -

192.168.0.1 68 1003     C,I,R      installed 10.10.10.2    private2     ipsec   -     None      None -

192.168.0.2 71 1005     R          installed 10.10.10.100  blue         ipsec   -     None      None -

192.168.0.2 72 1003     C,R        installed 10.10.10.2    default      ipsec   -     None      None -

192.168.0.2 74 1005     R          installed 10.10.10.100  blue         ipsec   -     None      None -

192.168.0.2 75 1003     C,R        installed 10.10.10.2    private1     ipsec   -     None      None -

192.168.0.2 77 1005     Inv,U      installed 10.10.10.100  blue         ipsec   -     None      None -

192.168.0.2 78 1003     C,R        installed 10.10.10.2    private2     ipsec   -     None      None -

Spoke 2#show sdwan system on-demand remote-system

SITE-ID SYSTEM-IP ON-DEMAND    STATUS IDLE-TIMEOUT-EXPIRY(sec)

---------------------------------------------------------------------------

1      10.10.10.1 yes         active 53

Spoke 2#show sdwan bfd sessions

SOURCE TLOC  REMOTE TLOC              DST PUBLIC    DST PUBLIC       DETECT TX

SYSTEM IP    SITE ID STATE COLOR        COLOR     SOURCE IP      IP            PORT     ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

10.10.10.100  100     up   blue          blue    10.10.10.2      10.100.100.1  12366    ipsec    7          1000 0:11:14:56 1

10.10.10.1    2       up   default       default 10.10.10.2      10.11.11.1    12366    ipsec    7          1000 0:00:00:53 1----> BFD session established due of interest traffic and on-demand configuration

10.10.10.1    2       up   blue          blue    10.10.10.2      10.11.11.1    12366    ipsec    7          1000 0:00:00:52

- After interst traffic stops and idle time-out expires BFD sessions between spokes disappear and on-demand status returns to inactive and route returns to C,I,R backup route status from Hub in OMP table

Spoke 1#show sdwan bfd sessions

SOURCE TLOC REMOTE TLOC               DST PUBLIC     DST PUBLIC     DETECT TX

SYSTEM IP     SITE ID STATE   COLOR        COLOR    SOURCE IP      IP              PORT    ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

10.10.10.100  100      up     blue         blue     10.10.10.1      10.100.100.1   12366   ipsec   7           1000 0:11:19:11 1

Spoke 1#show sdwan system on-demand remote-system

SITE-ID SYSTEM-IP ON-DEMAND STATUS IDLE-TIMEOUT-EXPIRY(sec)

---------------------------------------------------------------------------

2      10.10.10.2  yes      inactive -

Spoke 2#show sdwan bfd sessions

SOURCE TLOC REMOTE TLOC              DST PUBLIC       DST PUBLIC    DETECT TX

SYSTEM IP     SITE ID   STATE COLOR         COLOR   SOURCE IP       IP               PORT   ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

10.10.10.100  100       up    blue          blue    10.10.10.2      10.100.100.1    12366   ipsec  7            1000 0:11:19:11 1

Spoke 2#show sdwan system on-demand remote-system

SITE-ID SYSTEM-IP ON-DEMAND STATUS IDLE-TIMEOUT-EXPIRY(sec)

---------------------------------------------------------------------------

1 10.10.10.1 yes inactive -

Common Issue Scenarios

Topology Used

Scenario 1: Backup path through the hub considered invalid and unresolved by spokes

Symptom

- Destination prefix from Spoke 2 is unreachable, backup path from hub is seen but is considered invalid/uninstalled

Spoke 1#show sdwan omp routes vpn 10 10.2.2.2/32

Code:

C -> chosen

I -> installed

Red -> redistributed

Rej -> rejected

L -> looped

R -> resolved

S -> stale

Ext -> extranet

Inv -> invalid

Stg -> staged

IA -> On-demand inactive

U -> TLOC unresolved

BR-R -> border-router reoriginated

TGW-R -> transport-gateway reoriginated

AFFINITY

PATH                  ATTRIBUTE                                      GROUP

TENANT VPN PREFIX      FROM PEER    ID      LABEL  STATUS TYPE       TLOC IP    COLOR   ENCAP PREFERENCE NUMBER REGION ID REGION PATH

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

0     10   10.2.2.2/32 192.168.0.1  61  1005       Inv,U installed 10.10.10.100 blue    ipsec   -          None     None -

192.168.0.1  62  1003       U,IA  installed 10.10.10.2   default ipsec   -          None     None -

192.168.0.1  64  1005       Inv,U installed 10.10.10.100 blue    ipsec   -          None     None -

192.168.0.1  65  1003       U,IA  installed 10.10.10.2   private1ipsec   -          None     None -

192.168.0.1  67  1005       Inv,U installed 10.10.10.100 blue    ipsec   -          None     None -

192.168.0.1  68  1003       U,IA  installed 10.10.10.2   private2ipsec   -          None     None -

192.168.0.2  71  1005       Inv,U installed 10.10.10.100 blue    ipsec   -          None     None -

192.168.0.2  72  1003       U,IA  installed 10.10.10.2   default ipsec   -          None     None -

192.168.0.2  74  1005       Inv,U installed 10.10.10.100 blue    ipsec   -          None     None -

192.168.0.2  75  1003       U,IA  installed 10.10.10.2   private1ipsec   -          None     None -

192.168.0.2  77  1005       Inv,U installed 10.10.10.100 blue    ipsec   -          None     None -

192.168.0.2  78  1003       U,IA  installed 10.10.10.2   private2ipsec   -          None     None -

Troubleshoot

1. Check if hub BFD sessions towards spokes are established

Hub#show sdwan bfd sessions

SOURCE TLOC    REMOTE TLOC                    DST PUBLIC DST PUBLIC       DETECT     TX

SYSTEM IP  SITE ID  STATE COLOR          COLOR.          SOURCE IP      IP         PORT       ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

10.10.10.2  2       up   blue           blue             10.10.10.100   10.12.12.2 12366      ipsec 7          1000         1:23:58:15  2

10.10.10.1  1       up   default        default          10.10.10.100   10.11.11.1 12366      ipsec 7          1000         1:23:59:12  6

2. Check on-demand tunnel policy to confirm that all sites are included in the correct site lists according to their role (hub or spoke)

3. Confirm if on-demand feature is enabled and active in spokes using command show sdwan system on-demand

Spoke 1#show sdwan system on-demand

SITE-ID SYSTEM-IP  ON-DEMAND STATUS IDLE-TIMEOUT-CFG(min)

-------------------------------------------------------------------------

1       10.10.10.1 yes       active 10

Spoke 2#show sdwan system on-demand

SITE-ID SYSTEM-IP  ON-DEMAND STATUS IDLE-TIMEOUT-CFG(min)

-------------------------------------------------------------------------

2       10.10.10.2 yes       active 10

4. Confirm if Traffic Engineering service (service TE) is enabled in the hub site. Useful command could be show sdwan run | inc TE

hub#show sdwan run | inc TE

!

Solution

- In this case service TE is not enabled in the hub site. To fix, configure it in hub side:

hub#config-trans

hub(config)# sdwan

hub(config-vrf-global)# service TE vrf global

hub(config-vrf-global)# commit

- Check that in Spoke 1 OMP table has changed and now has this route as C,I,R for the entry that comes from hub 10.10.10.100 (before generate interest traffic) and gets C,I,R for the entry that comes from Spoke 2 10.10.10.2 (while interest traffic is generated). Also check that BFD session between spoke 1 and spoke 2, and on-demand tunnel is up with command show sdwan system on-demand remote-system <remote system ip> :

Before interest traffic

Spoke 1#show sdwan omp routes vpn 10 10.2.2.2/32

Generating output, this might take time, please wait ...

Code:

C -> chosen

I -> installed

Red -> redistributed

Rej -> rejected

L -> looped

R -> resolved

S -> stale

Ext -> extranet

Inv -> invalid

Stg -> staged

IA -> On-demand inactive

U -> TLOC unresolved

BR-R -> border-router reoriginated

TGW-R -> transport-gateway reoriginated

AFFINITY

PATH               ATTRIBUTE GROUP

TENANT   VPN PREFIX      FROM PEER ID LABEL STATUS    TYPE     TLOC IP        COLOR      ENCAP PREFERENCE NUMBER REGION ID REGION PATH

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

0        10 10.2.2.2/32 192.168.0.1 61 1005 C,I,R    installed 10.10.10.100   blue       ipsec   -          None None -

192.168.0.1 62 1003 I,U,IA   installed 10.10.10.2     default    ipsec   -          None None -

192.168.0.1 64 1005 C,R      installed 10.10.10.100   blue       ipsec   -          None None -

192.168.0.1 65 1003 I,U,IA   installed 10.10.10.2     private1   ipsec   -          None None -

192.168.0.1 67 1005 Inv,U    installed 10.10.10.100   blue       ipsec   -          None None -

192.168.0.1 68 1003 I,U,IA   installed 10.10.10.2     private2   ipsec   -          None None -

192.168.0.2 71 1005 C,R      installed 10.10.10.100   blue       ipsec   -          None None -

192.168.0.2 72 1003 U,IA     installed 10.10.10.2     default    ipsec   -          None None -

192.168.0.2 74 1005 C,R      installed 10.10.10.100   blue       ipsec   -          None None -

192.168.0.2 75 1003 U,IA     installed 10.10.10.2     private1   ipsec   -          None None -

192.168.0.2 77 1005 Inv,U    installed 10.10.10.100   blue       ipsec   -          None None -

192.168.0.2 78 1003 U,IA     installed 10.10.10.2     private2   ipsec   -          None None -

While interest traffic

Spoke 1#show sdwan omp routes vpn 10 10.2.2.2/32

Generating output, this might take time, please wait ...

Code:

C -> chosen

I -> installed

Red -> redistributed

Rej -> rejected

L -> looped

R -> resolved

S -> stale

Ext -> extranet

Inv -> invalid

Stg -> staged

IA -> On-demand inactive

U -> TLOC unresolved

BR-R -> border-router reoriginated

TGW-R -> transport-gateway reoriginated

AFFINITY

PATH             ATTRIBUTE                                GROUP

TENANT VPN PREFIX      FROM PEER  ID LABEL STATUS  TYPE      TLOC IP COLOR ENCAP PREFERENCE NUMBER REGION ID REGION PATH

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

0      10 10.2.2.2/32 192.168.0.1 61 1005 R        installed 10.10.10.100   blue          ipsec - None None -

192.168.0.1 62 1003 C,I,R    installed 10.10.10.2     default       ipsec - None None -

192.168.0.1 64 1005 R        installed 10.10.10.100   blue          ipsec - None None -

192.168.0.1 65 1003 C,I,R    installed 10.10.10.2     private1      ipsec - None None -

192.168.0.1 67 1005 Inv,U    installed 10.10.10.100   blue          ipsec - None None -

192.168.0.1 68 1003 C,I,R    installed 10.10.10.2     private2      ipsec - None None -

192.168.0.2 71 1005 R        installed 10.10.10.100   blue          ipsec - None None -

192.168.0.2 72 1003 C,R      installed 10.10.10.2     default       ipsec - None None -

192.168.0.2 74 1005 R        installed 10.10.10.100   blue          ipsec - None None -

192.168.0.2 75 1003 C,R      installed 10.10.10.2     private1      ipsec - None None -

192.168.0.2 77 1005 Inv,U    installed 10.10.10.100   blue          ipsec - None None -

192.168.0.2 78 1003 C,R      installed 10.10.10.2     private2      ipsec - None None -

Spoke 1#show sdwan bfd sessions

SOURCE TLOC REMOTE TLOC            DST PUBLIC      DST PUBLIC      DETECT TX

SYSTEM IP         SITE ID    STATE     COLOR       COLOR   SOURCE IP      IP              PORT    ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

10.10.10.100       100         up      blue        blue    10.10.10.1     10.100.100.1    12366   ipsec    7        1000     1:23:58:15 2

10.10.10.2         2           up      default     default 10.10.10.1     10.12.12.2      12366   ipsec    7        1000     0:00:01:50 2----> BFD session established due of on-demand tunnel configuration.

10.10.10.2         2           up      blue        blue    10.10.10.1     10.12.12.2      12366   ipsec    7        1000     0:00:01:52 2

Spoke 1#show sdwan system on-demand remote-system system-ip 10.10.10.2

SITE-ID SYSTEM-IP ON-DEMAND STATUS IDLE-TIMEOUT-EXPIRY(sec)

---------------------------------------------------------------------------

2       10.10.10.2 yes      active  41 ------>on-demand tunnel established to spoke 2 10.10.10.2 due of interest traffic

Scenario 2: BFD sessions between spokes remain up

Symptom

- In this case remote Spoke 2 endpoint is listed in the on-demand remote endpoints seen with command show sdwan system on-demand remote-system with a status of no on-demand, BFD session between Spoke 1 and Spoke 2 remains up even when no interest traffic is sent and destination prefix is learn directly from Spoke 2

Spoke 1#show sdwan system on-demand remote-system

SITE-ID SYSTEM-IP ON-DEMAND STATUS IDLE-TIMEOUT-EXPIRY(sec)

---------------------------------------------------------------------------

2       10.10.10.2 no        -     -

Spoke 1#show sdwan bfd sessions

SOURCE TLOC REMOTE TLOC           DST PUBLIC   DST PUBLIC       DETECT TX

SYSTEM IP    SITE ID STATE COLOR       COLOR      SOURCE IP  IP           PORT       ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

10.10.10.100 100      up   blue        blue       10.10.10.1 10.100.100.1 12366      ipsec 7           1000 1:23:58:15 3

10.10.10.2   2        up   default     default    10.10.10.1 10.12.12.2   12366      ipsec 7           1000 0:00:18:53 4

10.10.10.2   2        up   blue        blue       10.10.10.1 10.12.12.2   12366      ipsec 7           1000 0:00:18:52 3

Spoke 1#show sdwan omp route vpn 10 10.2.2.2/32

Generating output, this might take time, please wait ...

Code:

C -> chosen

I -> installed

Red -> redistributed

Rej -> rejected

L -> looped

R -> resolved

S -> stale

Ext -> extranet

Inv -> invalid

Stg -> staged

IA -> On-demand inactive

U -> TLOC unresolved

BR-R -> border-router reoriginated

TGW-R -> transport-gateway reoriginated

AFFINITY

PATH              ATTRIBUTE                                          GROUP

TENANT VPN PREFIX          FROM PEER ID LABEL STATUS   TYPE      TLOC IP      COLOR      ENCAP PREFERENCE NUMBER REGION ID REGION PATH

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

0      10 10.2.2.2/32     192.168.0.1 73 1005 R        installed 10.10.10.100  blue      ipsec  -          None       None      -

192.168.0.1 74 1003 C,I,R    installed 10.10.10.2    default   ipsec  -          None       None      -

192.168.0.1 76 1005 R        installed 10.10.10.100  blue      ipsec  -          None       None      -

192.168.0.1 77 1003 C,I,R    installed 10.10.10.2    private1  ipsec  -          None       None      -

192.168.0.1 79 1005 Inv,U    installed 10.10.10.100  blue      ipsec  -          None       None      -

192.168.0.1 80 1003 C,I,R    installed 10.10.10.2    private2  ipsec  -          None       None      -

192.168.0.2 89 1005 R        installed 10.10.10.100  blue      ipsec  -          None       None      -

192.168.0.2 90 1003 C,R      installed 10.10.10.2    default   ipsec  -          None       None      -

192.168.0.2 92 1005 R        installed 10.10.10.100  blue      ipsec  -          None       None      -

192.168.0.2 93 1003 C,R      installed 10.10.10.2    private1  ipsec  -          None       None      -

192.168.0.2 95 1005 Inv,U    installed 10.10.10.100  blue      ipsec  -          None       None      -

192.168.0.2 96 1003 C,R      installed 10.10.10.2    private2  ipsec  -          None       None      -

Troubleshoot

1. Check on-demand tunnel policy to confirm that all sites are included in the correct site lists according to their role (hub or spoke)

viptela-policy:policy

control-policy ondemand

sequence 1

match route

site-list Spokes

prefix-list _AnyIpv4PrefixList

!

action accept

set

tloc-action backup

tloc-list hub

!

!

!

default-action accept

!

lists

site-list Spokes

site-id 1-2

!

tloc-list hub

tloc 10.10.10.100 color blue encap ipsec

tloc 10.10.10.100 color default encap ipsec

tloc 10.10.10.100 color private1 encap ipsec

tloc 10.10.10.100 color private2 encap ipsec

!

prefix-list _AnyIpv4PrefixList

ip-prefix 0.0.0.0/0 le 32

!

!

!

apply-policy

site-list Spokes

control-policy ondemand out

!

!

2. Check if on-demand is enable with command show sdwan run | inc on-demand in Spokes and TE is enabled in hub with command show sdwan run | inc TE

Spoke 1#show sdwan run | inc on-demand

on-demand enable

on-demand idle-timeout 10

Spoke 2#show sdwan run | inc on-demand

Spoke 2#

Hub#show sdwan run | inc TE

service TE vrf global

Solution

- In this case on-demand is not enabled in Spoke 2. To fix, configure it in Spoke 2 side

Spoke 2#config-trans

Spoke 2(config)# system

Spoke 2(config-vrf-global)# on-demand enable

Spoke 2(config-vrf-global)# on-demand idle-timeout 10

Spoke 2(config-vrf-global)# commit

- Check that in Spoke 1 now Spoke 2 is seen as on-demand yes and the OMP table has changed and now has this route as C,I,R for the entry that comes from hub 10.10.10.100 (before generate interest traffic) and not directly from Spoke 2

Spoke 1#show sdwan system on-demand remote-system

SITE-ID SYSTEM-IP ON-DEMAND STATUS IDLE-TIMEOUT-EXPIRY(sec)

---------------------------------------------------------------------------

2       10.10.10.2 yes inactive -

Spoke 1#show sdwan omp routes vpn 10 10.2.2.2/32

Generating output, this might take time, please wait ...

Code:

C -> chosen

I -> installed

Red -> redistributed

Rej -> rejected

L -> looped

R -> resolved

S -> stale

Ext -> extranet

Inv -> invalid

Stg -> staged

IA -> On-demand inactive

U -> TLOC unresolved

BR-R -> border-router reoriginated

TGW-R -> transport-gateway reoriginated

AFFINITY

PATH ATTRIBUTE GROUP

TENANT VPN PREFIX       FROM PEER   ID LABEL STATUS TYPE     TLOC IP      COLOR       ENCAP PREFERENCE NUMBER REGION ID REGION PATH

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

0      10  10.2.2.2/32  192.168.0.1 61 1005 C,I,R  installed 10.10.10.100  blue       ipsec      -        None    None -

192.168.0.1 62 1003 I,U,IA installed 10.10.10.2    default    ipsec      -        None    None -

192.168.0.1 64 1005 C,R    installed 10.10.10.100  blue       ipsec      -        None    None -

192.168.0.1 65 1003 I,U,IA installed 10.10.10.2    private1   ipsec      -        None    None -

192.168.0.1 67 1005 Inv,U  installed 10.10.10.100  blue       ipsec      -        None    None -

192.168.0.1 68 1003 I,U,IA installed 10.10.10.2    private2   ipsec      -        None    None -

192.168.0.2 71 1005 C,R    installed 10.10.10.100  blue       ipsec      -        None    None -

192.168.0.2 72 1003 U,IA   installed 10.10.10.2    default    ipsec      -        None    None -

192.168.0.2 74 1005 C,R    installed 10.10.10.100  blue       ipsec      -        None    None -

192.168.0.2 75 1003 U,IA   installed 10.10.10.2    private1   ipsec      -        None    None -

192.168.0.2 77 1005 Inv,U  installed 10.10.10.100  blue       ipsec      -        None    None -

192.168.0.2 78 1003 U,IA   installed 10.10.10.2    private2   ipsec      -        None    None -

- When interest traffic is generated it gets C,I,R for the entry that comes from Spoke 2 10.10.10.2. Also check that BFD session between Spoke 1 and Spoke 2 is up, also check that on-demand tunnel is up with command show sdwan system on-demand remote-system <remote system ip>

Spoke 1#show sdwan omp routes vpn 10 10.2.2.2/32

Generating output, this might take time, please wait ...

Code:

C -> chosen

I -> installed

Red -> redistributed

Rej -> rejected

L -> looped

R -> resolved

S -> stale

Ext -> extranet

Inv -> invalid

Stg -> staged

IA -> On-demand inactive

U -> TLOC unresolved

BR-R -> border-router reoriginated

TGW-R -> transport-gateway reoriginated

AFFINITY

PATH               ATTRIBUTE                                     GROUP

TENANT  VPN        PREFIX          FROM PEER   ID LABEL STATUS    TYPE     TLOC IP      COLOR ENCAP PREFERENCE NUMBER REGION ID REGION PATH

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

0       10         10.2.2.2/32     192.168.0.1 61 1005 R         installed 10.10.10.100  blue     ipsec   - None None -

192.168.0.1 62 1003 C,I,R     installed 10.10.10.2    default  ipsec   - None None -

192.168.0.1 64 1005 R         installed 10.10.10.100  blue     ipsec   - None None -

192.168.0.1 65 1003 C,I,R     installed 10.10.10.2    private1 ipsec   - None None -

192.168.0.1 67 1005 Inv,U     installed 10.10.10.100  blue     ipsec   - None None -

192.168.0.1 68 1003 C,I,R     installed 10.10.10.2    private2 ipsec   - None None -

192.168.0.2 71 1005 R         installed 10.10.10.100  blue     ipsec   - None None -

192.168.0.2 72 1003 C,R       installed 10.10.10.2    default  ipsec   - None None -

192.168.0.2 74 1005 R         installed 10.10.10.100  blue     ipsec   - None None -

192.168.0.2 75 1003 C,R       installed 10.10.10.2    private1 ipsec   - None None -

192.168.0.2 77 1005 Inv,U     installed 10.10.10.100  blue     ipsec   - None None -

192.168.0.2 78 1003 C,R       installed 10.10.10.2    private2 ipsec   - None None -

Spoke 1#show sdwan bfd sessions

SOURCE TLOC REMOTE TLOC             DST PUBLIC  DST PUBLIC          DETECT      TX

SYSTEM IP SITE ID STATE COLOR       COLOR       SOURCE IP   IP          PORT         ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

10.10.10.100 100  up blue          blue         10.10.10.1 10.100.100.1 12366       ipsec     7        1000 2:04:34:11 2

10.10.10.2   2    up default       default      10.10.10.1 10.12.12.2   12366       ipsec     7        1000 0:00:02:10 2----> BFD session established due of on-demand tunnel configuration.

10.10.10.2   2    up blue          blue         10.10.10.1 10.12.12.2   12366       ipsec     7        1000 0:00:02:08 2

Spoke 1#show sdwan system on-demand remote-system system-ip 10.10.10.2

SITE-ID SYSTEM-IP ON-DEMAND STATUS IDLE-TIMEOUT-EXPIRY(sec)

---------------------------------------------------------------------------

2        10.10.10.2 yes      active   41 ------>on-demand tunnel established to Spoke 2 10.10.10.2 due of interest traffic

Scenario 3: No backup routes from hub are learened or installed in spokes

Symptom

- In this case there are no backup routes for prefix 10.2.2.2/32 originated in Spoke 2 in OMP table, only seen on-demand inactive entries. Confirmed that on-demand in spokes and TE in hub are configured

Spoke 1#show sdwan omp route vpn 10 10.2.2.2/32

Generating output, this might take time, please wait ...

Code:

C -> chosen

I -> installed

Red -> redistributed

Rej -> rejected

L -> looped

R -> resolved

S -> stale

Ext -> extranet

Inv -> invalid

Stg -> staged

IA -> On-demand inactive

U -> TLOC unresolved

BR-R -> border-router reoriginated

TGW-R -> transport-gateway reoriginated

AFFINITY

PATH ATTRIBUTE GROUP

TENANT VPN      PREFIX     FROM PEER   ID LABEL STATUS TYPE     TLOC IP   COLOR   ENCAP PREFERENCE NUMBER REGION ID REGION PATH

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

0      10      10.2.2.2/32 192.168.0.1 108 1003 U,IA installed 10.10.10.2 default  ipsec -             None  None -

192.168.0.1 113 1003 U,IA installed 10.10.10.2 private1 ipsec -             None  None -

192.168.0.1 141 1003 U,IA installed 10.10.10.2 private2 ipsec -             None  None -

192.168.0.2 112 1003 U,IA installed 10.10.10.2 default  ipsec -             None  None -

192.168.0.2 117 1003 U,IA installed 10.10.10.2 private1 ipsec -             None  None -

192.168.0.2 144 1003 U,IA installed 10.10.10.2 private2 ipsec -             None  None -

Spoke 1#show sdwan run | inc on-demand

on-demand enable

on-demand idle-timeout 10

Spoke 2#show sdwan run | inc on-demand

on-demand enable

on-demand idle-timeout 10

Hub#show sdwan run | inc TE

service TE vrf global

Troubleshoot

- Check the on-demand centralized policy and confirm if all the spokes are included on the correct site list

viptela-policy:policy

control-policy ondemand

sequence 1

match route

site-list Spokes

prefix-list _AnyIpv4PrefixList

!

action accept

set

tloc-action backup

tloc-list hub

!

!

!

default-action accept

!

lists

site-list Spokes

site-id 1

!

tloc-list hub

tloc 10.10.10.100 color blue encap ipsec

tloc 10.10.10.100 color default encap ipsec

tloc 10.10.10.100 color private1 encap ipsec

tloc 10.10.10.100 color private2 encap ipsec

!

prefix-list _AnyIpv4PrefixList

ip-prefix 0.0.0.0/0 le 32

!

!

!

apply-policy

site-list Spokes

control-policy ondemand out

!

Solution

- Notice that Spoke 2 site id 2 is missing from the site list spokes in the policy. After including it inside the site list, the backup paths are installed correctly, on-demand tunnel and BFD sessions between spokes comes up when interest traffic is sent.

Spokes site list from policy before

lists

site-list Spokes

site-id 1

!

Spokes site list from policy after

lists

site-list Spokes

site-id 1-2

!

Spoke 1#show sdwan omp routes vpn 10 10.2.2.2/32

Generating output, this might take time, please wait ...

Code:

C -> chosen

I -> installed

Red -> redistributed

Rej -> rejected

L -> looped

R -> resolved

S -> stale

Ext -> extranet

Inv -> invalid

Stg -> staged

IA -> On-demand inactive

U -> TLOC unresolved

BR-R -> border-router reoriginated

TGW-R -> transport-gateway reoriginated

AFFINITY

PATH ATTRIBUTE GROUP

TENANT VPN PREFIX           FROM PEER   ID LABEL STATUS TYPE      TLOC IP      COLOR    ENCAP PREFERENCE NUMBER REGION ID REGION PATH

-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

0      10  10.2.2.2/32      192.168.0.1 61 1005 C,I,R   installed 10.10.10.100 blue     ipsec    -        None     None -

192.168.0.1 62 1003 I,U,IA  installed 10.10.10.2   default  ipsec    -        None     None -

192.168.0.1 64 1005 C,R     installed 10.10.10.100 blue     ipsec    -        None     None -

192.168.0.1 65 1003 I,U,IA  installed 10.10.10.2   private1 ipsec    -        None     None -

192.168.0.1 67 1005 Inv,U   installed 10.10.10.100 blue     ipsec    -        None     None -

192.168.0.1 68 1003 I,U,IA  installed 10.10.10.2   private2 ipsec    -        None     None -

192.168.0.2 71 1005 C,R     installed 10.10.10.100 blue     ipsec    -        None     None -

192.168.0.2 72 1003 U,IA    installed 10.10.10.2   default  ipsec    -        None     None -

192.168.0.2 74 1005 C,R     installed 10.10.10.100 blue     ipsec    -        None     None -

192.168.0.2 75 1003 U,IA    installed 10.10.10.2   private1 ipsec    -        None     None -

192.168.0.2 77 1005 Inv,U   installed 10.10.10.100 blue     ipsec    -        None     None -

192.168.0.2 78 1003 U,IA    installed 10.10.10.2   private2 ipsec    -        None     None -

Spoke 1#show sdwan bfd sessions

SOURCE                     TLOC REMOTE TLOC DST PUBLIC     DST PUBLIC              DETECT        TX

SYSTEM IP    SITE ID STATE COLOR       COLOR    SOURCE IP  IP           PORT ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

10.10.10.100   100   up    blue         blue    10.10.10.1 10.100.100.1 12366 ipsec  7          1000         2:07:01:43 6

10.10.10.2      2    up    default      default 10.10.10.1 10.12.12.2   12366 ipsec  7          1000         0:00:00:56 2----> BFD session established due of on-demand tunnel configuration.

10.10.10.2      2    up    blue         blue    10.10.10.1 10.12.12.2   12366 ipsec  7          1000         0:00:00:56 2

Spoke 1#show sdwan system on-demand remote-system system-ip 10.10.10.2

SITE-ID SYSTEM-IP ON-DEMAND STATUS IDLE-TIMEOUT-EXPIRY(sec)

---------------------------------------------------------------------------

2       10.10.10.2 yes      active    56 ------>on-demand tunnel established to Spoke 2 10.10.10.2 due of interest traffic

Useful commands

- show sdwan system on-demand

- show sdwan system on-demand remote-system

- show sdwan system on-demand remote-system system-ip <system ip>

- show sdwan run | inc on-demand

- show sdwan run | inc TE

- show sdwan ompo routes vpn <vpn number>