Troubleshoot SD-WAN Dynamic On-Demand Tunnels

Available Languages

Download Options

ePub (142.9 KB)
View in various apps on iPhone, iPad, Android, Sony Reader, or Windows Phone
Mobi (Kindle) (140.3 KB)
View on Kindle device or Kindle app on multiple devices

Updated:October 5, 2023

Document ID:221045

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Introduction

This document describes troubleshoot commands that can be used when configuring or checking an issue related to SD-WAN dynamic on-demand tunnels.

Prerequistes

Components Used

This document is based on these configuration reference, software and hardware versions:

vManage version 20.9.3
Edge Router ISR4K version 17.9.3
All devices were configured for establish dynamic on-demand tunnels based in official documentation

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Note: Refer to this document for Dynamic On-Demand tunnels configuration.

Background Information

Cisco SD-WAN supports dynamic on-demand tunnels between any two Cisco SD-WAN spoke devices. These tunnels are triggered to be set up only when there is traffic between the two devices optimizing bandwidth usage and device performance.

Working Scenario

Topology Used

In a normal operation scenario the On-Demand tunnels trigger conditions are:

BFD sessions between spokes cannot be established or even appear as down in the show sdwan bfd sessions
BFD sessions can be triggered when interest traffic is sent between the endpoints
Basic Dynamic On-Demand tunnels configurations must be set and confirmed

Triger On-demand Tunnel Activation

Initially BFD sessions between spokes are not up, only sessions from Spokes to Hub are up and on-demand system status can be seen as inactive in both Spokes and in OMP table, backup route from Hub is set as C,I,R while route from Spoke 2 is set as I,U,IA

Spoke 1#show sdwan bfd sessions 
                            SOURCE TLOC REMOTE TLOC             DST PUBLIC    DST PUBLIC         DETECT TX
SYSTEM IP     SITE ID STATE COLOR       COLOR       SOURCE IP   IP            PORT         ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS 
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.10.10.100  100     up    blue        blue        10.10.10.1  10.100.100.1  12366        ipsec     7       1000          2:13:14:35 6

Spoke 1#show sdwan system on-demand remote-system 
SITE-ID SYSTEM-IP   ON-DEMAND STATUS   IDLE-TIMEOUT-EXPIRY(sec)
---------------------------------------------------------------------------
2       10.10.10.2  yes       inactive  - 

Spoke 1#show sdwan omp routes vpn 10 10.2.2.2/32 
Generating output, this might take time, please wait ...
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved
BR-R -> border-router reoriginated
TGW-R -> transport-gateway reoriginated

                                                                                                               AFFINITY 
                                         PATH                 ATTRIBUTE                                        GROUP 
TENANT    VPN       PREFIX      FROM PEER  ID LABEL STATUS    TYPE     TLOC IP      COLOR     ENCAP PREFERENCE NUMBER REGION ID REGION PATH 
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
0         10       10.2.2.2/32 192.168.0.1 61 1005   C,I,R   installed 10.10.10.100 blue      ipsec  -     None    None - 
                               192.168.0.1 62 1003   I,U,IA  installed 10.10.10.2   default   ipsec  -     None    None - 
                               192.168.0.1 64 1005   C,R     installed 10.10.10.100 blue      ipsec  -     None    None - 
                               192.168.0.1 65 1003   I,U,IA  installed 10.10.10.2   private1  ipsec  -     None    None - 
                               192.168.0.1 67 1005   Inv,U   installed 10.10.10.100 blue      ipsec  -     None    None - 
                               192.168.0.1 68 1003   I,U,IA  installed 10.10.10.2   private2  ipsec  -     None    None - 
                               192.168.0.2 71 1005   C,R     installed 10.10.10.100 blue      ipsec  -     None    None - 
                               192.168.0.2 72 1003   U,IA    installed 10.10.10.2   default   ipsec  -     None    None - 
                               192.168.0.2 74 1005   C,R     installed 10.10.10.100 blue      ipsec  -     None    None - 
                               192.168.0.2 75 1003   U,IA    installed 10.10.10.2   private1  ipsec  -     None    None - 
                               192.168.0.2 77 1005   Inv,U   installed 10.10.10.100 blue      ipsec  -     None    None - 
                               192.168.0.2 78 1003   U,IA    installed 10.10.10.2   private2  ipsec  -     None    None -
 
Spoke 2#show sdwan bfd sessions 
                            SOURCE TLOC REMOTE TLOC              DST PUBLIC    DST PUBLIC          DETECT TX
SYSTEM IP     SITE ID STATE COLOR       COLOR        SOURCE IP   IP            PORT         ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS 
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.10.10.100   100    up     blue        blue        10.10.10.2 10.100.100.1    12366       ipsec     7        1000        0:11:10:01 1

Spoke 2#show sdwan system on-demand remote-system 
SITE-ID SYSTEM-IP  ON-DEMAND STATUS IDLE-TIMEOUT-EXPIRY(sec)
---------------------------------------------------------------------------
1       10.10.10.1 yes       inactive -

To trigger on-demand tunnel activation interest traffic is needed. In this example ICMP traffic is used, after sending traffic the status of on-demand remote-system changes from status inactive to status active in both ends and destination prefix changes in OMP table from a C,I,R status from Hub to a C,I,R status from Spoke 2

Spoke 1#ping vrf 10 10.2.2.2 re 20
Type escape sequence to abort.
Sending 20, 100-byte ICMP Echos to 10.2.2.2, timeout is 2 seconds:
!!!!!!!!!!!!!!!!!!!!
Success rate is 100 percent (20/20), round-trip min/avg/max = 1/3/31 ms

Spoke 1#show sdwan system on-demand remote-system 
SITE-ID SYSTEM-IP   ON-DEMAND   STATUS IDLE-TIMEOUT-EXPIRY(sec)
---------------------------------------------------------------------------
2       10.10.10.2  yes         active 56

Spoke 1#show sdwan bfd sessions 
                            SOURCE TLOC REMOTE TLOC            DST PUBLIC   DST PUBLIC       DETECT TX
SYSTEM IP     SITE ID STATE COLOR       COLOR     SOURCE IP    IP           PORT       ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS 
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.10.10.100   100     up blue          blue      10.10.10.1   10.100.100.1 12366      ipsec    7        1000 0:11:14:51 1
10.10.10.2     2       up default       default   10.10.10.1   10.12.12.2   12366      ipsec    7        1000 0:00:00:53 1----> BFD session established due of interest traffic and on-demand configuration
10.10.10.2     2       up blue          blue      10.10.10.1   10.12.12.2   12366      ipsec    7        1000 0:00:00:52 1

Spoke 1#show sdwan omp routes vpn 10 10.2.2.2/32 
Generating output, this might take time, please wait ...
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved
BR-R -> border-router reoriginated
TGW-R -> transport-gateway reoriginated

                                                                                                                 AFFINITY 
                                   PATH                    ATTRIBUTE                                             GROUP 
TENANT   VPN PREFIX      FROM PEER  ID LABEL STATUS         TYPE      TLOC IP     COLOR         ENCAP PREFERENCE NUMBER REGION ID REGION PATH 
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
0        10 10.2.2.2/32 192.168.0.1 61 1005     R          installed 10.10.10.100  blue         ipsec   -     None      None - 
                        192.168.0.1 62 1003     C,I,R      installed 10.10.10.2    default      ipsec   -     None      None - 
                        192.168.0.1 64 1005     R          installed 10.10.10.100  blue         ipsec   -     None      None - 
                        192.168.0.1 65 1003     C,I,R      installed 10.10.10.2    private1     ipsec   -     None      None - 
                        192.168.0.1 67 1005     Inv,U      installed 10.10.10.100  blue         ipsec   -     None      None - 
                        192.168.0.1 68 1003     C,I,R      installed 10.10.10.2    private2     ipsec   -     None      None - 
                        192.168.0.2 71 1005     R          installed 10.10.10.100  blue         ipsec   -     None      None - 
                        192.168.0.2 72 1003     C,R        installed 10.10.10.2    default      ipsec   -     None      None - 
                        192.168.0.2 74 1005     R          installed 10.10.10.100  blue         ipsec   -     None      None - 
                        192.168.0.2 75 1003     C,R        installed 10.10.10.2    private1     ipsec   -     None      None - 
                        192.168.0.2 77 1005     Inv,U      installed 10.10.10.100  blue         ipsec   -     None      None - 
                        192.168.0.2 78 1003     C,R        installed 10.10.10.2    private2     ipsec   -     None      None -

Spoke 2#show sdwan system on-demand remote-system 
SITE-ID SYSTEM-IP ON-DEMAND    STATUS IDLE-TIMEOUT-EXPIRY(sec)
---------------------------------------------------------------------------
1      10.10.10.1 yes         active 53 

Spoke 2#show sdwan bfd sessions 
                           SOURCE TLOC  REMOTE TLOC              DST PUBLIC    DST PUBLIC       DETECT TX
SYSTEM IP    SITE ID STATE COLOR        COLOR     SOURCE IP      IP            PORT     ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS 
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.10.10.100  100     up   blue          blue    10.10.10.2      10.100.100.1  12366    ipsec    7          1000 0:11:14:56 1
10.10.10.1    2       up   default       default 10.10.10.2      10.11.11.1    12366    ipsec    7          1000 0:00:00:53 1----> BFD session established due of interest traffic and on-demand configuration
10.10.10.1    2       up   blue          blue    10.10.10.2      10.11.11.1    12366    ipsec    7          1000 0:00:00:52

After interst traffic stops and idle time-out expires BFD sessions between spokes disappear and on-demand status returns to inactive and route returns to C,I,R backup route status from Hub in OMP table

Spoke 1#show sdwan bfd sessions 
                             SOURCE TLOC REMOTE TLOC               DST PUBLIC     DST PUBLIC     DETECT TX
SYSTEM IP     SITE ID STATE   COLOR        COLOR    SOURCE IP      IP              PORT    ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS 
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.10.10.100  100      up     blue         blue     10.10.10.1      10.100.100.1   12366   ipsec   7           1000 0:11:19:11 1

Spoke 1#show sdwan system on-demand remote-system 
SITE-ID SYSTEM-IP ON-DEMAND STATUS IDLE-TIMEOUT-EXPIRY(sec)
---------------------------------------------------------------------------
2      10.10.10.2  yes      inactive - 

Spoke 2#show sdwan bfd sessions 
                              SOURCE TLOC REMOTE TLOC              DST PUBLIC       DST PUBLIC    DETECT TX
SYSTEM IP     SITE ID   STATE COLOR         COLOR   SOURCE IP       IP               PORT   ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS 
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.10.10.100  100       up    blue          blue    10.10.10.2      10.100.100.1    12366   ipsec  7            1000 0:11:19:11 1

Spoke 2#show sdwan system on-demand remote-system 
SITE-ID SYSTEM-IP ON-DEMAND STATUS IDLE-TIMEOUT-EXPIRY(sec)
---------------------------------------------------------------------------
1 10.10.10.1 yes inactive -

Common Issue Scenarios

Topology Used

Scenario 1: Backup path through the hub considered invalid and unresolved by spokes

Symptom

Destination prefix from Spoke 2 is unreachable, backup path from hub is seen but is considered invalid/uninstalled

Spoke 1#show sdwan omp routes vpn 10 10.2.2.2/32
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved
BR-R -> border-router reoriginated
TGW-R -> transport-gateway reoriginated

                                                                                                         AFFINITY 
                                    PATH                  ATTRIBUTE                                      GROUP 
TENANT VPN PREFIX      FROM PEER    ID      LABEL  STATUS TYPE       TLOC IP    COLOR   ENCAP PREFERENCE NUMBER REGION ID REGION PATH 
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
0     10   10.2.2.2/32 192.168.0.1  61  1005       Inv,U installed 10.10.10.100 blue    ipsec   -          None     None - 
                       192.168.0.1  62  1003       U,IA  installed 10.10.10.2   default ipsec   -          None     None - 
                       192.168.0.1  64  1005       Inv,U installed 10.10.10.100 blue    ipsec   -          None     None - 
                       192.168.0.1  65  1003       U,IA  installed 10.10.10.2   private1ipsec   -          None     None - 
                       192.168.0.1  67  1005       Inv,U installed 10.10.10.100 blue    ipsec   -          None     None - 
                       192.168.0.1  68  1003       U,IA  installed 10.10.10.2   private2ipsec   -          None     None - 
                       192.168.0.2  71  1005       Inv,U installed 10.10.10.100 blue    ipsec   -          None     None - 
                       192.168.0.2  72  1003       U,IA  installed 10.10.10.2   default ipsec   -          None     None - 
                       192.168.0.2  74  1005       Inv,U installed 10.10.10.100 blue    ipsec   -          None     None - 
                       192.168.0.2  75  1003       U,IA  installed 10.10.10.2   private1ipsec   -          None     None - 
                       192.168.0.2  77  1005       Inv,U installed 10.10.10.100 blue    ipsec   -          None     None - 
                       192.168.0.2  78  1003       U,IA  installed 10.10.10.2   private2ipsec   -          None     None -

Troubleshoot

Check if hub BFD sessions towards spokes are established

Hub#show sdwan bfd sessions 
                          SOURCE TLOC    REMOTE TLOC                    DST PUBLIC DST PUBLIC       DETECT     TX
SYSTEM IP  SITE ID  STATE COLOR          COLOR.          SOURCE IP      IP         PORT       ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS 
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.10.10.2  2       up   blue           blue             10.10.10.100   10.12.12.2 12366      ipsec 7          1000         1:23:58:15  2
10.10.10.1  1       up   default        default          10.10.10.100   10.11.11.1 12366      ipsec 7          1000         1:23:59:12  6

Check on-demand tunnel policy to confirm that all sites are included in the correct site lists according to their role (hub or spoke)

Confirm if on-demand feature is enabled and active in spokes using command show sdwan system on-demand

Spoke 1#show sdwan system on-demand 
SITE-ID SYSTEM-IP  ON-DEMAND STATUS IDLE-TIMEOUT-CFG(min)
-------------------------------------------------------------------------
1       10.10.10.1 yes       active 10

Spoke 2#show sdwan system on-demand 
SITE-ID SYSTEM-IP  ON-DEMAND STATUS IDLE-TIMEOUT-CFG(min)
-------------------------------------------------------------------------
2       10.10.10.2 yes       active 10

Confirm if Traffic Engineering service (service TE) is enabled in the hub site. Useful command could be show sdwan run | inc TE

hub#show sdwan run | inc TE
!

Solution

In this case service TE is not enabled in the hub site. To fix, configure it in hub side:

hub#config-trans
hub(config)# sdwan
hub(config-vrf-global)# service TE vrf global
hub(config-vrf-global)# commit

Check that in Spoke 1 OMP table has changed and now has this route as C,I,R for the entry that comes from hub 10.10.10.100 (before generate interest traffic) and gets C,I,R for the entry that comes from Spoke 2 10.10.10.2 (while interest traffic is generated). Also check that BFD session between spoke 1 and spoke 2, and on-demand tunnel is up with command show sdwan system on-demand remote-system <remote system ip> :

Before interest traffic

Spoke 1#show sdwan omp routes vpn 10 10.2.2.2/32 
Generating output, this might take time, please wait ...
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved
BR-R -> border-router reoriginated
TGW-R -> transport-gateway reoriginated

AFFINITY 
                                   PATH               ATTRIBUTE GROUP 
TENANT   VPN PREFIX      FROM PEER ID LABEL STATUS    TYPE     TLOC IP        COLOR      ENCAP PREFERENCE NUMBER REGION ID REGION PATH 
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
0        10 10.2.2.2/32 192.168.0.1 61 1005 C,I,R    installed 10.10.10.100   blue       ipsec   -          None None - 
                        192.168.0.1 62 1003 I,U,IA   installed 10.10.10.2     default    ipsec   -          None None - 
                        192.168.0.1 64 1005 C,R      installed 10.10.10.100   blue       ipsec   -          None None - 
                        192.168.0.1 65 1003 I,U,IA   installed 10.10.10.2     private1   ipsec   -          None None - 
                        192.168.0.1 67 1005 Inv,U    installed 10.10.10.100   blue       ipsec   -          None None - 
                        192.168.0.1 68 1003 I,U,IA   installed 10.10.10.2     private2   ipsec   -          None None - 
                        192.168.0.2 71 1005 C,R      installed 10.10.10.100   blue       ipsec   -          None None - 
                        192.168.0.2 72 1003 U,IA     installed 10.10.10.2     default    ipsec   -          None None - 
                        192.168.0.2 74 1005 C,R      installed 10.10.10.100   blue       ipsec   -          None None - 
                        192.168.0.2 75 1003 U,IA     installed 10.10.10.2     private1   ipsec   -          None None - 
                        192.168.0.2 77 1005 Inv,U    installed 10.10.10.100   blue       ipsec   -          None None - 
                        192.168.0.2 78 1003 U,IA     installed 10.10.10.2     private2   ipsec   -          None None -

While interest traffic

Spoke 1#show sdwan omp routes vpn 10 10.2.2.2/32 
Generating output, this might take time, please wait ...
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved
BR-R -> border-router reoriginated
TGW-R -> transport-gateway reoriginated

                                                                                            AFFINITY 
                                  PATH             ATTRIBUTE                                GROUP 
TENANT VPN PREFIX      FROM PEER  ID LABEL STATUS  TYPE      TLOC IP COLOR ENCAP PREFERENCE NUMBER REGION ID REGION PATH 
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
0      10 10.2.2.2/32 192.168.0.1 61 1005 R        installed 10.10.10.100   blue          ipsec - None None - 
                      192.168.0.1 62 1003 C,I,R    installed 10.10.10.2     default       ipsec - None None - 
                      192.168.0.1 64 1005 R        installed 10.10.10.100   blue          ipsec - None None - 
                      192.168.0.1 65 1003 C,I,R    installed 10.10.10.2     private1      ipsec - None None - 
                      192.168.0.1 67 1005 Inv,U    installed 10.10.10.100   blue          ipsec - None None - 
                      192.168.0.1 68 1003 C,I,R    installed 10.10.10.2     private2      ipsec - None None - 
                      192.168.0.2 71 1005 R        installed 10.10.10.100   blue          ipsec - None None - 
                      192.168.0.2 72 1003 C,R      installed 10.10.10.2     default       ipsec - None None - 
                      192.168.0.2 74 1005 R        installed 10.10.10.100   blue          ipsec - None None - 
                      192.168.0.2 75 1003 C,R      installed 10.10.10.2     private1      ipsec - None None - 
                      192.168.0.2 77 1005 Inv,U    installed 10.10.10.100   blue          ipsec - None None - 
                      192.168.0.2 78 1003 C,R      installed 10.10.10.2     private2      ipsec - None None -

Spoke 1#show sdwan bfd sessions 
                                      SOURCE TLOC REMOTE TLOC            DST PUBLIC      DST PUBLIC      DETECT TX
SYSTEM IP         SITE ID    STATE     COLOR       COLOR   SOURCE IP      IP              PORT    ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS 
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.10.10.100       100         up      blue        blue    10.10.10.1     10.100.100.1    12366   ipsec    7        1000     1:23:58:15 2
10.10.10.2         2           up      default     default 10.10.10.1     10.12.12.2      12366   ipsec    7        1000     0:00:01:50 2----> BFD session established due of on-demand tunnel configuration.
10.10.10.2         2           up      blue        blue    10.10.10.1     10.12.12.2      12366   ipsec    7        1000     0:00:01:52 2

Spoke 1#show sdwan system on-demand remote-system system-ip 10.10.10.2
SITE-ID SYSTEM-IP ON-DEMAND STATUS IDLE-TIMEOUT-EXPIRY(sec)
---------------------------------------------------------------------------
2       10.10.10.2 yes      active  41 ------>on-demand tunnel established to spoke 2 10.10.10.2 due of interest traffic

Scenario 2: BFD sessions between spokes remain up

Symptom

In this case remote Spoke 2 endpoint is listed in the on-demand remote endpoints seen with command show sdwan system on-demand remote-system with a status of no on-demand, BFD session between Spoke 1 and Spoke 2 remains up even when no interest traffic is sent and destination prefix is learn directly from Spoke 2

Spoke 1#show sdwan system on-demand remote-system 
SITE-ID SYSTEM-IP ON-DEMAND STATUS IDLE-TIMEOUT-EXPIRY(sec)
---------------------------------------------------------------------------
2       10.10.10.2 no        -     - 

Spoke 1#show sdwan bfd sessions 
                           SOURCE TLOC REMOTE TLOC           DST PUBLIC   DST PUBLIC       DETECT TX
SYSTEM IP    SITE ID STATE COLOR       COLOR      SOURCE IP  IP           PORT       ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS 
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.10.10.100 100      up   blue        blue       10.10.10.1 10.100.100.1 12366      ipsec 7           1000 1:23:58:15 3
10.10.10.2   2        up   default     default    10.10.10.1 10.12.12.2   12366      ipsec 7           1000 0:00:18:53 4
10.10.10.2   2        up   blue        blue       10.10.10.1 10.12.12.2   12366      ipsec 7           1000 0:00:18:52 3

Spoke 1#show sdwan omp route vpn 10 10.2.2.2/32
Generating output, this might take time, please wait ...
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved
BR-R -> border-router reoriginated
TGW-R -> transport-gateway reoriginated

                                                                                                          AFFINITY 
                                     PATH              ATTRIBUTE                                          GROUP 
TENANT VPN PREFIX          FROM PEER ID LABEL STATUS   TYPE      TLOC IP      COLOR      ENCAP PREFERENCE NUMBER REGION ID REGION PATH 
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
0      10 10.2.2.2/32     192.168.0.1 73 1005 R        installed 10.10.10.100  blue      ipsec  -          None       None      - 
                          192.168.0.1 74 1003 C,I,R    installed 10.10.10.2    default   ipsec  -          None       None      -  
                          192.168.0.1 76 1005 R        installed 10.10.10.100  blue      ipsec  -          None       None      - 
                          192.168.0.1 77 1003 C,I,R    installed 10.10.10.2    private1  ipsec  -          None       None      - 
                          192.168.0.1 79 1005 Inv,U    installed 10.10.10.100  blue      ipsec  -          None       None      - 
                          192.168.0.1 80 1003 C,I,R    installed 10.10.10.2    private2  ipsec  -          None       None      - 
                          192.168.0.2 89 1005 R        installed 10.10.10.100  blue      ipsec  -          None       None      - 
                          192.168.0.2 90 1003 C,R      installed 10.10.10.2    default   ipsec  -          None       None      - 
                          192.168.0.2 92 1005 R        installed 10.10.10.100  blue      ipsec  -          None       None      - 
                          192.168.0.2 93 1003 C,R      installed 10.10.10.2    private1  ipsec  -          None       None      - 
                          192.168.0.2 95 1005 Inv,U    installed 10.10.10.100  blue      ipsec  -          None       None      - 
                          192.168.0.2 96 1003 C,R      installed 10.10.10.2    private2  ipsec  -          None       None      -

Troubleshoot

Check on-demand tunnel policy to confirm that all sites are included in the correct site lists according to their role (hub or spoke)

viptela-policy:policy
 control-policy ondemand
    sequence 1
     match route
      site-list Spokes
      prefix-list _AnyIpv4PrefixList
     !
     action accept
      set
       tloc-action backup
       tloc-list hub
      !
     !
    !
  default-action accept
 !
 lists
  site-list Spokes
   site-id 1-2 
  !
  tloc-list hub
   tloc 10.10.10.100 color blue encap ipsec 
   tloc 10.10.10.100 color default encap ipsec 
   tloc 10.10.10.100 color private1 encap ipsec 
   tloc 10.10.10.100 color private2 encap ipsec 
  !
  prefix-list _AnyIpv4PrefixList
   ip-prefix 0.0.0.0/0 le 32 
  !
 !
!
apply-policy
 site-list Spokes
  control-policy ondemand out
 !
!

Check if on-demand is enable with command show sdwan run | inc on-demand in Spokes and TE is enabled in hub with command show sdwan run | inc TE

Spoke 1#show sdwan run | inc on-demand
on-demand enable
on-demand idle-timeout 10

Spoke 2#show sdwan run | inc on-demand
Spoke 2#

Hub#show sdwan run | inc TE
service TE vrf global

Solution

In this case on-demand is not enabled in Spoke 2. To fix, configure it in Spoke 2 side

Spoke 2#config-trans
Spoke 2(config)# system
Spoke 2(config-vrf-global)# on-demand enable
Spoke 2(config-vrf-global)# on-demand idle-timeout 10
Spoke 2(config-vrf-global)# commit

Check that in Spoke 1 now Spoke 2 is seen as on-demand yes and the OMP table has changed and now has this route as C,I,R for the entry that comes from hub 10.10.10.100 (before generate interest traffic) and not directly from Spoke 2

Spoke 1#show sdwan system on-demand remote-system 
SITE-ID SYSTEM-IP ON-DEMAND STATUS IDLE-TIMEOUT-EXPIRY(sec)
---------------------------------------------------------------------------
2       10.10.10.2 yes inactive - 

Spoke 1#show sdwan omp routes vpn 10 10.2.2.2/32
Generating output, this might take time, please wait ...
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved
BR-R -> border-router reoriginated
TGW-R -> transport-gateway reoriginated

AFFINITY 
        PATH ATTRIBUTE GROUP 
TENANT VPN PREFIX       FROM PEER   ID LABEL STATUS TYPE     TLOC IP      COLOR       ENCAP PREFERENCE NUMBER REGION ID REGION PATH 
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
0      10  10.2.2.2/32  192.168.0.1 61 1005 C,I,R  installed 10.10.10.100  blue       ipsec      -        None    None - 
                        192.168.0.1 62 1003 I,U,IA installed 10.10.10.2    default    ipsec      -        None    None - 
                        192.168.0.1 64 1005 C,R    installed 10.10.10.100  blue       ipsec      -        None    None - 
                        192.168.0.1 65 1003 I,U,IA installed 10.10.10.2    private1   ipsec      -        None    None - 
                        192.168.0.1 67 1005 Inv,U  installed 10.10.10.100  blue       ipsec      -        None    None - 
                        192.168.0.1 68 1003 I,U,IA installed 10.10.10.2    private2   ipsec      -        None    None - 
                        192.168.0.2 71 1005 C,R    installed 10.10.10.100  blue       ipsec      -        None    None - 
                        192.168.0.2 72 1003 U,IA   installed 10.10.10.2    default    ipsec      -        None    None - 
                        192.168.0.2 74 1005 C,R    installed 10.10.10.100  blue       ipsec      -        None    None - 
                        192.168.0.2 75 1003 U,IA   installed 10.10.10.2    private1   ipsec      -        None    None - 
                        192.168.0.2 77 1005 Inv,U  installed 10.10.10.100  blue       ipsec      -        None    None - 
                        192.168.0.2 78 1003 U,IA   installed 10.10.10.2    private2   ipsec      -        None    None -

When interest traffic is generated it gets C,I,R for the entry that comes from Spoke 2 10.10.10.2. Also check that BFD session between Spoke 1 and Spoke 2 is up, also check that on-demand tunnel is up with command show sdwan system on-demand remote-system <remote system ip>

Spoke 1#show sdwan omp routes vpn 10 10.2.2.2/32 
Generating output, this might take time, please wait ...
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved
BR-R -> border-router reoriginated
TGW-R -> transport-gateway reoriginated

                                                                                                                AFFINITY 
                                               PATH               ATTRIBUTE                                     GROUP 
TENANT  VPN        PREFIX          FROM PEER   ID LABEL STATUS    TYPE     TLOC IP      COLOR ENCAP PREFERENCE NUMBER REGION ID REGION PATH 
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
0       10         10.2.2.2/32     192.168.0.1 61 1005 R         installed 10.10.10.100  blue     ipsec   - None None - 
                                   192.168.0.1 62 1003 C,I,R     installed 10.10.10.2    default  ipsec   - None None - 
                                   192.168.0.1 64 1005 R         installed 10.10.10.100  blue     ipsec   - None None - 
                                   192.168.0.1 65 1003 C,I,R     installed 10.10.10.2    private1 ipsec   - None None - 
                                   192.168.0.1 67 1005 Inv,U     installed 10.10.10.100  blue     ipsec   - None None - 
                                   192.168.0.1 68 1003 C,I,R     installed 10.10.10.2    private2 ipsec   - None None - 
                                   192.168.0.2 71 1005 R         installed 10.10.10.100  blue     ipsec   - None None - 
                                   192.168.0.2 72 1003 C,R       installed 10.10.10.2    default  ipsec   - None None - 
                                   192.168.0.2 74 1005 R         installed 10.10.10.100  blue     ipsec   - None None - 
                                   192.168.0.2 75 1003 C,R       installed 10.10.10.2    private1 ipsec   - None None - 
                                   192.168.0.2 77 1005 Inv,U     installed 10.10.10.100  blue     ipsec   - None None - 
                                   192.168.0.2 78 1003 C,R       installed 10.10.10.2    private2 ipsec   - None None -

Spoke 1#show sdwan bfd sessions 
                        SOURCE TLOC REMOTE TLOC             DST PUBLIC  DST PUBLIC          DETECT      TX
SYSTEM IP SITE ID STATE COLOR       COLOR       SOURCE IP   IP          PORT         ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS 
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.10.10.100 100  up blue          blue         10.10.10.1 10.100.100.1 12366       ipsec     7        1000 2:04:34:11 2
10.10.10.2   2    up default       default      10.10.10.1 10.12.12.2   12366       ipsec     7        1000 0:00:02:10 2----> BFD session established due of on-demand tunnel configuration.
10.10.10.2   2    up blue          blue         10.10.10.1 10.12.12.2   12366       ipsec     7        1000 0:00:02:08 2

Spoke 1#show sdwan system on-demand remote-system system-ip 10.10.10.2
SITE-ID SYSTEM-IP ON-DEMAND STATUS IDLE-TIMEOUT-EXPIRY(sec)
---------------------------------------------------------------------------
2        10.10.10.2 yes      active   41 ------>on-demand tunnel established to Spoke 2 10.10.10.2 due of interest traffic

Scenario 3: No backup routes from hub are learened or installed in spokes

Symptom

In this case there are no backup routes for prefix 10.2.2.2/32 originated in Spoke 2 in OMP table, only seen on-demand inactive entries. Confirmed that on-demand in spokes and TE in hub are configured

Spoke 1#show sdwan omp route vpn 10 10.2.2.2/32 
Generating output, this might take time, please wait ...
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved
BR-R -> border-router reoriginated
TGW-R -> transport-gateway reoriginated

AFFINITY 
PATH ATTRIBUTE GROUP 
TENANT VPN      PREFIX     FROM PEER   ID LABEL STATUS TYPE     TLOC IP   COLOR   ENCAP PREFERENCE NUMBER REGION ID REGION PATH 
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
0      10      10.2.2.2/32 192.168.0.1 108 1003 U,IA installed 10.10.10.2 default  ipsec -             None  None - 
                           192.168.0.1 113 1003 U,IA installed 10.10.10.2 private1 ipsec -             None  None - 
                           192.168.0.1 141 1003 U,IA installed 10.10.10.2 private2 ipsec -             None  None - 
                           192.168.0.2 112 1003 U,IA installed 10.10.10.2 default  ipsec -             None  None - 
                           192.168.0.2 117 1003 U,IA installed 10.10.10.2 private1 ipsec -             None  None - 
                           192.168.0.2 144 1003 U,IA installed 10.10.10.2 private2 ipsec -             None  None - 



Spoke 1#show sdwan run | inc on-demand
on-demand enable
on-demand idle-timeout 10

Spoke 2#show sdwan run | inc on-demand
on-demand enable
on-demand idle-timeout 10

Hub#show sdwan run | inc TE
service TE vrf global

Troubleshoot

Check the on-demand centralized policy and confirm if all the spokes are included on the correct site list

viptela-policy:policy
 control-policy ondemand
    sequence 1
     match route
      site-list Spokes
      prefix-list _AnyIpv4PrefixList
     !
     action accept
      set
       tloc-action backup
       tloc-list hub
      !
     !
    !
  default-action accept
 !
 lists
  site-list Spokes
   site-id 1 
  !
  tloc-list hub
   tloc 10.10.10.100 color blue encap ipsec 
   tloc 10.10.10.100 color default encap ipsec 
   tloc 10.10.10.100 color private1 encap ipsec 
   tloc 10.10.10.100 color private2 encap ipsec 
  !
  prefix-list _AnyIpv4PrefixList
   ip-prefix 0.0.0.0/0 le 32 
  !
 !
!
apply-policy
 site-list Spokes
  control-policy ondemand out
 !

Solution

Notice that Spoke 2 site id 2 is missing from the site list spokes in the policy. After including it inside the site list, the backup paths are installed correctly, on-demand tunnel and BFD sessions between spokes comes up when interest traffic is sent.

Spokes site list from policy before
lists 
site-list Spokes 
site-id 1 
!

Spokes site list from policy after
lists
site-list Spokes 
site-id 1-2
!

Spoke 1#show sdwan omp routes vpn 10 10.2.2.2/32
Generating output, this might take time, please wait ...
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved
BR-R -> border-router reoriginated
TGW-R -> transport-gateway reoriginated

AFFINITY 
PATH ATTRIBUTE GROUP 
TENANT VPN PREFIX           FROM PEER   ID LABEL STATUS TYPE      TLOC IP      COLOR    ENCAP PREFERENCE NUMBER REGION ID REGION PATH 
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
0      10  10.2.2.2/32      192.168.0.1 61 1005 C,I,R   installed 10.10.10.100 blue     ipsec    -        None     None - 
                            192.168.0.1 62 1003 I,U,IA  installed 10.10.10.2   default  ipsec    -        None     None - 
                            192.168.0.1 64 1005 C,R     installed 10.10.10.100 blue     ipsec    -        None     None - 
                            192.168.0.1 65 1003 I,U,IA  installed 10.10.10.2   private1 ipsec    -        None     None - 
                            192.168.0.1 67 1005 Inv,U   installed 10.10.10.100 blue     ipsec    -        None     None - 
                            192.168.0.1 68 1003 I,U,IA  installed 10.10.10.2   private2 ipsec    -        None     None - 
                            192.168.0.2 71 1005 C,R     installed 10.10.10.100 blue     ipsec    -        None     None - 
                            192.168.0.2 72 1003 U,IA    installed 10.10.10.2   default  ipsec    -        None     None - 
                            192.168.0.2 74 1005 C,R     installed 10.10.10.100 blue     ipsec    -        None     None - 
                            192.168.0.2 75 1003 U,IA    installed 10.10.10.2   private1 ipsec    -        None     None - 
                            192.168.0.2 77 1005 Inv,U   installed 10.10.10.100 blue     ipsec    -        None     None - 
                            192.168.0.2 78 1003 U,IA    installed 10.10.10.2   private2 ipsec    -        None     None -

Spoke 1#show sdwan bfd sessions 
SOURCE                     TLOC REMOTE TLOC DST PUBLIC     DST PUBLIC              DETECT        TX
SYSTEM IP    SITE ID STATE COLOR       COLOR    SOURCE IP  IP           PORT ENCAP MULTIPLIER INTERVAL(msec) UPTIME TRANSITIONS 
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
10.10.10.100   100   up    blue         blue    10.10.10.1 10.100.100.1 12366 ipsec  7          1000         2:07:01:43 6
10.10.10.2      2    up    default      default 10.10.10.1 10.12.12.2   12366 ipsec  7          1000         0:00:00:56 2----> BFD session established due of on-demand tunnel configuration.
10.10.10.2      2    up    blue         blue    10.10.10.1 10.12.12.2   12366 ipsec  7          1000         0:00:00:56 2

Spoke 1#show sdwan system on-demand remote-system system-ip 10.10.10.2
SITE-ID SYSTEM-IP ON-DEMAND STATUS IDLE-TIMEOUT-EXPIRY(sec)
---------------------------------------------------------------------------
2       10.10.10.2 yes      active    56 ------>on-demand tunnel established to Spoke 2 10.10.10.2 due of interest traffic

Useful commands

show sdwan system on-demand
show sdwan system on-demand remote-system
show sdwan system on-demand remote-system system-ip <system ip>
show sdwan run | inc on-demand
show sdwan run | inc TE
show sdwan ompo routes vpn <vpn number>

Revision History

Revision	Publish Date	Comments
1.0	05-Oct-2023	Initial Release

Contributed by Cisco Engineers

Daniel Alva Vargas
Technical Consulting Engineer

Was this Document Helpful?

Feedback

Contact Cisco

Open a Support Case
(Requires a Cisco Service Contract)

Troubleshoot SD-WAN Dynamic On-Demand Tunnels

Available Languages

Download Options

Bias-Free Language

Contents

Introduction

Prerequistes

Components Used

Background Information

Working Scenario

Topology Used

Triger On-demand Tunnel Activation

Common Issue Scenarios

Topology Used

Scenario 1: Backup path through the hub considered invalid and unresolved by spokes

Symptom

Troubleshoot

Solution

Scenario 2: BFD sessions between spokes remain up

Symptom

Troubleshoot

Solution

Scenario 3: No backup routes from hub are learened or installed in spokes

Symptom

Troubleshoot

Solution

Useful commands

Revision History

Contributed by Cisco Engineers

Was this Document Helpful?

Contact Cisco

This Document Applies to These Products