Contents

Introduction

Prerequisites

Requirements

Components Used

Control Plane Information

Check Control Local Properties

Check Control Connections

Overlay Management Protocol

Verify OMP TLOCs are Advertised from the vEdges

Verify vSmart Receives and Advertises the TLOCs

Bidirectional Forwarding Detection

Understand the show bfd sessions command

Command show tunnel statistics

Access List

Network Address Translation

How to Use tools stun-client to Detect NAT Maps and Filters.

Supported NAT Types for Data Plane Tunnels“Sending” used in CLI

Firewalls

Security

ISP Problems with DSCP Marked Traffic

Debug BFD

Related Information

Introduction

This document describes vEdge data plane connection problems after a control plane connection however no data plane connectivity between sites.

Prerequisites

Requirements

Cisco recommends knowledge of Cisco Software Defined Wide Area Network (SDWAN) solution.

Components Used

This document is not restricted to specific software and hardware versions.  This document is focused on vEdge platforms.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

For Cisco Edge routers (Cisco IOS® XE routers in controller mode) , please read .

Control Plane Information

Check Control Local Properties

In order to check the status of the Wide Area Network (WAN) interfaces on a vEdge, use the command, show control local-properties wan-interface-list.

In this output, you can see the RFC 4787 Network Address Translation (NAT) Type.

When the vEdge is behind a NAT device (Firewall, Router, etc), Public and Private IPv4 address, Public and Private source User Datagram Protocol (UDP) ports are used to build the data plane tunnels.

You can also find the state of the tunnel interface, color, and maximum number of control connections configured.

vEdge1# show control local-properties wan-interface-list

NAT TYPE: E -- indicates End-point independent mapping

A -- indicates Address-port dependent mapping

N -- indicates Not learned

Requires minimum two vbonds to learn the NAT type

PUBLIC          PUBLIC PRIVATE         PRIVATE          PRIVATE                           MAX   RESTRICT/           LAST        SPI TIME   NAT  VM

INTERFACE  IPv4            PORT   IPv4            IPv6             PORT    VS/VM COLOR        STATE CNTRL  CONTROL/   LR/LB   CONNECTION   REMAINING  TYPE CON

STUN                                           PRF

---------------------------------------------------------------------------------------------------------------------------------------------------------------

ge0/0      203.0.113.225   4501   10.19.145.2     ::               12386    1/1  gold           up    2    no/yes/no   No/No  7:02:55:13   0:09:02:29  N    5

ge0/1      10.20.67.10     12426  10.20.67.10     ::               12426    0/0  mpls           up    2    yes/yes/no  No/No  0:00:00:01   0:11:40:16  N    5

With this data, you can identify certain information about how the data tunnels must be built and what ports you can expect (from the routers perspective) to use when you form the data tunnels.

Check Control Connections

It is important to ensure that the color that does not form data plane tunnels has a control connection established with the controllers in the overlay.

Otherwise, the vEdge does not send the Transport Locator (TLOC) information to the vSmart via Overlay Management Protocol (OMP).

You can verify if it is operational with the use of show control connections command, and look for the connect state.

vEdge1# show control connections

PEER                                          PEER                                              CONTROLLER

PEER    PEER PEER            SITE       DOMAIN PEER                                    PRIV  PEER                                    PUB                                               GROUP

TYPE    PROT SYSTEM IP       ID         ID     PRIVATE IP                              PORT  PUBLIC IP                               PORT  LOCAL COLOR     STATE           UPTIME      ID

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

vsmart  dtls 10.1.0.3         3          1      203.0.113.13                          12446 203.0.113.13                          12446 gold            up              7:03:18:31  0

vbond   dtls -                0          0      203.0.113.12                          12346 203.0.113.12                          12346 mpls            connect                     0

vmanage dtls 10.1.0.1         1          0      203.0.113.14                          12646 203.0.113.14                          12646 gold            up              7:03:18:31  0

If the interface (that does not form data tunnels) tries to connect, solve it with a successful start-up of the control connections via that color.

Or, set the max-control-connections 0 in the selected interface under the tunnel interface section.

vpn 0

interface ge0/1

ip address 10.20.67.10/24

tunnel-interface

encapsulation ipsec

color mpls restrict

max-control-connections 0

no allow-service bgp

allow-service dhcp

allow-service dns

allow-service icmp

no allow-service sshd

no allow-service netconf

no allow-service ntp

no allow-service ospf

no allow-service stun

!

no shutdown

!

Note: Sometimes, you can use the no control-connections command to achieve the same goal. However, that command does not establish a maximum number of control connections. This command is deprecated from version 15.4 and is not used on newer software.

Overlay Management Protocol

Verify OMP TLOCs are Advertised from the vEdges

OMP TLOCs cannot be sent because the interface tries to form control connections via that color and is not able to reach the controllers.

Check if the color (that the data tunnels) sends the TLOC for that particular color to the vSmarts.

Use the command show omp tlocs advertised in order to check the TLOCs that are sent to the OMP peers.

Example: Colors mpls and gold. No TLOC is sent to vSmart for color mpls.

vEdge1# show omp tlocs advertised

C   -> chosen

I   -> installed

Red -> redistributed

Rej -> rejected

L   -> looped

R   -> resolved

S   -> stale

Ext -> extranet

Stg -> staged

Inv -> invalid

PUBLIC           PRIVATE

ADDRESS                                                                      PSEUDO                   PUBLIC                   PRIVATE  PUBLIC  IPV6    PRIVATE  IPV6     BFD

FAMILY   TLOC IP          COLOR            ENCAP  FROM PEER        STATUS    KEY     PUBLIC IP        PORT    PRIVATE IP       PORT     IPV6    PORT    IPV6     PORT     STATUS

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

ipv4     10.1.0.5         gold             ipsec  0.0.0.0          C,Red,R   1       203.0.113.225    4501    10.19.145.2      12386    ::      0       ::       0        up

10.1.0.2         mpls             ipsec  10.1.0.3          C,I,R     1       10.20.67.20      12386   10.20.67.20      12386    ::      0       ::       0        down

10.1.0.2         blue             ipsec  10.1.0.3          C,I,R     1       198.51.100.187   12406   10.19.146.2      12406    ::      0       ::       0        up

10.1.0.30         mpls             ipsec  10.1.0.3          C,I,R     1       10.20.67.30      12346   10.20.67.30      12346    ::      0       ::       0        down

10.1.0.30         gold             ipsec  10.1.0.3          C,I,R     1       192.0.2.129      12386   192.0.2.129      12386    ::      0       ::       0        up

10.1.0.4         mpls             ipsec  10.1.0.3          C,I,R     1       10.20.67.40      12426   10.20.67.40      12426    ::      0       ::       0        down

10.1.0.4         gold             ipsec  10.1.0.3          C,I,R     1       203.0.113.226    12386   203.0.113.226    12386    ::      0       ::       0        up

Example: Colors mpls and gold. TLOC is sent for both the colors.

vEdge2# show omp tlocs advertised

C   -> chosen

I   -> installed

Red -> redistributed

Rej -> rejected

L   -> looped

R   -> resolved

S   -> stale

Ext -> extranet

Stg -> staged

Inv -> invalid

PUBLIC           PRIVATE

ADDRESS                                                                      PSEUDO                   PUBLIC                   PRIVATE  PUBLIC  IPV6    PRIVATE  IPV6     BFD

FAMILY   TLOC IP          COLOR            ENCAP  FROM PEER        STATUS    KEY     PUBLIC IP        PORT    PRIVATE IP       PORT     IPV6    PORT    IPV6     PORT     STATUS

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

ipv4     10.1.0.5         gold             ipsec  10.1.0.3          C,I,R     1       203.0.113.225    4501    10.19.145.2      12386    ::      0       ::       0        up

10.1.0.2         mpls             ipsec  0.0.0.0          C,Red,R   1       10.20.67.20      12386   10.20.67.20      12386    ::      0       ::       0        up

10.1.0.2         blue             ipsec  0.0.0.0          C,Red,R   1       198.51.100.187   12406   10.19.146.2      12406    ::      0       ::       0        up

10.1.0.30         mpls             ipsec  10.1.0.3          C,I,R     1       10.20.67.30      12346   10.20.67.30      12346    ::      0       ::       0        up

10.1.0.30         gold             ipsec  10.1.0.3          C,I,R     1       192.0.2.129      12386   192.0.2.129      12386    ::      0       ::       0        up

10.1.0.4         mpls             ipsec  10.1.0.3          C,I,R     1       10.20.67.40      12426   10.20.67.40      12426    ::      0       ::       0        up

10.1.0.4         gold             ipsec  10.1.0.3          C,I,R     1       203.0.113.226    12386   203.0.113.226    12386    ::      0       ::       0        up

Note: For any locally generated control plane information, "FROM PEER" field is set to 0.0.0.0. When you look for locally originated information, ensure to match based on this value.

Verify vSmart Receives and Advertises the TLOCs

TLOCs are now advertised out to the vSmart. Confirm that it receives TLOCs from the correct peer and advertises it to the other vEdge.

Example: vSmart receives the TLOCs from 10.1.0.2 vEdge1.

vSmart1# show omp tlocs received

C   -> chosen

I   -> installed

Red -> redistributed

Rej -> rejected

L   -> looped

R   -> resolved

S   -> stale

Ext -> extranet

Stg -> staged

Inv -> invalid

PUBLIC           PRIVATE

ADDRESS                                                                      PSEUDO                   PUBLIC                   PRIVATE  PUBLIC  IPV6    PRIVATE  IPV6     BFD

FAMILY   TLOC IP          COLOR            ENCAP  FROM PEER        STATUS    KEY     PUBLIC IP        PORT    PRIVATE IP       PORT     IPV6    PORT    IPV6     PORT     STATUS

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

ipv4     10.1.0.5         gold             ipsec  10.1.0.5         C,I,R     1       203.0.113.225  4501    10.19.145.2      12386    ::      0       ::       0        -

10.1.0.2         mpls             ipsec  10.1.0.2         C,I,R     1       10.20.67.20      12386   10.20.67.20      12386    ::      0       ::       0        -

10.1.0.2         blue             ipsec  10.1.0.2         C,I,R     1       198.51.100.187   12406   10.19.146.2      12406    ::      0       ::       0        -

10.1.0.30        mpls             ipsec  10.1.0.30         C,I,R     1       10.20.67.30      12346   10.20.67.30      12346    ::      0       ::       0        -

10.1.0.30        gold             ipsec  10.1.0.30         C,I,R     1       192.0.2.129      12386   192.0.2.129      12386    ::      0       ::       0        -

10.1.0.4         mpls             ipsec  10.1.0.4         C,I,R     1       10.20.67.40      12426   10.20.67.40      12426    ::      0       ::       0        -

10.1.0.4         gold             ipsec  10.1.0.4         C,I,R     1       203.0.113.226    12386   203.0.113.226    12386    ::      0       ::       0        -

If you do not see the TLOCs or you see any other codes here, check these:

vSmart-vIPtela-MEX# show omp tlocs received

C   -> chosen

I   -> installed

Red -> redistributed

Rej -> rejected

L   -> looped

R   -> resolved

S   -> stale

Ext -> extranet

Stg -> staged

Inv -> invalid

PUBLIC           PRIVATE

ADDRESS                                                                      PSEUDO                   PUBLIC                   PRIVATE  PUBLIC  IPV6    PRIVATE  IPV6     BFD

FAMILY   TLOC IP          COLOR            ENCAP  FROM PEER        STATUS    KEY     PUBLIC IP        PORT    PRIVATE IP       PORT     IPV6    PORT    IPV6     PORT     STATUS

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

ipv4     10.1.0.5         gold             ipsec  10.1.0.5         C,I,R     1       203.0.113.225    4501    10.19.145.2      12386    ::      0       ::       0        -

10.1.0.2         mpls             ipsec  10.1.0.2         C,I,R     1       10.20.67.20      12386   10.20.67.20      12386    ::      0       ::       0        -

10.1.0.2         blue             ipsec  10.1.0.2         Rej,R,Inv 1       198.51.100.187   12406   10.19.146.2      12406    ::      0       ::       0        -

10.1.0.30         mpls             ipsec  10.1.0.30         C,I,R     1       10.20.67.30      12346   10.20.67.30      12346    ::      0       ::       0        -

10.1.0.30         gold             ipsec  10.1.0.30         C,I,R     1       192.0.2.129      12386   192.0.2.129      12386    ::      0       ::       0        -

10.1.0.4         mpls             ipsec  10.1.0.4         C,I,R     1       10.20.67.40      12426   10.20.67.40      12426    ::      0       ::       0        -

10.1.0.4         gold             ipsec  10.1.0.4         C,I,R     1       203.0.113.226    12386   203.0.113.226    12386    ::      0       ::       0        -

Verify that there is no policy that blocks the TLOCs.

show run policy control-policy - look for any tloc-list that rejects your TLOCs as advertised or received in the vSmart.

vSmart1(config-policy)# sh config

policy

lists

tloc-list SITE20

tloc 10.1.0.2 color blue encap ipsec

!

!

control-policy SDWAN

sequence 10

match tloc

tloc-list SITE20

!

action reject ----> here we are rejecting the TLOC 10.1.0.2,blue,ipsec

!

!

default-action accept

!

apply-policy

site-list SITE20

control-policy SDWAN in -----> the policy is applied to control traffic coming IN the vSmart, it will filter the tlocs before adding it to the OMP table.

Note: If a TLOC is Rejected or Invalid, it is not advertised to the other vEdges.

Ensure that a policy does not filter the TLOC when it is advertised from the vSmart. You can see that the TLOC is received on the vSmart, but you do not see it on the other vEdge.

Example 1: vSmart with TLOC  in C,I,R.

vSmart1# show omp tlocs

C   -> chosen

I   -> installed

Red -> redistributed

Rej -> rejected

L   -> looped

R   -> resolved

S   -> stale

Ext -> extranet

Stg -> staged

Inv -> invalid

PUBLIC           PRIVATE

ADDRESS                                                                      PSEUDO                   PUBLIC                   PRIVATE  PUBLIC  IPV6    PRIVATE  IPV6     BFD

FAMILY   TLOC IP          COLOR            ENCAP  FROM PEER        STATUS    KEY     PUBLIC IP        PORT    PRIVATE IP       PORT     IPV6    PORT    IPV6     PORT     STATUS

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

ipv4     10.1.0.5         mpls             ipsec  10.1.0.5         C,I,R     1       10.20.67.10      12406   10.20.67.10      12406    ::      0       ::       0        -

10.1.0.5         gold             ipsec  10.1.0.5         C,I,R     1       203.0.113.225    4501    10.19.145.2      12386    ::      0       ::       0        -

10.1.0.2         mpls             ipsec  10.1.0.2         C,I,R     1       10.20.67.20      12386   10.20.67.20      12386    ::      0       ::       0        -

10.1.0.2         blue             ipsec  10.1.0.2         C,I,R     1       198.51.100.187   12426   10.19.146.2      12426    ::      0       ::       0        -

10.1.0.30         mpls             ipsec  10.1.0.30         C,I,R     1       10.20.67.30      12346   10.20.67.30      12346    ::      0       ::       0        -

10.1.0.30         gold             ipsec  10.1.0.30         C,I,R     1       192.0.2.129      12386   192.0.2.129      12386    ::      0       ::       0        -

10.1.0.4         mpls             ipsec  10.1.0.4         C,I,R     1       10.20.67.40      12426   10.20.67.40      12426    ::      0       ::       0        -

10.1.0.4         gold             ipsec  10.1.0.4         C,I,R     1       203.0.113.226    12386   203.0.113.226    12386    ::      0       ::       0        -

Example 2: vEdge1 does not see the TLOC from color blue that comes of vEdge2. It only sees MPLS TLOC.

vEdge1# show omp tlocs

C   -> chosen

I   -> installed

Red -> redistributed

Rej -> rejected

L   -> looped

R   -> resolved

S   -> stale

Ext -> extranet

Stg -> staged

Inv -> invalid

PUBLIC           PRIVATE

ADDRESS                                                                      PSEUDO                   PUBLIC                   PRIVATE  PUBLIC  IPV6    PRIVATE  IPV6     BFD

FAMILY   TLOC IP          COLOR            ENCAP  FROM PEER        STATUS    KEY     PUBLIC IP        PORT    PRIVATE IP       PORT     IPV6    PORT    IPV6     PORT     STATUS

----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

ipv4     10.1.0.5         mpls             ipsec  0.0.0.0          C,Red,R   1       10.20.67.10      12406   10.20.67.10      12406    ::      0       ::       0        up

10.1.0.5         gold             ipsec  0.0.0.0          C,Red,R   1       203.0.113.225    4501    10.19.145.2      12386    ::      0       ::       0        up

10.1.0.2         mpls             ipsec  10.1.0.3          C,I,R     1       10.20.67.20      12386   10.20.67.20      12386    ::      0       ::       0        up

10.1.0.30         mpls             ipsec  10.1.0.3          C,I,R     1       10.20.67.30      12346   10.20.67.30      12346    ::      0       ::       0        up

10.1.0.30         gold             ipsec  10.1.0.3          C,I,R     1       192.0.2.129      12386   192.0.2.129      12386    ::      0       ::       0        up

10.1.0.4         mpls             ipsec  10.1.0.3          C,I,R     1       10.20.67.40      12426   10.20.67.40      12426    ::      0       ::       0        up

10.1.0.4         gold             ipsec  10.1.0.3          C,I,R     1       203.0.113.226    12386   203.0.113.226    12386    ::      0       ::       0        up

When you check the policy, you can see why the TLOC does not appear on the vEdge1.

vSmart1# show running-config policy

policy

lists

tloc-list SITE20

tloc 10.1.0.2 color blue encap ipsec

!

site-list SITE10

site-id 10

!

!

control-policy SDWAN

sequence 10

match tloc

tloc-list SITE20

!

action reject

!

!

default-action accept

!

apply-policy

site-list SITE10

control-policy SDWAN out

!

!

Bidirectional Forwarding Detection

Understand the show bfd sessions command

These are the key things to look for in the output:

vEdge-2# show bfd sessions

SOURCE TLOC      REMOTE TLOC                                      DST PUBLIC                      DST PUBLIC         DETECT      TX

SYSTEM IP        SITE ID  STATE       COLOR            COLOR            SOURCE IP                       IP                              PORT        ENCAP  MULTIPLIER  INTERVAL(msec) UPTIME          TRANSITIONS

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

10.1.0.5         10       down        blue             gold             10.19.146.2                     203.0.113.225                   4501        ipsec  7           1000           NA              7

10.1.0.30         30       up          blue             gold             10.19.146.2                     192.0.2.129                     12386       ipsec  7           1000           0:00:00:22      2

10.1.0.4         40       up          blue             gold             10.19.146.2                     203.0.113.226                   12386       ipsec  7           1000           0:00:00:22      1

10.1.0.4         40       up          mpls             mpls             10.20.67.10                     10.20.67.40                     12426       ipsec  7           1000           0:00:10:11      0

- SYSTEM IP: Peers system-ip

- SOURCE and REMOTE TLOC COLOR: This is useful to know what TLOC is expected to receive and send.

- SOURCE IP: It is the private source IP. If you are behind a NAT, this information is displayed here (it can be seen with the use of show control local-properties <wan-interface-list).

- DST PUBLIC IP: It is the destination that the vEdge uses to form the Data Plane tunnel, whether or not it is behind NAT. (Example: vEdges directly attached to the Internet, or Multi-Protocol Label Switching (MPLS) links)

- DST PUBLIC PORT Public NAT-ed port that the vEdge uses in order to form the Data Plane tunnel to the remote vEdge.

- TRANSITIONS: Number of times the BFD session has changed its status, from NA to UP and vice versa.

Command show tunnel statistics

The show tunnel statistics can display information about the data plane tunnels. You can determine if you send or receive packets for a particular IPSEC tunnel between the vEdges.

This can help you understand if packets arrive on each end, and isolate connectivity issues between the nodes.

In the example, when you run the command multiple times, you can notice an increment or no increment in the tx-pkts or rx-pkts.

Tip: If your counter for tx-pkts increment, you transmit data out to the peer. If your rx-pkts does not increment, it means that data is not received from your peer. In this event, check the other end and confirm if the tx-pkts increments.

TCP

vEdge2# show tunnel statistics

TUNNEL                                   SOURCE  DEST                                                  TUNNEL                                          MSS

PROTOCOL  SOURCE IP      DEST IP         PORT    PORT   SYSTEM IP    LOCAL COLOR      REMOTE COLOR     MTU     tx-pkts  tx-octets  rx-pkts  rx-octets  ADJUST

---------------------------------------------------------------------------------------------------------------------------------------------------------------

ipsec     172.16.16.147  10.88.244.181   12386   12406  10.1.0.5     public-internet  default          1441    38282    5904968    38276    6440071    1361

ipsec     172.16.16.147  10.152.201.104  12386   63364  10.1.0.0  public-internet  default          1441    33421    5158814    33416    5623178    1361

ipsec     172.16.16.147  10.152.204.31   12386   58851  10.1.0.7     public-internet  public-internet  1441    12746    1975022    12744    2151926    1361

ipsec     172.24.90.129  10.88.244.181   12426   12406  10.1.0.5     biz-internet     default          1441    38293    5906238    38288    6454580    1361

ipsec     172.24.90.129  10.152.201.104  12426   63364  10.1.0.0  biz-internet     default          1441    33415    5157914    33404    5621168    1361

ipsec     172.24.90.129  10.152.204.31   12426   58851  10.1.0.7     biz-internet     public-internet  1441    12750    1975622    12747    2152446    1361

TUNNEL                                   SOURCE  DEST                                                  TUNNEL                                          MSS

PROTOCOL  SOURCE IP      DEST IP         PORT    PORT   SYSTEM IP    LOCAL COLOR      REMOTE COLOR     MTU     tx-pkts  tx-octets  rx-pkts  rx-octets  ADJUST

---------------------------------------------------------------------------------------------------------------------------------------------------------------

ipsec     172.16.16.147  10.88.244.181   12386   12406  10.1.0.5     public-internet  default          1441    39028    6020779    39022    6566326    1361

ipsec     172.16.16.147  10.152.201.104  12386   63364  10.1.0.0  public-internet  default          1441    34167    5274625    34162    5749433    1361

ipsec     172.16.16.147  10.152.204.31   12386   58851  10.1.0.7     public-internet  public-internet  1441    13489    2089069    13487    2276382    1361

ipsec     172.24.90.129  10.88.244.181   12426   12406  10.1.0.5     biz-internet     default          1441    39039    6022049    39034    6580835    1361

ipsec     172.24.90.129  10.152.201.104  12426   63364  10.1.0.0  biz-internet     default          1441    34161    5273725    34149    5747259    1361

ipsec     172.24.90.129  10.152.204.31   12426   58851  10.1.0.7     biz-internet     public-internet  1441    13493    2089669    13490    2276902    1361

Another useful command is show tunnel statistics bfd that can be used to check the number of BFD packets sent and received within particular data plane tunnel:

vEdge1# show tunnel statistics bfd

BFD   BFD   BFD     BFD

BFD      BFD                            PMTU  PMTU  PMTU    PMTU

TUNNEL                                  SOURCE  DEST   ECHO TX  ECHO RX  BFD ECHO   BFD ECHO   TX    RX    TX      RX

PROTOCOL  SOURCE IP      DEST IP        PORT    PORT   PKTS     PKTS     TX OCTETS  RX OCTETS  PKTS  PKTS  OCTETS  OCTETS

---------------------------------------------------------------------------------------------------------------------------

ipsec     192.168.109.4  192.168.109.5  4500    4500   0        0        0          0          0     0     0       0

ipsec     192.168.109.4  192.168.109.5  12346   12366  1112255  1112253  186302716  186302381  487   487   395939  397783

ipsec     192.168.109.4  192.168.109.7  12346   12346  1112254  1112252  186302552  186302210  487   487   395939  397783

ipsec     192.168.109.4  192.168.110.5  12346   12366  1112255  1112253  186302716  186302381  487   487   395939  397783

Access List

An access list is a useful and a necessary step after you look at the show bfd sessions output.

Now that the private, and public IPs and Ports are known, you can create an Access Control List (ACL) to match against the SRC_PORT, DST_PORT, SRC_IP, DST_IP.

This can help to verify sent and received BFD messages.

Here, you can find an example of an ACL configuration:

policy

access-list checkbfd-out

sequence 10

match

source-ip        192.168.0.92/32

destination-ip   198.51.100.187/32

source-port      12426

destination-port 12426

!

action accept

count bfd-out-to-dc1-from-br1

!

!

default-action accept

!

access-list checkbfd-in

sequence 20

match

source-ip        198.51.100.187/32

destination-ip   192.168.0.92/32

source-port      12426

destination-port 12426

!

action accept

count bfd-in-from-dc1-to-br1

!

!

default-action accept

!

vpn 0

interface ge0/0

access-list checkbfd-in in

access-list checkbfd-out out

!

!

!

In the example, this ACL uses two sequences. The sequence 10 matches the BFD messages that are sent from this vEdge to the peer. Sequence 20 does the opposite.

It matches against the source (Private) port and destination (Public) ports. If the vEdge uses NAT, ensure to check the right source and destination ports.

To check the hits on each sequence counter issue the show policy access-list counters <access-list name>

vEdge1# show policy access-list-counters

NAME      COUNTER NAME               PACKETS  BYTES

-----------------------------------------------------

checkbfd  bfd-out-to-dc1-from-br1    10       2048

bfd-in-from-dc1-to-br1        0        0

Network Address Translation

How to Use tools stun-client to Detect NAT Maps and Filters.

If you have done all the steps and you are behind NAT, the next step is to identify the UDP NAT Traversal (RFC 4787) Map and Filter behavior.

This tool is used to discover the local vEdge external IP address when that vEdge is located behind a NAT device.

This command obtains a port mapping for the device and optionally discovers properties about the NAT between the local device and a server (public server: example google stun server).

Note: For more detailed information visit: Docs Viptela - STUN Client

vEdge1# tools stun-client vpn 0 options "--mode full --localaddr 192.168.12.100 12386 --verbosity 2 stun.l.google.com 19302"

stunclient --mode full --localaddr 192.168.12.100 stun.l.google.com in VPN 0

Binding test: success

Local address: 192.168.12.100:12386

Mapped address: 203.0.113.225:4501

Behavior test: success

Nat behavior: Address Dependent Mapping

Filtering test: success

Nat filtering: Address and Port Dependent Filtering

On newer versions of software, syntax can be bit different:

vEdge1# tools stun-client vpn 0 options "--mode full --localaddr 192.168.12.100 --localport 12386 --verbosity 2 stun.l.google.com 19302"

In this example, you perform a full NAT detection test with the use of UDP source port 12386 to the Google STUN server.

The output of this command gives you NAT behavior and the NAT filter type based on RFC 4787.

Note: When you use tools stun, remember to allow the STUN service in the tunnel interface, otherwise it does not work. Use allow-service stun in order to let the stun data pass.

vEdge1# show running-config vpn 0 interface ge0/0

vpn 0

interface ge0/0

ip address 10.19.145.2/30

!

tunnel-interface

encapsulation ipsec

color gold

max-control-connections 1

no allow-service bgp

allow-service dhcp

allow-service dns

no allow-service icmp

no allow-service sshd

no allow-service netconf

no allow-service ntp

no allow-service ospf

allow-service stun

!

no shutdown

!

!

This shows the mapping between STUN terminology (Full-Cone NAT) and RFC 4787 (NAT Behavioral for UDP).

Supported NAT Types for Data Plane Tunnels“Sending” used in CLI

In most of the cases, your public colors like biz-internet or public-internet can be directly attached to the internet.

In other cases, there is a NAT device behind the vEdge WAN interface and the actual Internet Service Provider.

In this manner, the vEdge can have a private IP and the other device (Router, Firewall, etc) can be the device with the public facing IP addresses.

If you have an incorrect NAT type, then it could potentially be one of the most common reasons that do not allow the formation of Data Plane tunnels. These are the supported NAT types.

Firewalls

If you already checked the NAT and its not in the unsupported Source and Destination types, it is possible that a Firewall blocks the ports used to form the Data Plane tunnels.

Ensure that these ports are open in the Firewall for Data Plane connections: vEdge to vEdge Data Plane:

UDP 12346 to 13156

For control connections from vEdge to controllers:

UDP 12346 to 13156

TCP 23456 to 24156

Ensure that you open these ports in order to achieve successful connection of the Data Plane tunnels.

When you check the source and destination ports used for Data Plane tunnels, you can use show tunnel statistics or show bfd sessions | tab but not show bfd sessions.

It does not show any source ports, only destination ports as you can see:

vEdge1# show bfd sessions

SOURCE TLOC      REMOTE TLOC                                      DST PUBLIC                      DST PUBLIC         DETECT      TX

SYSTEM IP        SITE ID  STATE       COLOR            COLOR            SOURCE IP                       IP                              PORT        ENCAP  MULTIPLIER  INTERVAL(msec) UPTIME          TRANSITIONS

-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

192.168.30.105   50       up          biz-internet     biz-internet     192.168.109.181                 192.168.109.182                 12346       ipsec  7           1000           1:21:28:05      10

192.168.30.105   50       up          private1         private1         192.168.110.181                 192.168.110.182                 12346       ipsec  7           1000           1:21:26:13      2

vEdge1# show bfd sessions | tab

SRC    DST                    SITE                                     DETECT      TX

SRC IP           DST IP           PROTO  PORT   PORT   SYSTEM IP       ID    LOCAL COLOR   COLOR         STATE  MULTIPLIER  INTERVAL  UPTIME      TRANSITIONS

---------------------------------------------------------------------------------------------------------------------------------------------------------------

192.168.109.181  192.168.109.182  ipsec  12346  12346  192.168.30.105  50    biz-internet  biz-internet  up     7           1000      1:21:28:05  10

192.168.110.181  192.168.110.182  ipsec  12346  12346  192.168.30.105  50    private1      private1      up     7           1000      1:21:26:13  2

Note: More information about SD-WAN firewall ports used can be found here.

Security

If you observe that the ACL counter increases inbound and outbound, check several iterations show system statistics diff and ensure there are no drops.

vEdge1# show policy access-list-counters

NAME      COUNTER NAME               PACKETS  BYTES

-----------------------------------------------------

checkbfd  bfd-out-to-dc1-from-br1    55       9405

bfd-in-from-dc1-to-br1     54      8478

In this output, rx_replay_integrity_drops increase with every iteration of the show system statistics diff command.

vEdge1#show system statistics diff

rx_pkts : 5741427

ip_fwd : 5952166

ip_fwd_arp : 3

ip_fwd_to_egress : 2965437

ip_fwd_null_mcast_group : 26

ip_fwd_null_nhop : 86846

ip_fwd_to_cpu : 1413393

ip_fwd_from_cpu_non_local : 15

ip_fwd_rx_ipsec : 1586149

ip_fwd_mcast_pkts : 26

rx_bcast : 23957

rx_mcast : 304

rx_mcast_link_local : 240

rx_implicit_acl_drops : 12832

rx_ipsec_decap : 21

rx_spi_ipsec_drops : 16

rx_replay_integrity_drops : 1586035

port_disabled_rx : 2

rx_invalid_qtags : 212700

rx_non_ip_drops : 1038073

pko_wred_drops : 3

bfd_tx_record_changed : 23

rx_arp_non_local_drops : 19893

rx_arp_reqs : 294

rx_arp_replies : 34330

arp_add_fail : 263

tx_pkts : 4565384

tx_mcast : 34406

port_disabled_tx : 3

tx_ipsec_pkts : 1553753

tx_ipsec_encap : 1553753

tx_pre_ipsec_pkts : 1553753

tx_pre_ipsec_encap : 1553753

tx_arp_replies : 377

tx_arp_reqs : 34337

tx_arp_req_fail : 2

bfd_tx_pkts : 1553675

bfd_rx_pkts : 21

bfd_tx_octets : 264373160

bfd_rx_octets : 3600

bfd_pmtu_tx_pkts : 78

bfd_pmtu_tx_octets : 53052

rx_icmp_echo_requests : 48

rx_icmp_network_unreach : 75465

rx_icmp_other_types : 47

tx_icmp_echo_requests : 49655

tx_icmp_echo_replies : 48

tx_icmp_network_unreach : 86849

tx_icmp_other_types : 7

vEdge1# show system statistics diff

rx_pkts : 151

ip_fwd : 157

ip_fwd_to_egress : 75

ip_fwd_null_nhop : 3

ip_fwd_to_cpu : 43

ip_fwd_rx_ipsec : 41

rx_bcast : 1

rx_replay_integrity_drops : 41

rx_invalid_qtags : 7

rx_non_ip_drops : 21

rx_arp_non_local_drops : 2

tx_pkts : 114

tx_ipsec_pkts : 40

tx_ipsec_encap : 40

tx_pre_ipsec_pkts : 40

tx_pre_ipsec_encap : 40

tx_arp_reqs : 1

bfd_tx_pkts : 40

bfd_tx_octets : 6800

tx_icmp_echo_requests : 1

vEdge1# show system statistics diff

rx_pkts : 126

ip_fwd : 125

ip_fwd_to_egress : 58

ip_fwd_null_nhop : 3

ip_fwd_to_cpu : 33

ip_fwd_rx_ipsec : 36

rx_bcast : 1

rx_implicit_acl_drops : 1

rx_replay_integrity_drops : 35

rx_invalid_qtags : 6

rx_non_ip_drops : 22

rx_arp_replies : 1

tx_pkts : 97

tx_mcast : 1

tx_ipsec_pkts : 31

tx_ipsec_encap : 31

tx_pre_ipsec_pkts : 31

tx_pre_ipsec_encap : 31

bfd_tx_pkts : 32

bfd_tx_octets : 5442

rx_icmp_network_unreach : 3

tx_icmp_echo_requests : 1

tx_icmp_network_unreach : 3

vEdge1# show system statistics diff

rx_pkts : 82

ip_fwd : 89

ip_fwd_to_egress : 45

ip_fwd_null_nhop : 3

ip_fwd_to_cpu : 24

ip_fwd_rx_ipsec : 22

rx_bcast : 1

rx_implicit_acl_drops : 1

rx_replay_integrity_drops : 24

rx_invalid_qtags : 2

rx_non_ip_drops : 14

rx_arp_replies : 1

tx_pkts : 62

tx_mcast : 1

tx_ipsec_pkts : 24

tx_ipsec_encap : 24

tx_pre_ipsec_pkts : 24

tx_pre_ipsec_encap : 24

tx_arp_reqs : 1

bfd_tx_pkts : 23

bfd_tx_octets : 3908

rx_icmp_network_unreach : 3

tx_icmp_echo_requests : 1

tx_icmp_network_unreach : 3

vEdge1# show system statistics diff

rx_pkts : 80

ip_fwd : 84

ip_fwd_to_egress : 39

ip_fwd_to_cpu : 20

ip_fwd_rx_ipsec : 24

rx_replay_integrity_drops : 22

rx_invalid_qtags : 3

rx_non_ip_drops : 12

tx_pkts : 66

tx_ipsec_pkts : 21

tx_ipsec_encap : 21

tx_pre_ipsec_pkts : 21

tx_pre_ipsec_encap : 21

bfd_tx_pkts : 21

bfd_tx_octets : 3571

First, perform a request security ipsec-rekey on the vEdge. Then, go through several iterations of show system statistics diff and see if you still see rx_replay_integrity_drops.

If you do, check your security configuration.

vEdge1# show running-config security

security

ipsec

authentication-type sha1-hmac ah-sha1-hmac

!

!

ISP Problems with DSCP Marked Traffic

By default, all control and management traffic from the vEdge router to the controllers travels over DTLS or TLS connections and marked with a DSCP value of CS6 (48 decimal).

For data place tunnels traffic, vEdge routers use IPsec or GRE encapsulation to send data traffic to each other.

For data plane failure detection and performance measurement, routers periodically send each other BFD packets.

These BFD packets are also marked with a DSCP value of CS6 (48 decimal).

From the perspective of ISP, this type of traffic is seen as UDP traffic with DSCP value CS6 as well because vEdge routers and SD-WAN controllers copy DSCP that marks to the outer IP header by default.

Here is how it can look like if tcpdump runs on transit ISP router:

14:27:15.993766 IP (tos 0xc0, ttl 64, id 44063, offset 0, flags [DF], proto UDP (17), length 168)

192.168.109.5.12366 > 192.168.20.2.12346: [udp sum ok] UDP, length 140

14:27:16.014900 IP (tos 0xc0, ttl 63, id 587, offset 0, flags [DF], proto UDP (17), length 139)

192.168.20.2.12346 > 192.168.109.5.12366: [udp sum ok] UDP, length 111

14:27:16.534117 IP (tos 0xc0, ttl 63, id 0, offset 0, flags [DF], proto UDP (17), length 157)

192.168.109.5.12366 > 192.168.110.6.12346: [no cksum] UDP, length 129

14:27:16.534289 IP (tos 0xc0, ttl 62, id 0, offset 0, flags [DF], proto UDP (17), length 150)

192.168.110.6.12346 > 192.168.109.5.12366: [no cksum] UDP, length 122

As can be seen here, all packets are marked with TOS byte 0xc0 also known as DS field (that is equal to decimal 192, or 110 000 00 in binary.

First 6 high order bits correspond to DSCP bits value 48 in decimal or CS6).

First 2 packets in the output correspond to a control plane tunnel and the 2 that remain, to a data plane tunnel traffic.

Based on the packet length and the TOS mark, it can conclude with high confidence that it was BFD packets (RX and TX directions). These packets are marked with CS6 as well.

Sometimes some service providers (especially MPLS L3 VPN/MPLS L2 VPN service providers) maintain different SLA and can handle a different class of traffic based on DSCP marks differently.

For example, if you have premium service to prioritize DSCP EF and CS6 voice and signaling traffic.

Since priority traffic is almost always policed, even if the total bandwidth of an uplink is not exceeded, for this type of traffic packet loss can be seen and hence BFD sessions can be flapping as well.

It was seen in some cases that if dedicated priority queue on service provider router is starved, you do not see any drops for normal traffic (for example, when you run simple ping from vEdge router).

This is because such traffic is marked with default DSCP value 0 as can be seen here (TOS byte):

15:49:22.268044 IP (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto UDP (17), length 142)

192.168.110.5.12366 > 192.168.109.7.12346: [no cksum] UDP, length 114

15:49:22.272919 IP (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto UDP (17), length 142)

192.168.110.5.12366 > 192.168.109.7.12346: [no cksum] UDP, length 114

15:49:22.277660 IP (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto UDP (17), length 142)

192.168.110.5.12366 > 192.168.109.7.12346: [no cksum] UDP, length 114

15:49:22.314821 IP (tos 0x0, ttl 62, id 0, offset 0, flags [DF], proto UDP (17), length 142)

192.168.110.5.12366 > 192.168.109.7.12346: [no cksum] UDP, length 114

But at the same time, your BFD sessions flap:

show bfd history

DST PUBLIC       DST PUBLIC                                  RX      TX

SYSTEM IP        SITE ID  COLOR            STATE       IP               PORT        ENCAP  TIME                     PKTS    PKTS    DEL

---------------------------------------------------------------------------------------------------------------------------------------

192.168.30.4     13       public-internet  up          192.168.109.4    12346       ipsec  2019-05-01T03:54:23+0200 127     135     0

192.168.30.4     13       public-internet  up          192.168.109.4    12346       ipsec  2019-05-01T03:54:23+0200 127     135     0

192.168.30.4     13       public-internet  down        192.168.109.4    12346       ipsec  2019-05-01T03:55:28+0200 140     159     0

192.168.30.4     13       public-internet  down        192.168.109.4    12346       ipsec  2019-05-01T03:55:28+0200 140     159     0

192.168.30.4     13       public-internet  up          192.168.109.4    12346       ipsec  2019-05-01T03:55:40+0200 361     388     0

192.168.30.4     13       public-internet  up          192.168.109.4    12346       ipsec  2019-05-01T03:55:40+0200 361     388     0

192.168.30.4     13       public-internet  down        192.168.109.4    12346       ipsec  2019-05-01T03:57:38+0200 368     421     0

192.168.30.4     13       public-internet  down        192.168.109.4    12346       ipsec  2019-05-01T03:57:38+0200 368     421     0

192.168.30.4     13       public-internet  up          192.168.109.4    12346       ipsec  2019-05-01T03:58:05+0200 415     470     0

192.168.30.6     13       public-internet  up          192.168.109.4    12346       ipsec  2019-05-01T03:58:05+0200 415     470     0

192.168.30.6     13       public-internet  down        192.168.109.4    12346       ipsec  2019-05-01T03:58:25+0200 464063  464412  0

And here nping comes handy in order to troubleshoot:

vedge2# tools nping vpn 0 options "--tos 0x0c --icmp --icmp-type echo --delay 200ms -c 100 -q" 192.168.109.7

Nping in VPN 0

Starting Nping 0.6.47 ( http://nmap.org/nping ) at 2019-05-07 15:58 CEST

Max rtt: 200.305ms | Min rtt: 0.024ms | Avg rtt: 151.524ms

Raw packets sent: 100 (2.800KB) | Rcvd: 99 (4.554KB) | Lost: 1 (1.00%)

Nping done: 1 IP address pinged in 19.83 seconds

Debug BFD

If deeper investigation is required, run debugging of BFD on the vEdge router.

Forwarding Traffic Manager (FTM) is responsible for BFD operations on vEdge routers and hence you need debug ftm bfd.

All debugging output is stored in /var/log/tmplog/vdebug file and if you want to have those messages on the console (similar to Cisco IOS terminal monitor behavior), you can use monitor start /var/log/tmplog/vdebug.

In order to stop logging, you can use monitor stop /var/log/tmplog/vdebug

Here is how the output looks for BFD session that goes down because of the timeout (remote TLOC with IP address 192.168.110.6 is not reachable anymore):

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_update_state[1008]: BFD-session TNL 192.168.110.5:12366->192.168.110.6:12346,l-tloc(32771)->r-tloc(32772),TLOC 192.168.30.5:biz-internet->192.168.30.6:public-internet IPSEC: BFD Session STATE update, New_State :- DOWN, Reason :- LOCAL_TIMEOUT_DETECT Observed latency :- 7924, bfd_record_index :- 8, Hello timer :- 1000, Detect Multiplier :- 7

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: ftm_proc_tunnel_public_tloc_msg[252]: tun_rec_index 13 tloc_index 32772 public tloc 0.0.0.0/0

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: ftm_increment_wanif_bfd_flap[2427]: BFD-session TNL 192.168.110.5:12366->192.168.110.6:12346, : Increment the WAN interface counters by 1

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_update_state[1119]: BFD-session TNL 192.168.110.5:12366->192.168.110.6:12346,l-tloc(32771)->r-tloc(32772),TLOC 192.168.30.5:biz-internet->192.168.30.6:public-internet IPSEC BFD session history update, old state 3 new state 1 current  flap count 1 prev_index 1 current 2

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: ftm_tloc_add[1140]: Attempting to add TLOC : from_ttm 0 origin remote tloc-index 32772 pub 192.168.110.6:12346 pub v6 :::0 system_ip 192.168.30.6 color 5 spi 333

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_set_del_marker_internal[852]: (32771:32772) proto 50 src 192.168.110.5:12366 dst 192.168.110.6:12346 ref_count 1

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_set_del_marker_internal[852]: (32770:32772) proto 50 src 192.168.109.5:12366 dst 192.168.110.6:12346 ref_count 1

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_create[238]: Attempting BFD session creation. Remote-tloc: tloc-index 32772, system-ip 192.168.30.6, color 5 encap 2from local WAN Interface ge0_0

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_clear_delete_marker[828]: (32771:32772) proto 50 src 192.168.110.5:12366 dst 192.168.110.6:12346 ref_count 1

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_create[238]: Attempting BFD session creation. Remote-tloc: tloc-index 32772, system-ip 192.168.30.6, color 5 encap 2from local WAN Interface ge0_1

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_clear_delete_marker[828]: (32770:32772) proto 50 src 192.168.109.5:12366 dst 192.168.110.6:12346 ref_count 1

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_update_sa[1207]: BFD-session TNL 192.168.110.5:12366->192.168.110.6:12346,l-tloc(32771)->r-tloc(32772),TLOC 192.168.30.5:biz-internet->192.168.30.6:public-internet IPSEC: session sa index changed from 484 to 484

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: ftm_tloc_add[1653]: BFD (32771:32772) src 192.168.110.5:12366 dst 192.168.110.6:12346 record index 8 ref-count 1 sa-idx 484

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_update_sa[1207]: BFD-session TNL 192.168.109.5:12366->192.168.110.6:12346,l-tloc(32770)->r-tloc(32772),TLOC 192.168.30.5:public-internet->192.168.30.6:public-internet IPSEC: session sa index changed from 485 to 485

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: ftm_tloc_add[1653]: BFD (32770:32772) src 192.168.109.5:12366 dst 192.168.110.6:12346 record index 9 ref-count 1 sa-idx 485

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_update_state[1008]: BFD-session TNL 192.168.109.5:12366->192.168.110.6:12346,l-tloc(32770)->r-tloc(32772),TLOC 192.168.30.5:public-internet->192.168.30.6:public-internet IPSEC: BFD Session STATE update, New_State :- DOWN, Reason :- LOCAL_TIMEOUT_DETECT Observed latency :- 7924, bfd_record_index :- 9, Hello timer :- 1000, Detect Multiplier :- 7

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: ftm_proc_tunnel_public_tloc_msg[252]: tun_rec_index 14 tloc_index 32772 public tloc 0.0.0.0/0

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: ftm_increment_wanif_bfd_flap[2427]: BFD-session TNL 192.168.109.5:12366->192.168.110.6:12346, : Increment the WAN interface counters by 1

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_update_state[1119]: BFD-session TNL 192.168.109.5:12366->192.168.110.6:12346,l-tloc(32770)->r-tloc(32772),TLOC 192.168.30.5:public-internet->192.168.30.6:public-internet IPSEC BFD session history update, old state 3 new state 1 current  flap count 1 prev_index 1 current 2

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: ftm_tloc_add[1140]: Attempting to add TLOC : from_ttm 0 origin remote tloc-index 32772 pub 192.168.110.6:12346 pub v6 :::0 system_ip 192.168.30.6 color 5 spi 333

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_set_del_marker_internal[852]: (32771:32772) proto 50 src 192.168.110.5:12366 dst 192.168.110.6:12346 ref_count 1

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_set_del_marker_internal[852]: (32770:32772) proto 50 src 192.168.109.5:12366 dst 192.168.110.6:12346 ref_count 1

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_create[238]: Attempting BFD session creation. Remote-tloc: tloc-index 32772, system-ip 192.168.30.6, color 5 encap 2from local WAN Interface ge0_0

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_clear_delete_marker[828]: (32771:32772) proto 50 src 192.168.110.5:12366 dst 192.168.110.6:12346 ref_count 1

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_create[238]: Attempting BFD session creation. Remote-tloc: tloc-index 32772, system-ip 192.168.30.6, color 5 encap 2from local WAN Interface ge0_1

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_clear_delete_marker[828]: (32770:32772) proto 50 src 192.168.109.5:12366 dst 192.168.110.6:12346 ref_count 1

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_update_sa[1207]: BFD-session TNL 192.168.110.5:12366->192.168.110.6:12346,l-tloc(32771)->r-tloc(32772),TLOC 192.168.30.5:biz-internet->192.168.30.6:public-internet IPSEC: session sa index changed from 484 to 484

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: ftm_tloc_add[1653]: BFD (32771:32772) src 192.168.110.5:12366 dst 192.168.110.6:12346 record index 8 ref-count 1 sa-idx 484

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_update_sa[1207]: BFD-session TNL 192.168.109.5:12366->192.168.110.6:12346,l-tloc(32770)->r-tloc(32772),TLOC 192.168.30.5:public-internet->192.168.30.6:public-internet IPSEC: session sa index changed from 485 to 485

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: ftm_tloc_add[1653]: BFD (32770:32772) src 192.168.109.5:12366 dst 192.168.110.6:12346 record index 9 ref-count 1 sa-idx 485

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: ftm_send_bfd_msg[499]: Sending BFD notification Down notification to TLOC id 32772

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: ftm_tloc_add[1140]: Attempting to add TLOC : from_ttm 1 origin remote tloc-index 32772 pub 192.168.110.6:12346 pub v6 :::0 system_ip 192.168.30.6 color 5 spi 333

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_set_del_marker_internal[852]: (32771:32772) proto 50 src 192.168.110.5:12366 dst 192.168.110.6:12346 ref_count 1

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_set_del_marker_internal[852]: (32770:32772) proto 50 src 192.168.109.5:12366 dst 192.168.110.6:12346 ref_count 1

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: ftm_tloc_add[1285]: UPDATE local tloc

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_create[238]: Attempting BFD session creation. Remote-tloc: tloc-index 32772, system-ip 192.168.30.6, color 5 encap 2from local WAN Interface ge0_0

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_clear_delete_marker[828]: (32771:32772) proto 50 src 192.168.110.5:12366 dst 192.168.110.6:12346 ref_count 1

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_create[238]: Attempting BFD session creation. Remote-tloc: tloc-index 32772, system-ip 192.168.30.6, color 5 encap 2from local WAN Interface ge0_1

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_clear_delete_marker[828]: (32770:32772) proto 50 src 192.168.109.5:12366 dst 192.168.110.6:12346 ref_count 1

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_update_sa[1207]: BFD-session TNL 192.168.110.5:12366->192.168.110.6:12346,l-tloc(32771)->r-tloc(32772),TLOC 192.168.30.5:biz-internet->192.168.30.6:public-internet IPSEC: session sa index changed from 484 to 484

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: ftm_tloc_add[1653]: BFD (32771:32772) src 192.168.110.5:12366 dst 192.168.110.6:12346 record index 8 ref-count 1 sa-idx 484

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: bfdmgr_session_update_sa[1207]: BFD-session TNL 192.168.109.5:12366->192.168.110.6:12346,l-tloc(32770)->r-tloc(32772),TLOC 192.168.30.5:public-internet->192.168.30.6:public-internet IPSEC: session sa index changed from 485 to 485

log:local7.debug: May  7 16:23:09 vedge2 FTMD[674]: ftm_tloc_add[1653]: BFD (32770:32772) src 192.168.109.5:12366 dst 192.168.110.6:12346 record index 9 ref-count 1 sa-idx 485

log:local7.info: May  7 16:23:09 vedge2 FTMD[674]: %Viptela-vedge2-ftmd-6-INFO-1400002: Notification: 5/7/2019 14:23:9 bfd-state-change severity-level:major host-name:"vedge2" system-ip:192.168.30.5 src-ip:192.168.110.5 dst-ip:192.168.110.6 proto:ipsec src-port:12366 dst-port:12346 local-system-ip:192.168.30.5 local-color:"biz-internet" remote-system-ip:192.168.30.6 remote-color:"public-internet" new-state:down deleted:false flap-reason:timeout

log:local7.info: May  7 16:23:09 vedge2 FTMD[674]: %Viptela-vedge2-ftmd-6-INFO-1400002: Notification: 5/7/2019 14:23:9 bfd-state-change severity-level:major host-name:"vedge2" system-ip:192.168.30.5 src-ip:192.168.109.5 dst-ip:192.168.110.6 proto:ipsec src-port:12366 dst-port:12346 local-system-ip:192.168.30.5 local-color:"public-internet" remote-system-ip:192.168.30.6 remote-color:"public-internet" new-state:down deleted:false flap-reason:timeout

Another valuable debug in order to enable is Tunnel Traffic Manager (TTM) events debug is debug ttm events.

Here is how BFD DOWN event looks like from the perspective of TTM:

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[194]: Received TTM Msg LINK_BFD, Client: ftmd, AF: LINK

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[413]:     Remote-TLOC: 192.168.30.6 : public-internet : ipsec, Local-TLOC: 192.168.30.5 : biz-internet : ipsec, Status: DOWN, Rec Idx: 13 MTU: 1441, Loss: 77, Latency: 0, Jitter: 0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[194]: Received TTM Msg LINK_BFD, Client: ftmd, AF: LINK

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[413]:     Remote-TLOC: 192.168.30.6 : public-internet : ipsec, Local-TLOC: 192.168.30.5 : public-internet : ipsec, Status: DOWN, Rec Idx: 14 MTU: 1441, Loss: 77, Latency: 0, Jitter: 0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[194]: Received TTM Msg BFD, Client: ftmd, AF: TLOC-IPV4

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[402]:     TLOC: 192.168.30.6 : public-internet : ipsec, Status: DOWN

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_af_tloc_db_bfd_status[234]:   BFD message: I SAY WHAT WHAT tloc 192.168.30.6 : public-internet : ipsec status is 0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[194]: Sent TTM Msg TLOC_ADD, Client: ompd, AF: TLOC-IPV4

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[213]:     TLOC: 192.168.30.6 : public-internet : ipsec, Index: 32772, Origin: REMOTE, Status: DOWN, LR enabled: 0, LR hold time: 0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[217]:       Attributes: GROUP PREF WEIGHT GEN-ID VERSION TLOCv4-PUB TLOCv4-PRI TLOCv6-PUB TLOCv6-PRI SITE-ID CARRIER ENCAP RESTRICT

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[220]:         Preference: 0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[223]:         Weight: 1

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[226]:         Gen-ID: 2147483661

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[229]:         Version: 2

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[232]:         Site-ID: 13

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[235]:         Carrier: 4

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[241]:         Restrict: 0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[249]:         Group: Count: 1

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[262]:         Groups: 0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[269]:         TLOCv4-Public: 192.168.110.6:12346

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[273]:         TLOCv4-Private: 192.168.110.6:12346

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[277]:         TLOCv6-Public: :::0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[281]:         TLOCv6-Private: :::0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[285]:         TLOC-Encap: ipsec-tunnel

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[295]:           Authentication: unknown(0x98) Encryption: aes256(0xc) SPI 334 Proto ESP

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[312]:           SPI 334, Flags 0x1e           Integrity: 1, encrypt-keys: 1 auth-keys: 1

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[317]:           Number of protocols 0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[328]:           Number of encrypt types: 2

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[333]:           Encrypt type[0] AES256-GCM

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[333]:           Encrypt type[1] AES256-CBC

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[339]:           Number of integrity types: 1

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[344]:           integrity type[0] HMAC_SHA1

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[349]:       #Paths: 0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[194]: Sent TTM Msg TLOC_ADD, Client: ftmd, AF: TLOC-IPV4

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[213]:     TLOC: 192.168.30.6 : public-internet : ipsec, Index: 32772, Origin: REMOTE, Status: DOWN, LR enabled: 0, LR hold time: 0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[217]:       Attributes: GROUP PREF WEIGHT GEN-ID VERSION TLOCv4-PUB TLOCv4-PRI TLOCv6-PUB TLOCv6-PRI SITE-ID CARRIER ENCAP RESTRICT

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[220]:         Preference: 0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[223]:         Weight: 1

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[226]:         Gen-ID: 2147483661

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[229]:         Version: 2

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[232]:         Site-ID: 13

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[235]:         Carrier: 4

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[241]:         Restrict: 0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[249]:         Group: Count: 1

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[262]:         Groups: 0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[269]:         TLOCv4-Public: 192.168.110.6:12346

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[273]:         TLOCv4-Private: 192.168.110.6:12346

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[277]:         TLOCv6-Public: :::0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[281]:         TLOCv6-Private: :::0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[285]:         TLOC-Encap: ipsec-tunnel

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[295]:           Authentication: unknown(0x98) Encryption: aes256(0xc) SPI 334 Proto ESP

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[312]:           SPI 334, Flags 0x1e           Integrity: 1, encrypt-keys: 1 auth-keys: 1

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[317]:           Number of protocols 0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[328]:           Number of encrypt types: 2

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[333]:           Encrypt type[0] AES256-GCM

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[333]:           Encrypt type[1] AES256-CBC

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[339]:           Number of integrity types: 1

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[344]:           integrity type[0] HMAC_SHA1

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[349]:       #Paths: 0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[194]: Sent TTM Msg TLOC_ADD, Client: fpmd, AF: TLOC-IPV4

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[213]:     TLOC: 192.168.30.6 : public-internet : ipsec, Index: 32772, Origin: REMOTE, Status: DOWN, LR enabled: 0, LR hold time: 0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[217]:       Attributes: GROUP PREF WEIGHT GEN-ID VERSION TLOCv4-PUB TLOCv4-PRI TLOCv6-PUB TLOCv6-PRI SITE-ID CARRIER ENCAP RESTRICT

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[220]:         Preference: 0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[223]:         Weight: 1

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[226]:         Gen-ID: 2147483661

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[229]:         Version: 2

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[232]:         Site-ID: 13

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[235]:         Carrier: 4

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[241]:         Restrict: 0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[249]:         Group: Count: 1

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[262]:         Groups: 0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[269]:         TLOCv4-Public: 192.168.110.6:12346

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[273]:         TLOCv4-Private: 192.168.110.6:12346

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[277]:         TLOCv6-Public: :::0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[281]:         TLOCv6-Private: :::0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[285]:         TLOC-Encap: ipsec-tunnel

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[295]:           Authentication: unknown(0x98) Encryption: aes256(0xc) SPI 334 Proto ESP

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[312]:           SPI 334, Flags 0x1e           Integrity: 1, encrypt-keys: 1 auth-keys: 1

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[317]:           Number of protocols 0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[328]:           Number of encrypt types: 2

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[333]:           Encrypt type[0] AES256-GCM

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[333]:           Encrypt type[1] AES256-CBC

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[339]:           Number of integrity types: 1

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[344]:           integrity type[0] HMAC_SHA1

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[349]:       #Paths: 0

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[194]: Sent TTM Msg DATA_DEVICE_ADD, Client: pimd, AF: DATA-DEVICE-IPV4

log:local7.debug: May  7 16:58:19 vedge2 TTMD[683]: ttm_debug_announcement[431]:     Device: 192.168.30.6, Status: 2

log:local7.info: May  7 16:58:19 vedge2 FTMD[674]: %Viptela-vedge2-ftmd-6-INFO-1400002: Notification: 5/7/2019 14:58:19 bfd-state-change severity-level:major host-name:"vedge2" system-ip:192.168.30.5 src-ip:192.168.110.5 dst-ip:192.168.110.6 proto:ipsec src-port:12366 dst-port:12346 local-system-ip:192.168.30.5 local-color:"biz-internet" remote-system-ip:192.168.30.6 remote-color:"public-internet" new-state:down deleted:false flap-reason:timeout

log:local7.info: May  7 16:58:20 vedge2 FTMD[674]: %Viptela-vedge2-ftmd-6-INFO-1400002: Notification: 5/7/2019 14:58:19 bfd-state-change severity-level:major host-name:"vedge2" system-ip:192.168.30.5 src-ip:192.168.109.5 dst-ip:192.168.110.6 proto:ipsec src-port:12366 dst-port:12346 local-system-ip:192.168.30.5 local-color:"public-internet" remote-system-ip:192.168.30.6 remote-color:"public-internet" new-state:down deleted:false flap-reason:timeout

Related Information

- Technical Support & Documentation - Cisco Systems