Configure IP SLA Feature with L3out to Track Static Route

Introduction

This document describes how to configure the Internet Protocol Service Level Agreement (IPSLA) in Cisco Application Centric Infrastructure (ACI) to track static route learn from one L3out and advertise to another L3out only if the subnet is reachable from the first L3out.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• ACI software version 4.1 and later
• L3out toward External device or Server
• EX and -FX chassis
• Track the route to use Internet Control Message Protocol (ICMP) and TCP probes (in this example ICMP probe is used)

Note: ACI image IP SLA is supported in all Cisco Nexus second-generation switches, which includes -EX and -FX chassis. Please read Guidelines and Limitations for IP SLA.

Components Used

The information in this document is based on these software and hardware versions:

• ACI version 5.2(2f)
• N9K-C93180YC-FX

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

Some servers have multiple interfaces (like a  loopback) which is reachable from ACI via the physical IP address of the server. In such a case you can have a requirement to add a static route and advertise externally but only if the physical IP of the server is reachable. Hence IP SLA track feature is an unavoidable configuration that can only achieve through L3out configuration toward those servers. At this moment IP SLA track features are not supported for the static route on a Bridge Domain. In this document, we will look for server examples and transit route configurations that use IP SLA.

Configure

• L3out toward Server and toward N3K devices.
• Configure IP SLA track for the physical IP address of the server.
• Configure static route under L3out toward server which use IP SLA track and advertise from another L3out toward N3K.

Network Diagram

ACI Lab Topology

Configurations

Summary Steps:

ACI fabric policies:

• Create Contract (for this example, a common default filter that allows all traffic to be used, but you can use a specific filter locally created in the same Tenant to allow specific traffic. in such a case please ensure you allow protocol that we are used for IP SLA track).
• Create new L3out toward server 10.100.0.100/24 (ACI side SVI 550 with IP address 10.100.0.254)
• Create IP SLA Track Policies (IP SLA monitor policy, Track Members policy, Track List policy)
• Add static route under L3out toward server with IP SLA tracklist.
• Create a new L3out toward the N3K device which use BGP. (EBGP) ACI AS 65535 and N3K AS 65536
• Export static route from L3out toward N3K.
• Verify configuration and reachability.

1. Create Contract (for this example, use a common default filter that allows all traffic, however, you can use a specific filter locally created in the same tenant to allow specific traffic but in such case please ensure you allow protocol which we are used for IP SLA track).

Create Contract

2. Create a new L3out toward server 10.100.0.100/24 (ACI side SVI 550 with IP address 10.100.0.254).

Create L3out

Attaching node to L3out

Attaching interface to L3out

Configure External EPG

Attaching Contract to L3out

3. Create IP SLA Track Policies (IP SLA monitor policy, Track Members policy, Track List policy).

IP SLA Monitor Policy:

Configure IP SLA monitor policy

IP SLA Track Members:

Adding IP to monitor policy

Track List Policy:

Configure Track List

4. Configure static route under L3out toward server with newly created IP SLA tracklist policy.

Configure Static route under L3out

5. Create a new L3out toward the N3K device which uses Border Gateway Protocol (BGP). (EBGP) ACI AS 65535 and N3K AS 65536.

Configure BGP protocol

BGP Peer Profile

Configure BGP peer policy

Configure Logical interface profile under L3out

External EPG export subnet in transit L3out

Attaching Contract to External EPG

6. Export static route from L3out toward N3K.

switchname N3K
feature bgp
feature interface-vlan
interface Vlan550
  no shutdown
  vrf member BGP_L3out
  ip address 100.0.0.2/30
interface loopback200
  vrf member BGP_L3out
  ip address 30.30.30.1/32
interface Ethernet1/1
  switchport mode trunk
router bgp 65536
  address-family ipv4 unicast
  neighbor 100.0.0.1
  vrf BGP_L3out
    router-id 3.3.3.3
    address-family ipv4 unicast
      network 30.30.30.1/32
    neighbor 100.0.0.1
      remote-as 65535
      update-source Vlan550
      address-family ipv4 unicast

Verify

Use this section in order to confirm that your configuration works properly.

Nexus3K.

Transit route advertisement explained by topology

N3K# routing vrf BGP_L3out
N3K%BGP_L3out# show ip route
IP Route Table for VRF "BGP_L3out"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%' in via output denotes VRF 
30.30.30.1/32, ubest/mbest: 2/0, attached
    *via 30.30.30.1, Lo200, [0/0], 02:35:27, local
    *via 30.30.30.1, Lo200, [0/0], 02:35:27, direct
100.0.0.0/30, ubest/mbest: 1/0, attached
    *via 100.0.0.2, Vlan550, [0/0], 05:52:18, direct
100.0.0.2/32, ubest/mbest: 1/0, attached
    *via 100.0.0.2, Vlan550, [0/0], 05:52:18, local
200.0.0.1/32, ubest/mbest: 1/0
    *via 100.0.0.1, [20/0], 02:32:36, bgp-65536, external, tag 65535

Server Loopback is reachable with source as N3K loopback address.

N3K
interface loopback200
  vrf member BGP_L3out
  ip address 30.30.30.1/32

N3K# ping 200.0.0.1 vrf BGP_L3out source 30.30.30.1
PING 200.0.0.1 (200.0.0.1): 56 data bytes
64 bytes from 200.0.0.1: icmp_seq=0 ttl=252 time=0.94 ms
64 bytes from 200.0.0.1: icmp_seq=1 ttl=252 time=0.729 ms
64 bytes from 200.0.0.1: icmp_seq=2 ttl=252 time=0.658 ms
64 bytes from 200.0.0.1: icmp_seq=3 ttl=252 time=0.706 ms
64 bytes from 200.0.0.1: icmp_seq=4 ttl=252 time=0.655 ms
--- 200.0.0.1 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 0.655/0.737/0.94 ms

ACI Leaf 102 route table (which has L3out toward Nexus 3K).

Leaf102# show ip route vrf TN_D:VRF_S
IP Route Table for VRF "TN_D:VRF_S"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%' in via output denotes VRF 
10.100.0.0/24, ubest/mbest: 1/0
    *via 10.0.96.64%overlay-1, [200/0], 02:56:36, bgp-65535, internal, tag 65535
30.30.30.1/32, ubest/mbest: 1/0                                                      <<address of N3K. 
    *via 100.0.0.2%TN_D:VRF_S, [20/0], 02:44:34, bgp-65535, external, tag 65536
100.0.0.0/30, ubest/mbest: 1/0, attached, direct
    *via 100.0.0.1, vlan19, [0/0], 05:09:37, direct
100.0.0.1/32, ubest/mbest: 1/0, attached
    *via 100.0.0.1, vlan19, [0/0], 05:09:37, local, local
101.101.101.101/32, ubest/mbest: 1/0
    *via 10.0.96.64%overlay-1, [1/0], 02:56:36, bgp-65535, internal, tag 65535
102.102.102.102/32, ubest/mbest: 2/0, attached, direct
    *via 102.102.102.102, lo5, [0/0], 16:49:13, local, local
    *via 102.102.102.102, lo5, [0/0], 16:49:13, direct
200.0.0.1/32, ubest/mbest: 1/0
    *via 10.0.96.64%overlay-1, [1/0], 02:42:15, bgp-65535, internal, tag 65535

Leaf 101 IP SLA configuration verification from CLI.

Leaf101# show ip sla configuration
IP SLAs Infrastructure Engine-III
Entry number: 2000
Owner: owner-icmp-echo-dme
Tag:
Operation timeout (milliseconds): 900
Type of operation to perform: icmp-echo
Target address/Source address: 10.100.0.100/0.0.0.0
Traffic-Class parameter: 0x0
Type Of Service parameter: 0x0
Request size (ARR data portion): 28
Verify data: No
Vrf Name: TN_D:VRF_S
Schedule:
   Operation frequency (seconds): 5  (not considered if randomly scheduled)
   Next Scheduled Start Time: Start Time already passed
   Group Scheduled : FALSE
   Randomly Scheduled : FALSE
   Life (seconds): Forever
   Entry Ageout (seconds): 3600
   Recurring (Starting Everyday): FALSE
   Status of entry (SNMP RowStatus): Active
Threshold (milliseconds): 900
Distribution Statistics:
   Number of statistic hours kept: 2
   Number of statistic distribution buckets kept: 1
   Statistic distribution interval (milliseconds): 20
History Statistics:
   Number of history Lives kept: 0
   Number of history Buckets kept: 15
   History Filter Type: None

Leaf101# show track brief
 TrackId  Type     Instance       Parameter            State      Last Change
 4        IP SLA   2000           reachability         up         2021-09-16T18:08:42.364+00:00
 3        List     ---            percentage           up         2021-09-16T18:08:42.365+00:00

Leaf101# show track
Track 1
    List Threshold percentage
    Threshold percentage is up
    6 changes, last change 2021-09-16T00:01:50.339+00:00
    Threshold percentage up 1% down 0%
    Tracked List Members:
        Object 2 (100)% up
    Attached to:
        Route prefix 200.0.0.1/32
Track 2
    IP SLA 2000
    reachability is up
    6 changes, last change 2021-09-16T00:01:50.338+00:00
    Tracked by:
        Track List 1

Verification with Managed Object Query (Moquery) command:

apic1# moquery -c fvIPSLAMonitoringPol -f 'fv.IPSLAMonitoringPol.name=="ICMP_Monitor"'
Total Objects shown: 1

# fv.IPSLAMonitoringPol
name                 : ICMP_Monitor
annotation           :
childAction          :
descr                :
dn                   : uni/tn-TN_D/ipslaMonitoringPol-ICMP_Monitor
extMngdBy            :
httpMethod           : get
httpUri              : /
httpVersion          : HTTP10
ipv4Tos              : 0
ipv6TrfClass         : 0
lcOwn                : local
modTs                : 2021-09-15T21:18:48.195+00:00
monPolDn             : uni/tn-common/monepg-default
nameAlias            :
ownerKey             :
ownerTag             :
reqDataSize          : 28
rn                   : ipslaMonitoringPol-ICMP_Monitor
slaDetectMultiplier  : 3
slaFrequency         : 5
slaPort              : 0
slaType              : icmp
status               :
threshold            : 900
timeout              : 900
uid                  : 15374
userdom              : :all:


apic1# moquery -c fvTrackMember -f 'fv.TrackMember.name=="Server_Physical_IP"'
Total Objects shown: 1

# fv.TrackMember
name         : Server_Physical_IP
annotation   :
childAction  :
descr        :
dn           : uni/tn-TN_D/trackmember-Server_Physical_IP
dstIpAddr    : 10.100.0.100
extMngdBy    :
id           : 2000
lcOwn        : local
modTs        : 2021-09-15T21:16:22.992+00:00
monPolDn     : uni/tn-common/monepg-default
nameAlias    :
ownerKey     :
ownerTag     :
rn           : trackmember-Server_Physical_IP
scopeDn      : uni/tn-TN_D/out-L3out_Static_server
status       :
uid          : 15374
userdom      : :all:

apic1# moquery -c fvTrackList -f 'fv.TrackList.name=="Tracking_Server_Physical_IP"'
Total Objects shown: 1

# fv.TrackList
name           : Tracking_Server_Physical_IP
annotation     :
childAction    :
descr          :
dn             : uni/tn-TN_D/tracklist-Tracking_Server_Physical_IP
extMngdBy      :
lcOwn          : local
modTs          : 2021-09-15T07:41:15.958+00:00
monPolDn       : uni/tn-common/monepg-default
nameAlias      :
ownerKey       :
ownerTag       :
percentageDown : 0
percentageUp   : 1
rn             : tracklist-Tracking_Server_Physical_IP
status         :
type           : percentage
uid            : 15374
userdom        : :all:
weightDown     : 0
weightUp       : 1

Troubleshoot

There is currently no specific troubleshooting information available for this configuration.

In event of link disconnection or physical IP address is unreachable, ACI IP SLA shows destination IP 'timeout' after configured threshold reaches.

L3out interface down

IP SLA monitor link status after link down

Leaf 101 CLI verification (You can see timeout for "Last Operation return code").

Leaf101# show ip sla statistics
IPSLAs Latest Operation Statistics
IPSLA operation id: 2000
        Latest RTT: NoConnection/Busy/Timeout
Latest operation start time: 23:54:30 UTC Wed Sep 15 2021
Latest operation return code: Timeout
Number of successes: 658
Number of failures: 61
Operation time to live: forever

As soon as the server is reachable, it shows status OK.

IP SLA monitor status after link brought up

Leaf101# show ip sla statistics
IPSLAs Latest Operation Statistics
IPSLA operation id: 2000
        Latest RTT: 1 milliseconds
Latest operation start time: 00:03:15 UTC Thu Sep 16 2021
Latest operation return code: OK
Number of successes: 18
Number of failures: 86
Operation time to live: forever

Related Information

• Cisco APIC Layer 3 Networking Configuration Guide, Release 5.2(x)
• Technical Support & Documentation - Cisco Systems