Contents

Introduction

Prerequisites

Setup and Topology

Problem Statement

Issue Summary

Troubleshooting and Verification

Why EIGRP route is preferred over eBGP route?

Solution

Introduction

This document describes the routing behaviour in ACI leaf switch when it receives same route via EIGRP and eBGP.

Prerequisites

The reader must have good understanding of the ACI components, terminologies & operations along with the routing protocols (EIGRP & BGP).

Setup and Topology

1. This set up has been done using 2 different ACI fabrics connected as:

- Direct link between both DC Border Leaf switches (BGP).

- Extended via WAN network (EIGRP). SW1 & SW2 are WAN switches.

2. 192.168.10.0/24 is internal ACI subnet connected at Fabric-1 & advertised to Fabric-2 via eBGP as well as EIGRP.

Problem Statement

Fabric-2 Border Leaf switch is receiving the same route via EIGRP and eBGP where eBGP route gets installed in routing table of the switch as expected. When the eBGP session goes down, EIGRP routes gets installed in routing table of the switch. Switch retains EIGRP route even when eBGP comes up. Expectations here is that the eBGP route must get installed in routing table as soon as the eBGP session comes up since eBGP has lesser AD value [ 20 ] than EIGRP [ 90 ].

Issue Summary

- Fabric-1 & Fabric-2 data centres are connected via WAN network (EIGRP) and direct link between both sites BL switches running eBGP.

- Fabric-1 Border Leaf switch is advertising subnet 192.168.10.0/24 to Fabric-2 via eBGP and EIGRP.

- Both the L3Out are in same VRF.

- BGP route gets installed in the routing table of Fabric-2 Border Leaf switch on the basis of AD value.

- When eBGP session between both Fabric-1 & Fabric-2 goes down, EIGRP route gets installed in the routing table of Fabric-2_BL switch which is expected.

- When eBGP comes up, the expectation is that eBGP route must get re-installed & EIGRP route is to be removed from routing table which is not happening.

- Fabric-2 Border Leaf switch is retaining the EIGRP route in its routing table instead.

Troubleshooting and Verification

- Verify the eBGP neighborship between Fabric-1 & Fabric-2 Border Leaf switches.

Fabric-2_BL# show bgp sessions vrf snTn:snTn_VRF

Total peers 3, established peers 3

ASN 100

VRF snTn:snTn_VRF, local ASN 100

peers 1, established peers 1, local router-id 172.16.2.100

State: I-Idle, A-Active, O-Open, E-Established, C-Closing, S-Shutdown

Neighbor        ASN    Flaps LastUpDn|LastRead|LastWrit St Port(L/R)  Notif(S/R)

10.10.10.3     65001     2   1d23h   |never   |never    E  179/26051    45/6

- Verify EIGRP neighborship at Fabric-2.

Fabric-2_BL# show ip eigrp neighbors vrf snTn:snTn_VRF

EIGRP neighbors for process 500 VRF snTn:snTn_VRF

H   Address                 Interface       Hold  Uptime  SRTT   RTO  Q  Seq

(sec)         (ms)       Cnt Num

0   10.10.20.3              vlan7           13   2d00h     1    50    0   8

SW-2# show ip eigrp neighbors VRF default

IP-EIGRP neighbors for process 500 VRF default

H   Address                 Interface       Hold  Uptime  SRTT   RTO  Q  Seq

(sec)         (ms)       Cnt Num

0   10.10.20.2              Vlan776          14   2d00h    6    50    0   9

- Initially, BGP route gets installed in the routing table and same route is present in EIGRP topology table of Fabric-2 Border Leaf switch.

Fabric-2_BL# show ip route 192.168.10.0/24 vrf snTn:snTn_VRF

IP Route Table for VRF "snTn:snTn_VRF"

'*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

'%<string>' in via output denotes VRF <string>

192.168.10.0/24, ubest/mbest: 1/0

*via 10.10.10.3%snTn:snTn_VRF, [20/0], 00:00:17, bgp-100, external, tag 65005

recursive next hop: 10.10.10.3/32%snTn:snTn_VRF

Fabric-2_BL# show ip eigrp topology 192.168.10.0/24 vrf snTn:snTn_VRF

EIGRP (AS 500): VRF: snTn:snTn_VRF , Topology entry for 192.168.10.0/24

State is Passive, Query origin: Local origin, 0 Successor(s),  FD is Infinity

Routing Descriptor Blocks:

10.10.20.3(vlan7), from 10.10.20.3

Urib State: in-rib,up-to-date

Composite metric is (128576/128320), Route is Internal

Vector metric:

Minimum bandwidth is 8000000 Kbit

Total delay is 5010 microseconds

Reliability is 255/255

Load is 1/255

Minimum MTU is 1500

Hop count is 1

Internal tag is 0

- EIGRP route gets installed in routing table of Fabric-2 Border Leaf switch when eBGP session goes down between Fabric-1 & Fabric-2 Border Leaf switches and retains the EIGRP route even when the eBGP comes up.

Fabric-2_BL# show ip route 192.168.10.0/24 vrf snTn:snTn_VRF

IP Route Table for VRF "snTn:snTn_VRF

'*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

'%<string>' in via output denotes VRF <string>

192.168.10.0/24, ubest/mbest: 1/0

*via 10.10.20.3, vlan7, [90/128576], 2d00h, eigrp-default, internal

- The expectation here is that, eBGP route must get re-installed in routing table as soon as eBGP session comes up. But Fabric-2_BL switch keeps EIGRP route only.

Why EIGRP route is preferred over eBGP route?

- When eBGP session goes down, Fabric-2_BL switch installs EIGRP route in routing table and same gets redistributed into MP-BGP to forward it to other service Leaf switches in the Fabric-2.

- Since Fabric-2_BL switch is redistributing it, becomes an origin for that route with default weight value 32768. Whereas, route coming from eBGP holds weight 0.

- Since higher weight is the preferred one, Fabric-2_BL switch considers redistributed route as best route and does not install eBGP route.

- The output shown beneath is when the eBGP session came back up.

Fabric-2_BL# show ip bgp 192.168.10.0/24 vrf snTn:snTn_VRF

BGP routing table information for VRF snTn:snTn_VRF, address family IPv4 Unicast

BGP routing table entry for 192.168.10.0/24, version 28 dest ptr 0xa0fe0328

Paths: (2 available, best #1)

Flags: (0x80c0002 00000000) on xmit-list, is not in urib, exported

vpn: version 371, (0x100002) on xmit-list

Multipath: eBGP iBGP

Advertised path-id 1, VPN AF advertised path-id 1

Path type (0xa961d880): redist 0x408 0x1 ref 0 adv path ref 2, path is valid, is best path

AS-Path: NONE, path locally originated

Tx Domain path attribute Flag 0xc0,Code 36, Length 8, segment length 1

domain path: { <1:5345:128>}

0.0.0.0 (metric 0) from 0.0.0.0 (172.16.0.10)

Origin incomplete, MED 128576, localpref 100, weight 32768 tag 0, propagate 0

Extcommunity:

RT:100:2129921

VNID:2129921

COST:pre-bestpath:128:128576

COST:pre-bestpath:162:90

0x8800:32768:0 (Flags = 32768, Tag = 0)

0x8801:500:128256 (ASN = 500, Delay = 128256)

0x8802:65281:320 (Reliability = 255, Hop = 1, Bandwidth = 320)

0x8803:1:1500 (Reserve = 0, Load = 1, MTU = 1500)

0x8804:0:0 (Remote ASN = 0, Remote ID = 0)

0x8805:0:0 (Remote Prot = 0, Remote Metric = 0)

VPN AF advertised path-id 2

Path type (0xa961e0bc): external 0x28 0x0 ref 0 adv path ref 1, path is valid, not best reason: Weight

AS-Path: 65001 , path sourced external to AS

Source Domain: <1:16:128>

Tx Domain path attribute Flag 0xc0,Code 36, Length 15, segment length 2

domain path: { <1:5345:128>,<1:16:128>}

10.10.10.3 (metric 0) from 10.10.10.3 (172.16.1.100)

Origin IGP, MED not set, localpref 100, weight 0 tag 0, propagate 0

Extcommunity:

RT:100:2129921

VNID:2129921

VRF advertise information:

Path-id 1 not advertised to any peer

VPN AF advertise information:

Path-id 1 advertised to peers:

10.0.152.65        10.0.152.66

Path-id 2 not advertised to any peer

Solution

There are 2-ways to fix this issue:

- LPM is one of the solution:

1. Advertise the same subnet with /23 mask under EIGRP & /24 mask via eBGP so that both routes are present in routing table of Fabric-2_BL switch.

SW-2# show run interface vlan 776

!Command: show running-config interface Vlan776

!Time: Sun Jun 23 06:30:43 2024

version 7.0(3)I7(5) Bios:version 07.66

interface Vlan776

no shutdown

ip address 10.10.20.3/24

ip router eigrp 500

ip summary-address eigrp 500 192.168.10.0/23   >>>>>> Advertised /23 via EIGRP

Fabric-2_BL# show ip route vrf snTn:snTn_VRF

IP Route Table for VRF "snTn:snTn_VRF"

'*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

'%<string>' in via output denotes VRF <string>

192.168.10.0/23, ubest/mbest: 1/0

*via 10.10.20.3, vlan20, [90/128576], 00:24:11, eigrp-default, internal    >>>>>>>>> EIGRP Route

192.168.10.0/24, ubest/mbest: 1/0

*via 10.10.10.3%snTn:snTn_VRF, [20/0], 00:04:12, bgp-100, external, tag 65005   >>>>>>>> BGP Route

b. When, eBGP session goes down, EIGRP route is still present in routing table for redundancy.

c. As soon as BGP session comes up, BGP route gets re-installed in routing table and preferred for traffic forwarding.

- Applying Weight on eBGP route:

1. If you need to advertise the subnet with same subnet mask via both EIGRP & BGP, higher weight (than 32768) can be applied on eBGP route to be the preferred route always.

2. How to apply weight on ACI:

1. Create Route-map policy.

Tenant ----> Policies ----> Route Maps for Route Control (Right click and create new policy, fill all the required details)----> Create "Set Rule" policy ---> Select "Weight" attribute policy and enter value

ii. Apply Route Map to L3Out:

Tenant ---> Networking ---> L3Out ----> Logical Node Profiles ---> Node Profile ----> Logical Interface Profile ---> Interface Profile ---> Peer Profile ---> Click on "+" under "Route Control Profile" and select created new Route Map

Fabric-2_BL# show ip bgp 192.168.10.0/24 vrf snTn:snTn_VRF

BGP routing table information for VRF snTn:snTn_VRF, address family IPv4 Unicast

BGP routing table entry for 192.168.10.0/24, version 61 dest ptr 0xa0fa3f70

Paths: (1 available, best #1)

Flags: (0x80c001a 00000000) on xmit-list, is in urib, is best urib route, is in HW, exported

vpn: version 79, (0x100002) on xmit-list

Multipath: eBGP iBGP

Advertised path-id 1, VPN AF advertised path-id 1

Path type (0xa95a2d5c): external 0x28 0x0 ref 0 adv path ref 2, path is valid, is best path

AS-Path: 65005 65001 , path sourced external to AS

Source Domain: <1:16:128>

Tx Domain path attribute Flag 0xc0,Code 36, Length 15, segment length 2

domain path: { <1:5345:128>,<1:16:128>}

10.10.10.3 (metric 0) from 10.10.10.3 (172.16.0.10)

Origin IGP, MED not set, localpref 100, weight 32769 tag 0, propagate 0

Extcommunity:

RT:100:2129921

VNID:2129921

VRF advertise information:

Path-id 1 not advertised to any peer

VPN AF advertise information:

Path-id 1 advertised to peers:

10.0.152.65        10.0.152.66

c. The catch here is, you do not see redistributed EIGRP route in BGP table when BGP session is up. Reason is FD is set to Infinity for EIGRP external route.

Fabric-2_BL# show ip eigrp topology vrf snTn:snTn_VRF

EIGRP Topology Table for AS(500)/ID(172.16.2.100) VRF snTn:snTn_VRF

Codes: P - Passive, A - Active, U - Update, Q - Query, R - Reply,

r - reply Status, s - sia Status

P 192.168.10.0/24, 0 Successors, FD is Infinity

via 10.10.20.3(128576/128320), vlan20

d. The "FD is Infinity" message is actually an indicator within EIGRP that the RIB rejected the route due to a lower admin distance route already being present.

e. EIGRP route only gets redistributed into MP-BGP and installed in routing table of fabric-2_BL switch when BGP session goes down.

Fabric-2_BL# show ip bgp summary vrf snTn:snTn_VRF

BGP summary information for VRF snTn:snTn_VRF, address family IPv4 Unicast

BGP router identifier 172.16.2.100, local AS number 100

BGP table version is 65, IPv4 Unicast config peers 1, capable peers 0

6 network entries and 6 paths using 1248 bytes of memory

BGP attribute entries [4/704], BGP AS path entries [0/0]

BGP community entries [0/0], BGP clusterlist entries [2/8]

Neighbor        V    AS MsgRcvd MsgSent   TblVer  InQ OutQ Up/Down  State/PfxRcd

10.10.10.3      4 65001   18530   18554        0    0    0 00:04:25   Idle

Fabric-2_BL# show ip eigrp topology vrf snTn:snTn_VRF

IP-EIGRP Topology Table for AS(500)/ID(172.16.2.100) VRF snTn:snTn_VRF

Codes: P - Passive, A - Active, U - Update, Q - Query, R - Reply,

r - reply Status, s - sia Status

P 192.168.10.0/24, 1 successors, FD is 128576

via 10.10.20.3 (128576/128320), Vlan20

Fabric-2_BL# show ip route vrf snTn:snTn_VRF

IP Route Table for VRF "snTn:snTn_VRF"

'*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

'%<string>' in via output denotes VRF <string>

192.168.10.0/24, ubest/mbest: 1/0

*via 10.10.20.3, Vlan20, [90/128576], 02:31:52, eigrp-default, internal  >>>>>>> EIGRP Route

Fabric-2_BL# show ip bgp 192.168.10.0/24 vrf snTn:snTn_VRF

BGP routing table information for VRF snTn:snTn_VRF, address family IPv4 Unicast

BGP routing table entry for 192.168.10.0/24, version 65 dest ptr 0xa0fa3f70

Paths: (1 available, best #1)

Flags: (0x80c0002 00000000) on xmit-list, is not in urib, exported

vpn: version 83, (0x100002) on xmit-list

Multipath: eBGP iBGP

Advertised path-id 1, VPN AF advertised path-id 1

Path type (0xa95a2c64): redist 0x408 0x1 ref 0 adv path ref 2, path is valid, is best path

AS-Path: NONE, path locally originated

Tx Domain path attribute Flag 0xc0,Code 36, Length 8, segment length 1

domain path: { <1:5345:128>}

0.0.0.0 (metric 0) from 0.0.0.0 (172.16.0.10)

Origin incomplete, MED 128576, localpref 100, weight 32768 tag 0, propagate 0

Extcommunity:

RT:100:2129921

VNID:2129921

COST:pre-bestpath:128:128576

COST:pre-bestpath:162:90

0x8800:32768:0 (Flags = 32768, Tag = 0)

0x8801:500:128256 (ASN = 500, Delay = 128256)

0x8802:65281:320 (Reliability = 255, Hop = 1, Bandwidth = 320)

0x8803:1:1500 (Reserve = 0, Load = 1, MTU = 1500)

0x8804:0:0 (Remote ASN = 0, Remote ID = 0)

0x8805:0:0 (Remote Prot = 0, Remote Metric = 0)

VRF advertise information:

Path-id 1 not advertised to any peer

VPN AF advertise information:

Path-id 1 advertised to peers:

10.0.152.65        10.0.152.66