The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This document describes steps to understand and troubleshoot an L3out in ACI
The material from this document was extracted from the Troubleshooting Cisco Application Centric Infrastructure, Second Edition book specifically the External Forwarding - Overview, External Forwarding - Adjacencies, External Forwarding - Route advertisement, External Forwarding - Contract and L3out and External Forwarding - Share L3out chapters.
The following picture illustrates the major building blocks required to configure an L3 Outside (L3Out).
The following diagram shows the high-level operation involved for external routing.
In the L3Out EPG section, subnets can be defined with a series of 'Scope' and 'Aggregate' options as illustrated below:
'Scope' options:
'Aggregate' options:
The following topology will be used in this section:
This section explains how to troubleshoot and verify routing protocol adjacencies on L3Out interfaces.
Below are a few parameters to check that will be applicable for all ACI external routing protocols:
This section uses an example of an eBGP peering between the loopback on BL3, BL4, and R34 from the topology in the Overview section. The BGP AS on R34 is 65002.
Verify the following criteria when establishing a BGP adjacency.
The BGP AS number of a user L3Out will automatically be the same as the BGP AS for the infra-MP-BGP that is configured in the BGP Route Reflector policy. The 'Local AS' configuration in the BGP Peer Connectivity Profile is not required unless one needs to disguise the ACI BGP AS to the outside world. This means external routers should point to the BGP AS configured in the BGP Route Reflector.
NOTE — The scenario where Local AS configuration is required is the same as the standalone NX-OS 'local-as' command.
The Remote BGP AS number is required only for eBGP where the neighbor's BGP AS is different from ACI BGP AS.
ACI supports sourcing a BGP session from the loopback interface on top of a typical ACI L3Out interface type (routed, sub-interface, SVI).
When a BGP session needs to be sourced from a loopback, configure the BGP Peer Connectivity Profile under the Logical Node Profile.
When the BGP session needs to be sourced from a routed/sub-interface/SVI, configure the BGP Peer Connectivity Profile under the Logical Interface Profile.
When the BGP peer IPs are loopbacks, make sure the BL and the external router have reachability to the peer's IP address. Static routes or OSPF can be used to gain reachability to the peer IPs.
The CLI outputs for the following steps are collected from BL3 in the topology from the Overview section.
1. Check if the BGP session is established
'State/PfxRcd' in the following CLI output means the BGP session is established.
f2-leaf3# show bgp ipv4 unicast summary vrf Prod:VRF1 BGP summary information for VRF Prod:VRF1, address family IPv4 Unicast BGP router identifier 10.0.0.3, local AS number 65001 Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd 10.0.0.134 4 65002 10 10 10 0 0 00:06:39 0
If the 'State/PfxRcd' shows Idle or Active, BGP packets are not being exchanged with the peer yet. In such a scenario, check the following and move on to the next step.
2. Check BGP Neighbor details (BGP Peer Connectivity Profile)
The following command shows the parameters that are key for BGP neighbor establishment.
f2-leaf3# show bgp ipv4 unicast neighbors vrf Prod:VRF1 BGP neighbor is 10.0.0.134, remote AS 65002, ebgp link, Peer index 1 BGP version 4, remote router ID 10.0.0.134 BGP state = Established, up for 00:11:18 Using loopback3 as update source for this peer External BGP peer might be upto 2 hops away ... For address family: IPv4 Unicast ... Inbound route-map configured is permit-all, handle obtained Outbound route-map configured is exp-l3out-BGP-peer-3047424, handle obtained Last End-of-RIB received 00:00:01 after session start Local host: 10.0.0.3, Local port: 34873 Foreign host: 10.0.0.134, Foreign port: 179 fd = 64
Once the BGP peer is established correctly, the 'Local host' and 'Foreign host' appear at the bottom of the output.
3. Check IP reachability for the BGP peer
f2-leaf3# show ip route vrf Prod:VRF1 10.0.0.3/32, ubest/mbest: 2/0, attached, direct *via 10.0.0.3, lo3, [0/0], 02:41:46, local, local *via 10.0.0.3, lo3, [0/0], 02:41:46, direct 10.0.0.134/32, ubest/mbest: 1/0 *via 10.10.34.1, vlan27, [1/0], 02:41:46, static <--- neighbor IP reachability via static route 10.10.34.0/29, ubest/mbest: 2/0, attached, direct *via 10.10.34.3, vlan27, [0/0], 02:41:46, direct *via 10.10.34.2, vlan27, [0/0], 02:41:46, direct 10.10.34.2/32, ubest/mbest: 1/0, attached *via 10.10.34.2, vlan27, [0/0], 02:41:46, local, local 10.10.34.3/32, ubest/mbest: 1/0, attached *via 10.10.34.3, vlan27, [0/0], 02:41:46, local, local
Ensure ping to the neighbor IP works from ACI BGP's source IP.
f2-leaf3# iping 10.0.0.134 -V Prod:VRF1 -S 10.0.0.3 PING 10.0.0.134 (10.0.0.134) from 10.0.0.3: 56 data bytes 64 bytes from 10.0.0.134: icmp_seq=0 ttl=255 time=0.571 ms 64 bytes from 10.0.0.134: icmp_seq=1 ttl=255 time=0.662 ms
4. Check the same thing on the external router
The following is an example of configuration on the external router (standalone NX-OS).
router bgp 65002 vrf f2-bgp router-id 10.0.0.134 neighbor 10.0.0.3 remote-as 65001 update-source loopback134 ebgp-multihop 2 address-family ipv4 unicast neighbor 10.0.0.4 remote-as 65001 update-source loopback134 ebgp-multihop 2 address-family ipv4 unicast interface loopback134 vrf member f2-bgp ip address 10.0.0.134/32 interface Vlan2501 no shutdown vrf member f2-bgp ip address 10.10.34.1/29 vrf context f2-bgp ip route 10.0.0.0/29 10.10.34.2
5. Additional Step — tcpdump
On ACI leaf nodes, the tcpdump tool can sniff the 'kpm_inb' CPU interface to confirm if the protocol packets reached the leaf CPU. Use L4 port 179 (BGP) as a filter.
f2-leaf3# tcpdump -ni kpm_inb port 179 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on kpm_inb, link-type EN10MB (Ethernet), capture size 65535 bytes 20:36:58.292903 IP 10.0.0.134.179 > 10.0.0.3.34873: Flags [P.], seq 3775831990:3775832009, ack 807595300, win 3650, length 19: BGP, length: 19 20:36:58.292962 IP 10.0.0.3.34873 > 10.0.0.134.179: Flags [.], ack 19, win 6945, length 0 20:36:58.430418 IP 10.0.0.3.34873 > 10.0.0.134.179: Flags [P.], seq 1:20, ack 19, win 6945, length 19: BGP, length: 19 20:36:58.430534 IP 10.0.0.134.179 > 10.0.0.3.34873: Flags [.], ack 20, win 3650, length 0
This section uses an example of OSPF neighborships between BL3, BL4, and R34 from the topology in Overview section with OSPF AreaID 1 (NSSA).
The following are the common criteria to check for OSPF adjacency establishment.
Just like any routing device, OSPF Area ID and Type need to match on both neighbors. Some ACI specific limitations on OSPF Area ID configurations include:
Although the OSPF ID does not need to be backbone 0, in the case of Transit Routing it is required between two OSPF L3Outs on the same leaf; one of them must use OSPF Area 0 because any route exchange between OSPF areas must be through OSPF Area 0.
The default MTU on ACI is 9000 bytes, instead of 1500 bytes, which is typically the default used on traditional routing devices. Ensure the MTU matches with the external device. When OSPF neighbor establishment fails due to MTU, it gets stuck at EXCHANGE/DROTHER.
This is equivalent to 'ip router ospf <tag> area <area id>' on a standalone NX-OS config to enable OSPF on the interface. Without this, the leaf interfaces will not join OSPF.
OSPF requires the Hello and Dead Timers to match on each neighbor device. These are configured in the OSPF Interface Profile.
Ensure the OSPF Interface Network Type matches with the external device. When the external device is using type Point-to-Point, the ACI side needs to explicitly configure Point-to-Point as well. These are also configured in the OSPF Interface Profile.
The CLI outputs in the following steps are collected from BL3 in the topology from "Overview" section.
1. Check OSPF neighbor status
If the 'State' is 'FULL' in the following CLI, the OSPF neighbor is established correctly. Otherwise, move on to the next step to check parameters.
f2-leaf3# show ip ospf neighbors vrf Prod:VRF2 OSPF Process ID default VRF Prod:VRF2 Total number of neighbors: 2 Neighbor ID Pri State Up Time Address Interface 10.0.0.4 1 FULL/DR 00:47:30 10.10.34.4 Vlan28 <--- neighbor with BL4 10.0.0.134 1 FULL/DROTHER 00:00:21 10.10.34.1 Vlan28 <--- neighbor with R34
In ACI, BLs will form OSPF neighborships with each other on top of external routers when using the same VLAN ID with an SVI. This is because ACI has an internal flooding domain called L3Out BD (or External BD) for each VLAN ID in the L3Out SVIs. Note that the VLAN ID 28 is an internal VLAN called PI-VLAN (Platform-Independent VLAN) instead of the actual VLAN (Access Encap VLAN) used on wire. Use the following command to verify the access encap VLAN ('vlan-2502').
f2-leaf3# show vlan id 28 extended VLAN Name Encap Ports ---- -------------------------------- ---------------- ------------------------ 28 Prod:VRF2:l3out-OSPF:vlan-2502 vxlan-14942176, Eth1/13, Po1 vlan-2502
One could get the same output via access encap VLAN ID as well.
f2-leaf3# show vlan encap-id 2502 extended VLAN Name Encap Ports ---- -------------------------------- ---------------- ------------------------ 28 Prod:VRF2:l3out-OSPF:vlan-2502 vxlan-14942176, Eth1/13, Po1 vlan-2502
2. Check OSPF area
Ensure the OSPF area ID and Type is identical to the neighbors. If the OSPF interface profile is missing, the interface will not join OSPF and it will not show up in the OSPF CLI output.
f2-leaf3# show ip ospf interface brief vrf Prod:VRF2 OSPF Process ID default VRF Prod:VRF2 Total number of interface: 1 Interface ID Area Cost State Neighbors Status Vlan28 94 0.0.0.1 4 BDR 2 up f2-leaf3# show ip ospf vrf Prod:VRF2 Routing Process default with ID 10.0.0.3 VRF Prod:VRF2 ... Area (0.0.0.1) Area has existed for 00:59:14 Interfaces in this area: 1 Active interfaces: 1 Passive interfaces: 0 Loopback interfaces: 0 This area is a NSSA area Perform type-7/type-5 LSA translation SPF calculation has run 10 times Last SPF ran for 0.001175s Area ranges are Area-filter in 'exp-ctx-proto-3112960' Area-filter out 'permit-all' Number of LSAs: 4, checksum sum 0x0
3. Check OSPF interface details
Ensure interface level parameters meet the requirements for OSPF neighbor establishment such as IP subnet, Network Type, Hello/Dead Timer. Please note the VLAN ID to specify the SVI is PI-VLAN (vlan28)
f2-leaf3# show ip ospf interface vrf Prod:VRF2 Vlan28 is up, line protocol is up IP address 10.10.34.3/29, Process ID default VRF Prod:VRF2, area 0.0.0.1 Enabled by interface configuration State BDR, Network type BROADCAST, cost 4 Index 94, Transmit delay 1 sec, Router Priority 1 Designated Router ID: 10.0.0.4, address: 10.10.34.4 Backup Designated Router ID: 10.0.0.3, address: 10.10.34.3 2 Neighbors, flooding to 2, adjacent with 2 Timer intervals: Hello 10, Dead 40, Wait 40, Retransmit 5 Hello timer due in 0.000000 No authentication Number of opaque link LSAs: 0, checksum sum 0
f2-leaf3# show interface vlan28 Vlan28 is up, line protocol is up, autostate disabled Hardware EtherSVI, address is 0022.bdf8.19ff Internet Address is 10.10.34.3/29 MTU 9000 bytes, BW 10000000 Kbit, DLY 1 usec
4. Check IP reachability to the neighbor
Although OSPF Hello packets are Link Local Multicast packets, OSPF DBD packets required for the first OSPF LSDB exchange are unicast. Therefore, unicast reachability also needs to be verified for the OSPF neighborship establishment.
f2-leaf3# iping 10.10.34.1 -V Prod:VRF2 PING 10.10.34.1 (10.10.34.1) from 10.10.34.3: 56 data bytes 64 bytes from 10.10.34.1: icmp_seq=0 ttl=255 time=0.66 ms 64 bytes from 10.10.34.1: icmp_seq=1 ttl=255 time=0.653 ms
5. Check the same on the external router
The following are examples of configurations on the external router (standalone NX-OS)
router ospf 1 vrf f2-ospf router-id 10.0.0.134 area 0.0.0.1 nssa interface Vlan2502 no shutdown mtu 9000 vrf member f2-ospf ip address 10.10.34.1/29 ip router ospf 1 area 0.0.0.1
Make sure to verify the MTU as well on the physical interface.
6. Additional step — tcpdump
On ACI leaf nodes, the user can perform tcpdump on the 'kpm_inb' CPU interface to verify if the protocol packets have reached the leaf CPU. Although there are multiple filters for OSPF, the IP Protocol Number is the most comprehensive filter.
f2-leaf3# tcpdump -ni kpm_inb proto 89 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on kpm_inb, link-type EN10MB (Ethernet), capture size 65535 bytes 22:28:38.231356 IP 10.10.34.4 > 224.0.0.5: OSPFv2, Hello, length 52 22:28:42.673810 IP 10.10.34.3 > 224.0.0.5: OSPFv2, Hello, length 52 22:28:44.767616 IP 10.10.34.1 > 224.0.0.5: OSPFv2, Hello, length 52 22:28:44.769092 IP 10.10.34.3 > 10.10.34.1: OSPFv2, Database Description, length 32 22:28:44.769803 IP 10.10.34.1 > 10.10.34.3: OSPFv2, Database Description, length 32 22:28:44.775376 IP 10.10.34.3 > 10.10.34.1: OSPFv2, Database Description, length 112 22:28:44.780959 IP 10.10.34.1 > 10.10.34.3: OSPFv2, LS-Request, length 36 22:28:44.781376 IP 10.10.34.3 > 10.10.34.1: OSPFv2, LS-Update, length 64 22:28:44.790931 IP 10.10.34.1 > 224.0.0.6: OSPFv2, LS-Update, length 64
This section uses an example of EIGRP neighborship between BL3, BL4 and R34 from the topology in "Overview" section with EIGRP AS 10.
The following are the common criteria for EIGRP adjacency establishment.
This is equivalent to the 'ip router eigrp <as>' configuration on a standalone NX-OS device. Without this, the leaf interfaces will not join EIGRP.
Although this does not have to match to simply establish the EIGRP neighborship, the EIGRP topology exchange packets may become larger than the maximum MTU allowed on the interfaces between the peers, and since these packets are not allowed to be fragmented, they are dropped and as a result the EIGRP neighborship will flap.
The CLI outputs in the following steps are collected from BL3 in the topology from the "Overview" section.
1. Check EIGRP neighbor status
f2-leaf3# show ip eigrp neighbors vrf Prod:VRF3 EIGRP neighbors for process 10 VRF Prod:VRF3 H Address Interface Hold Uptime SRTT RTO Q Seq (sec) (ms) Cnt Num 0 10.10.34.4 vlan29 14 00:12:58 1 50 0 6 <--- neighbor with BL4 1 10.10.34.1 vlan29 13 00:08:44 2 50 0 4 <--- neighbor with R34
In ACI, BLs will form an EIGRP neighborship with each other on top of external routers when they use the same VLAN ID with SVI. This is because an ACI has an internal flooding domain called L3Out BD (or External BD) for each VLAN ID in L3Out SVIs.
Please note that the VLAN ID 29 is an internal VLAN called PI-VLAN (Platform-Independent VLAN) instead of the actual VLAN (Access Encap VLAN) used on wire. Use the following command to verify the access encap VLAN (vlan-2503).
f2-leaf3# show vlan id 29 extended VLAN Name Encap Ports ---- -------------------------------- ---------------- ------------------------ 29 Prod:VRF3:l3out-EIGRP:vlan-2503 vxlan-15237052, Eth1/13, Po1 vlan-2503
One could get the same output via access encap VLAN ID as well.
f2-leaf3# show vlan encap-id 2503 extended VLAN Name Encap Ports ---- -------------------------------- ---------------- ------------------------ 29 Prod:VRF3:l3out-EIGRP:vlan-2503 vxlan-15237052, Eth1/13, Po1 vlan-2503
2. Check EIGRP interface details
Ensure EIGRP is running on the expected interface. If not, check Logical Interface Profile and EIGRP Interface Profile.
f2-leaf3# show ip eigrp interfaces vrf Prod:VRF3 EIGRP interfaces for process 10 VRF Prod:VRF3 Xmit Queue Mean Pacing Time Multicast Pending Interface Peers Un/Reliable SRTT Un/Reliable Flow Timer Routes vlan29 2 0/0 1 0/0 50 0 Hello interval is 5 sec Holdtime interval is 15 sec Next xmit serial: 0 Un/reliable mcasts: 0/2 Un/reliable ucasts: 5/10 Mcast exceptions: 0 CR packets: 0 ACKs suppressed: 2 Retransmissions sent: 2 Out-of-sequence rcvd: 0 Classic/wide metric peers: 2/0
f2-leaf3# show int vlan 29 Vlan29 is up, line protocol is up, autostate disabled Hardware EtherSVI, address is 0022.bdf8.19ff Internet Address is 10.10.34.3/29 MTU 9000 bytes, BW 10000000 Kbit, DLY 1 usec
3. Check the same on the external router
The following the example config on the external router (standalone NX-OS).
router eigrp 10 vrf f2-eigrp interface Vlan2503 no shutdown vrf member f2-eigrp ip address 10.10.34.1/29 ip router eigrp 10
4. Additional step — tcpdump
On ACI leaf nodes, the user can perform tcpdump on the 'kpm_inb' CPU interface to confirm if the protocol packets reached the leaf's CPU. Use IP protocol number 88 (EIGRP) as a filter.
f2-leaf3# tcpdump -ni kpm_inb proto 88 tcpdump: verbose output suppressed, use -v or -vv for full protocol decode listening on kpm_inb, link-type EN10MB (Ethernet), capture size 65535 bytes 23:29:43.725676 IP 10.10.34.3 > 224.0.0.10: EIGRP Hello, length: 40 23:29:43.726271 IP 10.10.34.4 > 224.0.0.10: EIGRP Hello, length: 40 23:29:43.728178 IP 10.10.34.1 > 224.0.0.10: EIGRP Hello, length: 40 23:29:45.729114 IP 10.10.34.1 > 10.10.34.3: EIGRP Update, length: 20 23:29:48.316895 IP 10.10.34.3 > 224.0.0.10: EIGRP Hello, length: 40
This section focusses on the verification and troubleshooting of route advertisement in ACI. Specifically, it looks at examples involving:
This section discusses route-leaking as it pertains to shared L3Outs in later sections.
Before looking at common troubleshooting the user should familiarize themselves with how Bridge Domain advertisement is supposed to work.
BD advertisement, when the BD and L3Out are in the same VRF, involves:
In addition, it is also possible to control Bridge Domain advertisement using export route-profiles which prevent the need to associate the L3Out. However, 'Advertise Externally' should still be selected. This is a less common use-case so it won't be discussed here.
The contract relationship between the L3Out and the EPG is required in order to cause the BD pervasive static route to get pushed to the BL. The actual route-advertisement is handled via redistribution of the static route into the external protocol. Lastly, the redistribution route-maps will only be installed within the L3Outs that are associated to the BD. In this way the route isn't advertised out all L3Outs.
In this case, the BD subnet is 192.168.1.0/24 and it should be advertised via OSPF L3Out.
leaf103# show ip route 192.168.1.0/24 vrf Prod:Vrf1 IP Route Table for VRF "Prod:Vrf1" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%' in via output denotes VRF Route not found
Notice that the BD route isn't present yet on the BL.
At this point no other configuration has been done. The L3Out isn't yet associated to the BD and the 'Advertise Externally' flag isn't set.
leaf103# show ip route 10.0.1.0/24 vrf Prod:Vrf1 IP Route Table for VRF "Prod:Vrf1" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%' in via output denotes VRF 192.168.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive *via 10.0.120.34%overlay-1, [1/0], 00:00:08, static, tag 4294967294 recursive next hop: 10.0.120.34/32%overlay-1
Notice that the BD subnet route (indicated by the pervasive flag) is now deployed on the BL. Notice, however, that the route is tagged. This tag value is an implicit value assigned to BD routes before being configured with 'Advertise Externally'. All external protocols deny this tag from being redistributed.
The L3Out still hasn't been associated to the BD. However, notice that the tag has cleared.
leaf103# show ip route 192.168.1.0/24 vrf Prod:Vrf1 IP Route Table for VRF "Prod:Vrf1" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%' in via output denotes VRF
192.168.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive *via 10.0.120.34%overlay-1, [1/0], 00:00:06, static recursive next hop: 10.0.120.34/32%overlay-1
At this point the route still isn't being advertised externally because there is no route-map and prefix-list that matches this prefix for redistribution into the external protocol. This can be verified with the following commands:
leaf103# show ip ospf vrf Prod:Vrf1 Routing Process default with ID 10.0.0.3 VRF Prod:Vrf1 Stateful High Availability enabled Supports only single TOS(TOS0) routes Supports opaque LSA Table-map using route-map exp-ctx-2392068-deny-external-tag Redistributing External Routes from static route-map exp-ctx-st-2392068 direct route-map exp-ctx-st-2392068 bgp route-map exp-ctx-proto-2392068 eigrp route-map exp-ctx-proto-2392068 coop route-map exp-ctx-st-2392068
The BD route is programmed as a static route, so check the static redistribution route-map by running 'show route-map <route-map name>' and then 'show ip prefix-list <name>' on any prefix-lists that are present in the route-map. Do this in the next step.
As mentioned earlier, this step results in the prefix-list that matches the BD subnet being installed in the static to external protocol redistribution route-map.
leaf103# show route-map exp-ctx-st-2392068 route-map exp-ctx-st-2392068, deny, sequence 1 Match clauses: tag: 4294967294 Set clauses: ... route-map exp-ctx-st-2392068, permit, sequence 15803 Match clauses: ip address prefix-lists: IPv4-st16390-2392068-exc-int-inferred-export-dst ipv6 address prefix-lists: IPv6-deny-all Set clauses: tag 0
Verify the prefix-list:
leaf103# show ip prefix-list IPv4-st16390-2392068-exc-int-inferred-export-dst ip prefix-list IPv4-st16390-2392068-exc-int-inferred-export-dst: 1 entries seq 1 permit 192.168.1.1/24
The BD subnet is being matched to redistribute into OSPF.
At this point the configuration and verification workflow is complete for the advertisement of the BD subnet out of the L3Out. Past this point, verification would be protocol specific. For instance:
For BGP, all static routes are implicitly permitted for redistribution. The route-map which matches the BD subnet is applied at the BGP neighbor level.
leaf103# show bgp ipv4 unicast neighbor 10.0.0.134 vrf Prod:Vrf1 | grep Outbound Outbound route-map configured is exp-l3out-BGP-peer-2392068, handle obtained
In the above example, 10.0.0.134 is the BGP neighbor configured within the L3Out.
Like OSPF, a route-map is used to control Static to EIGRP redistribution. In this way only subnets associated to the L3Out and set to 'Advertise Externally' should be redistributed. This can be verified with this command:
leaf103# show ip eigrp vrf Prod:Vrf1 IP-EIGRP AS 100 ID 10.0.0.3 VRF Prod:Vrf1 Process-tag: default Instance Number: 1 Status: running Authentication mode: none Authentication key-chain: none Metric weights: K1=1 K2=0 K3=1 K4=0 K5=0 metric version: 32bit IP proto: 88 Multicast group: 224.0.0.10 Int distance: 90 Ext distance: 170 Max paths: 8 Active Interval: 3 minute(s) Number of EIGRP interfaces: 1 (0 loopbacks) Number of EIGRP passive interfaces: 0 Number of EIGRP peers: 2 Redistributing: static route-map exp-ctx-st-2392068 ospf-default route-map exp-ctx-proto-2392068 direct route-map exp-ctx-st-2392068 coop route-map exp-ctx-st-2392068 bgp-65001 route-map exp-ctx-proto-2392068
The final working BD configuration is shown below.
In this case, the typical symptom would normally be that a configured BD subnet is not being advertised out of an L3Out. Follow the previous workflow to understand which component is broken.
Start with the configuration before getting too low-level by verifying the following:
Possible Cause: BD Not Deployed
This case would be applicable in a couple of different scenarios, such as:
In both cases, the BD would not be deployed and, as a result, the BD static route would not get pushed to the BL. The solution here is to deploy some active resources within an EPG which is linked to this BD so that the subnet gets deployed.
Possible Cause: OSPF L3Out is configured as 'Stub' or 'NSSA' with No Redistribution
When OSPF is used as the L3Out protocol, basic OSPF rules must still be followed. Stub areas do not allow redistributed LSAs but can advertise a default route instead. NSSA areas do allow redistributed paths but 'Send Redistributed LSAs into NSSA Area' must be selected on the L3Out. Or NSSA can also advertise a default route instead by disabling 'Originate Summary LSA' as well which is a typical scenario where 'Send Redistributed LSA's into NSSA Area' would be disabled.
Possible Cause: 'Default-Export' Route-Profile with a 'Deny' Action configured under the L3Out
When route-profiles are configured under an L3Out with the names of 'default-export' or 'default-import' they are implicitly applied to the L3Out. In addition, if the default-export route-profile is set to a deny action and configured as 'Match Prefix and Routing Policy' then BD subnets should be advertised out of this L3Out and would be implicitly denied:
Prefix-matches within the default-export route-profile will not implicitly include BD Subnets if the 'Match Routing Policy Only' option is selected.
This section discusses how ACI learns external routes through an L3Out and distributes this to internal leaf nodes. It also covers transit and route-leaking use-cases in later sections
As with the previous section, the user should be aware of what happens at a higher-level.
By default, all routes learned through the external protocol are redistributed into the internal fabric BGP process. This is true regardless of what subnets are configured under the external EPG and what flags are selected. There are two examples where this is not true.
For an external route to be distributed to an internal leaf the following must happen:
If the internal EPG/BD is in the same VRF as the L3Out then the above three steps are all that is required for the internal EPG/BD to use external routes.
In this case the external route that should be learned on BLs 103 and 104 is 172.16.20.1/32.
leaf103# show ip route 172.16.20.1 vrf Prod:Vrf1 IP Route Table for VRF "Prod:Vrf1" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%' in via output denotes VRF 172.16.20.1/32, ubest/mbest: 1/0 *via 10.10.34.3, vlan347, [110/20], 00:06:29, ospf-default, type-2
It is evident that it is installed in the routing table as being learned through OSPF. If it wasn't seen here, check the individual protocol and ensure adjacencies are up. Route is redistributed into BGP The redistribution route-map can be verified, after checking that neither 'Import' enforcement or Interleak Route-Profiles are used, by looking at the route-map used for external protocol to BGP redistribution. See the following command:
leaf103# show bgp process vrf Prod:Vrf1 Information regarding configured VRFs: BGP Information for VRF Prod:Vrf1 VRF Type : System VRF Id : 85 VRF state : UP VRF configured : yes VRF refcount : 1 VRF VNID : 2392068 Router-ID : 10.0.0.3 Configured Router-ID : 10.0.0.3 Confed-ID : 0 Cluster-ID : 0.0.0.0 MSITE Cluster-ID : 0.0.0.0 No. of configured peers : 1 No. of pending config peers : 0 No. of established peers : 1 VRF RD : 101:2392068 VRF EVPN RD : 101:2392068 ... Redistribution direct, route-map permit-all static, route-map imp-ctx-bgp-st-interleak-2392068 ospf, route-map permit-all coop, route-map exp-ctx-st-2392068 eigrp, route-map permit-all
Here it is evident that the 'permit-all' route-map is used for OSPF to BGP redistribution. This is the default. From here, BL can be verified and the local route originating from BGP checked:
a-leaf101# show bgp ipv4 unicast 172.16.20.1/32 vrf Prod:Vrf1 BGP routing table information for VRF Prod:Vrf1, address family IPv4 Unicast BGP routing table entry for 172.16.20.1/32, version 25 dest ptr 0xa6f25ad0 Paths: (2 available, best #2) Flags: (0x80c0002 00000000) on xmit-list, is not in urib, exported vpn: version 16316, (0x100002) on xmit-list Multipath: eBGP iBGP Advertised path-id 1, VPN AF advertised path-id 1 Path type: redist 0x408 0x1 ref 0 adv path ref 2, path is valid, is best path AS-Path: NONE, path locally originated 0.0.0.0 (metric 0) from 0.0.0.0 (10.0.0.3) Origin incomplete, MED 20, localpref 100, weight 32768 Extcommunity: RT:65001:2392068 VNID:2392068 COST:pre-bestpath:162:110 VRF advertise information: Path-id 1 not advertised to any peer VPN AF advertise information: Path-id 1 advertised to peers: 10.0.64.64 10.0.72.66 Path-id 2 not advertised to any peer
In the above output, the 0.0.0.0/0 indicates it is originated locally. The list of peers advertised to are the spine nodes in the fabric which act as Route-Reflectors.
The BL should advertise it to the spine nodes through the VPNv4 BGP Address-Family. The spine nodes should advertise it to any leaf nodes with the VRF deployed (true of non-route-leaking example). On any of these leaf nodes run 'show bgp vpnv4 unicast <route> vrf overlay-1' to verify it is in VPNv4
Use the command below to verify the route on the internal leaf.
leaf101# show ip route 172.16.20.1 vrf Prod:Vrf1 IP Route Table for VRF "Prod:Vrf1" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%' in via output denotes VRF 172.16.20.1/32, ubest/mbest: 2/0 *via 10.0.72.64%overlay-1, [200/20], 00:21:24, bgp-65001, internal, tag 65001 recursive next hop: 10.0.72.64/32%overlay-1 *via 10.0.72.67%overlay-1, [200/20], 00:21:24, bgp-65001, internal, tag 65001 recursive next hop: 10.0.72.67/32%overlay-1
In the above output the route is being learned through BGP and the next-hops should be the Physical TEPs (PTEP) of the BLs.
leaf101# acidiag fnvread ID Pod ID Name Serial Number IP Address Role State LastUpdMsgId -------------------------------------------------------------------------------------------------------------- 103 1 a-leaf101 FDO20160TPS 10.0.72.67/32 leaf active 0 104 1 a-leaf103 FDO20160TQ0 10.0.72.64/32 leaf active 0
In this scenario the internal leaf (101) is not receiving an external route(s).
As always, first check the basics. Make sure that:
If the above criteria are correct, below are some more advanced examples of what could be causing the issue.
Possible Cause: VRF not deployed on the internal leaf
In this case, the issue would be that there are no EPGs with resources deployed on the internal leaf where the external route is expected. This could be caused by static path bindings only configured on down interfaces or only have On Demand mode VMM integrated EPGs present with no dynamic attachments detected.
Because the L3Out VRF is not deployed on the internal leaf (verify with 'show vrf' on internal leaf) the internal leaf will not import the BGP route from VPNv4.
To resolve this issue, the user should deploy resources within the L3Out VRF on the internal leaf.
Possible Cause: Import Route Enforcement is being used
As mentioned earlier, when import route-control enforcement is enabled the L3Out only accepts external routes that are explicitly permitted. Typically, the feature is implemented as a table-map. A table-map sits in between the protocol RIB and the actual routing table so that it only affects what is in the routing table.
In the output below the Import Route-Control is enabled, but there aren't any explicitly permitted routes. Notice that the LSA is in the OSPF database but not in the routing table on the BL:
leaf103# vsh -c "show ip ospf database external 172.16.20.1 vrf Prod:Vrf1" OSPF Router with ID (10.0.0.3) (Process ID default VRF Prod:Vrf1) Type-5 AS External Link States Link ID ADV Router Age Seq# Checksum Tag 172.16.20.1 10.0.0.134 455 0x80000003 0xb9a0 0
leaf103# show ip route 172.16.20.1 vrf Prod:Vrf1 IP Route Table for VRF "Prod:Vrf1" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%' in via output denotes VRF Route not found
Here is the table-map that is now installed causing this behavior:
leaf103# show ip ospf vrf Prod:Vrf1 Routing Process default with ID 10.0.0.3 VRF Prod:Vrf1 Stateful High Availability enabled Supports only single TOS(TOS0) routes Supports opaque LSA Table-map using route-map exp-ctx-2392068-deny-external-tag Redistributing External Routes from.. leaf103# show route-map exp-ctx-2392068-deny-external-tag route-map exp-ctx-2392068-deny-external-tag, deny, sequence 1 Match clauses: tag: 4294967295 Set clauses: route-map exp-ctx-2392068-deny-external-tag, deny, sequence 19999 Match clauses: ospf-area: 0.0.0.100 Set clauses:
Anything learning in area 100, which is the area configured on this L3Out, is implicitly denied by this table-map so that it is not installed in the routing table.
To resolve this issue, the user should define the subnet on the external EPG with the 'Import Route Control Subnet' flag or create an Import Route-Profile that matches the prefixes to be installed.
Possible Cause: an Interleak Profile is being used
Interleak Route-Profiles are used for EIGRP and OSPF L3Outs and intended to allow for control over what is redistributed from the IGP into BGP as well as allows the application of policy such as setting BGP attributes.
Without an interleak Route-Profile, all routes are implicitly imported to BGP.
Without an interleak Route-Profile:
leaf103# show bgp process vrf Prod:Vrf1 Information regarding configured VRFs: BGP Information for VRF Prod:Vrf1 VRF Type : System VRF Id : 85 VRF state : UP VRF configured : yes VRF refcount : 1 VRF VNID : 2392068 Router-ID : 10.0.0.3 Configured Router-ID : 10.0.0.3 Confed-ID : 0 Cluster-ID : 0.0.0.0 MSITE Cluster-ID : 0.0.0.0 No. of configured peers : 1 No. of pending config peers : 0 No. of established peers : 1 VRF RD : 101:2392068 VRF EVPN RD : 101:2392068 ... Peers Active-peers Routes Paths Networks Aggregates 1 1 7 11 0 0 Redistribution direct, route-map permit-all static, route-map imp-ctx-bgp-st-interleak-2392068 ospf, route-map permit-all coop, route-map exp-ctx-st-2392068 eigrp, route-map permit-all
With an interleak route-profile:
a-leaf103# show bgp process vrf Prod:Vrf1 Information regarding configured VRFs: BGP Information for VRF Prod:Vrf1 VRF Type : System VRF Id : 85 VRF state : UP VRF configured : yes VRF refcount : 1 VRF VNID : 2392068 Router-ID : 10.0.0.3 Configured Router-ID : 10.0.0.3 Confed-ID : 0 Cluster-ID : 0.0.0.0 MSITE Cluster-ID : 0.0.0.0 No. of configured peers : 1 No. of pending config peers : 0 No. of established peers : 1 VRF RD : 101:2392068 VRF EVPN RD : 101:2392068 ... Redistribution direct, route-map permit-all static, route-map imp-ctx-bgp-st-interleak-2392068 ospf, route-map imp-ctx-proto-interleak-2392068 coop, route-map exp-ctx-st-2392068 eigrp, route-map permit-all
The above highlighted route-map would only permit what is explicitly matched in the configured Interleak Profile. If the external route isn't matched it will not be redistributed into BGP.
This section discusses how routes from one L3Out are advertised out another L3Out. This would also cover the scenario where static routes that are configured directly on an L3Out need to be advertised. It will not go into every specific protocol consideration, but rather through how this is implemented in ACI. It will not go into inter-VRF transit routing at this time.
This scenario will use the following topology:
The high-level flow of how 172.16.20.1 would be learned from OSPF and then advertised into EIGRP, and verifications of the whole process and troubleshooting scenarios, are discussed below.
For the 172.16.20.1 route to get advertised into EIGRP, one of the following must be configured:
The above configurations would result in the transit route being advertised but it still needs to have a security policy in place to allow dataplane traffic to flow. As with any EPG to EPG communication, a contract must be in place before traffic is permitted.
Note that duplicate external subnets with the 'External Subnet for External EPG' cannot be configured in the same VRF. When configured, subnets need to be more specific than 0.0.0.0. It is important to configure 'External Subnet for External EPG' only for the L3Out where the route is being received. Don't configure this on the L3Out that should be advertising this route.
It's also important to understand that all transit routes are tagged with a specific VRF Tag. By default, this tag is 4294967295. The Route-Tag policy is configured under 'Tenant > Networking > Protocols > Route-Tag:
This Route Tag policy is then applied to the VRF. The purpose of this tag is essentially to prevent loops. This route tag is applied when the transit route is advertised back out of an L3Out. If these routes are then received back with the same route tag then the route is discarded.
Verify that the route is present on the receiving BL via OSPF
Like the last section, first verify that the BL that should initially receive the correct route.
leaf103# show ip route 172.16.20.1 vrf Prod:Vrf1 IP Route Table for VRF "Prod:Vrf1" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%' in via output denotes VRF 172.16.20.1/32, ubest/mbest: 1/0 *via 10.10.34.3, vlan347, [110/20], 01:25:30, ospf-default, type-2
For now, assume that the advertising L3Out is on a different BL (as in the topology) (later scenarios will discuss where it is on the same BL).
Verify that the route is present in BGP on the receiving OSPF BL
For the OSPF route to be advertised to the external EIGRP router, the route needs to be advertised into BGP on the receiving OSPF BL.
leaf103# show bgp ipv4 unicast 172.16.20.1/32 vrf Prod:Vrf1 BGP routing table information for VRF Prod:Vrf1, address family IPv4 Unicast BGP routing table entry for 172.16.20.1/32, version 30 dest ptr 0xa6f25ad0 Paths: (2 available, best #1) Flags: (0x80c0002 00000000) on xmit-list, is not in urib, exported vpn: version 17206, (0x100002) on xmit-list Multipath: eBGP iBGP Advertised path-id 1, VPN AF advertised path-id 1 Path type: redist 0x408 0x1 ref 0 adv path ref 2, path is valid, is best path AS-Path: NONE, path locally originated 0.0.0.0 (metric 0) from 0.0.0.0 (10.0.0.3) Origin incomplete, MED 20, localpref 100, weight 32768 Extcommunity: RT:65001:2392068 VNID:2392068 COST:pre-bestpath:162:110 VRF advertise information: Path-id 1 not advertised to any peer VPN AF advertise information: Path-id 1 advertised to peers: 10.0.64.64 10.0.72.66 Path-id 2 not advertised to any peer
The route is in BGP.
Verify on the EIGRP BL that should advertise the route that it is installed
leaf102# show ip route 172.16.20.1 vrf Prod:Vrf1 IP Route Table for VRF "Prod:Vrf1" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%' in via output denotes VRF 172.16.20.1/32, ubest/mbest: 2/0 *via 10.0.72.67%overlay-1, [200/20], 00:56:46, bgp-65001, internal, tag 65001 recursive next hop: 10.0.72.67/32%overlay-1 *via 10.0.72.64%overlay-1, [200/20], 00:56:46, bgp-65001, internal, tag 65001 recursive next hop: 10.0.72.64/32%overlay-1
It is installed in the routing table with overlay next-hops pointing to the originating border leaf nodes.
leaf102# acidiag fnvread ID Pod ID Name Serial Number IP Address Role State LastUpdMsgId -------------------------------------------------------------------------------------------------------------- 103 1 a-leaf101 FDO20160TPS 10.0.72.67/32 leaf active 0 104 1 a-leaf103 FDO20160TQ0 10.0.72.64/32 leaf active 0
Verify that the route is advertised on the BL
The route will be advertised by BL 102 as a result of the 'Export Route Control Subnet' flag being set on the configured subnet:
Use the following command to view the route-map that is created as a result of this 'Export Route Control' flag:
leaf102# show ip eigrp vrf Prod:Vrf1 IP-EIGRP AS 101 ID 10.0.0.2 VRF Prod:Vrf1 Process-tag: default Instance Number: 1 Status: running Authentication mode: none Authentication key-chain: none Metric weights: K1=1 K2=0 K3=1 K4=0 K5=0 metric version: 32bit IP proto: 88 Multicast group: 224.0.0.10 Int distance: 90 Ext distance: 170 Max paths: 8 Active Interval: 3 minute(s) Number of EIGRP interfaces: 1 (0 loopbacks) Number of EIGRP passive interfaces: 0 Number of EIGRP peers: 1 Redistributing: static route-map exp-ctx-st-2392068 ospf-default route-map exp-ctx-proto-2392068 direct route-map exp-ctx-st-2392068 coop route-map exp-ctx-st-2392068 bgp-65001 route-map exp-ctx-proto-2392068
To look for the 'BGP > EIGRP redistribution', look at the route-map. But, the route-map itself should be the same regardless of whether the source protocol is OSPF, EIGRP, or BGP. Static routes will be controlled with a different route-map.
leaf102# show route-map exp-ctx-proto-2392068 route-map exp-ctx-proto-2392068, permit, sequence 15801 Match clauses: ip address prefix-lists: IPv4-proto32771-2392068-exc-ext-inferred-export-dst ipv6 address prefix-lists: IPv6-deny-all Set clauses: tag 4294967295 a-leaf102# show ip prefix-list IPv4-proto32771-2392068-exc-ext-inferred-export-dst ip prefix-list IPv4-proto32771-2392068-exc-ext-inferred-export-dst: 1 entries seq 1 permit 172.16.20.1/32
In the above output, the VRF tag is set on this prefix for loop prevention and the subnet configured with 'Export Route Control' is explicitly matched.
As discussed earlier, when the receiving and advertising BLs are different, the route must be advertised through the fabric using BGP. When the BLs are the same, the redistribution or advertisement can be done directly between the protocols on the leaf.
Below are brief descriptions of how this is implemented:
This troubleshooting scenario involves routes that should be learned through one L3Out not being sent out the other L3Out.
As always, check the basics before looking at anything ACI specific.
If all the basic protocol verifications are configured correctly, below are some other common causes for a transit route that is not being advertised.
Possible Cause: No OSPF Area 0
If the affected topology involves two OSP L3Outs on the same border leaf, then there must be an Area 0 for routes to be advertised from one area to another. Look at the "Transit routing between two OSPF L3Outs on the same leaf" bullet above for more details.
Possible Cause: OSPF area is stub or NSSA
This would be seen if the OSPF L3Out is configured with a Stub or NSSA area that is not configured to advertise external LSAs. With OSPF, external LSAs are never advertised into Stub areas. They are advertised into NSSA areas if 'Send Redistributed LSAs into NSSA Area' is selected.
In this scenario the problem is that some routes advertised by an ACI L3Out are not being received back in another L3Out. This scenario could be applicable if the L3Outs are in two separate fabrics and are connected by external routers or if the L3Outs are in different VRFs and the routes are being passed between the VRFs by an external router.
Possible Cause: BL is Configured with the same Router ID in multiple VRFs
From a configuration perspective, a router-id cannot be duplicated within the same VRF. However, it is typically fine to use the same router-id in different VRFs as long as the two VRFs aren't attached to the same routing protocol domains.
Consider the following topology:
The problem here would be that the ACI leaf sees LSAs with its own Router-ID being received, resulting in these not being installed in the OSPF database.
In addition, if the same setup was seen with VPC pairs, LSAs would continuously be added and deleted on some routers. For example, the router would see LSAs coming from its VPC peer with VRF and LSAs coming from the same node (with same Router-ID) that were originated in the other VRF.
To resolve this issue, the user should make sure that a node will have a different, unique router-id within each VRF that it has an L3Out in.
Possible cause: routes from one L3Out in one ACI fabric received on another fabric with same VRF tag
The default route-tag in ACI is always the same unless it is changed. If routes are advertised from one L3Out in one VRF or ACI fabric to another L3Out in another VRF or ACI fabric without changing the default VRF tags, the routes will be dropped by the receiving BLs.
The solution to this scenario is simply to use a unique Route-Tag policy for each VRF in ACI.
This scenario would be seen when transit routes are advertised out an L3Out where they are not intended to be advertised.
Possible cause: usage of 0.0.0.0/0 with 'Aggregate Export'
When an external subnet is configured as 0.0.0.0/0 with 'Export Route Control Subnet' and 'Aggregate Export' the result is that a match all redistribution route-map is installed. In this case all routes on the BL that were learned through OSPF, EIGRP, or BGP are advertised out the L3Out where this is configured.
Below is the route-map that is deployed to the leaf as a result of the Aggregate Export:
leaf102# show ip eigrp vrf Prod:Vrf1 IP-EIGRP AS 101 ID 10.0.0.2 VRF Prod:Vrf1 Process-tag: default Instance Number: 1 Status: running Authentication mode: none Authentication key-chain: none Metric weights: K1=1 K2=0 K3=1 K4=0 K5=0 metric version: 32bit IP proto: 88 Multicast group: 224.0.0.10 Int distance: 90 Ext distance: 170 Max paths: 8 Active Interval: 3 minute(s) Number of EIGRP interfaces: 1 (0 loopbacks) Number of EIGRP passive interfaces: 0 Number of EIGRP peers: 1 Redistributing: static route-map exp-ctx-st-2392068 ospf-default route-map exp-ctx-proto-2392068 direct route-map exp-ctx-st-2392068 coop route-map exp-ctx-st-2392068 bgp-65001 route-map exp-ctx-proto-2392068 Tablemap: route-map exp-ctx-2392068-deny-external-tag , filter-configured Graceful-Restart: Enabled Stub-Routing: Disabled NSF converge time limit/expiries: 120/0 NSF route-hold time limit/expiries: 240/0 NSF signal time limit/expiries: 20/0 Redistributed max-prefix: Disabled selfAdvRtTag: 4294967295 leaf102# show route-map exp-ctx-proto-2392068 route-map exp-ctx-proto-2392068, permit, sequence 19801 Match clauses: ip address prefix-lists: IPv4-proto32771-2392068-agg-ext-inferred-export-dst ipv6 address prefix-lists: IPv6-deny-all Set clauses: tag 4294967295
leaf102# show ip prefix-list IPv4-proto32771-2392068-agg-ext-inferred-export-dst ip prefix-list IPv4-proto32771-2392068-agg-ext-inferred-export-dst: 1 entries seq 1 permit 0.0.0.0/0 le 32
This is the number one cause of routing loops that involve an ACI environment.
In an internal EPG (non-L3Out), contracts are enforced after deriving the pcTag of the source and the pcTag of the destination EPG. The encapsulation VLAN/VXLAN of the packet received on the downlink port is used to drive this pcTag by classing the packet into the EPG. Whenever learning a MAC address or an IP address, it is learnt along with its access encapsulation and the associated EPG pcTag. For more details on pcTag and contract enforcement, please refer to the "Security policies" chapter.
L3Outs also drive a pcTag using its L3Out EPG (External EPG) located under 'Tenant > Networking > L3OUT > Networks > L3OUT-EPG'. However, L3Outs do not rely on VLANs and interfaces to classify packets as such. Classification is instead based on source prefix/subnet in a 'Longest Prefix Match' fashion. Hence, an L3Out EPG can be referred to as a prefix-based EPG. After a packet is classified into an L3Out based on a subnet, it follows a similar policy enforcement pattern as a regular EPG.
The following diagram outlines where the pcTag of a given L3Out EPG can be found within the GUI.
The user is responsible for defining the prefix-based EPG table. This is done using the 'External Subnet for External EPG' subnet scope. Each subnet set with that scope will add an entry in a static Longest Prefix Match (LPM) table. This subnet will point to the pcTag value that will get used for any IP address falling within that prefix.
Th LPM table of prefix-based EPG subnets can be verified on leaf switches using the following command:
vsh -c 'show system internal policy-mgr prefix'
Remarks:
Scenario: A single BGP L3Out in vrf Prod:VRF1 with one L3Out EPG. Prefix 172.16.1.0/24 is being received from an external source so it must be classified into the L3Out EPG.
bdsol-aci32-leaf3# show ip route 172.16.1.0 vrf Prod:VRF1 IP Route Table for VRF "Prod:VRF1" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%' in via output denotes VRF 172.16.1.0/24, ubest/mbest: 1/0 *via 10.0.0.134%Prod:VRF1, [20/0], 00:56:14, bgp-132, external, tag 65002 recursive next hop: 10.0.0.134/32%Prod:VRF1
First, add the subnet to the prefix table.
Verify the programming of the prefix list on the leaf switches that have the VRF of the L3Out:
bdsol-aci32-leaf3# vsh -c ' show system internal policy-mgr prefix ' | egrep "Prod|==|Addr" Vrf-Vni VRF-Id Table-Id Table-State VRF-Name Addr Class Shared Remote Complete ======= ====== =========== ======= ============================ ================================= ====== ====== ====== ======== 2097154 35 0x23 Up Prod:VRF1 0.0.0.0/0 15 True True False 2097154 35 0x23 Up Prod:VRF1 172.16.1.0/24 32772 True True False
The pcTag of the L3Out EPG is 32772 in vrf scope 2097154.
Expanding on the previous example, in this scenario the L3Out is receiving multiple prefixes. While entering each prefix is functionally sound, an alternative option (depending on the intended design) is to accept all prefixes received on the L3Out.
This can be accomplished with the '0.0.0.0/0' prefix.
This results in the following policy-mgr prefix table entry:
bdsol-aci32-leaf3# vsh -c ' show system internal policy-mgr prefix ' | egrep "Prod|==|Addr" Vrf-Vni VRF-Id Table-Id Table-State VRF-Name Addr Class Shared Remote Complete ======= ====== =========== ======= ============================ ================================= ====== ====== ====== ======== 2097154 35 0x23 Up Prod:VRF1 0.0.0.0/0 15 True True False 2097154 35 0x23 Up Prod:VRF1 172.16.1.0/24 32772 True True False
Note that the pcTag assigned to 0.0.0.0/0 uses value 15, not 32772. pcTag 15 is a reserved system pcTag which is only used with 0.0.0.0/0 which acts as wildcard to match all prefixes on an L3Out.
If the VRF has a single L3Out with a single L3Out EPG using the 0.0.0.0/0, then the policy-prefix remains unique and is the easiest approach to catch all.
In this scenario there are multiple L3Out EPGs in the same VRF.
Note: From a prefix-based EPG perspective, the following two configurations will result in equivalent LPM policy-mgr prefix table entries:
In both scenarios, the total number of L3Out EPGs is 2. This means that each one will have its own pcTag and associated subnets.
All pcTags of a given L3Out EPG can be viewed in the GUI at 'Tenant > Operational > Resource id > L3Outs'
In this scenario, the ACI fabric is receiving multiple prefixes from the external routers and the L3Out EPG definition is as follows:
To match this, the config will be defined as follows:
The resulting prefix table entries will be:
bdsol-aci32-leaf3# vsh -c 'show system internal policy-mgr prefix' | egrep "Prod|==|Addr" Vrf-Vni VRF-Id Table-Id Table-State VRF-Name Addr Class Shared Remote Complete ======= ====== =========== ======= ============================ ================================= ====== ====== ====== ======== 2097154 35 0x23 Up Prod:VRF1 0.0.0.0/0 15 True True False 2097154 35 0x23 Up Prod:VRF1 172.16.1.0/24 32772 True True False 2097154 35 0x23 Up Prod:VRF1 172.16.0.0/16 32772 True True False 2097154 35 0x23 Up Prod:VRF1 172.16.2.0/24 32773 True True False
172.16.2.0/24 is assigned to pcTag 32773 (L3OUT-EPG2) and 172.16.0.0/16 is assigned to 32772 (L3OUT-EPG).
In this scenario, the entry for 172.16.1.0/24 is redundant as the /16 supernet is assigned to the same EPG.
Multiple L3Out EPGs is useful when the goal is to apply different contracts to groups of prefixes within a single L3Out. The next example will illustrate how contracts come into play with multiple L3Out EPGs.
This scenario contains the following setup:
The same policymgr prefixes from the previous example will be used:
policy-mgr prefix and zoning-rules:
bdsol-aci32-leaf3# vsh -c ' show system internal policy-mgr prefix ' | egrep "Prod|==|Addr" Vrf-Vni VRF-Id Table-Id Table-State VRF-Name Addr Class Shared Remote Complete ======= ====== =========== ======= ============================ ================================= ====== ====== ====== ======== 2097154 35 0x23 Up Prod:VRF1 0.0.0.0/0 15 True True False 2097154 35 0x23 Up Prod:VRF1 172.16.1.0/24 32772 True True False 2097154 35 0x23 Up Prod:VRF1 172.16.0.0/16 32772 True True False 2097154 35 0x23 Up Prod:VRF1 172.16.2.0/24 32773 True True False
bdsol-aci32-leaf3# show zoning-rule scope 2097154 +---------+--------+--------+----------+----------------+---------+---------+------+----------+----------------------+ | Rule ID | SrcEPG | DstEPG | FilterID | Dir | operSt | Scope | Name | Action | Priority | +---------+--------+--------+----------+----------------+---------+---------+------+----------+----------------------+ | 4326 | 0 | 0 | implicit | uni-dir | enabled | 2097154 | | deny,log | any_any_any(21) | | 4335 | 0 | 16387 | implicit | uni-dir | enabled | 2097154 | | permit | any_dest_any(16) | | 4334 | 0 | 0 | implarp | uni-dir | enabled | 2097154 | | permit | any_any_filter(17) | | 4333 | 0 | 15 | implicit | uni-dir | enabled | 2097154 | | deny,log | any_vrf_any_deny(22) | | 4332 | 0 | 16386 | implicit | uni-dir | enabled | 2097154 | | permit | any_dest_any(16) | | 4342 | 32771 | 32773 | 5 | uni-dir-ignore | enabled | 2097154 | ICMP | permit | fully_qual(7) | | 4343 | 32773 | 32771 | 5 | bi-dir | enabled | 2097154 | ICMP | permit | fully_qual(7) | | 4340 | 32770 | 32772 | 38 | uni-dir | enabled | 2097154 | HTTP | permit | fully_qual(7) | | 4338 | 32772 | 32770 | 37 | uni-dir | enabled | 2097154 | HTTP | permit | fully_qual(7) | +---------+--------+--------+----------+----------------+---------+---------+------+----------+----------------------+
With an ICMP flow between 172.16.2.1 on the external network and 192.168.3.1 in EPG2, fTriage can be used to catch and analyze the flow. In this case, start fTriage on both leaf switch 103 and 104 as traffic may enter either of them:
admin@apic1:~> ftriage route -ii LEAF:103,104 -sip 172.16.2.1 -dip 192.168.3.1 fTriage Status: {"dbgFtriage": {"attributes": {"operState": "InProgress", "pid": "14454", "apicId": "1", "id": "0"}}} Starting ftriage Log file name for the current run is: ftlog_2019-10-02-22-30-41-871.txt 2019-10-02 22:30:41,874 INFO /controller/bin/ftriage route -ii LEAF:103,104 -sip 172.16.2.1 -dip 192.168.3.1 2019-10-02 22:31:28,868 INFO ftriage: main:1165 Invoking ftriage with default password and default username: apic#fallback\\admin 2019-10-02 22:32:15,076 INFO ftriage: main:839 L3 packet Seen on bdsol-aci32-leaf3 Ingress: Eth1/12 (Po1) Egress: Eth1/12 (Po1) Vnid: 11365 2019-10-02 22:32:15,295 INFO ftriage: main:242 ingress encap string vlan-2551 2019-10-02 22:32:17,839 INFO ftriage: main:271 Building ingress BD(s), Ctx 2019-10-02 22:32:20,583 INFO ftriage: main:294 Ingress BD(s) Prod:VRF1:l3out-BGP:vlan-2551 2019-10-02 22:32:20,584 INFO ftriage: main:301 Ingress Ctx: Prod:VRF1 2019-10-02 22:32:20,693 INFO ftriage: pktrec:490 bdsol-aci32-leaf3: Collecting transient losses snapshot for LC module: 1 2019-10-02 22:32:38,933 INFO ftriage: nxos:1404 bdsol-aci32-leaf3: nxos matching rule id:4343 scope:34 filter:5 2019-10-02 22:32:39,931 INFO ftriage: main:522 Computed egress encap string vlan-2502 2019-10-02 22:32:39,933 INFO ftriage: main:313 Building egress BD(s), Ctx 2019-10-02 22:32:41,796 INFO ftriage: main:331 Egress Ctx Prod:VRF1 2019-10-02 22:32:41,796 INFO ftriage: main:332 Egress BD(s): Prod:BD2 2019-10-02 22:32:48,636 INFO ftriage: main:933 SIP 172.16.2.1 DIP 192.168.3.1 2019-10-02 22:32:48,637 INFO ftriage: unicast:973 bdsol-aci32-leaf3: <- is ingress node 2019-10-02 22:32:51,257 INFO ftriage: unicast:1202 bdsol-aci32-leaf3: Dst EP is local 2019-10-02 22:32:54,129 INFO ftriage: misc:657 bdsol-aci32-leaf3: EP if(Po1) same as egr if(Po1) 2019-10-02 22:32:55,348 INFO ftriage: misc:657 bdsol-aci32-leaf3: DMAC(00:22:BD:F8:19:FF) same as RMAC(00:22:BD:F8:19:FF) 2019-10-02 22:32:55,349 INFO ftriage: misc:659 bdsol-aci32-leaf3: L3 packet getting routed/bounced in SUG 2019-10-02 22:32:55,596 INFO ftriage: misc:657 bdsol-aci32-leaf3: Dst IP is present in SUG L3 tbl 2019-10-02 22:32:55,896 INFO ftriage: misc:657 bdsol-aci32-leaf3: RW seg_id:11365 in SUG same as EP segid:11365 2019-10-02 22:33:02,150 INFO ftriage: main:961 Packet is Exiting fabric with peer-device: bdsol-aci32-n3k-3 and peer-port: Ethernet1/16
fTriage confirms the zoning-rule hit against the ICMP rule from L3OUT_EPG2 to EPG:
2019-10-02 22:32:38,933 INFO ftriage: nxos:1404 bdsol-aci32-leaf3: nxos matching rule id:4343 scope:34 filter:5
With ICMP traffic sourced from 172.16.1.1 (L3OUT-EPG) towards 192.168.3.1 (EPG2), expect a policy drop.
admin@apic1:~> ftriage route -ii LEAF:103,104 -sip 172.16.1.1 -dip 192.168.3.1 fTriage Status: {"dbgFtriage": {"attributes": {"operState": "InProgress", "pid": "15139", "apicId": "1", "id": "0"}}} Starting ftriage Log file name for the current run is: ftlog_2019-10-02-22-39-15-050.txt 2019-10-02 22:39:15,056 INFO /controller/bin/ftriage route -ii LEAF:103,104 -sip 172.16.1.1 -dip 192.168.3.1 2019-10-02 22:40:03,523 INFO ftriage: main:1165 Invoking ftriage with default password and default username: apic#fallback\\admin 2019-10-02 22:40:43,338 ERROR ftriage: unicast:234 bdsol-aci32-leaf3: L3 packet getting fwd dropped, checking drop reason 2019-10-02 22:40:43,339 ERROR ftriage: unicast:234 bdsol-aci32-leaf3: L3 packet getting fwd dropped, checking drop reason SECURITY_GROUP_DENY condition setcast:236 bdsol-aci32-leaf3: Drop reason - SECURITY_GROUP_DENY condition set 2019-10-02 22:40:43,340 INFO ftriage: unicast:252 bdsol-aci32-leaf3: policy drop flow sclass:32772 dclass:32771 sg_label:34 proto:1 2019-10-02 22:40:43,340 INFO ftriage: main:681 : Ftriage Completed with hunch: None fTriage Status: {"dbgFtriage": {"attributes": {"operState": "Idle", "pid": "0", "apicId": "0", "id": "0"}}}
fTriage confirms that the packet is dropped with the SECURITY_GROUP_DENY (policy drop) reason and that the derived source pcTag is 32772 and destination pcTag is 32771. Checking this against zoning-rules, there are clearly no entries between those EPG.
bdsol-aci32-leaf3# show zoning-rule scope 2097154 src-epg 32772 dst-epg 32771 +---------+--------+--------+----------+-----+--------+-------+------+--------+----------+ | Rule ID | SrcEPG | DstEPG | FilterID | Dir | operSt | Scope | Name | Action | Priority | +---------+--------+--------+----------+-----+--------+-------+------+--------+----------+ +---------+--------+--------+----------+-----+--------+-------+------+--------+----------+
The scenario is setup similarly to example 3 (L3Out and L3Out EPG definitions), but the network defined on both L3Out EPGs is 0.0.0.0/0.
Contract configuration is the following:
This configuration may look ideal in the case where the external network is advertising many prefixes, but there are at least two chunks of prefixes that follow different allowed flow patterns. In this example, one prefix should only allow ICMP1 and the other should only allow ICMP2.
Despite to using '0.0.0.0/0' twice in the same VRF, only one prefix gets programmed in the policy-mgr prefix table:
bdsol-aci32-leaf3# vsh -c ' show system internal policy-mgr prefix ' | egrep "Prod|==|Addr" Vrf-Vni VRF-Id Table-Id Table-State VRF-Name Addr Class Shared Remote Complete ======= ====== =========== ======= ============================ ================================= ====== ====== ====== ======== 2097154 35 0x23 Up Prod:VRF1
Two flows reexamined below. Based on the contract configuration above, the following is expected:
Run fTriage with an ICMP flow from 172.16.2.1 (L3OUT-EPG2) to 192.168.3.1 (EPG2 — pcTag 32771).
Starting ftriage Log file name for the current run is: ftlog_2019-10-02-23-11-14-298.txt 2019-10-02 23:11:14,302 INFO /controller/bin/ftriage route -ii LEAF:103,104 -sip 172.16.2.1 -dip 192.168.3.1 2019-10-02 23:12:00,887 INFO ftriage: main:1165 Invoking ftriage with default password and default username: apic#fallback\\admin 2019-10-02 23:12:44,565 INFO ftriage: main:839 L3 packet Seen on bdsol-aci32-leaf3 Ingress: Eth1/12 (Po1) Egress: Eth1/12 (Po1) Vnid: 11365 2019-10-02 23:12:44,782 INFO ftriage: main:242 ingress encap string vlan-2551 2019-10-02 23:12:47,260 INFO ftriage: main:271 Building ingress BD(s), Ctx 2019-10-02 23:12:50,041 INFO ftriage: main:294 Ingress BD(s) Prod:VRF1:l3out-BGP:vlan-2551 2019-10-02 23:12:50,042 INFO ftriage: main:301 Ingress Ctx: Prod:VRF1 2019-10-02 23:12:50,151 INFO ftriage: pktrec:490 bdsol-aci32-leaf3: Collecting transient losses snapshot for LC module: 1 2019-10-02 23:13:08,595 INFO ftriage: nxos:1404 bdsol-aci32-leaf3: nxos matching rule id:4336 scope:34 filter:5 2019-10-02 23:13:09,608 INFO ftriage: main:522 Computed egress encap string vlan-2502 2019-10-02 23:13:09,609 INFO ftriage: main:313 Building egress BD(s), Ctx 2019-10-02 23:13:11,449 INFO ftriage: main:331 Egress Ctx Prod:VRF1 2019-10-02 23:13:11,449 INFO ftriage: main:332 Egress BD(s): Prod:BD2 2019-10-02 23:13:18,383 INFO ftriage: main:933 SIP 172.16.2.1 DIP 192.168.3.1 2019-10-02 23:13:18,384 INFO ftriage: unicast:973 bdsol-aci32-leaf3: <- is ingress node 2019-10-02 23:13:21,078 INFO ftriage: unicast:1202 bdsol-aci32-leaf3: Dst EP is local 2019-10-02 23:13:23,926 INFO ftriage: misc:657 bdsol-aci32-leaf3: EP if(Po1) same as egr if(Po1) 2019-10-02 23:13:25,216 INFO ftriage: misc:657 bdsol-aci32-leaf3: DMAC(00:22:BD:F8:19:FF) same as RMAC(00:22:BD:F8:19:FF) 2019-10-02 23:13:25,217 INFO ftriage: misc:659 bdsol-aci32-leaf3: L3 packet getting routed/bounced in SUG 2019-10-02 23:13:25,465 INFO ftriage: misc:657 bdsol-aci32-leaf3: Dst IP is present in SUG L3 tbl 2019-10-02 23:13:25,757 INFO ftriage: misc:657 bdsol-aci32-leaf3: RW seg_id:11365 in SUG same as EP segid:11365 2019-10-02 23:13:32,235 INFO ftriage: main:961 Packet is Exiting fabric with peer-device: bdsol-aci32-n3k-3 and peer-port: Ethernet1/16
This flow is allowed (as expected) by zoning-rule 4336.
Run fTriage with an ICMP flow from 172.16.2.1 (L3OUT-EPG2) to 192.168.1.1 (EPG1 — pcTag 32770):
admin@apic1:~> ftriage route -ii LEAF:103,104 -sip 172.16.2.1 -dip 192.168.1.1 fTriage Status: {"dbgFtriage": {"attributes": {"operState": "InProgress", "pid": "31500", "apicId": "1", "id": "0"}}} Starting ftriage Log file name for the current run is: ftlog_2019-10-02-23-53-03-478.txt 2019-10-02 23:53:03,482 INFO /controller/bin/ftriage route -ii LEAF:103,104 -sip 172.16.2.1 -dip 192.168.1.1 2019-10-02 23:53:50,014 INFO ftriage: main:1165 Invoking ftriage with default password and default username: apic#fallback\\admin 2019-10-02 23:54:39,199 INFO ftriage: main:839 L3 packet Seen on bdsol-aci32-leaf3 Ingress: Eth1/12 (Po1) Egress: Eth1/12 (Po1) Vnid: 11364 2019-10-02 23:54:39,417 INFO ftriage: main:242 ingress encap string vlan-2551 2019-10-02 23:54:41,962 INFO ftriage: main:271 Building ingress BD(s), Ctx 2019-10-02 23:54:44,765 INFO ftriage: main:294 Ingress BD(s) Prod:VRF1:l3out-BGP:vlan-2551 2019-10-02 23:54:44,766 INFO ftriage: main:301 Ingress Ctx: Prod:VRF1 2019-10-02 23:54:44,875 INFO ftriage: pktrec:490 bdsol-aci32-leaf3: Collecting transient losses snapshot for LC module: 1 2019-10-02 23:55:02,905 INFO ftriage: nxos:1404 bdsol-aci32-leaf3: nxos matching rule id:4341 scope:34 filter:5 2019-10-02 23:55:04,525 INFO ftriage: main:522 Computed egress encap string vlan-2501 2019-10-02 23:55:04,526 INFO ftriage: main:313 Building egress BD(s), Ctx 2019-10-02 23:55:06,390 INFO ftriage: main:331 Egress Ctx Prod:VRF1 2019-10-02 23:55:06,390 INFO ftriage: main:332 Egress BD(s): Prod:BD1 2019-10-02 23:55:13,571 INFO ftriage: main:933 SIP 172.16.2.1 DIP 192.168.1.1 2019-10-02 23:55:13,572 INFO ftriage: unicast:973 bdsol-aci32-leaf3: <- is ingress node 2019-10-02 23:55:16,159 INFO ftriage: unicast:1202 bdsol-aci32-leaf3: Dst EP is local 2019-10-02 23:55:18,949 INFO ftriage: misc:657 bdsol-aci32-leaf3: EP if(Po1) same as egr if(Po1) 2019-10-02 23:55:20,126 INFO ftriage: misc:657 bdsol-aci32-leaf3: DMAC(00:22:BD:F8:19:FF) same as RMAC(00:22:BD:F8:19:FF) 2019-10-02 23:55:20,126 INFO ftriage: misc:659 bdsol-aci32-leaf3: L3 packet getting routed/bounced in SUG 2019-10-02 23:55:20,395 INFO ftriage: misc:657 bdsol-aci32-leaf3: Dst IP is present in SUG L3 tbl 2019-10-02 23:55:20,687 INFO ftriage: misc:657 bdsol-aci32-leaf3: RW seg_id:11364 in SUG same as EP segid:11364 2019-10-02 23:55:26,982 INFO ftriage: main:961 Packet is Exiting fabric with peer-device: bdsol-aci32-n3k-3 and peer-port: Ethernet1/16
This flow is allowed (unexpected) by zoning-rule 4341. The zoning-rules must now be analyzed to understand why.
The zoning-rules corresponding to the last 2 tests are below:
+---------+--------+--------+----------+---------+---------+---------+-------+----------+----------------------+ | Rule ID | SrcEPG | DstEPG | FilterID | Dir | operSt | Scope | Name | Action | Priority | +---------+--------+--------+----------+---------+---------+---------+-------+----------+----------------------+ | 4326 | 0 | 0 | implicit | uni-dir | enabled | 2097154 | | deny,log | any_any_any(21) | | 4335 | 0 | 16387 | implicit | uni-dir | enabled | 2097154 | | permit | any_dest_any(16) | | 4334 | 0 | 0 | implarp | uni-dir | enabled | 2097154 | | permit | any_any_filter(17) | | 4333 | 0 | 15 | implicit | uni-dir | enabled | 2097154 | | deny,log | any_vrf_any_deny(22) | | 4332 | 0 | 16386 | implicit | uni-dir | enabled | 2097154 | | permit | any_dest_any(16) | | 4339 | 32770 | 15 | 5 | uni-dir | enabled | 2097154 | ICMP2 | permit | fully_qual(7) | | 4341 | 49153 | 32770 | 5 | uni-dir | enabled | 2097154 | ICMP2 | permit | fully_qual(7) | | 4337 | 32771 | 15 | 5 | uni-dir | enabled | 2097154 | ICMP1 | permit | fully_qual(7) | | 4336 | 49153 | 32771 | 5 | uni-dir | enabled | 2097154 | ICMP1 | permit | fully_qual(7) | +---------+--------+--------+----------+---------+---------+---------+-------+----------+----------------------+
Both flows derive the src pcTag of 49153. This is the pcTag of the VRF. This can be verified in the UI:
The following happens when the 0.0.0.0/0 prefix is in use with an L3Out:
The contract_parser script gives a holistic view of the zoning-rules:
bdsol-aci32-leaf3# contract_parser.py --vrf Prod:VRF1 Key: [prio:RuleId] [vrf:{str}] action protocol src-epg [src-l4] dst-epg [dst-l4] [flags][contract:{str}] [hit=count] [7:4339] [vrf:Prod:VRF1] permit ip icmp tn-Prod/ap-App/epg-EPG1(32770) pfx-0.0.0.0/0(15) [contract:uni/tn-Prod/brc-ICMP2] [hit=0] [7:4337] [vrf:Prod:VRF1] permit ip icmp tn-Prod/ap-App/epg-EPG2(32771) pfx-0.0.0.0/0(15) [contract:uni/tn-Prod/brc-ICMP] [hit=0] [7:4341] [vrf:Prod:VRF1] permit ip icmp tn-Prod/vrf-VRF1(49153) tn-Prod/ap-App/epg-EPG1(32770) [contract:uni/tn-Prod/brc-ICMP2] [hit=270] [7:4336] [vrf:Prod:VRF1] permit ip icmp tn-Prod/vrf-VRF1(49153) tn-Prod/ap-App/epg-EPG2(32771) [contract:uni/tn-Prod/brc-ICMP] [hit=0]
The ELAM Assistant App gives another method to confirm the source and destination pcTag of live traffic flows.
The screen shot below shows the ELAM result for traffic from pcTag 32771 to pcTag 49153.
The usage of 0.0.0.0/0 must be carefully tracked within a VRF as every L3Out using that subnet will inherit the contracts applied to every other L3Out using it. This will likely lead to unplanned permit flows.
This section will discuss how to troubleshoot route-advertisement in Shared L3Out configurations. The term 'Shared L3Out' refers to the scenario where an L3Out is in one VRF but an internal EPG having a contract with the L3Out is in a another VRF. With Shared L3Outs, the route-leaking is being done internally to the ACI fabric.
This section will not go into deep detail about security policy troubleshooting. For that refer to the "Security Policies" chapter of this book. This section will also not talk in detail about External Policy Prefix classification for security purposes. Refer to the "Contract and L3Out" section in the "external forwarding" chapter.
This section uses the following topology for our examples.
At a high level, the following configurations must be in place for a Shared L3Out to function:
The next section will go into detail about how leaked routes are advertised and learned in ACI.
This section will outline the path of a learned external route as it is advertised into the fabric.
This command will show the external route learned from OSPF:
leaf103# show ip route 172.16.20.1/32 vrf Prod:Vrf1 IP Route Table for VRF "Prod:Vrf1" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%' in via output denotes VRF 172.16.20.1/32, ubest/mbest: 1/0 *via 10.10.34.3, vlan347, [110/20], 03:59:59, ospf-default, type-2
Next, the route must be imported into BGP. By default, all external routes should be imported into BGP.
The route must be in the BGP VPNv4 Address-family with a route-target to be distributed throughout the fabric. The route-target is a BGP extended community exported by the external VRF and imported by any internal VRFs that needs to receive the path.
Next, verify the route-target that is being exported by the external VRF on the BL.
leaf103# show bgp process vrf Prod:Vrf1 Information regarding configured VRFs: BGP Information for VRF Prod:Vrf1 VRF Type : System VRF Id : 85 VRF state : UP VRF configured : yes VRF refcount : 1 VRF VNID : 2392068 Router-ID : 10.0.0.3 Configured Router-ID : 10.0.0.3 Confed-ID : 0 Cluster-ID : 0.0.0.0 MSITE Cluster-ID : 0.0.0.0 No. of configured peers : 1 No. of pending config peers : 0 No. of established peers : 0 VRF RD : 101:2392068 VRF EVPN RD : 101:2392068 ... Wait for IGP convergence is not configured Export RT list: 65001:2392068 Import RT list: 65001:2392068 Label mode: per-prefix
The above output shows that any paths advertised from the external VRF into VPNv4 should receive a route-target of 65001:2392068.
Next, verify the bgp path:
leaf103# show bgp ipv4 unicast 172.16.20.1/32 vrf Prod:Vrf1 BGP routing table information for VRF Prod:Vrf1, address family IPv4 Unicast BGP routing table entry for 172.16.20.1/32, version 30 dest ptr 0xa6f25ad0 Paths: (2 available, best #1) Flags: (0x80c0002 00000000) on xmit-list, is not in urib, exported vpn: version 17206, (0x100002) on xmit-list Multipath: eBGP iBGP Advertised path-id 1, VPN AF advertised path-id 1 Path type: redist 0x408 0x1 ref 0 adv path ref 2, path is valid, is best path AS-Path: NONE, path locally originated 0.0.0.0 (metric 0) from 0.0.0.0 (10.0.0.3) Origin incomplete, MED 20, localpref 100, weight 32768 Extcommunity: RT:65001:2392068 VNID:2392068 COST:pre-bestpath:162:110 VRF advertise information: Path-id 1 not advertised to any peer VPN AF advertise information: Path-id 1 advertised to peers: 10.0.64.64 10.0.72.66 Path-id 2 not advertised to any peer
The above output shows that the path has the correct route-target. The VPNv4 path can also be verified by using 'show bgp vpnv4 unicast 172.16.20.1 vrf overlay-1' command.
For the internal EPG leaf to install the BL-advertised route, it must import the route-target (mentioned above) into the internal VRF. The internal VRF's BGP process can be checked to validate this:
leaf101# show bgp process vrf Prod:Vrf2 Information regarding configured VRFs: BGP Information for VRF Prod:Vrf2 VRF Type : System VRF Id : 54 VRF state : UP VRF configured : yes VRF refcount : 0 VRF VNID : 2916352 Router-ID : 192.168.1.1 Configured Router-ID : 0.0.0.0 Confed-ID : 0 Cluster-ID : 0.0.0.0 MSITE Cluster-ID : 0.0.0.0 No. of configured peers : 0 No. of pending config peers : 0 No. of established peers : 0 VRF RD : 102:2916352 VRF EVPN RD : 102:2916352 ... Wait for IGP convergence is not configured Import route-map 2916352-shared-svc-leak Export RT list: 65001:2916352 Import RT list: 65001:2392068 65001:2916352
The above output shows the internal VRF importing the route-target that is exported by the external VRF. Additionally, there is an 'Import Route-Map' that is referenced. The import route-map includes the specific prefixes that are defined in the shared L3Out with the 'Shared Route Control Subnet' flag.
The route-map contents can be checked to ensure it includes the external prefix:
leaf101# show route-map 2916352-shared-svc-leak route-map 2916352-shared-svc-leak, deny, sequence 1 Match clauses: pervasive: 2 Set clauses: route-map 2916352-shared-svc-leak, permit, sequence 2 Match clauses: extcommunity (extcommunity-list filter): 2916352-shared-svc-leak Set clauses: route-map 2916352-shared-svc-leak, permit, sequence 1000 Match clauses: ip address prefix-lists: IPv4-2392068-16387-5511-2916352-shared-svc-leak ipv6 address prefix-lists: IPv6-deny-all Set clauses: a-leaf101# show ip prefix-list IPv4-2392068-16387-5511-2916352-shared-svc-leak ip prefix-list IPv4-2392068-16387-5511-2916352-shared-svc-leak: 1 entries seq 1 permit 172.16.20.1/32
The above output shows the import route-map which includes the subnet to be imported.
The final verifications include checking that the route is in the BGP table and that it is installed in the routing table.
BGP table on server leaf:
leaf101# show bgp ipv4 unicast 172.16.20.1/32 vrf Prod:Vrf2 BGP routing table information for VRF Prod:Vrf2, address family IPv4 Unicast BGP routing table entry for 172.16.20.1/32, version 3 dest ptr 0xa763add0 Paths: (2 available, best #1) Flags: (0x08001a 00000000) on xmit-list, is in urib, is best urib route, is in HW vpn: version 10987, (0x100002) on xmit-list Multipath: eBGP iBGP Advertised path-id 1, VPN AF advertised path-id 1 Path type: internal 0xc0000018 0x40 ref 56506 adv path ref 2, path is valid, is best path Imported from 10.0.72.64:5:172.16.20.1/32 AS-Path: NONE, path sourced internal to AS 10.0.72.64 (metric 3) from 10.0.64.64 (192.168.1.102) Origin incomplete, MED 20, localpref 100, weight 0 Received label 0 Received path-id 1 Extcommunity: RT:65001:2392068 VNID:2392068 COST:pre-bestpath:162:110 Originator: 10.0.72.64 Cluster list: 192.168.1.102
The route is imported into the internal VRF BGP table and has the expected route-target.
The installed routes can be verified:
leaf101# vsh -c "show ip route 172.16.20.1/32 detail vrf Prod:Vrf2" IP Route Table for VRF "Prod:Vrf2" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%' in via output denotes VRF 172.16.20.1/32, ubest/mbest: 2/0 *via 10.0.72.64%overlay-1, [200/20], 01:00:51, bgp-65001, internal, tag 65001 (mpls-vpn) MPLS[0]: Label=0 E=0 TTL=0 S=0 (VPN) client-specific data: 548 recursive next hop: 10.0.72.64/32%overlay-1 extended route information: BGP origin AS 65001 BGP peer AS 65001 rw-vnid: 0x248004 table-id: 0x36 rw-mac: 0 *via 10.0.72.67%overlay-1, [200/20], 01:00:51, bgp-65001, internal, tag 65001 (mpls-vpn) MPLS[0]: Label=0 E=0 TTL=0 S=0 (VPN) client-specific data: 54a recursive next hop: 10.0.72.67/32%overlay-1 extended route information: BGP origin AS 65001 BGP peer AS 65001 rw-vnid: 0x248004 table-id: 0x36 rw-mac: 0
The above output uses a specific 'vsh -c' command to get the 'detail' output. The 'detail' flag includes the rewrite VXLAN VNID. This is the VXLAN VNID of the external VRF. When the BL receives dataplane traffic with this VNID, it knows to make the forwarding decision in the external VRF.
The rw-vnid value is in hex, so converting to decimal will get the VRF VNID of 2392068. Search for the corresponding VRF using 'show system internal epm vrf all | grep 2392068' on the leaf. A global search can be performed on an APIC using the 'moquery -c fvCtx -f 'fv.Ctx.seg=="2392068"'' command.
The next-hop's IP should also point to the BL PTEPs and the '%overlay-1' indicates that the route lookup for the next-hop is in the overlay VRF.
As in previous sections, advertising internal BD subnets out a shared L3Out is handled by the following:
leaf103# vsh -c "show ip route 192.168.1.0 detail vrf Prod:Vrf1" IP Route Table for VRF "Prod:Vrf1" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] '%' in via output denotes VRF 192.168.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive *via 10.0.120.34%overlay-1, [1/0], 00:55:27, static, tag 4294967292 recursive next hop: 10.0.120.34/32%overlay-1 vrf crossing information: VNID:0x2c8000 ClassId:0 Flush#:0
Notice that in the above output the VNID of the internal VRF is set for the rewrite. The next-hop is also set to the proxy-v4-anycast address.
The above route is advertised externally through the same route-maps that are demonstrated in the "Route Advertisement" section.
If a BD subnet is set to 'Advertise Externally', it is redistributed into every L3Out's external protocol that the internal EPG has a contract relationship with.
This scenario has multiple L3Outs in the external VRF and an internal EPG is receiving a route from an L3Out where the network is not defined with the 'shared' scope options.
Consider the following figure:
The BGP import-map with the prefix-list programmed from the 'Shared Route Control Subnet' flags is applied at the VRF level. If one L3Out in VRF1 has a subnet with 'Shared Route Control Subnet', then all routes received on L3Outs within VRF1 that match this Shared Route Control Subnet will get imported into VRF2.
The above design can result in unexpected traffic flows. If there are no contracts between the internal EPG and the unexpected advertising L3Out EPG, then there will be traffic drops.
Revision | Publish Date | Comments |
---|---|---|
1.0 |
10-Aug-2022 |
Initial Release |