Troubleshoot ACI L3Out - Subnet 0.0.0.0/0 and System PcTag 15
Available Languages
Contents
Introduction
This document describes the PcTag derivation of the 0.0.0.0/0 subnet when defined in an L3Out EPG.
Background Information
The "L3Out EPG with 0.0.0.0/0 subnet" section of the ACI Contract Guide summarizes 0.0.0.0/0 with "External Subnets for the External EPG" scope traffic classification as:
- Traffic sourced from an L3Out which is Longest Prefix Matched to a configured 0.0.0.0/0 subnet is assigned the source class ID (sclass) of the VRF PcTag.
- Traffic destined to an L3Out EPG which is Longest Prefix Matched to a configured 0.0.0.0/0 subnet is assigned the destination class ID (dclass) of 15, a System PcTag.
The "An exception for 0.0.0.0/0 with External Subnets for the External EPG" section of the ACI L3Out Whitepaper contains a warning:
"...Although it is not recommended, you can configure 0.0.0.0/0 with 'External Subnets for the External EPG' in multiple L3Out EPGs in the same VRF... While this configuration is allowed, an unintended contract deployment occurs..."
This article dives into that unintended contract deployment.
Configure
Topology Diagram
Configuration Highlights
- Leaf Nodes 301 and 303 are Border Leaf Nodes
- Leaf Node 302 is a Non-Border Leaf
- L3Out-1-EEPG, on Border Leaf 301, has a 0.0.0.0/0 subnet with "External Subnets for the External EPG"
- L3Out-1-EEPG provides a contract
- EPG, on Non-Border Leaf 302, consumes the same contract
Verify
VRF with "Ingress" Policy Enforcement
Non-Border Leaf Zoning-Rules
As highlighted in the Background Information section, traffic destined to networks behind this L3Out which Longest Prefix Match on the configured 0.0.0.0/0 subnet get a destination class (pcTag) of 15.
This is the zoning-rules table on Non-Border Leaf 302 for VRF "v1" (segment ID 2129920):
Leaf-302# show zoning-rule scope 2129920 +---------+--------+--------+----------+---------+---------+---------+------------------+----------+----------------------+ | Rule ID | SrcEPG | DstEPG | FilterID | Dir | operSt | Scope | Name | Action | Priority | +---------+--------+--------+----------+---------+---------+---------+------------------+----------+----------------------+ | 4107 | 0 | 0 | implarp | uni-dir | enabled | 2129920 | | permit | any_any_filter(17) | | 4106 | 0 | 0 | implicit | uni-dir | enabled | 2129920 | | deny,log | any_any_any(21) | | 4105 | 0 | 49155 | implicit | uni-dir | enabled | 2129920 | | permit | any_dest_any(16) | | 4108 | 0 | 15 | implicit | uni-dir | enabled | 2129920 | | deny,log | any_vrf_any_deny(22) | | 4112 | 16386 | 49156 | default | uni-dir | enabled | 2129920 | tn1:EPG_to_L3Out | permit | src_dst_any(9) | | 4111 | 49156 | 15 | default | uni-dir | enabled | 2129920 | tn1:EPG_to_L3Out | permit | src_dst_any(9) | +---------+--------+--------+----------+---------+---------+---------+------------------+----------+----------------------+
There are two rules installed as a result of the contract between L3Out-1-EEPG and EPG (49156):
- Rule 4112 is for external traffic sourced from the L3Out EPG with 0.0.0.0/0 LPM destined to the EPG.
- The traffic flow is classified with the sclass of the VRF PcTag (16386) and dclass of EPG (49156) .
- Rule 4111 is for traffic sourced from the EPG destined to the L3Out EPG with 0.0.0.0/0 LPM.
- The traffic flow is classified with the sclass of EPG (49156) and the dclass of System PcTag 15
Border Leaf Zoning-Rules
Border Leaf Node 301 does not have the same Zoning-Rules as Non-Border Leaf Node 302 due to VRF Policy Enforcement set to 'Ingress' (default value). Policy for these types of flows is expected to be applied on Non-Border Leaf Nodes.
Leaf-301# show zoning-rule scope 2129920 +---------+--------+--------+----------+---------+---------+---------+------+----------+----------------------+ | Rule ID | SrcEPG | DstEPG | FilterID | Dir | operSt | Scope | Name | Action | Priority | +---------+--------+--------+----------+---------+---------+---------+------+----------+----------------------+ | 4105 | 0 | 0 | implarp | uni-dir | enabled | 2129920 | | permit | any_any_filter(17) | | 4107 | 0 | 0 | implicit | uni-dir | enabled | 2129920 | | deny,log | any_any_any(21) | | 4106 | 0 | 15 | implicit | uni-dir | enabled | 2129920 | | deny,log | any_vrf_any_deny(22) | | 4108 | 0 | 16387 | implicit | uni-dir | enabled | 2129920 | | permit | any_dest_any(16) | +---------+--------+--------+----------+---------+---------+---------+------+----------+----------------------+
*No entry for 16386 to 49156 , or 49156 to 15*
EPG to L3Out ELAM
A ping from EPG endpoint 192.168.1.1 to the IP behind L3Out-1-EEPG is successful:
Host# ping 10.1.1.1 count 10000 int 1 PING 10.1.1.1 (10.1.1.1): 56 data bytes 64 bytes from 10.1.1.1: icmp_seq=0 ttl=252 time=1.063 ms 64 bytes from 10.1.1.1: icmp_seq=1 ttl=252 time=0.92 ms 64 bytes from 10.1.1.1: icmp_seq=2 ttl=252 time=0.963 ms
An ELAM for EPG to L3Out traffic on Non-Border Leaf 302 (EPG gateway) confirms:
- The packet has the expected source and destination IPs: Source IP:192.168.1.1, Destination IP: 10.1.1.1
- The source class (sclass) is the EPG PcTag 49156
- The destination class (dclass) is System PcTag 15, as 10.1.1.0/24 Longest Prefix Matches the 0.0.0.0/0 Subnet on L3Out-1-EEPG
- Policy was applied on this node 302, the Non-Border Leaf Node.
Leaf-302# ereport
======================================================================================================================================================
Captured Packet
======================================================================================================================================================
...snip...
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L2 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
Destination MAC : 0022.BDF8.19FF
Source MAC : AAAA.AAAA.2222
802.1Q tag is valid : yes( 0x1 )
CoS : 0( 0x0 )
Access Encap VLAN : 192( 0xC0 )
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
L3 Type : IPv4
...
IP Protocol Number : ICMP
IP CheckSum : 63781( 0xF925 )
Destination IP : 10.1.1.1
Source IP : 192.168.1.1
...
======================================================================================================================================================
Contract Lookup ( FPC )
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Lookup Key
------------------------------------------------------------------------------------------------------------------------------------------------------
IP Protocol : ICMP( 0x1 )
L4 Src Port : 2048( 0x800 )
L4 Dst Port : 43014( 0xA806 )
sclass (src pcTag) : 49156( 0xC004 )
dclass (dst pcTag) : 15( 0xF )
src pcTag is from local table : yes
...
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Result
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Drop : no
Contract Logging : no
Contract Applied : yes
Contract Hit : yes
Contract Aclqos Stats Index : 81875
( show sys int aclqos zoning-rules | grep -B 9 "Idx: 81875" )
The command given by ereport can be entered for additional validation of the Zoning-Rule that was hit:
module-1(DBG-elam-insel6)# show sys int aclqos zoning-rules | grep -B 9 "Idx: 81875"
===========================================
Rule ID: 4111 Scope 6 Src EPG: 49156 Dst EPG: 15 Filter 65535
unit_id: 0
=== Region priority: 2462 (rule prio: 9 entry: 158)===
sw_index = 46 | hw_index = 45 | stats_idx = 81875
Curr TCAM resource:
=============================
=== SDK Info ===
Result/Stats Idx: 81875
L3Out to EPG ELAM
The return flow gets policy applied on the Non-Border Leaf Node 302. This is expected when VRF Policy Enforcement is set to "Ingress".
Leaf-302# ereport
... ------------------------------------------------------------------------------------------------------------------------------------------------------ Inner L3 Header ------------------------------------------------------------------------------------------------------------------------------------------------------ L3 Type : IPv4 DSCP : 0 Don't Fragment Bit : 0x0 TTL : 254 IP Protocol Number : ICMP Destination IP : 192.168.1.1 Source IP : 10.1.1.1 ====================================================================================================================================================== Contract Lookup ( FPC ) ====================================================================================================================================================== ------------------------------------------------------------------------------------------------------------------------------------------------------ Contract Lookup Key ------------------------------------------------------------------------------------------------------------------------------------------------------ IP Protocol : ICMP( 0x1 ) L4 Src Port : 0( 0x0 ) L4 Dst Port : 60691( 0xED13 ) sclass (src pcTag) : 16386( 0x4002 ) dclass (dst pcTag) : 49156( 0xC004 ) src pcTag is from local table : no derived from group-id in iVxLAN header of incoming packet Unknown Unicast / Flood Packet : no If yes, Contract is not applied here because it is flooded ------------------------------------------------------------------------------------------------------------------------------------------------------ Contract Result ------------------------------------------------------------------------------------------------------------------------------------------------------ Contract Drop : no Contract Logging : no Contract Applied : yes Contract Hit : yes Contract Aclqos Stats Index : 81874 ( show sys int aclqos zoning-rules | grep -B 9 "Idx: 81874" )
Further validation:
module-1(DBG-elam-insel14)# show sys int aclqos zoning-rules | grep -B 9 "Idx: 81874"
===========================================
Rule ID: 4112 Scope 6 Src EPG: 16386 Dst EPG: 49156 Filter 65535
unit_id: 0
=== Region priority: 2462 (rule prio: 9 entry: 158)===
sw_index = 47 | hw_index = 46 | stats_idx = 81874
Curr TCAM resource:
=============================
=== SDK Info ===
Result/Stats Idx: 81874
module-1(DBG-elam-insel14)#
VRF with "Egress" Policy Enforcement
Non-Border Leaf Zoning-Rules
With VRF Policy Enforcement set to "Egress", contract rules for an L3Out are deployed on both Border Leaf and Non–Border Leaf Nodes. As a result, this configuration consumes additional TCAM space compared to "Ingress" Enforcement. This configuration is not the default value, and if used, must be carefully considered.
Non-Border Leaf Node 302 has two Zoning-Rules, one per flow directionality:
Leaf-302# show zoning-rule scope 2129920 +---------+--------+--------+----------+---------+---------+---------+------------------+----------+----------------------+ | Rule ID | SrcEPG | DstEPG | FilterID | Dir | operSt | Scope | Name | Action | Priority | +---------+--------+--------+----------+---------+---------+---------+------------------+----------+----------------------+ | 4107 | 0 | 0 | implarp | uni-dir | enabled | 2129920 | | permit | any_any_filter(17) | | 4106 | 0 | 0 | implicit | uni-dir | enabled | 2129920 | | deny,log | any_any_any(21) | | 4105 | 0 | 49155 | implicit | uni-dir | enabled | 2129920 | | permit | any_dest_any(16) | | 4108 | 0 | 15 | implicit | uni-dir | enabled | 2129920 | | deny,log | any_vrf_any_deny(22) | | 4112 | 16386 | 49156 | default | uni-dir | enabled | 2129920 | tn1:EPG_to_L3Out | permit | src_dst_any(9) | | 4111 | 49156 | 15 | default | uni-dir | enabled | 2129920 | tn1:EPG_to_L3Out | permit | src_dst_any(9) | +---------+--------+--------+----------+---------+---------+---------+------------------+----------+----------------------+
Border Leaf Zoning-Rules
With "Egress" Policy Enforcement, Border Leaf Node 301 also has two additional Zoning-Rules:
Leaf-301# show zoning-rule scope 2129920 +---------+--------+--------+----------+---------+---------+---------+------------------+----------+----------------------+ | Rule ID | SrcEPG | DstEPG | FilterID | Dir | operSt | Scope | Name | Action | Priority | +---------+--------+--------+----------+---------+---------+---------+------------------+----------+----------------------+ | 4105 | 0 | 0 | implarp | uni-dir | enabled | 2129920 | | permit | any_any_filter(17) | | 4107 | 0 | 0 | implicit | uni-dir | enabled | 2129920 | | deny,log | any_any_any(21) | | 4106 | 0 | 15 | implicit | uni-dir | enabled | 2129920 | | deny,log | any_vrf_any_deny(22) | | 4108 | 0 | 16387 | implicit | uni-dir | enabled | 2129920 | | permit | any_dest_any(16) | | 4109 | 16386 | 49156 | default | uni-dir | enabled | 2129920 | tn1:EPG_to_L3Out | permit | src_dst_any(9) | | 4110 | 49156 | 15 | default | uni-dir | enabled | 2129920 | tn1:EPG_to_L3Out | permit | src_dst_any(9) | +---------+--------+--------+----------+---------+---------+---------+------------------+----------+----------------------+
EPG to L3Out ELAM
A ping from the endpoint 192.168.1.1 to the network behind the L3Out is successful:
Host# ping 10.1.1.1 count 10000 int 1 PING 10.1.1.1 (10.1.1.1): 56 data bytes 64 bytes from 10.1.1.1: icmp_seq=0 ttl=252 time=1.319 ms 64 bytes from 10.1.1.1: icmp_seq=1 ttl=252 time=0.962 ms 64 bytes from 10.1.1.1: icmp_seq=2 ttl=252 time=0.958 ms 64 bytes from 10.1.1.1: icmp_seq=3 ttl=252 time=1.093 ms
The ELAM on Non-Border Leaf Node 302 indicates that policy was not applied on this leaf. Additionally, it picked up a dclass of System PcTag 1 to allow the flow to hit the next leaf node in the flow:
Leaf-302# ereport
======================================================================================================================================================
Captured Packet
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
...
IP Protocol Number : ICMP
IP CheckSum : 26943( 0x693F )
Destination IP : 10.1.1.1
Source IP : 192.168.1.1
======================================================================================================================================================
Contract Lookup ( FPC )
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Lookup Key
------------------------------------------------------------------------------------------------------------------------------------------------------
IP Protocol : ICMP( 0x1 )
L4 Src Port : 2048( 0x800 )
L4 Dst Port : 27360( 0x6AE0 )
sclass (src pcTag) : 49156( 0xC004 )
dclass (dst pcTag) : 1( 0x1 )
...
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Result
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Drop : no
Contract Logging : no
Contract Applied : no
Contract Hit : yes
Contract Aclqos Stats Index : 81903
( show sys int aclqos zoning-rules | grep -B 9 "Idx: 81903" )
The ELAM on Border Leaf Node 301 indicates that policy was applied on this node. It also picked up a dclass of System PcTag 15. This means it Longest-Prefix Matched on the 0.0.0.0/0 L3Out Subnet entry:
Leaf-301# ereport
======================================================================================================================================================
Captured Packet
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
...
IP Protocol Number : ICMP
Destination IP : 10.1.1.1
Source IP : 192.168.1.1
======================================================================================================================================================
Contract Lookup ( FPC )
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Lookup Key
------------------------------------------------------------------------------------------------------------------------------------------------------
IP Protocol : ICMP( 0x1 )
L4 Src Port : 2048( 0x800 )
L4 Dst Port : 40498( 0x9E32 )
sclass (src pcTag) : 49156( 0xC004 )
dclass (dst pcTag) : 15( 0xF )
src pcTag is from local table : no
derived from group-id in iVxLAN header of incoming packet
Unknown Unicast / Flood Packet : no
If yes, Contract is not applied here because it is flooded
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Result
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Drop : no
Contract Logging : no
Contract Applied : yes
Contract Hit : yes
Contract Aclqos Stats Index : 81874
( show sys int aclqos zoning-rules | grep -B 9 "Idx: 81874" )
...
module-1(DBG-elam-insel14)# show sys int aclqos zoning-rules | grep -B 9 "Idx: 81874"
===========================================
Rule ID: 4110 Scope 6 Src EPG: 49156 Dst EPG: 15 Filter 65535
unit_id: 0
=== Region priority: 2462 (rule prio: 9 entry: 158)===
sw_index = 47 | hw_index = 46 | stats_idx = 81874
Curr TCAM resource:
=============================
=== SDK Info ===
Result/Stats Idx: 81874
L3Out to EPG ELAM
There is a caveat with the return flow in this setup:
- Border Leaf Node 301 does not have an endpoint learn for 192.168.1.1.
Leaf-301# show endpoint ip 192.168.1.1
Legend:
S - static s - arp L - local O - peer-attached
V - vpc-attached a - local-aged p - peer-aged M - span
B - bounce H - vtep R - peer-attached-rl D - bounce-to-proxy
E - shared-service m - svc-mgr
+-----------------------------------+---------------+-----------------+--------------+-------------+
VLAN/ Encap MAC Address MAC Info/ Interface
Domain VLAN IP Address IP Info
+-----------------------------------+---------------+-----------------+--------------+-------------+
...empty...
As a result, policy is not applied on Border Leaf Node 301 for this flow and it must be implicitly allowed to reach the next leaf:
Leaf-301# ereport
======================================================================================================================================================
Captured Packet
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
...
IP Protocol Number : ICMP
IP CheckSum : 25157( 0x6245 )
Destination IP : 192.168.1.1
Source IP : 10.1.1.1
======================================================================================================================================================
Contract Lookup ( FPC )
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Lookup Key
------------------------------------------------------------------------------------------------------------------------------------------------------
IP Protocol : ICMP( 0x1 )
L4 Src Port : 0( 0x0 )
L4 Dst Port : 33570( 0x8322 )
sclass (src pcTag) : 16386( 0x4002 )
dclass (dst pcTag) : 1( 0x1 )
src pcTag is from local table : yes
derived from a local table on this node by the lookup of src IP or MAC
Unknown Unicast / Flood Packet : no
If yes, Contract is not applied here because it is flooded
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Result
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Drop : no
Contract Logging : no
Contract Applied : no
Contract Hit : yes
Contract Aclqos Stats Index : 81903
( show sys int aclqos zoning-rules | grep -B 9 "Idx: 81903" )
Instead, policy is applied on Non-Border Leaf Node 302:
Leaf-302# ereport
======================================================================================================================================================
Captured Packet
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Inner L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
...
IP Protocol Number : ICMP
Destination IP : 192.168.1.1
Source IP : 10.1.1.1
======================================================================================================================================================
Contract Lookup ( FPC )
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Lookup Key
------------------------------------------------------------------------------------------------------------------------------------------------------
IP Protocol : ICMP( 0x1 )
L4 Src Port : 0( 0x0 )
L4 Dst Port : 61057( 0xEE81 )
sclass (src pcTag) : 16386( 0x4002 )
dclass (dst pcTag) : 49156( 0xC004 )
src pcTag is from local table : no
derived from group-id in iVxLAN header of incoming packet
Unknown Unicast / Flood Packet : no
If yes, Contract is not applied here because it is flooded
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Result
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Drop : no
Contract Logging : no
Contract Applied : yes
Contract Hit : yes
Contract Aclqos Stats Index : 81874
( show sys int aclqos zoning-rules | grep -B 9 "Idx: 81874" )
...
module-1(DBG-elam-insel14)# show sys int aclqos zoning-rules | grep -B 9 "Idx: 81874"
===========================================
Rule ID: 4112 Scope 6 Src EPG: 16386 Dst EPG: 49156 Filter 65535
unit_id: 0
=== Region priority: 2462 (rule prio: 9 entry: 158)===
sw_index = 47 | hw_index = 46 | stats_idx = 81874
Curr TCAM resource:
=============================
=== SDK Info ===
Result/Stats Idx: 81874
If Border Leaf Node 301 had an endpoint learn 192.168.1.1, policy would have been applied on that node.
Troubleshoot
Scenario - Unintended Allows
A deployment with multiple L3Outs in the same VRF configured with the 0.0.0.0/0 Subnet with “External Subnets for the External EPG” can allow traffic to pass to external destinations unexpectedly.
To induce this, add the 0.0.0.0/0 Subnet under L3Out-2-EEPG which is in the same VRF as L3Out-1-EEPG.
There are no contracts on L3Out-2-EEPG, so we would expect all traffic to be dropped by default:
However, a ping from EPG endpoint 192.168.1.1 to destination 10.2.2.2 behind L3Out-2-EEPG is successful. This is unexpected!
Host# ping 10.2.2.2 PING 10.2.2.2 (10.2.2.2): 56 data bytes 64 bytes from 10.2.2.2: icmp_seq=0 ttl=252 time=0.881 ms 64 bytes from 10.2.2.2: icmp_seq=1 ttl=252 time=0.801 ms 64 bytes from 10.2.2.2: icmp_seq=2 ttl=252 time=0.877 ms 64 bytes from 10.2.2.2: icmp_seq=3 ttl=252 time=0.827 ms
The forward route and policy-mgr prefix both show that traffic destined to 10.2.2.2 in this VRF is assigned System PcTag 15
Leaf-302# vsh_lc -c "show forward route 10.2.2.2 platform vrf tn1:v1" ... Policy Prefix 0.0.0.0/0 SDK Information: vrf: 7(0x7), routed_if: 0x0 epc_class: 15(0xf) ... Leaf-302# vsh -c "show system internal policy-mgr prefix" Requested prefix data Vrf-Vni VRF-Id Table-Id Table-State VRF-Name Addr Class Shared Remote Complete Svc_ena ======= ====== =========== ======= ============================ ================================= ====== ====== ====== ======== ======== ... 2129920 7 0x7 Up tn1:v1 0.0.0.0/0 15 False False False False 2129920 7 0x80000007 Up tn1:v1 ::/0 15 False False False False Leaf-302#
An ELAM on Non-Border Leaf Node 302 validates that traffic is classified with a dclass of System PcTag 15.
Leaf-302# ereport
====================================================================================================================================================== Captured Packet ====================================================================================================================================================== ------------------------------------------------------------------------------------------------------------------------------------------------------ Outer L3 Header ------------------------------------------------------------------------------------------------------------------------------------------------------ ... IP Protocol Number : ICMP IP CheckSum : 14444( 0x386C ) Destination IP : 10.2.2.2 Source IP : 192.168.1.1 ====================================================================================================================================================== Contract Lookup ( FPC ) ====================================================================================================================================================== ------------------------------------------------------------------------------------------------------------------------------------------------------ Contract Lookup Key ------------------------------------------------------------------------------------------------------------------------------------------------------ IP Protocol : ICMP( 0x1 ) L4 Src Port : 2048( 0x800 ) L4 Dst Port : 33134( 0x816E ) sclass (src pcTag) : 49156( 0xC004 ) dclass (dst pcTag) : 15( 0xF ) src pcTag is from local table : yes derived from a local table on this node by the lookup of src IP or MAC Unknown Unicast / Flood Packet : no If yes, Contract is not applied here because it is flooded ------------------------------------------------------------------------------------------------------------------------------------------------------ Contract Result ------------------------------------------------------------------------------------------------------------------------------------------------------ Contract Drop : no Contract Logging : no Contract Applied : yes Contract Hit : yes Contract Aclqos Stats Index : 81875 ( show sys int aclqos zoning-rules | grep -B 9 "Idx: 81875" ) ...
module-1(DBG-elam-insel6)# show sys int aclqos zoning-rules | grep -B 9 "Idx: 81875" =========================================== Rule ID: 4111 Scope 6 Src EPG: 49156 Dst EPG: 15 Filter 65535 unit_id: 0 === Region priority: 2462 (rule prio: 9 entry: 158)=== sw_index = 46 | hw_index = 45 | stats_idx = 81875 Curr TCAM resource: ============================= === SDK Info === Result/Stats Idx: 81875
The Zoning-Rules for VRF "v1" do not show any new entries for EPG and L3Out-2:
Leaf-302# show zoning-rule scope 2129920 +---------+--------+--------+----------+---------+---------+---------+------------------+----------+----------------------+ | Rule ID | SrcEPG | DstEPG | FilterID | Dir | operSt | Scope | Name | Action | Priority | +---------+--------+--------+----------+---------+---------+---------+------------------+----------+----------------------+ | 4107 | 0 | 0 | implarp | uni-dir | enabled | 2129920 | | permit | any_any_filter(17) | | 4106 | 0 | 0 | implicit | uni-dir | enabled | 2129920 | | deny,log | any_any_any(21) | | 4105 | 0 | 49155 | implicit | uni-dir | enabled | 2129920 | | permit | any_dest_any(16) | | 4108 | 0 | 15 | implicit | uni-dir | enabled | 2129920 | | deny,log | any_vrf_any_deny(22) | | 4112 | 16386 | 49156 | default | uni-dir | enabled | 2129920 | tn1:EPG_to_L3Out | permit | src_dst_any(9) | | 4111 | 49156 | 15 | default | uni-dir | enabled | 2129920 | tn1:EPG_to_L3Out | permit | src_dst_any(9) | +---------+--------+--------+----------+---------+---------+---------+------------------+----------+----------------------+ Leaf-302#
As L3Out-2-EEPG only has the 0.0.0.0/0 Subnet configured, all traffic destined to it is classified with dclass of System PcTag 15.
Zoning-Rules ID 4111 and 4112 are programmed as L3Out-1-EEPG has both the 0.0.0.0/0 Subnet and provides a contract that is consumed by EPG.
Flows to L3Out-2-EEPG are unexpectedly allowed due to this configuration!
Solution - Unintended Allows
To prevent this behavior:
- It is strongly recommended to only use the 0.0.0.0/0 Subnet on one L3Out EPG per VRF
- Where possible, use specific subnets for other L3Outs in the same VRF. This allows traffic to pull in the unique L3Out PcTag values as their dclass.
Apply these changes to mitigate the unexpected allow:
- On L3Out-2-EEPG, replace the 0.0.0.0/0 Subnet with a 10.2.2.0/24 Subnet
- On L3Out-2-EEPG, provide a contract
- On EPG, consume the same contract
Once completed, observe these changes on Non-Border Leaf Node 302:
- There is a more specific policy-mgr prefix for 10.2.2.0/24 tied to L3Out-2-EEPG PcTag 32771
- There is a Zoning-Rules ID 4109 entry
- This entry allows a flow from EPG PcTag 49156 to L3Out-2-EEPG PcTag 32771
- There is a Zoning-Rules ID 4110 entry
- This entry allows a flow from L3Out-2-EEPG PcTag 32771 to EPG PcTag 49156
The updated forward route and policy-mgr prefix which show that 10.2.2.2 is assigned the L3Out-2-EEPG PgTag of 32771:
Leaf-302# vsh_lc -c "show forward route 10.2.2.2 platform vrf tn1:v1" ... Policy Prefix 10.2.2.0/24 ... SDK Information: vrf: 7(0x7), routed_if: 0x0 epc_class: 32771(0x8003) attributes: SUP_CP DST_POL_IC SRC_POL_IC
Leaf-302# vsh -c "show system internal policy-mgr prefix" Requested prefix data Vrf-Vni VRF-Id Table-Id Table-State VRF-Name Addr Class Shared Remote Complete Svc_ena ======= ====== =========== ======= ============================ ================================= ====== ====== ====== ======== ======== ... 2129920 7 0x7 Up tn1:v1 0.0.0.0/0 15 False False False False 2129920 7 0x80000007 Up tn1:v1 ::/0 15 False False False False 2129920 7 0x7 Up tn1:v1 10.2.2.0/24 32771 False True False False
Leaf-302# show zoning-rule scope 2129920 +---------+--------+--------+----------+----------------+---------+---------+------------------+----------+----------------------+ | Rule ID | SrcEPG | DstEPG | FilterID | Dir | operSt | Scope | Name | Action | Priority | +---------+--------+--------+----------+----------------+---------+---------+------------------+----------+----------------------+ | 4107 | 0 | 0 | implarp | uni-dir | enabled | 2129920 | | permit | any_any_filter(17) | | 4106 | 0 | 0 | implicit | uni-dir | enabled | 2129920 | | deny,log | any_any_any(21) | | 4105 | 0 | 49155 | implicit | uni-dir | enabled | 2129920 | | permit | any_dest_any(16) | | 4108 | 0 | 15 | implicit | uni-dir | enabled | 2129920 | | deny,log | any_vrf_any_deny(22) | | 4112 | 16386 | 49156 | default | uni-dir | enabled | 2129920 | tn1:EPG_to_L3Out | permit | src_dst_any(9) | | 4111 | 49156 | 15 | default | uni-dir | enabled | 2129920 | tn1:EPG_to_L3Out | permit | src_dst_any(9) | | 4109 | 49156 | 32771 | default | bi-dir | enabled | 2129920 | tn1:EPG_to_L3Out | permit | src_dst_any(9) | | 4110 | 32771 | 49156 | default | uni-dir-ignore | enabled | 2129920 | tn1:EPG_to_L3Out | permit | src_dst_any(9) | +---------+--------+--------+----------+----------------+---------+---------+------------------+----------+----------------------+
Ping from the EPG host to the external destination behind L3Out-2-EEPG is successful:
Host# ping 10.2.2.2 PING 10.2.2.2 (10.2.2.2): 56 data bytes
64 bytes from 10.2.2.2: icmp_seq=0 ttl=252 time=0.854 ms
64 bytes from 10.2.2.2: icmp_seq=1 ttl=252 time=0.669 ms
64 bytes from 10.2.2.2: icmp_seq=2 ttl=252 time=0.716 ms
64 bytes from 10.2.2.2: icmp_seq=3 ttl=252 time=0.669 ms
64 bytes from 10.2.2.2: icmp_seq=4 ttl=252 time=0.666 ms
The ELAM for the icmp request on Non-Border Leaf Node 302 indicates the dclass is now 32771 - the PcTag of L3Out-2-EEPG.
Leaf-302# ereport
======================================================================================================================================================
Captured Packet
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Outer L3 Header
------------------------------------------------------------------------------------------------------------------------------------------------------
...
IP Protocol Number : ICMP
IP CheckSum : 4095( 0xFFF )
Destination IP : 10.2.2.2
Source IP : 192.168.1.1
======================================================================================================================================================
Contract Lookup ( FPC )
======================================================================================================================================================
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Lookup Key
------------------------------------------------------------------------------------------------------------------------------------------------------
IP Protocol : ICMP( 0x1 )
L4 Src Port : 2048( 0x800 )
L4 Dst Port : 49837( 0xC2AD )
sclass (src pcTag) : 49156( 0xC004 )
dclass (dst pcTag) : 32771( 0x8003 )
src pcTag is from local table : yes
derived from a local table on this node by the lookup of src IP or MAC
Unknown Unicast / Flood Packet : no
If yes, Contract is not applied here because it is flooded
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Result
------------------------------------------------------------------------------------------------------------------------------------------------------
Contract Drop : no
Contract Logging : no
Contract Applied : yes
Contract Hit : yes
Contract Aclqos Stats Index : 81873
( show sys int aclqos zoning-rules | grep -B 9 "Idx: 81873" )
...
The ereport provided aclqos command shows that this flow hits one of the new Zoning-Rules, specifically Rule ID 4109:
module-1(DBG-elam-insel6)# show sys int aclqos zoning-rules | grep -B 9 "Idx: 81873"
===========================================
Rule ID: 4109 Scope 6 Src EPG: 49156 Dst EPG: 32771 Filter 65535
unit_id: 0
=== Region priority: 2462 (rule prio: 9 entry: 158)===
sw_index = 48 | hw_index = 47 | stats_idx = 81873
Curr TCAM resource:
=============================
=== SDK Info ===
Result/Stats Idx: 81873
Revision History
| Revision | Publish Date | Comments |
|---|---|---|
2.0 |
01-Sep-2022 |
Added forward route and policy-mgr prefix output |
1.0 |
30-Aug-2022 |
Initial Release |
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)