Contents

Introduction

Concept

Prerequisites

Setup and Topology

Scenarios

Traffic sourced from overlapping subnets

Fabric with overlapping subnets declared as external on separate external EPGs

Fabric with 0.0.0.0/0 prefix declared as external on multiple external EPGs

Further Reading

Introduction

Cisco's Application Centric Infrastructure (ACI) facilitates for communication between internal Tenants and external routed networks, via L3outs(Layer 3 out). Such L3outs can also be configured to have one or more End Point Groups (EPGs). For ACI to know how to classify taffic coming in, as an L3out's EPG, explicit subnets need to be defined with certain flags enabled. This article aims to shed some light on the hardware implementation of L3out EPG's in the context of contract based policy application. We will specifically explore the flag 'external subnets for external EPGs' and the unexpected consequences of declaring overlapping prefixes as 'external' on separate EPGs.

Concept

The rule of thumb is: when deploying L3outs, separate EPGs in the same Virtual Routing and Forwarding (VRF) instance, should not have overlapping subnets marked as 'external subnet for external EPGs'. This also means that traffic sourced from a specific subnet should not come in through different EPGs. This may cause unexpected classification of traffic based on longest prefix match against subnets declared against unrelated EPGs. Let's look at a few scenarios to understand this in detail

Prerequisites

Basic understanding of ACI: L3outs, contracts and policy enforcement. Some useful terms are briefly explained below, more detailed information on these is beyong the scope of this document:

pcTag: ACI classifies traffic into pcTags and these are internal representations of EPGs. These values, by default have a scope of VRF - i.e, they are unique within a VRF, but may be reused across VRFs. However, if one EPG has a contract with another EPG in a differenet VRF / Tenant , then the pcTag value has a global scope - i.e, you will not find any other EPG in ACI with the same pcTag.

ELAM: Embedded Logic Analyser Module. This tool is used to capture one packet on ASIC based on filters and to check the headers/flags set on the packet. This tool also helps understand lookups / logic made by hardware based

sclass/dclass: when traffic comes in to a leaf, based on direction of policy enforcement and locally available prefix knowledge the leaf will mark source and destination traffic into EPGs - in ELAM captures this will be seen as sclass and dclass respectively

zoning-rule: These are internal representations of contracts and are similar to lines of an ACL. The SrcEpg and DstEpg values should match with sclass/dclass for traffic to hit a given rule and be allowed. By default in an enforced vrf there is an implicit deny as the last line , so any traffic not matching against a certain rule will hit the implicit deny and be dropped.

Setup and Topology

Two leafs - 101 and 102 , model: N9K-C93180YC-EX

- Version 3.2(4e)

- One VRF used -

- Policy Enforcement Preference : Enforced

- Policy Enforcement Direction: Ingress.

- VRF VNID(VxLAN Network Identifier): 2752513 ; pcTag: 32770

- L3out in Leaf1 (101) -

- Protocol: Open Shortest Path First (OSPF)

- L3 interface user for neighborship- eth1/22 (10.27.48.1/24)

- external EPG pcTag: 16387

- Application EPG on Leaf101

- Trunk - eth1/24

- pcTag: 49153

- IP End Point: 172.16.1.17

- Gateway: 172.16.1.254/24 - deployed on Bridge Domain (BD)

- BD has pcTag 32771

- L3out on Leaf2 (202) -

- Protocol: Enhanced Interior Gateway Routing Protocol (EIGRP)

- SVI used for neighborship with Path 1/16 - vlan 2747 (10.27.47.1/24)

- external EPG pcTag: 163869

Scenarios

Traffic sourced from overlapping subnets

In this scenario we look at potential mis-classification when traffic is sourced from overlapping subnets (from ACI's perspective)

OSPF advertises:

10.9.9.6/32

EIGRP advertises:

10.9.9.1/32

We start with the topology in Diagram 1, but without any contracts. For EPG on OSPF we define subnet 0.0.0.0/0 as 'external subnet for external EPGs' and 10.9.9.0/24 with the same flag for EIGRP's EPG. Here's what the tables on Leaf1 and 2 look like:

Leaf1:

leaf101# show end int eth1/24

Legend:

s - arp              H - vtep             V - vpc-attached     p - peer-aged

R - peer-attached-rl B - bounce           S - static           M - span

D - bounce-to-proxy  O - peer-attached    a - local-aged       L - local

+-----------------------------------+---------------+-----------------+--------------+-------------+

VLAN/                           Encap           MAC Address       MAC Info/       Interface

Domain                          VLAN            IP Address        IP Info

+-----------------------------------+---------------+-----------------+--------------+-------------+

48                                        vlan-2743    dcce.c15b.1e47 L                     eth1/24

shparanj:eigrp-test                       vlan-2743       172.16.1.17 L                     eth1/24

leaf101# show ip route vrf shparanj:eigrp-test

IP Route Table for VRF "shparanj:eigrp-test"

'*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

'%<string>' in via output denotes VRF <string>

10.9.9.1/32, ubest/mbest: 1/0

*via 10.0.248.0%overlay-1, [200/128576], 05:31:49, bgp-65003, internal, tag 65003

10.9.9.6/32, ubest/mbest: 1/0

*via 10.27.48.2, eth1/22, [110/5], 05:09:51, ospf-default, intra

10.27.47.0/24, ubest/mbest: 1/0

*via 10.0.248.0%overlay-1, [200/0], 05:31:49, bgp-65003, internal, tag 65003

10.27.48.0/24, ubest/mbest: 1/0, attached, direct

*via 10.27.48.1, eth1/22, [1/0], 05:31:46, direct

10.27.48.1/32, ubest/mbest: 1/0, attached

*via 10.27.48.1, eth1/22, [1/0], 05:31:46, local, local

172.16.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive

*via 10.0.240.34%overlay-1, [1/0], 05:27:43, static

172.16.1.254/32, ubest/mbest: 1/0, attached, pervasive

*via 172.16.1.254, vlan47, [1/0], 05:31:52, local, local

leaf101# show zoning-rule scope 2752513

Rule ID         SrcEPG          DstEPG          FilterID        operSt          Scope           Action                              Priority

=======         ======          ======          ========        ======          =====           ======                              ========

4173            0               0               implicit        enabled         2752513         deny,log                            any_any_any(21)

4174            0               0               implarp         enabled         2752513         permit                              any_any_filter(17)

4175            0               15              implicit        enabled         2752513         deny,log                            any_vrf_any_deny(22)

4207            0               32771           implicit        enabled         2752513         permit                              any_dest_any(16)

<<vsh>>  (to go into vsh propmt , type: #vsh )

leaf101# show system internal policy-mgr prefix | grep shparanj:eigrp-test

2752513 26     0x1a          Up     shparanj:eigrp-test                                  0.0.0.0/0   15      False  True   False

2752513 26     0x8000001a    Up     shparanj:eigrp-test                                       ::/0   15      False  True   False

Leaf2:

leaf102# show ip route vrf shparanj:eigrp-test

IP Route Table for VRF "shparanj:eigrp-test"

'*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

'%<string>' in via output denotes VRF <string>

10.9.9.1/32, ubest/mbest: 1/0

*via 10.27.47.10, vlan78, [90/128576], 06:13:41, eigrp-default, internal

10.9.9.6/32, ubest/mbest: 1/0

*via 10.0.0.64%overlay-1, [200/5], 05:20:27, bgp-65003, internal, tag 65003

10.27.47.0/24, ubest/mbest: 1/0, attached, direct

*via 10.27.47.2, vlan78, [1/0], 3d21h, direct

10.27.47.2/32, ubest/mbest: 1/0, attached

*via 10.27.47.2, vlan78, [1/0], 3d21h, local, local

10.27.48.0/24, ubest/mbest: 1/0

*via 10.0.0.64%overlay-1, [200/0], 05:35:06, bgp-65003, internal, tag 65003

leaf102# show zoning-rule scope 2752513

Rule ID         SrcEPG          DstEPG          FilterID        operSt          Scope           Action                              Priority

=======         ======          ======          ========        ======          =====           ======                              ========

4472            0               0               implicit        enabled         2752513         deny,log                            any_any_any(21)

4471            0               0               implarp         enabled         2752513         permit                              any_any_filter(17)

4470            0               15              implicit        enabled         2752513         deny,log                            any_vrf_any_deny(22)

<<vsh>>

leaf102# show system internal policy-mgr prefix | grep shparanj:eigrp-test

2752513 37     0x80000025    Up     shparanj:eigrp-test                                       ::/0   15      False  True   False

2752513 37     0x25          Up     shparanj:eigrp-test                                  0.0.0.0/0   15      False  True   False

2752513 37     0x25          Up     shparanj:eigrp-test                                 10.9.9.0/24  16386   False  True   False

Let's add contract B (contract in tenant , scope vrf - filer: common:default)

As soon as we add contract B - we see the eigrp EPG prefix added on leaf1:

leaf101# show system internal policy-mgr prefix | grep shparanj:eigrp-test

2752513 26     0x1a          Up     shparanj:eigrp-test                                 10.9.9.0/24  16386   False  True   False

2752513 26     0x1a          Up     shparanj:eigrp-test                                  0.0.0.0/0   15      False  True   False

2752513 26     0x8000001a    Up     shparanj:eigrp-test                                       ::/0   15      False  True   False

Lets look at other polices:

Leaf 1 contracts:

leaf101# show zoning-rule scope 2752513

Rule ID         SrcEPG          DstEPG          FilterID        operSt          Scope           Action                              Priority

=======         ======          ======          ========        ======          =====           ======                              ========

4173            0               0               implicit        enabled         2752513         deny,log                            any_any_any(21)

4174            0               0               implarp         enabled         2752513         permit                              any_any_filter(17)

4175            0               15              implicit        enabled         2752513         deny,log                            any_vrf_any_deny(22)

4207            0               32771           implicit        enabled         2752513         permit                              any_dest_any(16)

4604            49153           16386           default         enabled         2752513         permit                              src_dst_any(9)

4605            16386           49153           default         enabled         2752513         permit                              src_dst_any(9)

Leaf 2 contracts (remain unchanged):

leaf102# show zoning-rule scope 2752513

Rule ID         SrcEPG          DstEPG          FilterID        operSt          Scope           Action                              Priority

=======         ======          ======          ========        ======          =====           ======                              ========

4472            0               0               implicit        enabled         2752513         deny,log                            any_any_any(21)

4471            0               0               implarp         enabled         2752513         permit                              any_any_filter(17)

4470            0               15              implicit        enabled         2752513         deny,log                            any_vrf_any_deny(22)

In this scenario traffic coming in from ospf l3out , which we expect to be tagged with 16387 gets tagged with 16386 instead. This is because traffic hits the new prefix entry on Leaf1.

Ping from 10.9.9.6 to end-point 172.16.1.17:

# ping 172.16.1.17 vrf shp-ospf source 10.9.9.6 count 1000 interval 1

PING 172.16.1.17 (172.16.1.17) from 10.9.9.6: 56 data bytes

64 bytes from 172.16.1.17: icmp_seq=0 ttl=253 time=2.207 ms

64 bytes from 172.16.1.17: icmp_seq=1 ttl=253 time=1.443 ms

64 bytes from 172.16.1.17: icmp_seq=2 ttl=253 time=1.312 ms

Ping works even without a contract between ospf epg and app-epg. This is because it hits against policy for eigrp-epg and gets allowed.

ELAM:

module-1(DBG-elam)# trigger init in-select 6 out-select 0

module-1(DBG-elam-insel6)# set outer ipv4 src_ip 10.9.9.6

module-1(DBG-elam-insel6)# start

module-1(DBG-elam-insel6)# stat

ELAM STATUS

===========

Asic 0 Slice 0 Status Armed

Asic 0 Slice 1 Status Triggered

module-1(DBG-elam-insel6)# report | grep sclass

sug_lurw_vec.info.nsh_special.sclass: 0x4002

sug_lurw_vec.info.ifabric_spine.sclass: 0x4002

sug_lurw_vec.info.ifabric_leaf.sclass: 0x4002

#dec 0x4002

16386

In this scenario the traffic ends up working due to classification into a pcTag that has a contract with the intended destination. However, if, for example, the compute leaf was a separate 3rd leaf, then our traffic would fail - as the entry for contract would only exist on the third leaf (ingress policy) or on leaf102 (egress policy).

Fabric with overlapping subnets declared as external on separate external EPGs

In this scenario we look at policy conflict and potential mis-classification due to overlapping or same subnets declared as external on different external EPGs.

OSPF advertises network:

10.9.1.0/24

EIGRP advertises network:

10.9.2.0/24

We start with the topology in Diagram 1, but without any contracts. We define subnet 10.9.0.0/16 as 'external subnet for external EPGs'  for EPG on both L3outs.

Here's what the tables on Leaf1 and 2 look like:

Leaf 1:

leaf101# show ip route vrf shparanj:eigrp-test

IP Route Table for VRF "shparanj:eigrp-test"

'*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

'%<string>' in via output denotes VRF <string>

10.9.1.0/24, ubest/mbest: 1/0

*via 10.27.48.2, eth1/22, [110/5], 00:01:50, ospf-default, intra

10.9.2.0/24, ubest/mbest: 1/0

*via 10.0.248.0%overlay-1, [200/128576], 00:00:32, bgp-65003, internal, tag 65003

10.27.47.0/24, ubest/mbest: 1/0

*via 10.0.248.0%overlay-1, [200/0], 01:54:45, bgp-65003, internal, tag 65003

10.27.48.0/24, ubest/mbest: 1/0, attached, direct

*via 10.27.48.1, eth1/22, [1/0], 1d09h, direct

10.27.48.1/32, ubest/mbest: 1/0, attached

*via 10.27.48.1, eth1/22, [1/0], 1d09h, local, local

172.16.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive

*via 10.0.240.34%overlay-1, [1/0], 1d09h, static

172.16.1.254/32, ubest/mbest: 1/0, attached, pervasive

*via 172.16.1.254, vlan47, [1/0], 1d09h, local, local

leaf101# show zoning-rule scope 2752513

Rule ID         SrcEPG          DstEPG          FilterID        operSt          Scope           Action                              Priority

=======         ======          ======          ========        ======          =====           ======                              ========

4173            0               0               implicit        enabled         2752513         deny,log                            any_any_any(21)

4174            0               0               implarp         enabled         2752513         permit                              any_any_filter(17)

4175            0               15              implicit        enabled         2752513         deny,log                            any_vrf_any_deny(22)

4207            0               32771           implicit        enabled         2752513         permit                              any_dest_any(16)

<<vsh>>

leaf101#  show system internal policy-mgr prefix | grep shparanj:eigrp-test

2752513 26     0x1a          Up     shparanj:eigrp-test                                 10.9.0.0/16  16387   False  True   False

2752513 26     0x1a          Up     shparanj:eigrp-test                                  0.0.0.0/0   15      False  True   False

2752513 26     0x8000001a    Up     shparanj:eigrp-test                                       ::/0   15      False  True   False

Leaf2:

leaf102# show ip route vrf shparanj:eigrp-test

IP Route Table for VRF "shparanj:eigrp-test"

'*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

'%<string>' in via output denotes VRF <string>

10.9.1.0/24, ubest/mbest: 1/0

*via 10.0.0.64%overlay-1, [200/5], 00:05:29, bgp-65003, internal, tag 65003

10.9.2.0/24, ubest/mbest: 1/0

*via 10.27.47.10, vlan80, [90/128576], 00:04:10, eigrp-default, internal

10.27.47.0/24, ubest/mbest: 1/0, attached, direct

*via 10.27.47.2, vlan80, [1/0], 01:58:24, direct

10.27.47.2/32, ubest/mbest: 1/0, attached

*via 10.27.47.2, vlan80, [1/0], 01:58:24, local, local

10.27.48.0/24, ubest/mbest: 1/0

*via 10.0.0.64%overlay-1, [200/0], 1d09h, bgp-65003, internal, tag 65003

leaf102# show zoning-rule scope 2752513

Rule ID         SrcEPG          DstEPG          FilterID        operSt          Scope           Action                              Priority

=======         ======          ======          ========        ======          =====           ======                              ========

4472            0               0               implicit        enabled         2752513         deny,log                            any_any_any(21)

4471            0               0               implarp         enabled         2752513         permit                              any_any_filter(17)

4470            0               15              implicit        enabled         2752513         deny,log                            any_vrf_any_deny(22)

<<vsh>>

leaf102# show system internal policy-mgr prefix | grep shparanj:eigrp-test

2752513 37     0x80000025    Up     shparanj:eigrp-test                                       ::/0   15      False  True   False

2752513 37     0x25          Up     shparanj:eigrp-test                                  0.0.0.0/0   15      False  True   False

2752513 37     0x25          Up     shparanj:eigrp-test                                 10.9.0.0/16  16386   False  True   False

In this state, without any contracts, we see no faults on either EPGs. No overlap in prefixes is detected yet!

If we add Contract B, we see a fault in the app-EPG (which consumes Contract B).

Topology:

Let's look at the change in tables:

leaf101# show zoning-rule scope 2752513

Rule ID         SrcEPG          DstEPG          FilterID        operSt          Scope           Action                              Priority

=======         ======          ======          ========        ======          =====           ======                              ========

4173            0               0               implicit        enabled         2752513         deny,log                            any_any_any(21)

4174            0               0               implarp         enabled         2752513         permit                              any_any_filter(17)

4175            0               15              implicit        enabled         2752513         deny,log                            any_vrf_any_deny(22)

4207            0               32771           implicit        enabled         2752513         permit                              any_dest_any(16)

4605            49153           16386           default         enabled         2752513         permit                              src_dst_any(9)

4604            16386           49153           default         enabled         2752513         permit                              src_dst_any(9)

<<vsh>>

leaf101# show system internal policy-mgr prefix | grep shparanj:eigrp-test

2752513 26     0x1a          Up     shparanj:eigrp-test                                 10.9.0.0/16  16387   False  True   False

2752513 26     0x1a          Up     shparanj:eigrp-test                                  0.0.0.0/0   15      False  True   False

2752513 26     0x8000001a    Up     shparanj:eigrp-test                                       ::/0   15      False  True   False

Leaf2 remains unchanged.

This shows us that the zoning-rule corresponding to Contract B is installed. However the prefix cannot be added, as it already exists - marked against the OSPF EPG!

And that is exactly what the fault warns us, "prefix entry already used in another EPG" - the fault is only raised when there is a conflict on a particular leaf between policy (zoning-rules) and its application. The fault is raised on the consumer EPG.

If we start traffic from 10.9.2.1 , it gets dropped on Leaf101 due to policy deny:

# show logging ip access-list internal packet-log deny

[ Tue Feb 19 19:31:33 2019 234270 usecs]: CName: shparanj:eigrp-test(VXLAN: 2752513), VlanType: FD_VLAN, Vlan-Id: 48, SMac: 0xdccec15b1e47, DMac:0x0022bdf819ff, SIP: 172.16.1.17, DIP: 10.9.2.1, SPort: 0, DPort: 0, Src Intf: Ethernet1/24, Proto: 1, PktLen: 98

[ Tue Feb 19 19:31:31 2019 234310 usecs]: CName: shparanj:eigrp-test(VXLAN: 2752513), VlanType: FD_VLAN, Vlan-Id: 48, SMac: 0xdccec15b1e47, DMac:0x0022bdf819ff, SIP: 172.16.1.17, DIP: 10.9.2.1, SPort: 0, DPort: 0, Src Intf: Ethernet1/24, Proto: 1, PktLen: 98

We see that replies from EP 172.16.1.17 to 10.9.2.1 are dropped. This is because:

- Requests from 10.9.2.1 coming in from fabric are already classified with sclass 16386 - these hit the Rule ID 4604 and are allowed through

- Replies from 172.16.1.17 get marked with dclass 16387 - this is picked up based on policy-mgr prefix rules. There is no rule corresponding to 16387 and these are denied.

In this situation misclassification causes traffic to be dropped even though we seem to have the right config in place (if fault is ignored).

Fabric with 0.0.0.0/0 prefix declared as external on multiple external EPGs

In this scenario we look at potential mis-classification and unexpected security violations due to the application of 0.0.0.0/0 subnet as external on different external EPGs.

OSPF advertises network:

10.7.7.0/24

EIGRP advertises network:

10.8.8.0/24

We start with the topology in Diagram 1, but without any contracts. We define subnet 0.0.0.0/0 as 'external subnet for external EPGs'  for EPG on both L3outs.

Here's what the tables on Leaf1 and 2 look like:

Leaf1:

leaf101# show zoning-rule scope 2752513

Rule ID         SrcEPG          DstEPG          FilterID        operSt          Scope           Action                              Priority

=======         ======          ======          ========        ======          =====           ======                              ========

4173            0               0               implicit        enabled         2752513         deny,log                            any_any_any(21)

4174            0               0               implarp         enabled         2752513         permit                              any_any_filter(17)

4175            0               15              implicit        enabled         2752513         deny,log                            any_vrf_any_deny(22)

4207            0               32771           implicit        enabled         2752513         permit                              any_dest_any(16)

leaf101# show ip route vrf shparanj:eigrp-test

IP Route Table for VRF "shparanj:eigrp-test"

'*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

'%<string>' in via output denotes VRF <string>

10.7.7.0/24, ubest/mbest: 1/0

*via 10.27.48.2, eth1/22, [110/5], 00:23:29, ospf-default, intra

10.8.8.0/24, ubest/mbest: 1/0

*via 10.0.248.0%overlay-1, [200/128576], 00:02:30, bgp-65003, internal, tag 65003

10.27.47.0/24, ubest/mbest: 1/0

*via 10.0.248.0%overlay-1, [200/0], 00:02:33, bgp-65003, internal, tag 65003

10.27.48.0/24, ubest/mbest: 1/0, attached, direct

*via 10.27.48.1, eth1/22, [1/0], 1d07h, direct

10.27.48.1/32, ubest/mbest: 1/0, attached

*via 10.27.48.1, eth1/22, [1/0], 1d07h, local, local

172.16.1.0/24, ubest/mbest: 1/0, attached, direct, pervasive

*via 10.0.240.34%overlay-1, [1/0], 1d07h, static

172.16.1.254/32, ubest/mbest: 1/0, attached, pervasive

*via 172.16.1.254, vlan47, [1/0], 1d07h, local, local

<<vsh>>

leaf101# show system internal policy-mgr prefix | grep shparanj:eigrp-test

2752513 26     0x1a          Up     shparanj:eigrp-test                                  0.0.0.0/0   15      False  True   False

2752513 26     0x8000001a    Up     shparanj:eigrp-test                                       ::/0   15      False  True   False

Leaf2:

leaf102# show ip route vrf shparanj:eigrp-test

IP Route Table for VRF "shparanj:eigrp-test"

'*' denotes best ucast next-hop

'**' denotes best mcast next-hop

'[x/y]' denotes [preference/metric]

'%<string>' in via output denotes VRF <string>

10.7.7.0/24, ubest/mbest: 1/0

*via 10.0.0.64%overlay-1, [200/5], 00:26:07, bgp-65003, internal, tag 65003

10.8.8.0/24, ubest/mbest: 1/0

*via 10.27.47.10, vlan80, [90/128576], 00:05:08, eigrp-default, internal

10.27.47.0/24, ubest/mbest: 1/0, attached, direct

*via 10.27.47.2, vlan80, [1/0], 00:05:11, direct

10.27.47.2/32, ubest/mbest: 1/0, attached

*via 10.27.47.2, vlan80, [1/0], 00:05:11, local, local

10.27.48.0/24, ubest/mbest: 1/0

*via 10.0.0.64%overlay-1, [200/0], 1d07h, bgp-65003, internal, tag 65003

leaf102#  show zoning-rule scope 2752513

Rule ID         SrcEPG          DstEPG          FilterID        operSt          Scope           Action                              Priority

=======         ======          ======          ========        ======          =====           ======                              ========

4472            0               0               implicit        enabled         2752513         deny,log                            any_any_any(21)

4471            0               0               implarp         enabled         2752513         permit                              any_any_filter(17)

4470            0               15              implicit        enabled         2752513         deny,log                            any_vrf_any_deny(22)

<<vsh>>

leaf102# show system internal policy-mgr prefix | grep shparanj:eigrp-test

2752513 37     0x80000025    Up     shparanj:eigrp-test                                       ::/0   15      False  True   False

2752513 37     0x25          Up     shparanj:eigrp-test                                  0.0.0.0/0   15      False  True   False

If we add both contracts A & B, we still don't see any faults.

Let's look at the tables on Leafs:

Leaf1:

leaf101# show zoning-rule scope 2752513

Rule ID         SrcEPG          DstEPG          FilterID        operSt          Scope           Action                              Priority

=======         ======          ======          ========        ======          =====           ======                              ========

4173            0               0               implicit        enabled         2752513         deny,log                            any_any_any(21)

4174            0               0               implarp         enabled         2752513         permit                              any_any_filter(17)

4175            0               15              implicit        enabled         2752513         deny,log                            any_vrf_any_deny(22)

4207            0               32771           implicit        enabled         2752513         permit                              any_dest_any(16)

4616            49153           15              default         enabled         2752513         permit                              src_dst_any(9)

4617            32770           49153           default         enabled         2752513         permit                              src_dst_any(9)

<<vsh>>

leaf101# show system internal policy-mgr prefix | grep shparanj:eigrp-test

2752513 26     0x1a          Up     shparanj:eigrp-test                                  0.0.0.0/0   15      False  True   False

2752513 26     0x8000001a    Up     shparanj:eigrp-test                                       ::/0   15      False  True   False

Tables on Leaf2 remain unchanged.

We don't see any faults as there is actually no policy conflict from each leaf's perspective. The Rule IDs added when using 0.0.0.0/0 as external EPG are special.

- Traffic coming in to either border leaf from its respective EPG is marked with sclass 32770 - this is the VRF's pcTag.

- dclass on this traffic is 49153 - the app-EPG's pcTag.

- Return traffic from app-EPG has dclass of 15

ELAM  on Leaf1:

module-1(DBG-elam)# trigger init in-select 6 out-select 0

module-1(DBG-elam-insel6)# set outer ipv4 src_ip 10.7.7.1

module-1(DBG-elam-insel6)# start

module-1(DBG-elam-insel6)# stat

ELAM STATUS

===========

Asic 0 Slice 0 Status Armed

Asic 0 Slice 1 Status Triggered

module-1(DBG-elam-insel6)# report | grep sclass

sug_lurw_vec.info.nsh_special.sclass: 0x8002

sug_lurw_vec.info.ifabric_spine.sclass: 0x8002

sug_lurw_vec.info.ifabric_leaf.sclass: 0x8002

module-1(DBG-elam-insel6)# dec 0x8002

32770

module-1(DBG-elam-insel6)# reset

module-1(DBG-elam-insel6)# set outer ipv4 dst_ip 10.7.7.1

module-1(DBG-elam-insel6)# start

module-1(DBG-elam-insel6)# stat

ELAM STATUS

===========

Asic 0 Slice 0 Status Armed

Asic 0 Slice 1 Status Armed

module-1(DBG-elam-insel6)# stat

ELAM STATUS

===========

Asic 0 Slice 0 Status Armed

Asic 0 Slice 1 Status Triggered

module-1(DBG-elam-insel6)# report | grep dclass

sug_lurw_vec.info.nsh_special.dclass: 0xF

sug_lurw_vec.info.ifabric_leaf.dclass: 0xF

Even if we remove Contract A, 10.7.7.1 can continue communication with 172.16.1.17.

This is because removal of Contract A does not result in any changes on the zoning-rules on Leaf1.

leaf101# show system internal policy-mgr prefix | grep shparanj:eigrp-test

2752513 26     0x1a          Up     shparanj:eigrp-test                                  0.0.0.0/0   15      False  True   False

2752513 26     0x8000001a    Up     shparanj:eigrp-test                                       ::/0   15      False  True   False

leaf101# exit

leaf101# show zoning-rule scope 2752513

Rule ID         SrcEPG          DstEPG          FilterID        operSt          Scope           Action                              Priority

=======         ======          ======          ========        ======          =====           ======                              ========

4173            0               0               implicit        enabled         2752513         deny,log                            any_any_any(21)

4174            0               0               implarp         enabled         2752513         permit                              any_any_filter(17)

4175            0               15              implicit        enabled         2752513         deny,log                            any_vrf_any_deny(22)

4207            0               32771           implicit        enabled         2752513         permit                              any_dest_any(16)

4616            49153           15              default         enabled         2752513         permit                              src_dst_any(9)

4617            32770           49153           default         enabled         2752513         permit                              src_dst_any(9)

Further, traffic coming in on OSPF external EPG continues to be tagged with VRF pcTag, as the EPG still has 0.0.0.0/0 marked as external subnet.

This leads to a breach in security policy, i.e, two EPGs able to communicate without a contract in an enforced VRF.

Further Reading

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/aci/apic/sw/1-x/ACI_Best_Practices/b_ACI_Best_Practices/b_ACI_Best_Practices_chapter_010010.html