Contents

Introduction

Background Information

Intersight Connected ACI Fabrics

Quick Start Video

Quick Start to Address Fault

ACI Pre-upgrade Validation Script Example

Additional Details

Issues Induced by fabric-encap-mismatch

Future Prevention

Related Information

Introduction

This document describes next steps for remediation of ACI fault F3274: fabric-encap-mismatch.

Background Information

This specific fault is checks for encap VLAN VNID mismatches between VPC peer nodes as this is always a problem.

For Non-VPC nodes, your expected re-usage of VLANs throughout an ACI fabric ultimately determines if a VNID mismatch across non-VPC nodes for a given VLAN can result in datapath/loop issues. Non-VPC node VNID validation is outside the scope of fault F3274.

"Code" : "F3274",

"Description" : "VNID mismatch between peers detected for encap vlans (<vlanId>).",

"Dn" : "topology/pod-1/node-<leafNodeId>/sys/vpc/inst/dom-<domainId>/if-<ifId>/fault-F3274",

Intersight Connected ACI Fabrics

This fault is actively monitored as part of Proactive ACI Engagements.

If you have an Intersight connected ACI fabric, a Service Request was generated on your behalf to indicate that instances of this fault were found within your Intersight Connected ACI fabric.

Quick Start Video

Video: Address ACI Fault Code F3274: fabric-encap-mismatch

Quick Start to Address Fault

1. Copy the ACI Pre-Upgrade Validation Script onto an APIC within the ACI Fabric where this fault was flagged

2. Run the script

3. Look for the "Overlapping VLAN Pools" check results to identify which EPGs were found to have multiple domains related to distinct but overlapped VLAN Pools

4. Based on the output, plan for an outage window^1 to address the multiple domains related to distinct but overlapped VLAN pools config on each Identified EPG

5. At the time of the outage window, update the Access Policies associated with the overlapped VLAN pools config on highlighted EPGs. Corrected config can be achieved by any of these approaches:

- Approach 1 - Each identified EPG with Domains each related to a distinct VLAN Pool with a distinct set of VLANs, overlaps removed

- Approach 2 - Each identified EPG with Domains that have converged to VLAN Pools that have converged on non-overlapped VLANs

- Approach 3 - Each identified EPG with associated Domains converged to a single Domain with all required VLANs

6. If Access Policy correction results in a switch no longer having a reference to a given VLAN Pool, then the VLAN is automatically redeployed with a new Fabric Encap from remaining VLAN pools. A brief outage occurs when the VLAN is reprogrammed. Otherwise, the VLAN must be manually redeployed^2 to allocate a new VXLAN ID.

1A brief outage occurs when the VLAN is redeployed.

2A VLAN is reprogrammed when the VLAN declaration config is redeployed;a static port binding, vmm domain assignment, or AEP EPG binding. This results in an outage until the VLAN has completed redeployment. If a large number of bindings need to be reprogrammed on a given leaf node, all VLANs can be reprogrammed with a clean reload of the switch(es) in question after Access Policies correction. A clean reload is performed when you issue an "acidiag touch clean" prior to a reload.

ACI Pre-upgrade Validation Script Example

When the ACI Pre-Upgrade Validation Script is run on an APIC, identified EPGs are flagged under the "Overlapping VLAN Pools" check:

Example Output:

[Check 29/36] Overlapping VLAN Pools...                                          FAIL - OUTAGE WARNING!!

Tenant   AP   EPG     VLAN Pool (Domain) 1     VLAN Pool (Domain) 2

------   --   ---     --------------------     --------------------

MY_T     AP1  EPG1-1  VLAN_POOL_1 (DOM_1)      VLAN_POOL_2 (DOM_2)

Reference Document: "Overlapping VLAN Pool" from from Pre-Upgrade Check Lists

[Check 30/37] VNID Mismatch...                                                   FAIL - OUTAGE WARNING!!

EPG                                 Access Encap  Node ID  Fabric Encap

---                                 ------------  -------  ------------

uni/tn-MY_T/ap-AP1/epg-EPG1-1       vlan-768      101      vxlan-8660

uni/tn-MY_T/ap-AP1/epg-EPG1-1       vlan-768      103      vxlan-8492

Recommended Action: Remove any domains with overlapping VLAN Pools from above EPGs, then redeploy VLAN

Reference Document: "Overlapping VLAN Pool" from Pre-Upgrade Check Lists

Given the example output, EPG EPG1-1 must have both DOM_1 and DOM_2 domains evaluated to identify which VLAN blocks within VLAN_POOL_1 and VLAN_POOL_2 contain overlap and why.

Additional Details

Please note that this specific condition is documented within the Cisco APIC Installation and ACI Upgrade and Downgrade Guide: Overlapping VLAN Pool section.

As the condition outlined from this fault can lead to datapath issues post-upgrade, logic to identify overlapped VLAN pools already exists within the ACI Pre-Upgrade Validation Script which is currently available on github.

Issues Induced by fabric-encap-mismatch

An ACI fabric with fabric-encap-mismatches derived from an overlap of VLAN blocks can result in:

- Packet loss due to issues in endpoint learns

- Spanning tree loops due to mismatched BPDU forwarding domains

These issues might not manifest until after an upgrade or clean reload of the affected switches. Leaf switches fetch policy from the APICs after an upgrade or clean-reload and may or may not not apply the same VLAN ID from the same pool that was used prior. As a result, the VLAN ID can get mapped to a different VXLAN VNID compared to other switch nodes. Remediation of this issue removes the uncertainty involved in a VLAN reprogram event.

Future Prevention

It is critical to ensure that there are no overlapped VLAN pools in your fabric unless it is an intentional design choice to re-use VLANs for distinct customers. This type of design requires additional config considerations not outlined in this document. If unsure, consider the "Enforce EPG VLAN Validation" setting under System > System Settings > Fabric Wide Settings within the APIC GUI. Available with release 3.2(6) and up, this setting prevents the most common problematic configuration: two domains with overlapped VLAN pools associated to the same EPG.

Related Information

These documents contain additional information on Overlapped VLAN pools, why its an issue and how this scenario occurs:

- Overlap VLAN pool Lead Intermittent Packet Drop to VPC Endpoints and Spanning-tree Loop

- ACI: Common migration issue / Overlapping VLAN pools

- Validating Overlapping VLANs in the Cisco APIC Layer 2 Networking Configuration Guide, Release 4.2(x)