ACI fabric can be partitioned into groups of 4K VLANs to allow a large number of layer 2 domains across the fabric, which can be used by multiple tenants. A VLAN domain represents a set of VLANs that can be configured on group of nodes and ports. VLAN domains let multiple tenants share and independently manage common fabric resources such as nodes, ports, and VLANs. A tenant can be provided access to one or more VLAN domains. For more information about VLAN pools, see Endpoint Groups in the ACI Policy Model chapter in Cisco Application Centric Infrastructure Fundamentals.

VLAN domains can be static or dynamic. Static VLAN domains support static VLAN pools, while dynamic VLAN domains can support both static and dynamic VLAN pools. VLANs in static pools are managed by the user and are used for applications such as connectivity to bare metal hosts. VLANs in the dynamic pool are allocated and managed by the APIC without user intervention and are used for applications such as VMM. The default type for VLAN domains and VLAN pools within the domain is static.

The fabric administrator performs the following steps before tenants can start using the fabric resources for their L2/L3 configurations:

1. Create VLAN domains and assign VLANs in each VLAN domain.

2. Assign the external facing ports on the leaf switches to one or more VLAN domains.

3. Convert a port to L2/L3 by using the [no] switchport command. The default state of a port is L2(switchport) in trunk mode.

4. For an L2 port, set the scope of a VLAN on a port to be global or local. The default is global.

The fabric administrator can update any configuration in these steps even after VLAN domains are assigned to tenants and are in use by tenant applications.

A Note About Spanning Tree and VLAN Domains

Although the ACI fabric does not participate in spanning tree, it can partition a spanning tree domain based on access policy configuration. ACI does not rely on a bridge domain or its settings to determine spanning tree domains. Instead, leaf switches flood BPDUs within the same VLAN encapsulation, if a VLAN Pool is assigned to EPG domains. The VLAN pool assigned to EPG domains ultimately serves as the spanning tree domain.

Using multiple EPG domains tied to different VLAN Pools does not allow BPDUs to flood across endpoints properly, even if they are all using the same VLAN ID. The type of EPG domain, (physical or Layer 2 external), does not change this behavior.

Because the ACI Fabric floods all BPDUs from all devices within a spanning-tree domain, this may trigger behaviors on external devices that are verifying BPDU info, such as the MAC address per interface. An example of a feature that activates is "spanning-tree EtherChannel misconfig guard" found on IOS devices. These features should be taken into account when utilizing ACI as a Layer 2 Tunnel.

Note	Multiple Spanning Tree (MST) is not supported on interfaces configured with the Per Port VLAN feature (configuring multiple EPGs on a leaf switch using the same VLAN ID with localPort scope).

Using Cisco APIC, you can map double-tagged VLAN traffic ingressing on a regular interface, PC, or vPC to an EPG. When this feature is enabled, when double-tagged traffic enters the network for an EPG, both tags are processed individually in the fabric and restored to double-tags when egressing the ACI switch. Ingressing single-tagged and untagged traffic is dropped.

This feature is only supported on Nexus 9300-FX platform switches.

Both the outer and inner tag must be of EtherType 0x8100.

MAC learning and routing are based on the EPG port, sclass, and VRF, not on the access encapsulations.

QoS priority settings are supported, derived from the outer tag on ingress, and rewritten to both tags on egress.

EPGs can simultaneously be associated with other interfaces on a leaf switch, that are configured for single-tagged VLANs.

Service graphs are supported for provider and consumer EPGs that are mapped to Q-in-Q encapsulated interfaces. You can insert service graphs, as long as the ingress and egress traffic on the service nodes is in single-tagged encapsulated frames.

The following features and options are not supported with this feature:

• Per-Port VLAN feature

• FEX connections

• Mixed Mode is not supported. For example, an interface in Q-in-Q encapsulation mode can have a static path binding to an EPG with double-tagged encapsulation only, not with regular VLAN encapsulation.

• STP and the “Flood in Encapsulation” option

• Untagged and 802.1p mode

• Multi-pod and Multi-Site

• Legacy bridge domain

• L2Out and L3Out connections

• VMM integration

• Changing a port mode from routed to Q-in-Q encapsulation mode is not supported

• Per-vlan MCP is not supported between ports in Q-in-Q encapsulation mode and ports in regular trunk mode.

• When vPC ports are enabled for Q-in-Q encapsulation mode, VLAN consistency checks are not performed.

A switch is in NPV mode after enabling NPV. NPV mode applies to an entire switch. All end devices connected to a switch that are in NPV mode must log in as an N port to use this feature (loop-attached devices are not supported). All links from the edge switches (in NPV mode) to the NPV core switches are established as NP ports (not E ports), which are used for typical inter-switch links.

FC Topology

The topology of a typical configuration supporting FC traffic over the ACI fabric consists of the following components:

• A Leaf can be connected to a FC switch by using FCoE NP port or native FC NP port.

• An ACI Leaf can be directly connected with a server/Storage using FCoE links.

• FC/FCoE traffic is not sent to fabric/spine. A Leaf switch does not do local switching for FCoE traffic. The switching is done by a core switch which is connected with a leaf switch via FC/FCoE NPV link.

• Multiple FDISC followed by Flogi is supported with FCoE host and FC/FCoE NP links.

With Cisco ACI and Cisco APIC Release 2.2(1x) and higher, you can configure 802.1Q tunnels on edge (tunnel) ports to enable point-to-multi-point tunneling of Ethernet frames in the fabric, with Quality of Service (QoS) priority settings. A Dot1q Tunnel transports untagged, 802.1Q tagged, and 802.1ad double-tagged frames as-is across the fabric. Each tunnel carries the traffic from a single customer and is associated with a single bridge domain. ACI front panel ports can be part of a Dot1q Tunnel. Layer 2 switching is done based on Destination MAC (DMAC) and regular MAC learning is done in the tunnel. Edge-port Dot1q Tunnels are supported on second-generation (and later) Cisco Nexus 9000 series switches with "EX" on the end of the switch model name.

With Cisco ACI and Cisco APIC Release 2.3(x) and higher, you can also configure multiple 802.1Q tunnels on the same core port to carry double-tagged traffic from multiple customers, each distinguished with an access encapsulation configured for each 802.1Q tunnel. You can also disable MAC Address Learning on 802.1Q tunnels. Both edge ports and core ports can belong to an 802.1Q tunnel with access encapsulation and disabled MAC Address Learning. Both edge ports and core ports in Dot1q Tunnels are supported on third-generation Cisco Nexus 9000 series switches with "FX" on the end of the switch model name.

Terms used in this document may be different in the Cisco Nexus 9000 Series documents.

The following guidelines and restrictions apply:

• Layer 2 tunneling of VTP, CDP, LACP, LLDP, and STP protocols is supported with the following restrictions:

  • Link Aggregation Control Protocol (LACP) tunneling functions as expected only with point-to-point tunnels using individual leaf interfaces. It is not supported on port-channels (PCs) or virtual port-channels (vPCs).

  • CDP and LLDP tunneling with PCs or vPCs is not deterministic; it depends on the link it chooses as the traffic destination.

  • To use VTP for Layer 2 protocol tunneling, CDP must be enabled on the tunnel.

  • STP is not supported in an 802.1Q tunnel bridge domain when Layer 2 protocol tunneling is enabled and the bridge domain is deployed on Dot1q Tunnel core ports.

  • ACI leaf switches react to STP TCN packets by flushing the end points in the tunnel bridge domain and flooding them in the bridge domain.

  • CDP and LLDP tunneling with more than two interfaces flood packets on all interfaces.

  • With Cisco APIC Release 2.3(x) or higher, the destination MAC address of Layer 2 protocol packets tunneled from edge to core ports is rewritten as 01-00-0c-cd-cd-d0 and the destination MAC address of Layer 2 protocol packets tunneled from core to edge ports is rewritten with the standard default MAC address for the protocol.

• If a PC or vPC is the only interface in a Dot1q Tunnel and it is deleted and reconfigured, remove the association of the PC/VPC to the Dot1q Tunnel and reconfigure it.

• With Cisco APIC Release 2.2(x) the Ethertypes for double-tagged frames must be 0x9100 followed by 0x8100.

  However, with Cisco APIC Release 2.3(x) and higher, this limitation no longer applies for edge ports, on third-generation Cisco Nexus switches with "FX" on the end of the switch model name.

• For core ports, the Ethertypes for double-tagged frames must be 0x8100 followed by 0x8100.

• You can include multiple edge ports and core ports (even across leaf switches) in a Dot1q Tunnel.

• An edge port may only be part of one tunnel, but a core port can belong to multiple Dot1q tunnels.

• With Cisco APIC Release 2.3(x) and higher, regular EPGs can be deployed on core ports that are used in 802.1Q tunnels.

• L3Outs are not supported on interfaces enabled for Dot1q Tunnels.

• FEX interfaces are not supported as members of a Dot1q Tunnel.

• Interfaces configured as breakout ports do not support 802.1Q tunnels.

• Interface-level statistics are supported for interfaces in Dot1q Tunnels, but statistics at the tunnel level are not supported.

Breakout cables are suitable for very short links and offer a cost effective way to connect within racks and across adjacent racks.

Breakout enables a 40 Gigabit (Gb) port to be split into four independent and logical 10Gb ports or a 100Gb port to be split into four independent and logical 25Gb ports.

Before you configure breakout ports, connect a 40Gb port to four 10Gb ports or a 100Gb port to four 25Gb ports with one of the following cables:

• Cisco QSFP-4SFP10G

• Cisco QSFP-4SFP25G

The 40Gb to 10Gb dynamic breakout feature is supported on the access facing ports of the following switches:

• N9K-C9332PQ

• N9K-C93180LC-EX

• N9K-C9336C-FX

Observe the following guidelines and restrictions:

• In general, breakouts and port profiles (ports changed from uplink to downlink) are not supported on the same port.

• Fast Link Failover policies are not supported on the same port with the dynamic breakout feature.

• Breakout subports can be used in the same way other port types in the policy model are used.

• When a port is enabled for dynamic breakout, other policies (expect monitoring policies) on the parent port are no longer valid.

• When a port is enabled for dynamic breakout, other EPG deployments on the parent port are no longer valid.

• A breakout sub-port can not be further broken out using a breakout policy group.

Microsegmentation with the Cisco Application Centric Infrastructure (ACI) provides the ability to automatically assign endpoints to logical security zones called endpoint groups (EPGs) based on various network-based or virtual machine (VM)-based attributes. This section contains instructions for configuring microsegment (uSeg) EPGs on virtual switches.

Microsegmentation with Cisco ACI provides support for virtual endpoints attached to the following:

• VMware vSphere Distributed Switch (VDS)

• Cisco Application Virtual Switch (AVS)

• Microsoft vSwitch

See the Cisco ACI Virtualization Guide for information about how Microsegmentation with Cisco ACI works, prerequisites, guidelines, and scenarios.

You can use Cisco APIC to configure Microsegmentation with Cisco ACI to create a new, attribute-based EPG using a network-based attribute, a MAC address or one or more IP addresses. You can configure Microsegmentation with Cisco ACI using network-based attributes to isolate VMs or physical endpoints within a single base EPG or VMs or physical endpoints in different EPGs.

IGMP snooping is the process of listening to Internet Group Management Protocol (IGMP) network traffic. The feature allows a network switch to listen in on the IGMP conversation between hosts and routers and filter multicasts links that do not need them, thus controlling which ports receive specific multicast traffic.

Cisco APIC provides support for the full IGMP snooping feature included on a traditional switch such as the N9000 standalone.

• Policy-based IGMP snooping configuration per bridge domain

  APIC enables you to configure a policy in which you enable, disable, or customize the properties of IGMP Snooping on a per bridge-domain basis. You can then apply that policy to one or multiple bridge domains.

• Static port group implementation

  IGMP static port grouping enables you to pre-provision ports, already statically-assigned to an application EPG, as the switch ports to receive and process IGMP multicast traffic. This pre-provisioning prevents the join latency which normally occurs when the IGMP snooping stack learns ports dynamically.

  Static group membership can be pre-provisioned only on static ports (also called, static-binding ports) assigned to an application EPG.

• Access group configuration for application EPGs

  An “access-group” is used to control what streams can be joined behind a given port.

  An access-group configuration can be applied on interfaces that are statically assigned to an application EPG in order to ensure that the configuration can be applied on ports that will actually belong to the that EPG.

  Only Route-map-based access groups are allowed.

Note

You can use vzAny to enable protocols such as IGMP Snooping for all the EPGs in a VRF. For more information about vzAny, see Use vzAny to Automatically Apply Communication Rules to all EPGs in a VRF. To use vzAny, navigate to Tenants > tenant-name > Networking > VRFs > vrf-name > EPG Collection for VRF.

IGMP static port grouping enables you to pre-provision ports, that were previously statically-assigned to an application EPG, to enable the switch ports to receive and process IGMP multicast traffic. This pre-provisioning prevents the join latency which normally occurs when the IGMP snooping stack learns ports dynamically.

Static group membership can be pre-provisioned only on static ports assigned to an application EPG.

Static group membership can be configured through the APIC GUI, CLI, and REST API interfaces.

An “access-group” is used to control what streams can be joined behind a given port.

An access-group configuration can be applied on interfaces that are statically assigned to an application EPG in order to ensure that the configuration can be applied on ports that will actually belong to the that EPG.

Only Route-map-based access groups are allowed.

IGMP snoop access groups can be configured through the APIC GUI, CLI, and REST API interfaces.

The port security feature protects the ACI fabric from being flooded with unknown MAC addresses by limiting the number of MAC addresses learned per port. The port security feature support is available for physical ports, port channels, and virtual port channels.

The guidelines and restrictions are as follows:

• Port security is available per port.

• Port security is supported for physical ports, port channels, and virtual port channels (vPCs).

• Static and dynamic MAC addresses are supported.

• MAC address moves are supported from secured to unsecured ports and from unsecured ports to secured ports.

• The MAC address limit is enforced only on the MAC address and is not enforced on a MAC and IP address.

• Port security is not supported with the Fabric Extender (FEX).

In the APIC, the user can configure the port security on switch ports. Once the MAC limit has exceeded the maximum configured value on a port, all traffic from the exceeded MAC addresses is forwarded. The following attributes are supported:

• Port Security Timeout—The current supported range for the timeout value is from 60 to 3600 seconds.

• Violation Action—The violation action is available in protect mode. In the protect mode, MAC learning is disabled and MAC addresses are not added to the CAM table. Mac learning is re-enabled after the configured timeout value.

• Maximum Endpoints—The current supported range for the maximum endpoints configured value is from 0 to 12000. If the maximum endpoints value is 0, the port security policy is disabled on that port.

IEEE 802.1x is a port-based authentication mechanism to prevent unauthorized devices from gaining access to the network. You can configure 802.1x port and node authentication using the NX-OS style CLI.

Proxy ARP in Cisco ACI enables endpoints within a network or subnet to communicate with other endpoints without knowing the real MAC address of the endpoints. Proxy ARP is aware of the location of the traffic destination, and offers its own MAC address as the final destination instead.

To enable Proxy ARP, intra-EPG endpoint isolation must be enabled on the EPG see the following figure for details. For more information about intra-EPG isolation and Cisco ACI, see the Cisco ACI Virtualization Guide.

Proxy ARP within the Cisco ACI fabric is different from the traditional proxy ARP. As an example of the communication process, when proxy ARP is enabled on an EPG, if an endpoint A sends an ARP request for endpoint B and if endpoint B is learned within the fabric, then endpoint A will receive a proxy ARP response from the bridge domain (BD) MAC. If endpoint A sends an ARP request for endpoint B, and if endpoint B is not learned within the ACI fabric already, then the fabric will send a proxy ARP request within the BD. Endpoint B will respond to this proxy ARP request back to the fabric. At this point, the fabric does not send a proxy ARP response to endpoint A, but endpoint B is learned within the fabric. If endpoint A sends another ARP request to endpoint B, then the fabric will send a proxy ARP response from the BD MAC.

The following example describes the proxy ARP resolution steps for communication between clients VM1 and VM2:

1. VM1 to VM2 communication is desired.

   

   Table 2. ARP Table State

   Device	State
   VM1	IP = * MAC = *
   ACI fabric	IP = * MAC = *
   VM2	IP = * MAC = *

2. VM1 sends an ARP request with a broadcast MAC address to VM2.

   

   Table 3. ARP Table State

   Device	State
   VM1	IP = VM2 IP; MAC = ?
   ACI fabric	IP = VM1 IP; MAC = VM1 MAC
   VM2	IP = * MAC = *

3. The ACI fabric floods the proxy ARP request within the bridge domain (BD).

   

   Table 4. ARP Table State

   Device	State
   VM1	IP = VM2 IP; MAC = ?
   ACI fabric	IP = VM1 IP; MAC = VM1 MAC
   VM2	IP = VM1 IP; MAC = BD MAC

4. VM2 sends an ARP response to the ACI fabric.

   

   Table 5. ARP Table State

   Device	State
   VM1	IP = VM2 IP; MAC = ?
   ACI fabric	IP = VM1 IP; MAC = VM1 MAC
   VM2	IP = VM1 IP; MAC = BD MAC

5. VM2 is learned.

   

   Table 6. ARP Table State

   Device	State
   VM1	IP = VM2 IP; MAC = ?
   ACI fabric	IP = VM1 IP; MAC = VM1 MAC IP = VM2 IP; MAC = VM2 MAC
   VM2	IP = VM1 IP; MAC = BD MAC

6. VM1 sends an ARP request with a broadcast MAC address to VM2.

   

   Table 7. ARP Table State

   Device	State
   VM1	IP = VM2 IP MAC = ?
   ACI fabric	IP = VM1 IP; MAC = VM1 MAC IP = VM2 IP; MAC = VM2 MAC
   VM2	IP = VM1 IP; MAC = BD MAC

7. The ACI fabric sends a proxy ARP response to VM1.

   

   Table 8. ARP Table State

   Device	State
   VM1	IP = VM2 IP; MAC = BD MAC
   ACI fabric	IP = VM1 IP; MAC = VM1 MAC IP = VM2 IP; MAC = VM2 MAC
   VM2	IP = VM1 IP; MAC = BD MAC

Consider these guidelines and limitations when using Proxy ARP:

• Proxy ARP is supported only on isolated EPGs. If an EPG is not isolated, a fault will be raised. For communication to happen within isolated EPGs with proxy ARP enabled, you must configure uSeg EPGs. For example, within the isolated EPG, there could be multiple VMs with different IP addresses, and you can configure a uSeg EPG with IP attributes matching the IP address range of these VMs.

• ARP requests from isolated endpoints to regular endpoints and from regular endpoints to isolated endpoints do not use proxy ARP. In such cases, endpoints communicate using the real MAC addresses of destination VMs.

A traffic storm occurs when packets flood the LAN, creating excessive traffic and degrading network performance. You can use traffic storm control policies to prevent disruptions on Layer 2 ports by broadcast, unknown multicast, or unknown unicast traffic storms on physical interfaces.

By default, storm control is not enabled in the ACI fabric. ACI bridge domain (BD) Layer 2 unknown unicast flooding is enabled by default within the BD but can be disabled by an administrator. In that case, a storm control policy only applies to broadcast and unknown multicast traffic. If Layer 2 unknown unicast flooding is enabled in a BD, then a storm control policy applies to Layer 2 unknown unicast flooding in addition to broadcast and unknown multicast traffic.

Traffic storm control (also called traffic suppression) allows you to monitor the levels of incoming broadcast, multicast, and unknown unicast traffic over a one second interval. During this interval, the traffic level, which is expressed either as percentage of the total available bandwidth of the port or as the maximum packets per second allowed on the given port, is compared with the traffic storm control level that you configured. When the ingress traffic reaches the traffic storm control level that is configured on the port, traffic storm control drops the traffic until the interval ends. An administrator can configure a monitoring policy to raise a fault when a storm control threshold is exceeded.

Configure traffic storm control levels according to the following guidelines and limitations:

• Typically, a fabric administrator configures storm control in fabric access policies on the following interfaces:

  • A regular trunk interface.

  • A direct port channel on a single leaf switch.

  • A virtual port channel (a port channel on two leaf switches).

• Beginning with the APIC Release 4.2(1), support is now available for triggering SNMP traps from Cisco ACI when storm control thresholds are met, with the following restrictions:

  • There are two actions associated with storm control: drop and shutdown. With the shutdown action, interface traps will be raised, but the storm control traps to indicate that the storm is active or clear is not determined by the shutdown action. Storm control traps with the shutdown action on the policy should therefore be ignored.

  • If the ports flap with the storm control policy on, clear and active traps are seen together when the stats are collected. Clear and active traps are typically not seen together, but this is expected behavior in this case.

• For port channels and virtual port channels, the storm control values (packets per second or percentage) apply to all individual members of the port channel. Do not configure storm control on interfaces that are members of a port channel.

  Note

  On switch hardware starting with the APIC 1.3(x) and switch 11.3(x) release, for port channel configurations, the traffic suppression on the aggregated port may be up to two times the configured value. The new hardware ports are internally subdivided into these two groups: slice-0 and slice-1. To check the slicing map, use the vsh_lc command show platform internal hal l2 port gpd and look for slice 0 or slice 1 under the Sl column. If port-channel members fall on both slice-0 and slice-1, allowed storm control traffic may become twice the configured value because the formula is calculated based on each slice.

• When configuring by percentage of available bandwidth, a value of 100 means no traffic storm control and a value of 0.01 suppresses all traffic.

• Due to hardware limitations and the method by which packets of different sizes are counted, the level percentage is an approximation. Depending on the sizes of the frames that make up the incoming traffic, the actual enforced level might differ from the configured level by several percentage points. Packets-per-second (PPS) values are converted to percentage based on 256 bytes.

• Maximum burst is the maximum accumulation of rate that is allowed when no traffic passes. When traffic starts, all the traffic up to the accumulated rate is allowed in the first interval. In subsequent intervals, traffic is allowed only up to the configured rate. The maximum supported is 65535 KB. If the configured rate exceeds this value, it is capped at this value for both PPS and percentage.

• The maximum burst that can be accumulated is 512 MB.

• On an egress leaf switch in optimized multicast flooding (OMF) mode, traffic storm control will not be applied.

• On an egress leaf switch in non-OMF mode, traffic storm control will be applied.

• On a leaf switch for FEX, traffic storm control is not available on host-facing interfaces.

• Traffic storm control unicast/multicast differentiation is not supported on Cisco Nexus C93128TX, C9396PX, C9396TX, C93120TX, C9332PQ, C9372PX, C9372TX, C9372PX-E, or C9372TX-E switches.

• SNMP traps for traffic storm control are not supported on Cisco Nexus C93128TX, C9396PX, C9396TX, C93120TX, C9332PQ, C9372PX, C9372TX, C9372PX-E, C9372TX-E switches.

• Traffic storm control traps is not supported on Cisco Nexus C93128TX, C9396PX, C9396TX, C93120TX, C9332PQ, C9372PX, C9372TX, C9372PX-E, or C9372TX-E switches.

• Storm Control Action is supported only on physical Ethernet interfaces and port-channel interfaces.

  Starting with release 4.1(1), Storm Control Shutdown option is supported. When the shutdown action is selected for an interface with the default Soak Instance Count, the packets exceeding the threshold are dropped for 3 seconds and the port is shutdown on the 3rd second. The default action is Drop. When Shutdown action is selected, the user has the option to specify the soaking interval. The default soaking interval is 3 seconds. The configurable range is from 3 to 10 seconds.

• Starting with release 4.1(1), Error Disable Recovery option for storm-control is supported for the ports which are in-error-disabled state due to storm shutdown action.

MACsec is an IEEE 802.1AE standards based Layer 2 hop-by-hop encryption that provides data confidentiality and integrity for media access independent protocols.

MACsec, provides MAC-layer encryption over wired networks by using out-of-band methods for encryption keying. The MACsec Key Agreement (MKA) Protocol provides the required session keys and manages the required encryption keys.

The 802.1AE encryption with MKA is supported on all types of links, that is, host facing links (links between network access devices and endpoint devices such as a PC or IP phone), or links connected to other switches or routers.

MACsec encrypts the entire data except for the Source and Destination MAC addresses of an Ethernet packet. The user also has the option to skip encryption up to 50 bytes after the source and destination MAC address.

To provide MACsec services over the WAN or Metro Ethernet, service providers offer Layer 2 transparent services such as E-Line or E-LAN using various transport layer protocols such as Ethernet over Multiprotocol Label Switching (EoMPLS) and L2TPv3.

The packet body in an EAP-over-LAN (EAPOL) Protocol Data Unit (PDU) is referred to as a MACsec Key Agreement PDU (MKPDU). When no MKPDU is received from a participants after 3 hearbeats (each hearbeat is of 2 seconds), peers are deleted from the live peer list. For example, if a client disconnects, the participant on the switch continues to operate MKA until 3 heartbeats have elapsed after the last MKPDU is received from the client.

APIC Fabric MACsec

The APIC will be responsible for the MACsec keychain distribution to all the nodes in a Pod or to particular ports on a node. Below are the supported MACsec keychain and MACsec policy distribution supported by the APIC.

• A single user provided keychain and policy per Pod

• User provided keychain and user provided policy per fabric interface

• Auto generated keychain and user provided policy per Pod

A node can have multiple policies deployed for more than one fabric link. When this happens, the per fabric interface keychain and policy are given preference on the affected interface. The auto generated keychain and associated MACsec policy are then given the least preference.

APIC MACsec supports two security modes. The MACsec must secure only allows encrypted traffic on the link while the should secure allows both clear and encrypted traffic on the link. Before deploying MACsec in must secure mode, the keychain must be deployed on the affected links or the links will go down. For example, a port can turn on MACsec in must secure mode before its peer has received its keychain resulting in the link going down. To address this issue the recommendation is to deploy MACsec in should secure mode and once all the links are up then change the security mode to must secure.

Note	Any MACsec interface configuration change will result in packet drops.

MACsec policy definition consists of configuration specific to keychain definition and configuration related to feature functionality. The keychain definition and feature functionality definitions are placed in separate policies. Enabling MACsec per Pod or per interface involves deploying a combination of a keychain policy and MACsec functionality policy.

Note	Using internal generated keychains do not require the user to specify a keychain.