Understanding Fabric Policies
Now that ACME has been provisioned with ACI fabric and infrastructure space has been configured between the leaf and spine switches, access privileges, and basic management policies, it is time to start creating connectivity policies within the ACI fabric. The fabric tab in the APIC GUI is used to configure system-level features including, but not limited to, device discovery and inventory management, diagnostic tools, configuring domains, and switch and port behavior. The fabric pane is split into three sections: inventory, fabric policies, and access policies. Understanding how fabric and access policies configure the fabric is key for the ACME network teams, as they will be maintaining these policies for the purposes of internal connections between fabric leaf nodes, connections to external entities such as servers, networking equipment, and storage arrays. It is important that other teams such as server teams understand these concepts as well, as they will be interacting with them, particularly in the case of their build processes for adding additional capacity.
This chapter will review the key objects in the access policies subsection of the fabric tab -- many of which are reusable; demonstrate how to add and pre-provision switches, and walk through the steps and objects required when new devices are added to the fabric to effectively operate an ACI fabric. While many policies are reusable, it is also key to understand the implications of deleting policies on the ACI fabric.
The access policies subsection is split into folders separating out different types of policies and objects that affect fabric behavior. For example, the interface policies folder is where port behavior is configured, like port speed, or whether or not to run protocols like LACP on leaf switch interfaces is set. Domains and AEPs are also created in the access policies view. The fabric access policies provide the fabric with the base configuration of the access ports on the leaf switches.
Fabric - Access Policies
Endpoint groups are considered the "who" in ACI; contracts are considered the "what/when/why"; AEPs can be considered the "where" and domains can be thought of as the "how" of the fabric. Different domain types are created depending on how a device is connected to the leaf switch. There are four different domain types: physical domains, external bridged domains, external routed domains, and VMM domains.
Physical domains are generally used for bare metal servers or servers where hypervisor integration is not an option.
External bridged domains are used for Layer 2 connections. For example, an external bridged domain could be used to connect an existing switch trunked-up to a leaf switch.
External routed domains are used for Layer 3 connections. For example, an external routed domain could be used to connect a WAN router to the leaf switch.
Domains act as the glue between the configuration done in the fabric tab to the policy model and endpoint group configuration found in the tenant pane. The fabric operator creates the domains, and the tenant administrators associate domains to endpoint groups.
Ideally, policies should be created once and reused when connecting new devices to the fabric. Maximizing the reusability of policy and objects makes day-to-day operations exponentially faster and easier to make large-scale changes. The usage of these policies can be viewed by clicking the Show Usage button in the Application Policy Infrastructure Controller (APIC) GUI. Use this to determine what objects are using a certain policy to understand the impact when making changes.
For an in-depth whiteboard explanation on domains, watch the following video titled "How Devices Connect to the Fabric: Understanding Cisco ACI Domains": https://www.youtube.com/watch?v=_iQvoC9zQ_A.
VLAN pools contain the VLANs used by the EPGs the domain will be tied to. A domain is associated to a single VLAN pool. VXLAN and multicast address pools are also configurable. VLANs are instantiated on leaf switches based on AEP configuration. Allow/deny forwarding decisions are still based on contracts and the policy model, not subnets and VLANs.
Attachable Access Entity Profiles
Attachable Access Entity Profiles (AEPs) can be considered the "where" of the fabric configuration, and are used to group domains with similar requirements. AEPs are tied to interface policy groups. One or more domains can be added to an AEP. By grouping domains into AEPs and associating them, the fabric knows where the various devices in the domain live and the Application Policy Infrastructure Controller (APIC) can push the VLANs and policy where it needs to be. AEPs are configured under the global policies section.
Most of the policies folders have subfolders. For example, under the interface policies folder there are folders for configuration called policies, policy groups, and profiles.
There are also policies for switches - for example, configuring vPC domains, which are called explicit vPC protection groups in the Application Policy Infrastructure Controller (APIC) GUI and vPC policies. Ideally, policies should be created once and reused when connecting new devices to the fabric. Maximizing reusability of policy and objects makes day-to-day operations exponentially faster and easier to make large-scale changes.
Switch Policy Groups
Switch policy groups allow leveraging of existing switch policies like Spanning Tree and monitoring policies
Switch profiles allow the selection of one or more leaf switches and associate interface profiles to configure the ports on that specific node. This association pushes the configuration to the interface and creates a Port Channel or vPC if one has been configured in the interface policy.
The following figure highlights the relationship between the various global, switch, and interface policies:
Interface policies dictate interface behavior, and are later tied to interface policy groups. For example, there should be a policy that dictates if CDP is disabled and a policy that dictates if CDP is enabled; these can be reused as new devices are connected to the leaf switches.
Interface Policy Groups
Interface policy groups are templates to dictate port behavior and are associated to an AEP. Interface policy groups use the policies described in the previous paragraph to specify how links should behave. These are also reusable objects as many devices are likely to be connected to ports that will require the same port configuration. There are three types of interface policy groups depending on link type: Access Port, Port Channel, and vPC.
The ports on the leaf switches default to 10GE, and a 1GE link level policy must be created for devices connected at that speed. Regarding Port Channels and vPC, each policy group designated a single logical interface on the switches. If you want to create 10 PCs/vPCs, you must create 10 policy groups. Access port policy groups can be reused between interfaces. Policy groups do not actually specify where the protocols and port behavior should be implemented. The "where" happens by associating one or more interface profiles to a switch profile, covered in the following paragraphs.
Interface profiles help tie the pieces together. Interface profiles contain blocks of ports - interface selectors - and are also tied to the interface policy groups described in the previous paragraphs. Again, this is just an arbitrary port, such as e1/1, the profile must be associated to a specific switch profile to configure the ports.
Layer 2 Interface Policy
In the Cisco APIC release 1.1, a configurable interface policy was added to allow a per port-VLAN significance.
To connect devices to the Cisco Application Centric Infrastructure (Cisco ACI) fabric we can use untagged traffic, 802.1Q tagged traffic, or VXLAN encapsulation. A per-port VLAN allows the same VLAN to be used for different endpoint groups, providing that traffic is coming in on a different port. Prior to release 1.1, a VLAN could only be tied to one endpoint group per leaf.
In traditional networking one of the limitations related to VLAN encapsulation is scalability and reusability due to the limit of 4096 VLANs in networking devices.
In Cisco ACI, with the default configuration (global), EPGs can use the same VLAN encapsulation as long as EPGs are bound to separate switches. This allows tenants to re-use VLAN encapsulation IDs through the fabric without allowing communication between tenants. However, global configuration assumes that tenants do not share leaf switches and therefore there is no VLAN overlapping within the same leaf.
When per port-VLAN is used, a port and VLAN pair (P,V) is registered internally instead of just a VLAN encapsulation ID. This increases the consumption of hardware resources at a per switch level.
Two EPGs belonging to a single bridge domain cannot share the same encapsulation ID on a given leaf switch.
It is expected that the port will flap when the Layer 2 interface policy changes between global and local. That is, traffic will get affected.
DWDM-SFP10G-C Optic Interface Policy
The Cisco APIC release 3.1(1) added support for the DWDM-SFP10G-C optic, which includes an interface policy for the optic. When a DWDM-SFP10G-C port is inserted, the port by default has channel number 32. In the DWDM-SFP10G-C optic interface policy, you can change the channel number to any number between 1 to 96, which tunes the optic to the corresponding wavelength.
Cisco has established several best practices for fabric configuration. These are not requirements and might not work for all environments or applications, but might help simplify day-to-day operation of the Cisco Application Centric Infrastructure (ACI) fabric.
- Reuse policies whenever possible. For example, there should be policies for LACP active/passive/off, 1GE port speed, and 10GE port speed.
- When naming policies, use names that clearly describe the setting. For example, a policy that enables LACP in mode active could be called "LACP-Active". There are many "default" policies out of the box. However, it can be hard to remember what all the defaults are, which is why policies should be clearly named to avoid making a mistake when adding new devices to the fabric.
- Create a switch profile for each leaf switch individually, and additionally, create a switch profile for each vPC pair (if using vPC).
- Build one physical domain per tenant for bare metal servers or servers without hypervisor integration requiring similar treatment.
- Build one physical domain per tenant for external connectivity.
- If a VMM domain needs to be leveraged across multiple tenants, a single VMM domain can be created and associated with all leaf ports where VMware ESXi servers are connected.
- Multiple domains can be associated to a single AEP for simplicity's sake. There are some cases where multiple AEPs may need to be configured to enable the infrastructure VLAN, such as overlapping VLAN pools, or to limit the scope of the presence of VLANs across the fabric.