This document explains how Nexus 9000 ternary content-addressable memory (TCAM) carving works. It cover the current and most common concepts, configuration, and error messages.
This document is not comprehensive - there are too many TCAM carving combinations to cover. The purpose of this document is to help users understand how the TCAM allocation works so they can come up with valid configurations that meet their needs.
If you want to use a non-default feature for Nexus 9000 Series switches, you must manually carve out TCAM space for the features. By default all TCAM space is allocated.
Feature Width - There are single-width and double-width features. A single-width feature requires at minimum one slice. A double-width feature at minimum requires two slices. For both single- and double-width features, the total size, if greater than 256, must be a multiple of 512. A slice can be allocated to one region only. For example, you cannot use a 512-size slice in order to configure two features of size 256 each nor can you use a 512-size slice in order to configure a single double-width feature.
Slice - A unit of memory allocation. Slices can be of size 256 or of size 512, measured in bytes.
TCAM - Ternary Content Addressable Memory. This is the space in hardware where access-lists (ACLs) are stored. This is a specialized piece of memory that stores complex tabular data and supports very rapid parallel lookups.
ACL TCAM Regions
You can change the size of the ACL TCAM regions in the hardware. The egress TCAM size is 1K, divided into four 256 entries. The ingress TCAM size is 4K, divided into eight 256 slices and four 512 slices.
The IPv4 TCAM regions are single wide. The IPv6, quality of service (QoS), MAC, control-plane policing (CoPP) , and system TCAM regions are double wide and consume double the physical TCAM entries. For example, a logical region size of 256 entries actually consumes 512 physical TCAM entries.
You can create IPv6, port ACLs (PACLs), VLAN ACLs (VACLs), and router ACLs (RACLs), and you can match IPv6 and MAC addresses for QoS. However, Cisco NX-OS cannot support all of them simultaneously. You must remove or reduce the size of the current TCAM regions (TCAM carving) in order to enable the IPv6 and MAC TCAM regions. For every TCAM region configuration command, the system evaluates if the new change can fit in the TCAM. If not, it reports an error, and the command is rejected. You must remove or reduce the size of current TCAM regions in order to make room for new requirements.
ACL TCAM region sizes have these guidelines and limitations:
On Cisco Nexus 9500 Series switches, the default ingress TCAM region configuration has one free 256-entry slice in Cisco NX-OS Release 6.1(2)I1(1). This slice is allocated to the switch port anaylzer (SPAN) region in Cisco NX-OS Release 6.1(2)I2(1). Similarly, the RACL region is reduced from 2K to 1.5K in Cisco NX-OS Release 6.1(2)I2(1) in order to make room for the virtual port-chanel (vPC) convergence region with 512 entries.
On Cisco Nexus 9300 Series switches, the Application Centric Infrastructure (ACI) leaf line card is used in order to enforce the QoS classification policies applied on 40G ports. It has 768 TCAM entries available for carving in 256-entry granularity. These region names are prefixed with "ns-".
For the ACI leaf line card on Cisco Nexus 9300 Series switches, only the IPv6 TCAM regions consume double-wide entries. The rest of the TCAM regions consume single-wide entries.
When a VACL region is configured, it is configured with the same size in both the ingress and egress directions. If the region size cannot fit in either direction, the configuration is rejected.
Both the Nexus 9300 and 9500 Series switches have four slices of size 512 bytes and eight slices of size 256 bytes. By default, all slices and all space are used, though the default allocation is different between the Nexus 9300 series and the 9500 series.
Note: The Nexus 9332PQ uses the same default allocation as the Nexus 9500.
Nexus 9500 Series TCAM Allocation
The Nexus 9500 Series switches have this TCAM allocation by default:
Nexus9500# show system internal access-list globals
slot 1 =======
Atomic Update : ENABLED Default ACL : DENY Bank Chaining : DISABLED Fabric path DNL : DISABLED NS Buffer Profile: Mesh optimized Min Buffer Profile: all EOQ Class Stats: qos-group-0 NS MCQ3 Alias: qos-group-3 Ing PG Share: ENABLED
In order to reconfigure a TCAM region, use the hardware access-list tcam region <feature_name> <feature_size> command in the configuration terminal. Once you have changed the regions to be the intended sizes, you must reload the device.
You have a Nexus 9300 and want to allocate the TCAM space in order to best fit your needs. You need to free up 512 bytes of TCAM. This allows you to add more to IPv4 PACL. However, you decide that you do not need 512 VACL or 512 RACL, but need some of both so you decide to unallocate 256 bytes from VACL and RACL. This frees up 512 space as these commands show:
Nexus9300(config)# hardware access-list tcam region vacl 256 Warning: Please save config and reload the system for the configuration to take effect
Nexus9300(config)# hardware access-list tcam region racl 256 Warning: Please save config and reload the system for the configuration to take effect
With 512 bytes free, you try to allocate an additional 512 to IPv4 PACL, but see this output:
Nexus9300(config)# hardware access-list tcam region ifacl 1024 ERROR: Aggregate TCAM region configuration exceeded the available Ingress TCAM slices. Please re-configure.
Even though 512 bytes were freed up, both the VACL and RACL space, which 256 were pulled from, were size 512 blocks. As such, the previous commands unallocated space, but did not unallocate any slices. In order to increase IPv4 PACL's size to 1024, you need to take 512 bytes from a single feature which frees up both a slice and space:
Nexus9300(config)# hardware access-list tcam region vacl 512 Warning: Please save config and reload the system for the configuration to take effect
Nexus9300(config)# hardware access-list tcam region racl 0 Warning: Please save config and reload the system for the configuration to take effect
Nexus9300(config)# hardware access-list tcam region ifacl 1024 Warning: Please save config and reload the system for the configuration to take effect
show hardware access-list tcam region - Verifies the current software configuration
show system internal access-list globals - Verifies the current hardware configuration
show system internal access-list input entries detail - Shows the specific ACLs configured for each instance
show hardware access-list resource utilization - Shows the current utilization of each configured TCAM region
show hardware access-list resource entries - Shows the number of ACL entries configured for each instance
Errors and Solutions
These are the common errors you can see in a TCAM configuration:
ERROR: Aggregate TCAM region configuration exceeded the available Ingress TCAM slices. Please re-configure.
This error occurs when you try to configure a valid amount of TCAM space with regards to the 4k limit, but your allocation consumes more slices than are available. The only solution for this error is to revisit your intended TCAM carving design in order to free up slices. This error is more common when you try to configure a new double-width feature, as they require at least two slices of 256 or 512.
ERROR: Aggregate TCAM region configuration exceeded the available Ingress TCAM space. Please re-configure.
As with the slice error, the only solution is to reconfigure. This error message is only seen when all TCAM slices have already been allocated and you try to allocate more space.
ERROR: TCAM regions with size more than 256, should have size in multiple of 512 entries
Due to hardware limitations, TCAM sizes above 256 cannot be combined in any manner that combines an odd number of 256 blocks with a 512 block. For this reason, when you configure a TCAM region that is larger than 512, the only valid sizes are multiples of 512.
Design Guidelines and Limitations
TCAM space is limited. The choice for what is best for you depends entirely on the specific use case. By default, all TCAM space is already allocated, so you need to decide where you want to 'steal' TCAM space from in order to allocate elsewhere.
In the case of ingress, four of the eight available size-256 slices cannot be unallocated (used by CoPP and ingress system).
One 256 slice is used by SPAN. If you borrow from this, it removes the ability to use the SPAN and Packet-Tracer features completely (not recommended to remove for troubleshooting purposes).
A size 256 or 512 slice is used for vPC on the Nexus 9300 and 9500 platforms respectively. Stealing from this will remove the ability to use vPC
A size 512 or 256 slice is used for Redirect on the Nexus 9300 and 9500 platforms respectively. If you borrow from this, it removes the ability to use DHCPv4, DHCPv6, or BFD.
If atomic updates are enabled, and you are over 50% utilization for one TCAM feature, you cannot remove a line from any ACL due to lack of space.
By default QoS policy applied on multiple interfaces does not share the label since statistics are enabled by default. In order to share the label for the same QoS policy applied on multiple interfaces, you have to configure the QoS policy with the no-stats option as this example shows:
(config-if)# service-policy type qos input my-policy no-stats
Where possible users should use the 'lite' version of features. With the 'lite' versions, the switch uses half of the TCAM space for that feature. This causes a double-width feature to be single-width. The cost is that the feature does not keep track of confirmed policer traffic. It only keeps track of violated policer packets. Since most users are only concerned with drop traffic, this is usually the better option as it saves TCAM space.
Users cannot reduce the default amount of Ingress System and CoPP TCAM. These are already at the minimum value and cannot be reduced.