Understanding unusual TCAM utilization levels on Nexus 7000

Introduction

This document explains how TCAM is unexpectedly overutilized depending on the feature sets enabled on a Nexus 7000 series switch.

Problem description

With atomic updates enabled, in case of more than one feature such as RACL, NetFlow is applied on different interfaces, it can cause TCAM overuse error although the 50% limit has not been reached.

Example: Unable to apply multiple features onto the subsequent sub-interfaces, despite the utilization being 29.57 whereas the upper limit is 50% (as atomic updates are enabled) and the below error is thrown.

ERROR: Module 1 returned status: TCAM will be overused, please enable bank chaining and/or turn off the atomic update. If bank-chaining is enabled on other modules and this is a new line card insertion, please enable bank-chaining prior to reloading this module.

Also, after reload, all interfaces lose configuration and get assigned to VDC 0 for this reason.

Background

Atomic updates which are enabled by default on Nexus 7000 allow only 50% of the entire TCAM to be utilized. The other 50% is reserved to accommodate ACL changes so as to provide non-disruptive updates of ACLs. More information about this is available through links provided in the section: Recommended reading.

Prerequisites

It is recommended to know the following topics:

Atomic updates
TCAM on N7000

Components used

The information in this document is based on the lab testing performed on N7718 chassis on the module: N77-F312CK-26 running on version 8.3(2)
All of the devices used in this document started with a default configuration and the features exhibited are RACL, NetFlow.

Observation

Egress RACL configured on main and subinterfaces whereas Netflow enabled only on main interfaces

Initial configuration:

Interface Ethernet 11/2: Layer-3; RACL 
Interface Ethernet 11/2.300-302: Layer-3; RACL

TCAM utilization with just RACL applied is:

N7718(config-if)# show system internal access-list resource utilization module 11 | in "Tcam 1, Bank 0" across all instances:
Tcam 1, Bank 0 1211 2885 29.57 
Tcam 1, Bank 0 1211 2885 29.57
Tcam 1, Bank 0 1211 2885 29.57

Note: TCAM space utilization for the Egress RACL in this document is assumed to be 29.57%

Once, the NetFlow is applied only on the main interface, the interface utilization doubles although NetFlow configuration in this situation barely takes 1% space.

Interface Ethernet 11/2: Layer-3; RACL; NetFlow
Interface Ethernet 11/2.300-302: Layer-3; RACL

Here the main interface has NetFlow configured whereas sub interface has no NetFlow( Atomic updates to be disabled to observe this behavior)

N7718(config-if)# show system internal access-list resource utilization module 11 | in "Tcam 1, Bank 0"
Tcam 1, Bank 0 2394 1702 58.45
Tcam 1, Bank 0 2394 1702 58.45
Tcam 1, Bank 0 2394 1702 58.45

Note: In case, atomic updates were still enabled, this doubling behavior wouldn't be possible since the limit with atomic updates is only 50% and the following error would be seen:

ERROR: Module 1 returned status: TCAM will be overused, please enable bank chaining and/or turn off the atomic update. If bank-chaining is enabled on other modules and this is a new line card insertion, please enable bank-chaining prior to reloading this module.

Explanation:

In this case, two different policy sets are present here. One destination has RACL alone and another destination has RACL +NF, Hence two sets of TCAM entries are being allocated for the same features, causing the doubling behavior that we see as the actual consumption should only be 29.57 behavior.

The device achieves this by generating two separate labels for both interfaces as observed below:

module-11# show system internal access-list interface e11/2 out statistics

INSTANCE 0x0
---------------

Tcam 1 resource usage:
----------------------
Label_b = 0x801 >>> LABEL is 0x801

module-11# show system internal access-list interface e11/2.300 out statistics

INSTANCE 0x0
---------------

Tcam 1 resource usage:
----------------------
Label_b = 0x802 >>> NEW LABEL 0x802 IS GENERATED

When both RACL and Netflow are configured on main interface and sub-interface

Already present configuration from Case 1:

Interface Ethernet 11/2: Layer-3; RACL; NetFlow
Interface Ethernet 11/2.300-302: Layer-3; RACL

Now, Apply NetFlow on the rest of the sub-interfaces as well:

Interface Ethernet 11/2: Layer-3; RACL; NetFlow
Interface Ethernet 11/2.300-302: Layer-3; RACL; NetFlow

Since all the destinations now have RACL + Netflow configured, the same label is being shared b/w destination(single set of TCAM entries being referenced by all interfaces).

N7718(config-if)# show system internal access-list resource utilization module 11 | in "Tcam 1, Bank 0"
Tcam 1, Bank 0 1211 2885 29.57
Tcam 1, Bank 0 1211 2885 29.57
Tcam 1, Bank 0 1211 2885 29.57


module-11# show system internal access-list interface ethernet11/2 out statistics |in Label_b p 5 n 4


INSTANCE 0x0
Tcam 1 resource usage:
----------------------
Label_b = 0x802 >>> LABEL is 0x802

module-11# show system internal access-list interface ethernet11/2.300 out statistics |in Label_b p 5 n 4


INSTANCE 0x0
Tcam 1 resource usage:
----------------------
Label_b = 0x802 >>> SAME LABEL IS MAINTAINED

Note: This behavior is extended to physical interfaces, sub-interfaces as well. Only, when all destinations in concern have the same set of configuration, will the TCAM utilization not be doubled.

Do observe that the only after the application of Netflow on all the interfaces already having RACL will the TCAM reduce to original 29.57%

1. Apply RACL on interface "1": 29.57%

2. Apply RACL on subsequent interfaces: 29.57%

3. Apply NF after RACL application on interface "1": 58.45%

4. Apply NF on subsequent interfaces: 58.45%

5. Apply NF on last interface: 29.57%

Resolution

1. Disable atomic updates.

<OR>

2. Decrease ACL size so as to keep the limit to <25%.

Summary

With Atomic updates:

Once, Netflow is applied on the first interface, the creation of a separate TCAM instance is attempted as the first interface now has both ACL, NF configured but the second interface has only RACL configured.
However, due to atomic updates being enabled, a separate instance creation fails as doing so will increase the utilization to >50. Consequently, TCAM over-used error is thrown.

Without Atomic updates:

1. On applying ACL on all interfaces: It remains at 29 since there are no other features.
2. Apply NetFlow to the first interface: Switch assumes this to be a separate configuration of features/combination of features (maintains a separate internal label) and hence, creates a separate instance on the same bank.
3. Once NetFlow is applied to all other interfaces that have ACL configured for, the configuration/combination of features are the same (Label is now the same for both interfaces) and hence, re-shuffling occurs
4. TCAM has now shared for both interfaces and the utilization drops back to 29.57%.

This is a method of optimization when combinations of features are used on different interfaces.

Related Defects

CSCvs50014 ACL and Netflow on subinterface occupy double TCAM entries

Related Articles

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus7000/sw/security/config/cisco_nexus7000_security_config_guide_8x/configuring_ip_acls.html

https://www.cisco.com/c/en/us/td/docs/switches/datacenter/sw/6_x/nx-os/security/configuration/guide/b_Cisco_Nexus_7000_NX-OS_Security_Configuration_Guide__Release_6-x/b_Cisco_Nexus_7000_NX-OS_Security_Configuration_Guide__Release_6-x_chapter_01110.html#con_1458580