Catalyst 6500 Series Switches Netflow TCAM Utilization Management

Introduction

This document describes a problem encountered on Cisco Catalyst 6500 Series switches when the Netflow Ternary Content Addressable Memory (TCAM) threshold is exceeded and provides a solution to the problem.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on Cisco Catalyst 6500 Series switches that run Supervisor Engine 720.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Background Information

Netflow is a feature used in order to collect statistics on the traffic that traverses a switch. The statistics are then stored in the Netflow table until they are exported by Netflow Data Expert (NDE). There is a Netflow table on the Policy Feature Card (PFC), as well as on each Distributed Forwarding Card (DFC). Some features, such as Network Address Translation (NAT), require the flow to be processed in the software initially, and then hardware-accelerated. The Netflow table on the PFC and DFC collects statistics for traffic that is hardware-accelerated or flow-switched.

Some features use Netflow, such as NAT and Quality of Service (QoS). NAT uses Netflow in order to make forwarding decisions, while QoS uses Netflow in order to monitor flows for micropolicing. With use of Netflow Data Export (NDE), you have the ability to export these statistics to an external Netflow collector for further analysis of the network behavior.

The Supervisor Engine 720 polls how full the NetFlow table is at each poll interval and activates aggressive aging when the table size reaches a set threshold.

When the table is nearly full, there are new active flows that cannot be created because of the lack of available space in the TCAM. At this point, it makes sense to more aggressively age-out the less-active or non-active flows in the table in order to create space for new flows. The flow can be reinserted into the table, as long as it meets the configured timeout and packet threshold values, which are discussed later in this document.

Problem

The Cisco Catalyst 6500 Series switch might report this log:

EARL_NETFLOW-4-TCAM_THRLD: Netflow TCAM threshold exceeded, TCAM Utilization [[dec]%] 

Here is the console output that is displayed when this problem occurs:

Aug 24 12:30:53: %EARL_NETFLOW-SP-4-TCAM_THRLD:
 Netflow TCAM threshold exceeded, TCAM Utilization [97%]

Aug 24 12:31:53: %EARL_NETFLOW-SP-4-TCAM_THRLD:
 Netflow TCAM threshold exceeded, TCAM Utilization [97%]

Solution

Complete these steps in order to assess and optimize Netflow TCAM utilization:

1. Disable service internal if it is enabled on the switch:

   6500(config)#no service internal

2. Check the hardware limits for Netflow TCAM.
   • Use the show mls netflow ip count command in order to check the number of flows present in the TCAM. 
   • Use the show platform hardware pfc mode command in order to check the PFC operating mode.

   Note: The capacity for NetFlow TCAM (IPv4) for PFC3A, PFC3B, and PFC3C is 128,000 entries. For  PFC3BXL and PFC3CXL, the capacity is 256,000 entries.

3. Prepare to alter the flowmask. Netflow uses the concept of masks. The Netflow mask allows you to control the volume and granularity of the statistics collected. This allows you to control the impact on the Supervisor Engine processors. The more specific the mask used, the more Netflow table entries used.
   
   For example, if you configure to have the Statistics set to flows per interface-source IP address, you use fewer entries than if you kept flows per interface-destination-source.
   
   If the flowmask is set to interface-full mode, then the TCAM for NetFlow can overflow, depending on how many intefaces for which it is enabled. Issue the show mls netflow ip count command in order to check this information. Even though you can change masks, the interface-full mode provides the most granular statistics, such as information about Layers 2, 3, and 4.
4. Check the current flowmask:

   6500#show mls netflow flowmask
    current ip   flowmask for unicast:   if-full
    current ipv6 flowmask for unicast:    null  

   Alter the flowmask as required (interface-full flow keyword sets the maximum TCAM entries used):

   6500(config)#mls flow ip ?
     interface-destination         interface-destination flow keyword
     interface-destination-source  interface-destination-source flow keyword
     interface-full                interface-full flow keyword
     interface-source              interface-source only flow keyword

5. Check the aging timers. There are three different timers for Netflow TCAM aging: Normal, Fast, and Long.
   • The Normal timer is used in order to clear inactive TCAM entries. By default, any entry that is not matched in 300 seconds is cleared.
   • The Long timer is used in order to clear entries that are in the table for more than 1,920 seconds (32 minutes). The main purpose of the Long timer is to prevent the incorrect statistics caused by counters that wrap.
   • The Fast timer, by default, is not enabled. In order to enable the Fast timer, use the mls aging fast [{time seconds} [{threshold packet-count}]] global command. The Fast timer clears any entry that does not see the configured number of packets within the configured time.

   6500#show mls netflow aging
   
                enable timeout  packet threshold
                ------ -------  ----------------
   normal aging true      300         N/A
   fast aging   true       32         100
   long aging   true     1920         N/A

6. Change the aging timers:

   6500(config)#mls aging normal ?
     <32-4092> L3 aging timeout in second
   
   6500(config)#mls aging long ?
     <64-1920> long aging timeout
   
   6500(config)#mls aging fast ?
     threshold fast aging threshold
     time fast aging timeout value
   
   6500(config)#mls aging fast threshold ?
     <1-128> L3 fast aging theshold packet count
     time fast aging timeout value
   
   6500(config)#mls aging fast time ?
     <1-128> L3 fast aging time in seconds
     threshold fast aging threshold

   If you enable the Fast timer, set the value to 128 seconds initially. If the size of the MLS cache continues to grow over 32,000 entries, then decrease the setting until the cache size remains less than 32,000. If the cache still continues to grow over 32,000 entries, then decrease the Normal MLS aging timer. Any aging-timer value that is not a multiple of eight seconds is adjusted to the closest multiple of eight seconds.

   6500(config)#mls aging fast threshold 64 time 30

Related Information

• Catalyst 6500 Release 12.2SX Software Configuration Guide
• Introduction to Cisco IOS NetFlow - A Technical Overview
• Technical Support & Documentation - Cisco Systems