The management capabilities of a switch contribute greatly to the ease of deployment and maintenance of a network. Cisco® Catalyst® 6500 Series Switches include several monitoring features for Supervisor Engine 720-based systems. The next-generation Cisco Catalyst 6500 Series Supervisor Engine 2T, recently released with 12.2(50)SY software, introduces a host of new features in this area. Supervisor Engine 2T brings in the capability to view traffic statistics on a more granular level, which extends to interface counters, quality of service (QoS) packet counts and statistics related to control-plane policing. These enhancements, along with Cisco IOS NetFlow developments, help network administrators monitor traffic flows through the switch more closely. In addition, the new supervisor engine’s baseboard allows for remote console access and manageability to the switch via a Connectivity Management Processor (CMP). The Supervisor Engine 2T software also incorporates Embedded Event Manager (EEM) Version 3.0, an enhanced version of EEM. This paper discusses the key enhancements related to manageability, delivered by the Supervisor Engine 2T with Cisco IOS 12.2(50)SY.
A basic requirement in setting up and maintaining a switch is to have access to the device and to ensure reliable connectivity to it. The primary methods of accessing a switch in order to configure, verify or troubleshoot it are through a console or a Telnet/Secure Shell (SSH) connection. It is possible, however, for network issues to render both of these links inaccessible. For example, a high CPU condition or a denial-of-service (DoS) attack can effectively take down a Telnet connection. A hung Route Processor (RP) could lock up the console, leaving it impossible to recover or determine the state of the switch with no way to login to it. In such situations, network administrators may have to resort to resetting the switch to recover it at the cost of losing the ability to efficiently correct the problem. This, in turn, significantly contributes to network downtime.
A solution in such critical situations is to have a ‘backdoor’ access into the switch to help restore it. This is addressed on next-generation supervisor engines through a new processor called the Connectivity Management Processor (CMP), which exists in conjunction with the primary Route Processor (RP).
The CMP is an independent processor dedicated to switch management and has its own RAM, bootflash and front panel management Ethernet port. While the CMP and RP share the same console, a multiplexer enables switching between the two, providing access to the CMP even if the primary RP is hung. From within the CMP, a host of corrective actions can be taken to restore the switch. Examples of how the CMP can be used include system recovery (of the control plane), system resets and reboots and the copying of Cisco IOS image files in the event that the primary IOS image gets corrupted or deleted.
The Supervisor Engine 2T is the first supervisor in the Cisco Catalyst 6500 Series to provide this functionality. As part of the Multilayer Switch Feature Card 5 (MSFC5) complex that houses the CPU and serves as the control plane on the supervisor, CMP functionality has been added on the board (Figure 1). More details on the hardware can be found at the Cisco Catalyst 6500White Paper.
The CMP and the RP share the same console through a programmable multiplexer, which by default has the RP console active. The multiplexer intercepts specific escape sequences that instruct it to switch from one console to the other if required. The CMP runs its own image based on Linux, and the CMP and RP software run independent of each other. The block diagram of the MSFC5 in Figure 2 illustrates the structure of the RP and CMP.
The CMP on the Supervisor Engine 2T supports a front panel 10/100/1000 management port. This port supports auto negotiation, detection, and correction of pair swaps (MDI crossover) and support for jumbo frames up to 9216 bytes in size. It also uses a bi-color LED on the front panel port to show activity and link status.
The following sections highlight some of the device management functions that the CMP brings to the table.
- Ability to access the console remotely (SSH/Telnet) without a terminal server for troubleshooting and recovery (even when the RP is not responsive).
This can enable the CMP to be used as a console of last resort if the primary console locks up. Access to the CMP can be via either the console or the management port. On the console, Ctrl-C, Shift-M three times switches to the CMP console, and Ctrl-R, Shift-M three times switches back to the RP console. The dedicated out-of-band Ethernet management port can also be configured (from within the CMP) with an IP address, to enable SSH or Telnet access to the CMP. Access to the CMP requires a username and password. Once connected to the CMP, it is possible to attach to the RP as well.
Note: SSH is enabled by default on the management interface. Telnet is disabled by default, and has to be enabled as shown below before a Telnet session can be started.
The following example shows how to log in to the CMP:
From within the CMP, the management interface can be configured with an IP address and set up for Telnet access, as in the following example:
You can Telnet directly to the CMP and attach to the RP from there if required. The attach option only works with Telnet or SSH, and not from the front panel console.
Here is a list of CMP commands available from enable mode and config mode:
- Ability to quickly recover the IOS image remotely via a Trivial File Transfer Protocol (TFTP) server without the need for a terminal server or on-site personnel.
If IOS crashes and the IOS image on the Compact Flash is corrupted or deleted, the CMP can be used to boot an image from ROM monitor (ROMmon) mode. Once the cmpmgmt port is configured with an IP address and connectivity is established to a TFTP server, you can boot the image from the TFTP server using the following command in ROMmon:
boot tftp://<tftp server address>/<file path>/<file name>
- Ability to reset the Supervisor Engine 2T remotely without any external power management devices.
The Supervisor Engine 2T can be reset from the CMP using the following commands:
The reset reason can be verified using the show version command on the RP, and with the show system reset-reason command on the CMP.
- Ability to log messages from the RP to the CMP
In a situation where the primary console is unresponsive, CMP provides the ability to access the switch logs, in order to determine the possible causes. This would help in taking appropriate corrective action to restore the switch. The capability to view RP logs exists even when the RP is hung, since the messages previously sent by the RP are stored by the CMP in a flash partition reserved for syslogs.
While exporting logs to a syslog server is a feature available only from the RP, the CMP stores these files using a Linux syslog server. Linux "logrotate" is used for archiving the old files in a compressed format. It is then possible to transfer the archived files or the current file to a TFTP or Secure Copy (SCP) server using the copy command.
CMP implementation does not support features such as SNMP and Network Time Protocol (NTP). However, the CMP gets regular clock updates from Cisco IOS Software, which itself uses NTP.
Cisco IOS Embedded Event Manager (EEM) is a unique subsystem within Cisco IOS Software. EEM is a powerful and flexible tool to automate tasks and customize the behavior of Cisco IOS Software and the operation of the device. Customers can use EEM to create and run programs or scripts directly on a router or switch. The scripts are referred to as EEM policies and can be programmed using a simple CLI-based interface or using a scripting language called Tool Command Language (Tcl). EEM allows customers to harness the significant intelligence within Cisco IOS Software to respond to real-time events, automate tasks, create customer commands and take local automated action based on conditions detected by the Cisco IOS Software itself.
EEM was first introduced on the Cisco Catalyst 6500 Series in Cisco IOS Software Release 12.2(18)SXF4. The software version 12.2(33)SXI on the Supervisor 720 supports EEM Version 2.4. The Supervisor Engine 2T with 12.2(50)SY software supports EEM Version 3.0 which includes a set of new event detectors, among other features.
The Cisco IOS Embedded Event Manager is a product-independent software feature consisting of a series of event detectors, an Embedded Event Manager Server and interfaces that allow action routines, called policies, to be invoked. There are also internal application programming interfaces that enable other Cisco IOS subsystems to take advantage of the EEM subsystem. Figure 3 illustrates the components of EEM.
Notice that there are two types of EEM policies:
● Applet policies: Applet policies have an easy-to-use interface and are defined using the configuration CLI. They can be saved to the startup configuration file.
● Tcl policies: These policies are more flexible and have extensive capabilities. Tcl policies are defined using the Tcl programming language.
Once one or more policies are defined, the Event Detector software will watch for the conditions that match those defined by the policy. When a condition occurs, the event is passed to the Event Manager Server. The server in turn invokes any policy that has registered for that particular event. The actions defined within the policy are then carried out.
The current version of the EEM subsystem supported on the Supervisor Engine 2T is EEM Version 3.0. This version ushers in a significant number of enhancements over previous versions, including enhanced performance, increased feature integration, new capabilities and extended flexibility, enabling EEM to be used in new and exciting ways.
A list of event detectors originally supported on the platform can be found here:.
Included in EEM 3.0 are four new event detectors:
Routing Event Detector
This event detector monitors the events relative to the Routing Information Base (RIB). Events are raised for conditions such as when a particular route is added or removed, or when a route is modified.
For example, here is how a notification can be logged if a static route is added:
Trigger the policy by configuring a static route:
Verify that the log message is generated:
Flexible NetFlow (FNF) Event Detector
This event detector can be used to detect events related to Flexible NetFlow. It provides a powerful set of triggers to detect and react to real-time network activity. It triggers policies based on the detection of flows that match particular criteria, such as when a new flow is seen with a particular destination IP address and port number. This event detector also detects conditions such as when the rate of new flow entries exceeds a defined threshold.
Figure 4 shows a Flexible NetFlow Event Detector that is configured to be triggered any time a TTL=0 packet is detected through NetFlow. Once this packet is detected, a syslog message is generated to inform the network operations team that such a flow record has been detected.
IP SLA Event Detector
This event detector provides event triggers based on Cisco IOS IP Service Level Agreements (SLA) operation results. It integrates IP SLA directly with the EEM subsystem and provides an event-driven mechanism to take immediate action when an IP SLA operation fails. Figure 5 shows a sample use case with the IP SLA Routing Event Detector.
Enhanced CLI Event Detector
This event detector offers enhancements to make creation of custom CLI commands easier and more powerful. It provides new event triggers when special characters like “Tab”, “?”, and the “Enter” key are detected. It also provides a way to offer “Help” for the new commands and make them similar to commands developed by Cisco.
Once the CLI Event Detector is configured, there are commands available to verify that the EEM applet has been registered successfully. Here is an example to verify the registration of the applet, with the event CLI for the “?” character:
EEM 3.0 introduces several other features, noteworthy among which are the applet enhancements and high performance Tcl policies. Applets now include support for variables and logical functions, and if-then-else constructs. Here is a list of the major new functions added for customer usability:
● Class based scheduling
● High-performance Tcl policies
● Interactive CLI support with applets
● Variable logic for applets
● Digital signature support
● Support authenticating e-mail servers
● SMTP IPv6 support
● SNMP Tcl extensions (Get, Set and Notify for local and remote hosts)
● SNMP Proxy IPv6 support
● CLI Library XML-PI support
More information on these features can be found in the following document:.
Traditionally, interface counters viewed under the show interfaces command have reflected packet and byte counts for the following classes of traffic:
● L2 Bridged Unicast
● L2 Bridged Multicast
● L3 IN Unicast
● L3 IN Multicast
● L3 OUT Unicast
● L3 OUT Multicast
Here, L2 packets refer to those that arrive and leave on the same Layer 2 VLAN, and L3 packets imply that the input packet before forwarding and the output packet after forwarding are on different Layer 2 VLANs. These six counters reflect packet and byte counts, as seen in the following output:
This structure has been changed to now include eight counters on the Supervisor Engine 2T with per-protocol statistics, due to the implementation of Logical Interfaces (LIFs).
Two new concepts are introduced on the Supervisor Engine 2T: Bridge Domains (BDs) and Logical Interfaces (LIFs). Bridge domains are used to represent Layer 2 broadcast domains, including VLANs. Logical interfaces are used to represent all types of Layer 3 interfaces, including (but not limited to) Switched Virtual Interfaces (SVIs), tunnel interfaces, routed ports, subinterfaces and Layer 3 EtherChannel interfaces.
A LIF on the Supervisor Engine 2T essentially maps to a port-VLAN combination, allowing for the capability to provide different services for different customers per interface. Prior to the introduction of Supervisor Engine 2T, VLANs were used internally by the system to represent not only Layer 2 VLANs but also Layer 3 interfaces, and the number of available VLANs was limited to 4K. With the introduction of LIFs on the Supervisor Engine 2T, the number of available routed interfaces has increased to 128K and the number of broadcast domains (equivalent of VLANs) has increased to 16K (4K at first software release). Figures 6 and 7 show the differences between Supervisor Engine 2T, and Supervisor Engines 32 and 720.
Compared to its Supervisor Engine 720 counterpart where every Layer 3 interface was associated with a VLAN, the Supervisor Engine 2T associates each Layer 3 interface with a unique logical interface value or LIF. So while statistics were previously collected on a per-VLAN basis, statistics are now collected on a per-LIF basis.
There are 128K primary LIFs available on the Supervisor Engine 2T, and the forwarding engine has a component dedicated to capturing LIF statistics. The Policy Feature Card 4 (PFC4) on the Supervisor Engine 2T contains a Layer 2 forwarding engine and a Layer 3 forwarding engine, with additional components as shown in Figure 8.
The LIF statistics module is attached to the Layer 2 forwarding engine and is used to collect packet counts for all Layer 3 hardware switched non-recirculated packets. This includes statistics for the following:
● Physical routed ports
● Layer 3 port-channels
The support for counters under subinterfaces is a new feature on the Supervisor Engine 2T. It is important to note that LIF statistics are not available for tunnel interfaces due to the above definition regarding non-recirculated packets. Recirculated packets refer to packets that require multiple lookups to be forwarded through the switch (for example, GRE-encapsulated packets). Also, since a loopback interface does not have a LIF associated with it, these statistics are not available for loopback interfaces either.
For each of the primary LIFs, there are now eight pairs of counters (packet and byte) being maintained, for IPv4, IPv6, multicast and others. These counters are explained in detail in the section. The counters are obtained from the LIF statistics module shown in Figure 8.
For the first 16K LIFs that map to the bridge domains (BDs), there are an additional two pairs of counters (packet and byte) obtained from the adjacency statistics module, attached to the Layer 3 forwarding engine, as shown in Figure 8. This includes counters for Layer 2 bridged packets (one pair of counters for unicast, and a second pair for multicast).
Thus, a Layer 3 interface would be mapped to eight pairs of counters (from the LIF stats) and a VLAN and SVI would be associated with 10 pairs of counters (eight pairs from LIF stats and two pairs from adjacency stats).
The logical flow in collecting LIF statistics is as follows: When a packet arrives on the ingress port, a LIF is associated with the (port, VLAN) combination. This is called the Ingress LIF. The packet then passes through Layer 2 and Layer 3 engine look-ups for the forwarding decisions. For Layer 2 packets, since they are switched within the same VLAN (or BD), the bridged counters will be incremented. For Layer 3 packets, the FIB (Forwarding Information Base) table returns an Adjacency pointer which contains the Egress LIF information. The Egress LIF consists of a VLAN and a destination index (which maps to the egress port). For packets that egress on an SVI, the Egress LIF corresponds to the BD-LIF of the VLAN. After processing by the forwarding engines, the LIF statistics module, which is attached to the Layer 2 forwarding engine, collects the statistics for both the Ingress LIF and Egress LIF. The adjacency statistics module maintains the counters for Layer 2 packets.
For each of the primary LIFs, there are now eight pairs of counters being maintained, which can be viewed through CLI outputs. Each pair of counters consists of a packet counter and a byte counter, and the following classes of traffic are now identified by default with the new structure:
● L3 IN IPv4 Unicast
● L3 OUT IPv4 Unicast
● L3 IN IPv4 + IPv6 Multicast
● L3 OUT IPv4 + IPv6 Multicast
● L3 IN IPv6 Unicast
● L3 OUT IPv6 Unicast
● L3 IN MPLS
● L3 OUT MPLS
Figure 9 shows a logical view of the LIF statistics counters. Each pair of counters is represented with 72 bits, comprising a 31-bit packet counter, a 40-bit byte counter and 1-bit parity for data integrity.
Correspondingly, the interface counters reflect the new fields as can be seen in the subsequent output. Packet and byte counts can be viewed for Layer 2 and Layer 3 switched traffic in both directions. Counters are also available for unicast IPv4 and IPv6, and for multicast traffic (IPv4 and IPv6 combined). Customers who are running a dual-stack IPv4 and IPv6 environment can now monitor these flows individually. In addition, support for Multiprotocol Label Switching (MPLS) counters has been added.
The following CLI outputs are grouped into three sets:
a) Layer 3 interface counters
b) VLAN counters
c) Interface accounting and interface stats
The counters for a Layer 2 switchport remain the same as in previous implementations.
Layer 3 Interface Counters
The following output highlights the new counters available:
VLAN counters consist of 10 pairs of counters: eight pairs from LIF statistics that map to the same counters as above, and two pairs from adjacency statistics. In the following outputs, the highlighted counters (Layer 3) are from LIF statistics, and the remaining counters (Layer 2) are from adjacency statistics:
The show vlan counters detail output shows all the eight LIF counters:
Interface Accounting and Interface Stats
Another command that makes use of LIF statistics is the show interfaces accounting command. The command can be used to get ingress Layer 3 switched protocol counters for IPv4, IPv6 and MPLS, as the following output shows:
The totals shown above include both hardware switched and software switched packets.
The show interfaces stats command includes packet counts that tally with ‘show interfaces accounting’, with the ingress counters for Distributed cache being pulled from LIF statistics.
Here, the breakdown of the switching paths is as follows:
Processor: Packets that are software processed (non-interrupt path)
Route cache: Packets that take the software interrupt path (CEF switched packets)
Distributed cache: Packets forwarded in hardware
For the Distributed cache statistics, Ingress stats (Pkts In and Chars In) are obtained from LIF statistics and Egress stats (Pkts Out and Chars Out) are obtained from Adjacency statistics.
NetFlow is a process designed to collect information about traffic flows that pass through the switch. The Supervisor Engine 2T is equipped with a PFC4 that supports all the NetFlow features that existed on the PFC3 (Supervisor Engine 720) and adds these new capabilities:
● Sampled NetFlow: Allows users to have NetFlow records created based on a sample of traffic matching the flow. The Supervisor Engine 2T with PFC4 performs NetFlow sampling in hardware, whereas the PFC3-based supervisors performed NetFlow sampling in software.
● Increased support for NetFlow entries: Up to 512K NetFlow entries (no ingress or egress limitation) can now be stored in the PFC4. Up to 1 million NetFlow entries (512K for ingress and 512K for egress) can now be stored in the PFC4XL. Both of these represent a quadrupling of the capabilities of their PFC3-based predecessors.
● Improved NetFlow hash: The hash efficiency is improved to 99 percent, allowing a greater percentage of the NetFlow table to be utilized.
● Egress NetFlow: Provides support for collecting flow statistics for packets after they have had ingress processing applied to them and the final destination of the packets has been determined.
● Layer 2 NetFlow: NetFlow hardware is capable of creating and tracking bridged IP flows as opposed to traditional NetFlow where flows would be created and tracked only for IP traffic that gets routed.
● Flexible NetFlow: This is Cisco’s next-generation NetFlow technology, featuring customized traffic identification and simultaneous tracking of multiple NetFlow applications. It supports the NetFlow Version 9 record format, including new fields for IPv6 and multicast information.
● TCP flags: TCP flags (SYN, FIN, RST, ACK, URGENT, PUSH) are now collected as part of a flow record.
● NetFlow with Embedded Event Manager: As described earlier, EEM Version 3.0 also supports a new event detector for Flexible NetFlow.
More details on the NetFlow features available on Supervisor Engine 2T can be found in thepaper on Cisco.com.
Control plane policing (CoPP) is a critical feature in ensuring that the CPU of the supervisor engine is protected and regulated from traffic spikes. The Supervisor Engine 2T introduces new hardware rate limiters (HWRL) and CoPP features to further aid in control-plane protection. Detailed information on CoPP and HWRL can be found at thepaper on Cisco.com.
From the monitoring point of view, the Supervisor Engine 2T adds packet-based counters to CoPP in addition to byte-based counters. The HWRL have been enhanced with several new rate limiters to make a total of 31 Layer 3 rate limiters and 26 Layer 2 rate limiters on the platform (as compared to eight Layer 3 rate limiters and four Layer 2 rate limiters on PFC3-based systems). Supervisor Engine 2T includes HWRL counters for forwarded, dropped, and leaked packets. Another feature that has been introduced is the ability to apply CoPP on exceptions such as IP-option, TTL-failure, etc. The rate-limiters available are shown in the output below.
It is also possible to monitor control-plane traffic on a per-flow basis using Flexible NetFlow to develop realistic traffic rates, which can then be used in developing custom control-plane service policies. This is described in detail in the CoPP paper referenced above.
With Cisco IOS Software Release 12.2(50)SY, the Supervisor Engine 2T adds enhancements to several of the existing MIBs, and introduces a few new ones, as described in Table 1.
Table 1. New Platform-Specific MIBs Introduced with Cisco IOS Software Release 12.2(50)SY
Provides SNMP access to the Switch NetFlow component
Provides SNMP access to the configuration and monitoring of traffic statistics on Cisco’s switching devices
Provides SNMP access to the Cisco TrustSec® component
Provides SNMP access to the Cisco TrustSec component
Provides SNMP access to the Cisco TrustSec Role-based Access Control (RBAC), Layer 3 transport and policy feature
Provides SNMP access to the Cisco TrustSec component
Provides SNMP access to the Cisco TrustSec component
The Cisco Catalyst 6500 Series Supervisor Engine 2T with Cisco IOS Software Release 12.2(50)SY is a platform with an extensive set of monitoring features. The forwarding engine on Supervisor Engine 2T allows for increased capabilities and traffic analysis in areas such as QoS, NetFlow and SNMP. The platform’s support for EEM 3.0 puts it on par with router platforms that run Cisco IOS Software, paving the way for a consistent and scalable method to customize device monitoring and automate tasks. Release 12.2(50)SY has also enhanced existing Cisco management innovations such as, Generic Online Diagnostics , and Onboard Failure Logging ( ), which can be used to simplify monitoring, fault detection, and fault isolation. Overall, these features can be used to assist in comprehensive capacity planning, application assessment, network troubleshooting and security operations.