Congestion in networks happens more often than expected. Identifying which application or flow is experiencing congestion can be difficult.
The new Cisco® SPAN-on-Drop feature in the Cisco Nexus® 5600 switches allows the user to identify the applications experiencing congestion in the network and correlate packet drop with applications.
Note: Everything discussed in this document paper applies to both the Cisco Nexus 5600 platform and Cisco Nexus 6000 Series of Switches.
SPAN-on-Drop enables the Cisco Switched Port Analyzer (SPAN) feature to be applied to packets that would normally be dropped due to lack of available buffer or queue space on ingress. With SPAN-on-Drop, instead of dropping a packet when congestion occurs, the system stores the packet in a separate SPAN-on-Drop buffer and then sends the packet to the specified SPAN-on-Drop destination port. SPAN-on-Drop with Encapsulated Remote SPAN (ERSPAN) is an extension of this feature in which the dropped frames are spanned and sent to a remote IP address instead of a local port.
The Cisco Nexus 5600 platform supports high-performance SPAN.
Figure 1 shows the use of line-rate SPAN in this switch.
Figure 1. Line-Rate SPAN
The Cisco Nexus 5600 switches are designed to have twice the bandwidth from the port application-specific integrated circuit (ASIC) to the fabric, making line-rate SPAN possible (120-Gbps of regular traffic plus 120-Gbps of SPAN replicated packets). The switches have four times the bandwidth from the fabric to the port ASIC, increasing the likelihood that a downlink will be available from the fabric to the egress unified port controller (UPC) and avoiding fabric congestion.
The switches also have a dedicated buffer space for SPAN (which is very small compared to the buffer space for data traffic), so SPAN-on-Drop does not affect production traffic. Data traffic is always given higher priority than SPAN traffic.
How SPAN-on-Drop Works
As shown in Figure 2, if the ingress data buffer becomes full, the network will start to experience tail drop. The ingress data buffer may become full if the port is oversubscribed (that is, if the port receives much more data that it can handle). Port-speed mismatch, buffer discrepancies, and congestion in the network are some of the many reasons that a port may be oversubscribed.
Figure 2. SPAN-on-Drop
Cisco Nexus 5600 platform switches have a separate SPAN buffer. The data packets are stored in the data buffer and then transported to the destination, until the buffer becomes full. When the data buffer is completely full, the data packets start to be dropped. All the packets that are dropped due to buffer exhaustion are moved to the SPAN buffer, which can then be spanned to a local or remote host. In this way, all the applications suffering from packet loss can be identified by running a packet analyzer such as Wireshark at the monitored destination.
SPAN-on-Drop uses the existing SPAN configuration in Cisco NX-OS Software.
The configuration examples presented below use this scenario: A lot of ports are sending data to port 3/1. At some point, the buffers for port 3/1 start to fill up, leading to tail drops. To identify which application is experiencing loss, you can configure a SPAN-on-Drop session using port 3/1 as the source, send the SPAN data to a local or remote host running Wireshark, and then decode all the packets that were dropped. With this approach, you can easily debug drops and identify the application being degraded by congestion.
SPAN-on-Drop with Local Destination SPAN Port
This configuration creates a SPAN session with type SPAN-on-DROP. In the following example, the source interface, where congestion may be present, is port e3/1. The destination port is e3/2, which must be in switchport monitor mode.
switch(config)# monitor session <session_number>
switch(config-SPAN-on-DROP)# source interface e3/1
switch(config-SPAN-on-DROP)# destination interface e3/2
SPAN-on-Drop with ERSPAN
This configuration creates a SPAN session with type SPAN-on-DROP-erspan. In the following example, the source interface, where congestion may be present, is port e3/1. The destination port is a remote host with IP address 220.127.116.11.
switch(config)# monitor session <session_number>
switch(config-SPAN-on-DROP-erspan)# source interface e3/1
switch(config-SPAN-on-DROP-erspan)# destination ip 18.104.22.168
Note: The destination port configuration and restrictions are the same as for a local SPAN session. Access control list (ACL)-based SPAN is not supported for SPAN-on-Drop sessions. Maximum transmission unit (MTU) truncation also is not supported with SPAN-on-Drop.
To verify that the monitor session was correctly configured, use the command show monitor session <session_number>.
Note the following guidelines when using the SPAN-on-Drop feature:
● The feature works for unicast packets and not multicast because packet drops can be monitored on ingress only. For Multicast, packets are dropped at Egress in case of congestion.
● The source interfaces can only be Ethernet, although they can be part of a PortChannel. Sources also can be a part of a SPAN-on-Drop session and a local SPAN session simultaneously. Fabric extender (HIF) interfaces are not supported as sources; however, fabric (NIF) interfaces are supported. Setting a fabric interface as a source allows SPAN-on-Drop to be enabled on all fabric extender ports associated with that fabric interface.
● Only one SPAN-on-drop or SPAN-on-drop ERSPAN session can be active at a time. However, you can have multiple source ports and multiple destination ports. Source ports can be part of a SPAN-on-Drop session and a local SPAN session simultaneously.
● Direction on the source interface is not supported. The command-line interface (CLI) does not support direction for source interfaces such as local SPAN or ERSPAN; however, as an option, you can specify that an interface is an ingress interface.
The SPAN-on-Drop feature can be helpful in troubleshooting applications that are experiencing loss in the network. By applying SPAN to packets that are dropped, the user can easily identify the packets that are being dropped and determine whether the packets are being dropped because of network congestion or misconfiguration and so on.
For More Information