Use Wireshark to Troubleshoot OTV Solutions

Introduction

This document demonstrates the use of Wireshark, a well-known freeware packet capture and analysis tool, in troubleshooting Cisco OTV solution. 

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Overlay Transport Virtualization (OTV) on Nexus Series Switches
• Basics of Multiprotocol Label Switching (MPLS) Layer 2 Virtual Private Netwoks (VPNs)
• Wireshark, a free and open source packet analyzer (https://www.wireshark.org)

Components Used

The information in this document is based on the Nexus 7000 Series Switch platform.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Problem Description

When troubleshooting network issues in VPN environments, one of the techniques involves the capture and analysis of encapsulated packets. However, in Cisco OTV network environments this approach is met with a certain challenge. Commonly used packet analysis tools, like Wireshark, a free and open source packet analyzer, may not correctly interpret the content of OTV-encapsulated traffic. Hence laborious workarounds, such as the extraction of encapsulated data from an OTV packet, are usually required to successfully perform data analysis.

OTV Packet Format

OTV encapsulation increases the overall MTU size of the packet by 42 bytes. This is the result of the operation of the OTV Edge Device that removes the CRC and the 802.1Q fields from the original Layer 2 frame and adds an OTV Shim (containing also the VLAN and Overlay ID information) and an external IP header.

In MPLS L2VPN solutions, devices in the underlay network don't have enough information to correctly decode the MPLS packet payload. Typically, this is not a problem, as packet forwarding in an MPLS core network is performed based on labels, therefore an in-depth analysis of the content of MPLS packets in the underlay network is not required.

However this presents a challenge if data analysis of OTV packets is required for troubleshooting and/or monitoring purposes.

Packet analysis tools, such as Wireshark, attempt to decode packet data which follows the MPLS header by applying regular MPLS packet parsing rules. However, since it may not have information about the results of Control Word negotiation, which would be normally performed between MPLS L2VPN Head-end and Tail-end routers, packet analysis tools falls back to default parsing behaviour and applies it to packet data which follows MPLS header.

Note: In MPLS L2VPN solutions, such as Any Transport Over MPLS (ATOM), pseudowire end-points negotiate the use of Control Word parameter.  A control word is an optional 4-byte field located between the MPLS label stack and the Layer 2 payload in the pseudowire packet. The control word carries generic and Layer 2 payload-specific information. If the C-bit is set to 1, the advertising Provider Edge (PE) expects the control word to be present in every pseudowire packet on the pseudowire that is being signaled. If the C-bit is set to 0, no control word is expected to be present.

As a result, default Wireshark parsing behaviour may not interpret content of OTV packets correctly, thus making troubleshooting process of OTV network more complex.

Topology

The following is a network diagram of a simple OTV network. Routers in Vlan 100 and Vlan 200 establish OSPF and EIGRP adjacencies between two DataCenters, DataCenter1 and DataCenter2, respectively. DataCenter Interconnect (DCI) is implemented with OTV tunnel between N7k switches, shown on the diagram as AED1 and AED2.

Note: Cisco OTV solution uses the concept of Authoritative Edge Device (AED) role, assigned to network device which encapsulates and decapsulates OTV traffic at a particular site. 

The challenge often seen in tunneling solutions is to verify whether a particular type of overlay packets (IGP, FHRP, etc.) makes it to certain points in underlay network. OSPF and EIGRP overlay traffic is used as an example.

Packet Capture

There are multiple ways to perform a packet capture in the network. One option is to use Cisco Switched Port Analyzer (SPAN) feature, available on Cisco Catalyst and Cisco Nexus switching platforms.

As part of troubleshooting process, packet captures at multiple points may need to be performed. OTV Join interfaces and interfaces in the underlay network can be used as SPAN packet capture point.

Solution

Wireshark default parsing engine may mis-interpret first few bytes of an OTV-encapsulated overlay packets as if they are a part of Pseudowire Emulation Edge-to-Edge (PWE3) Control Word,which is typically used in MPLS L2VPNs over an MPLS packet switched network.

Note: MPLS Pseudowire Emulation Edge-to-Edge (PWE3) Control Word is referred to as Control Word in the rest of this document.

To ensure Wireshark packet analysis tool interprets content of OTV-encapsulated packets correctly, manual adjustment to packet decode process is needed.

Note: MPLS label used in OTV header equals overlay vlan number + 32.

Decode Packets in Vlan 100

As a first step of decode process, display only OTV-encapsulated packets which carry content of OTV-extended vlan 100. Filter used is mpls.label == 132, which represents vlan 100.

Note: To display OTV-encapsulated packets for a particular vlan extended over OTV, use the following Wireshark display filter: mpls.label == <<vlan number extended over OTV> + 32>

Display OTV-encapsulated packets for Vlan 100, extended over OTV

By default Wireshark interprets the first four bytes of the content of MPLS L2VPN packets as Control Word. This needs to be corrected for OTV-encapsulated packets. To do this, right-click on MPLS label field of any of the packets, and chose Decode As... option.

Right-click on MPLS label field and choose Decode As... option

 The next step is to tell Wireshark that encapsulated content has no Control Word.

Choose "no CW" option

Once this change has been submitted by clicking OK button, Wireshark analysis tool will display content of OTV-encapsulated packets correctly.

Wireshark correctly displays content of OTV-encapsulated packets

Decode Packets in Vlan 200

Above steps are applicable for any vlan extended over OTV. For example, using Wireshark filter to display only packets of vlan 200, we get the following output in analysis tool.

Display packets for vlan 200, extended over OTV

 Once Wireshark is instructed not to interpret first few bytes of MPLS packet as PW Control Word, decode process can complete successfully.

WIreshark correctly displays Vlan 200 traffic as EIGRP packets

Use Editcap to Remove OTV Header

Typically, Wireshark installations come with a command line packet editing tool called Editcap. This tool can permanently remove OTV overhead from captured packets. This allows easy display and analysis of captured packets in the Wireshark Graphical User Interface (GUI), without the need to manually adjust Wireshark's parsing behaviour. 

Run Editcap on Windows Platform

On Windows operating system, editcap.exe is installed by default in c:\Program Files\Wireshark> directory.

Run this tool with -C flag to remove OTV overhead and save the result in a .pcap file.

c:\Users\cisco\Desktop> "c:\Program Files\Wireshark\editcap.exe" -C 42 otv-underlay-capture.pcap otv-underlay-capture-no-header.pcap
c:\Users\cisco\Desktop>

Run Editcap on Mac OS Platform

On Mac OS operating system, editcap is available in /usr/local/bin folder.

CISCO:cisco$ /usr/local/bin/editcap -C 42 otv-underlay-capture.pcap otv-underlay-capture-no-header.pcap
CISCO:cisco$

By removing the OTV header from captured  packets withEditcaptool, one loses Vlan information which is encoded as part of MPLS header, which is in turn a part of OTV shim. Remember to use 'mpls.label == <<vlan number extended over OTV> + 32>' Wireshark GUI filter before removing the OTV header with  Editcap tool, if analysis of traffic only of a particular Vlan is required.

Conclusion

Troubleshooting Cisco OTV solutions requires a good understanding of the technology, both from the control plane operation and data plane encapsulation perspective. Effectively applying the knowledge, freeware packet analysis tools such as Wireshark can prove very powerful in OTV packet analysis. In addition to various packet display options, typical Wireshark installation offers a packet editing tool which can simplify the packet analysis. This allows troubleshooting to be focused on the parts of the packet content which are most relevant to a particular troubleshooting session.