Configure Nexus 9000: Packet Tracer Tool

Introduction

This document describes how to configure Cisco Nexus 9000 packet tracer utility.

Prerequisites

Requirements

Cisco recommends that you have basic knowledge of these topics:

• Cisco Nexus 9000 hardware architecture

Components Used

The information in this document is based on these software and hardware versions:

• Cisco Nexus 9500
• SW Version 7.0(3)I2(2a)

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

Packet-tracer is an inbuilt utility  on the Nexus 9000 that can be used to trace the path of the packet through the switch. It can be invoked using the command line and can be configured to match IP address and or layer 4 attributes. It cannot be used to match ARP traffic.

This tool provides confirmation on whether a flow is traversing through the switch. It also provides a counter to track flow statistics which can be useful for scenerios of intermittent/complete packet loss.

Use Case Scenerios

• Applicable for IPv4 flows only (IPv6 and non IP not supported)
• This tool does not display the packet inner details as shown by wireshark.
• Intermittent packet loss : Ping or any other utility can provide a definite symptom of lost packets
• Complete packet loss

Supported Hardware

Only line cards/Fabric Modules or TORs with Broadcom Trident II asics are supported. List is below:

• N9K-C9372TX
• N9K-C9372PX
• N9K-C9332PQ
• N9K-C9396TX
• N9K-C9396PX
• N9K-C93128TX
• N9K-C9336PQ
• N9K-X9564PX
• N9K-X9564TX
• N9K-X9636PQ

Unsupported Hardware

• N9K-C93180YC-EX
• N9K-X9732C-EX
• N9K-C9232C
• N9k-C9272Q
• N9k-C92160YC

Note: Please reach out to TAC if a specific line card/TOR is not listed

How to Use Packet Tracer

Configuration

Packet-tracer commands are EXEC level commands.

N9K-9508#test packet-tracer src_ip <src_ip> dst_ip <dst_ip> <==== provide your src and dst ip 
N9K-9508#test packet-tracer start <==== Start packet tracer
N9K-9508#test packet-tracer stop <==== Stop packet tracer
N9K-9508#test packet-tracer show <==== Check for packet matches

The previous commands programs the trigger on every Broadcom Trident II Asic that exist on the line card or the fabric modules. When a flow with the matching attributes passes through these modules, it shows the counters being hit thereby helping identify the path within the switch (Ingress module--->One of the fabric module---->egress module).

The counters can be used to corelate drops.

Background Information

Fabric modules interconenct I/O module slots. All fabric moduels are active and carry traffic. Two Broadcom Trident II ASIC (T2) instances per fabric module.

Problem

PACL (Port Access-list)  is used to see if a particular physical interface received our interested traffic. However on Nexus platform, some of the linecards do not have TCAM carved for PACL. TCAM carving requires module reload. In those cases, use packet tracer to match the interested traffic. You can also trace the packet going up to fabric ports and going towards egress module. So packet tracer gives you more insight into how traffic is being forwarded within the switch.

Packet tracer uses TCAM entries carved for SPAN.

Solution

NS - North Star ASIC
T2 - Trident II ASIC
NFE - Network Forwarding Engine
ALE - ACI Leaf Engine

For more infomation about Nexus 9000 Switch Architecture, refer to this white paper.

Note: There are up to six fabric modules on a 9500 chassis. Showing only one fabric in previous picture to make it simple. Traffic from modules can hit any fabric module.

USE CASE:  Match traffic on ingress module, traffic ingressing on a fabric module and traffic ingressing T2 ASIC on egress module

Here are the basic steps that needs to be configured to match our interested traffic:

switch#test packet-tracer {<src-ip>|<dst-ip>|<src-l4-port>|<dst-l4-port>} [<protocol>] [detail-fp|detail-hg]

Here is the config you need :

switch#test packet-tracer src_ip <src_ip> dst_ip <dst_ip> protocol <>   <==== provide your src and dst ip and protocol (protocol option 1 is for icmp) 
switch#test packet-tracer start                                          <==== Start packet tracer
switch#test packet-tracer show    <==== Check for packet match statistics

You do not need to apply it to any paritcular interface. These config installs filter ACL across all LC’s/FM’s on all instances of T2 ASIC.
It shows packet count on the module on which traffic ingressed. This matches our interested traffic ingressing on a module, both linecard and Fabric.

Here is a configuraiton example:

N9K-9508# test packet-tracer src-ip 10.1.1.1 dst-ip 10.2.2.1 protocol 1 <=== Protocol 1 matches ICMP traffic
N9K-9508# test packet-tracer start

 Here is how to interpret 'test packet-tracer show" output:

N9K-9508# test packet-tracer show
Packet-tracer stats
---------------------
Module 1:                                           <=== Slot #. Same output will be displayed for other Linecards's and Fabric modules.
Filter 1 installed: src-ip 10.1.1.1 dst-ip 10.2.2.1 <==== Our filter #1
ASIC instance 0:                                    <==== Trident ASIC instance #0
Entry 0: id = 7425, count = 0, active, fp,          <==== pakcet match count on front panel port. it could be any port 
Entry 1: id = 7426, count = 0, active, hg,          <==== packet match count from fabric module to T2 ASIC on the linecard
ASIC instance 1:
Entry 0: id = 7425, count = 0, active, fp,
Entry 1: id = 7426, count = 0, active, hg,
Filter 2 uninstalled:
Filter 3 uninstalled:
Filter 4 uninstalled:
Filter 5 uninstalled:

 Example Configuration:

 Configure Packet Tracer:

N9K-9508# test packet-tracer src-ip 10.1.1.1 dst-ip 10.2.2.1 protocol 1 <==== Filter to match echo traffic. Protocol 1 to match icmp traffic
N9K-9508# test packet-tracer src-ip 10.2.2.1 dst-ip 10.1.1.1 protocol 1 <=== Filter to match echo reply traffic 
N9K-9508# test packet-tracer start                                      <==== Start packet tracer
N9K-9508# test packet-tracer show non-zero                              <==== Command to see packet statistics
Packet-tracer stats
---------------------
Module 1:
Filter 1 installed: src-ip 10.1.1.1 dst-ip 10.2.2.1
Filter 2 installed: src-ip 10.2.2.1 dst-ip 10.1.1.1
Filter 3 uninstalled:
Filter 4 uninstalled:
Filter 5 uninstalled:
Module 2:
Filter 1 installed: src-ip 10.1.1.1 dst-ip 10.2.2.1
Filter 2 installed: src-ip 10.2.2.1 dst-ip 10.1.1.1
Filter 3 uninstalled:
Filter 4 uninstalled:
Filter 5 uninstalled:
Module 22:
Filter 1 installed: src-ip 10.1.1.1 dst-ip 10.2.2.1
Filter 2 installed: src-ip 10.2.2.1 dst-ip 10.1.1.1
Filter 3 uninstalled:
Filter 4 uninstalled:
Filter 5 uninstalled:
Module 23:
Filter 1 installed: src-ip 10.1.1.1 dst-ip 10.2.2.1
Filter 2 installed: src-ip 10.2.2.1 dst-ip 10.1.1.1
Filter 3 uninstalled:
Filter 4 uninstalled:
Filter 5 uninstalled:
Module 24:
Filter 1 installed: src-ip 10.1.1.1 dst-ip 10.2.2.1
Filter 2 installed: src-ip 10.2.2.1 dst-ip 10.1.1.1
Filter 3 uninstalled:
Filter 4 uninstalled:
Filter 5 uninstalled:
Module 25:
Filter 1 installed: src-ip 10.1.1.1 dst-ip 10.2.2.1
Filter 2 installed: src-ip 10.2.2.1 dst-ip 10.1.1.1
Filter 3 uninstalled:
Filter 4 uninstalled:
Filter 5 uninstalled:

 Test: Run ping from SRC IP Connected Off of Module 1 to a DST IP Connected Off of Module 2:

Router# ping 10.1.1.1 source 10.2.2.1
PING 10.1.1.1 (10.1.1.1) from 10.2.2.1: 56 data bytes
64 bytes from 10.1.1.1: icmp_seq=0 ttl=253 time=0.77 ms
64 bytes from 10.1.1.1: icmp_seq=1 ttl=253 time=0.43 ms
64 bytes from 10.1.1.1: icmp_seq=2 ttl=253 time=0.408 ms
64 bytes from 10.1.1.1: icmp_seq=3 ttl=253 time=0.398 ms
64 bytes from 10.1.1.1: icmp_seq=4 ttl=253 time=0.383 ms
--- 10.1.1.1 ping statistics ---
5 packets transmitted, 5 packets received, 0.00% packet loss
round-trip min/avg/max = 0.383/0.477/0.77 ms

Verify: Check packet tracer count:

N9K-9508# test packet-tracer show non-zero <==== Command to see packet statistics 

 Packet-tracer stats
---------------------

Module 1:
Filter 1 installed: src-ip 10.1.1.1 dst-ip 10.2.2.1 protocol 1
ASIC instance 0:
Entry 0: id = 7425, count = 5, active, fp, <===== 5 Echo packets ingress on Module 1
Filter 2 installed: src-ip 10.2.2.1 dst-ip 10.1.1.1 protocol 1
Filter 3 uninstalled:
Filter 4 uninstalled:
Filter 5 uninstalled:

Module 2:
Filter 1 installed: src-ip 10.1.1.1 dst-ip 10.2.2.1 protocol 1
Filter 2 installed: src-ip 10.2.2.1 dst-ip 10.1.1.1 protocol 1
ASIC instance 0:
Entry 0: id = 7457, count = 5, active, fp, <===== 5 Echo reply packets ingress on Module 2
Filter 3 uninstalled:
Filter 4 uninstalled:
Filter 5 uninstalled:

Module 3:
Filter 1 installed: src-ip 10.1.1.1 dst-ip 10.2.2.1 protocol 1
Filter 2 installed: src-ip 10.2.2.1 dst-ip 10.1.1.1 protocol 1
Filter 3 uninstalled:
Filter 4 uninstalled:
Filter 5 uninstalled:

Module 4:
Filter 1 installed: src-ip 10.1.1.1 dst-ip 10.2.2.1 protocol 1
Filter 2 installed: src-ip 10.2.2.1 dst-ip 10.1.1.1 protocol 1
Filter 3 uninstalled:
Filter 4 uninstalled:
Filter 5 uninstalled:

Module 22:
Filter 1 installed: src-ip 10.1.1.1 dst-ip 10.2.2.1 protocol 1
ASIC instance 0:
Entry 0: id = 7425, count = 4, active, hg, <==== Fabric module 22 received 4 echo packets
Filter 2 installed: src-ip 10.2.2.1 dst-ip 10.1.1.1 protocol 1
Filter 3 uninstalled:
Filter 4 uninstalled:
Filter 5 uninstalled:

Module 23:
Filter 1 installed: src-ip 10.1.1.1 dst-ip 10.2.2.1 protocol 1
ASIC instance 0:
Entry 0: id = 7425, count = 1, active, hg, <==== Fabric module 23 received 1 echo packets 
Filter 2 installed: src-ip 10.2.2.1 dst-ip 10.1.1.1 protocol 1
ASIC instance 0:
Entry 0: id = 7425, count = 3, active, hg, <==== Fabric module 23 received 3 echo reply packets
Filter 3 uninstalled:
Filter 4 uninstalled:
Filter 5 uninstalled:

Module 24:
Filter 1 installed: src-ip 10.1.1.1 dst-ip 10.2.2.1 protocol 1
Filter 2 installed: src-ip 10.2.2.1 dst-ip 10.1.1.1 protocol 1
ASIC instance 0:
Entry 0: id = 7425, count = 2, active, hg, <==== Fabric module 23 received 2 echo reply packets
Filter 3 uninstalled:
Filter 4 uninstalled:
Filter 5 uninstalled:

Module 26:
Filter 1 installed: src-ip 10.1.1.1 dst-ip 10.2.2.1 protocol 1
Filter 2 installed: src-ip 10.2.2.1 dst-ip 10.1.1.1 protocol 1
Filter 3 uninstalled:
Filter 4 uninstalled:
Filter 5 uninstalled:
N9K-9508#

Other Useful Commands:

test packet-tracer remove-all                                                                                                              <=== Removes all configured filters
test packet-tracer clear <filter #>                                                                                                        <=== Clear counters for all filters or specified filter
test packet-tracer src_ip <.> dst_ip <> l4-dst-port <dst_port> | l4-src-port <src_port> | protocol    <=== Matches based on L4 src_port, L4 dst_port or protocol.