Configure and Troubleshoot Selective Q-in-Q on Nexus 9000

Introduction

This document describes how to configure and troubleshoot Selective Q-in-Q in Nexus 9300 (EX-FX-GX) model. 

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Q-in-Q encapsulation
• Dot1q switching
• Nexus 9000

Components Used

The information in this document is based on these software and hardware versions:

• Hardware: N9K-C9336C-FX2
• Version: 10.2(3)

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

A Q-in-Q VLAN tunnel enables a service provider to segregate the traffic of different users in their infrastructure, while still giving the you a full range of VLANs for their internal use by adding a second 802.1Q tag to an already tagged frame.

Selective Q-in-Q with multiple provider VLANs is a tunneling feature that allows user-specific range of VLANs on a port to be associated with one specific provider VLAN.

Q-in-Q allows you to have multiple user VLAN-to-provider VLAN mappings on a port.

Packets that come in with a VLAN tag that matches any of the configured user VLANs on the port are tunneled across the fabric using the properties of the service provider VLAN. The encapsulated packet carries the user VLAN tag as part of the Layer 2 header of the inner packet.

Network Diagram

Configure

The goal of the configuration of Selective Q-in-Q is to map the VLANs from the traffic received in N9KFX2 and N9KFX2-1 on interfaces eth1/1.

The Nexus encapsulates the traffic and sends it with the DOT1Q header VLAN 20 to the provider, simulated by Nexus 93180FX.

Host-A

interface Ethernet1/1
  switchport
  switchport mode trunk
  no shutdown

interface Vlan10
  no shutdown
  ip address 192.168.1.1/24

Host-B

interface Ethernet1/1
  switchport
  switchport mode trunk
  no shutdown

interface Vlan10
  no shutdown
  ip address 192.168.1.2/24

N9KFX2

interface Ethernet1/1
  switchport
  switchport mode trunk
  switchport vlan mapping all dot1q-tunnel 20
  switchport trunk allowed vlan 20
  spanning-tree port type edge trunk
  spanning-tree bpdufilter enable
  no shutdown

interface Ethernet1/35
  switchport
  switchport mode trunk
  no shutdown

N93180FX

system dot1q-tunnel transit 

interface Ethernet1/17
  switchport
  switchport mode trunk
  no shutdown

interface Ethernet1/18
  switchport
  switchport mode trunk
  no shutdown

N9KFX2-1

interface Ethernet1/1
  switchport
  switchport mode trunk
  switchport vlan mapping all dot1q-tunnel 20
  switchport trunk allowed vlan 20
  spanning-tree port type edge trunk
  spanning-tree bpdufilter enable
  no shutdown

interface Ethernet1/35
  switchport
  switchport mode trunk
  no shutdown

Troubleshoot

The Q-in-Q programming moves from the control plane to the hardware through the components:

:

Consistency Checker

Recent versions of the Nexus switch have consistency checker to use in case misprogramming.

N9KFX2# show consistency-checker selective-qinq

Box level hardware programming checks      : PASS
Fetching ingressVlanXlate entries from slice: 0 HW
Fetching ingressVlanXlate entries from slice: 1 HW
Fetching ingressVlanXlate TCAM entries
Performing port specific checks for intf Eth1/1
Port specific selective QinQ checks for interface  Eth1/1 : PASS
Selective QinQ Overall status      : PASS

Control plane

Confirm if the VLAN is active on the ports.

N9KFX2#sh vlan

VLAN Name                             Status    Ports
---- -------------------------------- --------- -------------------------------
1    default                          active    Eth1/1
20   VLAN0020                         active    Eth1/1, Eth1/35

Confirm that the VLAN mapping is clearly seen.

N9KFX2# sh vlan internal info mapping

Per Port Vlan Translation Details

================
Entry No                :  0
----------------------
  ifindex Eth1/1(0x1a000000)
  vlan mapping enabled: TRUE
  vlan translation mapping information (count=1):
    Original Vlan      Translated Vlan
    -------------      ---------------
    all                  20

Entry No                :  1
----------------------
  ifindex Eth1/35(0x1a004400)
  vlan mapping enabled: FALSE
  vlan translation mapping information (count=0):
    Original Vlan      Translated Vlan
    -------------      ---------------

ELTM

Confirm if there is any errors increasing or non zero.

N9KFX2# show system internal eltm error count
interface config error: 0
dce error: 0
port struct error: 0
port channel error: 0
tunnel error: 0
tlv count error: 0
vpn count error: 0
SI add to DB error: 0

Run the command attach module X. The X is the module where the physical interface is allocated.

N9KFX2# attach module 1
module-1# show system internal eltm info vlan 20
IFTMC Detailed info for VLAN = 20
cr_flags = VLAN , bd = 21(0x15)  stg-id = 0 type = 1 stp:0
shut_state = 2 , oper_state = 1
vni = 0, ipmc_idx = 0x0 ipmc_remap_idx = 0x0 ipmc_remap_idx2 = 0x0 dvif: 0
ipmc_mrouter_idx = 0x0 ipmc_mrouter_remap_idx = 0x0 ipmc_mrouter_remap_idx2 = 0x0
pbp_idx=0
eid[0] = 0, alt_eid[0] = 0eid[1] = 0, alt_eid[1] = 0eid[2] = 0, alt_eid[2] = 0eid[3] = 0, alt_eid[3] = 0eid[4] = 0, alt_eid[4] = 0eid[5] = 0, alt_eid[5] = 0eid[6] = 0, alt_eid[6] = 0eid[7] = 0, alt_eid[7] = 0


Active ports:                                             <<<<< Confirm Active ports are listed for the vlan
  Ethernet1/1(0x1a000000)  Ethernet1/35(0x1a004400)

Active Local ports:
  Ethernet1/1(0x1a000000)  Ethernet1/35(0x1a004400)

Ports using this VLAN as Provider:                          <<<<< Displays the interface that is doing the tunneling
  Ethernet1/1(0x1a000000)

SDB Active ports:
      Ethernet1/1

pbp_idx:0

VLAN Features:
  prim_vlan      : 0, prim_intf_vlan  : 0, vlan mode      : 0(CE)
  pt_cam_en      : 1, ipv4_igmp_snoop  : 1, ipv4_pim_snoop  : 0
  ipv6_mld_snoop : 0, ipv6_pim_snoop  : 0, l2l3_lkup_cfg  : 1
  v4_table_id    : 1        (0x00000001), v4_vpn_id        : 1 (0x1)
  v6_table_id    : 2147483649 (0x80000001), v6_vpn_id      : 1 (0x1)
  age_tmr_sel    : 1, sm_en  : 0 fcf  : 0 omf : 1, dhcp_snoop_en  : 0
  proxy_learn    : 0, iic_en : 0, use_oxid : 0, age_grp_sel        : 0
  disable_learn  : 0, otv_en : 0, global  : FALSE
  bc_uc_ftag    : 4095  , mcast_ftag_num : 1, arp_sup: 0x0 ,icmpv6 nd local proxy: 0x0
  isdot1qvlan    : 0  , num_dot1q_ports: 0  num_sel_dot1q_qinvni_ports: 0
  num_multi_tag_ports: 0,  eid: 0,  xconnect: 0,  mcast_ftag[1]  : {4095}
  is_transit    : 0    mdns_en: 0,

TAH:VLAN_CONTROL:
------------------
vrf: 0 forwarding_vlan: 0 forwarding mode: 0
Flags: 0x0
up: 0 vni: 0 is_sr_evpn: 0

You can check for interface programming. The output displays information about the port programming.

N9KFX2# attach module 1
module-1# show system internal eltm info interface e1/1
IFTMC Detailed info for Interface Ethernet1/1(0x1a000000)
CR_flags = INTF LIF  (0x3),  LTL = 6144 (0x1800), (S 0x0, P 0x0)
IF_INDEX = Ethernet1/1(0x1a000000), LIF = 4100 (0x1004), SDB LIF = 4100 (0x1004)

State = UP
Layer = L2, Mode = TRUNK, native_vlan = 1 tag_native_vlan = 0   <<< Confirm mode and Layer
AdminMode = 0x100000 is_monitor = 0 src_if_pruning 1
Operational VLAN's (2):                                         <<< Confirm if the VLAN for tunnel and native are correct
  1,20
voice vlan = 4096
MVR Configured VLAN's (0):

local_port = 1, ldb_sharing = 1, ilm_sharing = 0x1
ldb_port_flags = 0x0, dsm = 0, dnl = 0
pc_ifindx = 0x0, fpc = 0x0
Interface Features:
  ipv4_en        : 0, ipv4_mcast_en : 0, df_mask : 0, mpls_en : 0
  ipv6_en        : 0, ipv6_mcast_en : 0  pim_dr_en :0 ipsg_en  : 0
  FC port mode:0 vsan: 0 trunk mode: 0 speed: 0
  FCF Enabled : 0 FCF MAC:0000:0000:0000
  PTP fid = 0
  mpls_strip_en    :0
external port :0
Operational VSAN's (0):

  v4_table_id    : 1          (0x00000001), v4_vpn_id    : 1 (0x1)
  v6_table_id    : 2147483649 (0x80000001), v6_vpn_id    : 1 (0x1)
  per_pkt_ls_en  : 0,  bd              : N/A (0x0)
  icmp_redirect  : 1, ipv6_redirect    : 1
  v4_same_if_chk : 0, v6_same_if_check : 0
  mtu_index      : 1    (0x0001), new_mtu_index  : 1    (0x0001)
  mtu            : 1500 (0x05dc), port_trust    : 0 
  v4_rpf_mode    : 0  , v6_rpf_mode            : 0
  v4_rpf_mode (sdb) : 0  , v6_rpf_mode (sdb) : 0 
  intf_type      : 0x0, intf_flags  : 0x20
  sub_flags      : 0x0, port_type  : 0x0, is_aida  : 0x0, vpc_dvp_no  : 0x0
  fcoe_mac_ip_chk: 0 buffer_boost: 0 openflow_en: 0 span_buffer_cfg: 0 span_mode: 0 outer_bd = 512
  interface mac: 00:00:00:00:00:00 old_mac: 00:00:00:00:00:00 mac_index:0
  Src-Idx:1
  mc_index : 4100, egress_vsl_drop: 0, pv_en : 0, is_selective_qinvni_port 1 ether_type 0x8100 vlan_bmp_State_change : False, policy flood_l2_local 1.  <<<Confirm the ether type is 8100 and is_selective_qinvni_port is 1
  is_multi_tag_port 0, nat_realm_id 0
global_id 0
  enm_if_type  : 0
  pbp_idx  : 0
  pinned_ifindex : 0x0
normal_vlan_bmp (0):

arp_suppress inst[0] 0 arp_suppress inst[1] 0 arp_suppress inst[2] 0 arp_suppress inst[3] 0 arp_suppress inst[4] 0 arp_suppress inst[5] 0 arp_suppress inst[6] 0 arp_suppress inst[7] 0 icmpv6 nd local proxy en inst[0] 0 icmpv6 nd local proxy en inst[1] 0 icmpv6 nd local proxy en inst[2] 0 icmpv6 nd local proxy en inst[3] 0 icmpv6 nd local proxy en inst[4] 0 icmpv6 nd local proxy en inst[5] 0 icmpv6 nd local proxy en inst[6] 0 icmpv6 nd local proxy en inst[7] 0 mdns_en inst[0] 0 mdns_en inst[1] 0 mdns_en ins
t[2] 0 mdns_en inst[3] 0 mdns_en inst[4] 0 mdns_en inst[5] 0 mdns_en inst[6] 0 mdns_en inst[7] 0 arp_suppress inst[0] 0 arp_suppress inst[1] 0 arp_suppress inst[2] 0 arp_suppress inst[3] 0 arp_suppress inst[4] 0 arp_suppress inst[5] 0 arp_suppress inst[6] 0 arp_suppress inst[7] 0 icmpv6 nd local proxy en inst[0] 0 icmpv6 nd local proxy en inst[1] 0 icmpv6 nd local proxy en inst[2] 0 icmpv6 nd local proxy en inst[3] 0 icmpv6 nd local proxy en inst[4] 0 icmpv6 nd local proxy en inst[5] 0 icmpv6 nd local prox
y en inst[6] 0 icmpv6 nd local proxy en inst[7] 0 mdns_en inst[0] 0 mdns_en inst[1] 0 mdns_en inst[2] 0 mdns_en inst[3] 0 mdns_en inst[4] 0 mdns_en inst[5] 0 mdns_en inst[6] 0 mdns_en inst[7] 
service provider Vlans 0

vlan_xlt_tlb_en_ingress : 1  num_vlan_xlt_ingr : 1

  Vlan Translation Table (INGRESS)         <<< This table displays xlt the vlan used to send to the provider
  --------------------------------
    in_vlan  xlt_vlan
        1      20


SDB INFO:
--------------------------------
MVR VLAN's (0):

STATE = UP, is_mcec = 0, is_mct = 0
TAH:L2 INFO: 4100
----------
Flags : 0x41
NIV Index: 1 Source Chip: 1 Source Port: 72
slice: 1 slice_port: 0
is_monitor: 0 lif: 4100 phy_intf_id: 1 vnic_if: 0 learn_en: 1 native_vlan 1
trunk_port: 1 src_if_check: 1
FC info: intf_type = 0x0 port_mode = 0x0 port_vsan = 0 port_speed = 0 fcf_mac = 0:0:0:0:0:0

IFTMC

For IFTMC you can run the 2 commands. Run the attach module 1 command before running the next commands.

N9KFX2# attach module 1
module-1# show system internal iftmc info interface e1/1
module-1# show system internal iftmc info vlan 20

Both Commands

• show system internal iftmc info interface ethernet1/1
• show system internal iftmc info vlan 20

Displays similar information as ELTM outputs, make sure to verify you have same values between ELTM and IFTMC modules.

Hardware

To confirm the outputs are correctly programmed on the hardware you can run the command show hardware internal tah interface ethernet 1/1. In the output of the command, you can go to the BdStateTable part.

N9KFX2-1(config-if)# show hardware internal tah interface ethernet 1/1 
BdStateTable:
ENTRY: 1
        info_leaf_flood_dst_ptr : 1                 <<<Displays native vlan      
        info_leaf_igmp_mld_dst_ptr : 4098
        info_leaf_fid : 1                           <<<Displays native vlan
        info_leaf_vrf : 1
        info_leaf_igmp_snp_en : 1
        info_leaf_l3_bind_check_en : 1
        info_leaf_v4_omf : 1
        info_leaf_unknown_uc_flood : 1
        info_leaf_unknown_mc_flood : 1
        info_leaf_ecn_mark_en : 1
        info_leaf_l2_bind_check_en : 1
        info_leaf_bd_profile_idx : 1
        info_leaf_analytics_en : 1
        info_leaf_flow_collect_en : 1
        info_spine_info0_gipo_idx : 1              <<<Displays native vlan
        info_spine_info0_scope : 64
        info_spine_info0_spare : 0x00000000:0x00100044
        info_spine_info1_spare : 0x00001000:0x00000000
        info_spine_info2_gipo_idx : 512
        info_spine_info2_scope : 12416
        info_spine_info2_spare : 0x00005000:0x10000000
        info_spine_info3_gipo_idx : 384
        info_spine_info3_spare : 0x00000000:0x00000000
=========================================

Useful commands to gather and check for any Q-in-Q issue

• show vlan internal info mapping
• show consistency-checker selective-qinq
• show consistency-checker selective-qinq interface <ethX/Y, port-channel X>
• show hardware internal tah interface ethernet x/y
• show hardware internal tah interface all       
• show hardware internal tah sdk event-history error       
• show system internal iftmc info interface <>

On Nexus platform you can collect show tech-support more specific than the general, which provides a series of commands:

• show tech details
• show tech vlan
• show tech usd-all
• show tech eltm
• attach module 1 ; show tech iftmc

Note: Look at the guidelines and limitations sections on the configuration guide listed on Related Information for the version your Nexus is running to confirm that your configuration is not hitting any hardware or software limitations.

Related Information

• Release 9.3.X
• Release 10.1.X
• Release 10.2.X
• Release 10.3.X
• Release 10.4.X
• Cisco Technical Support & Downloads