Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Understanding the Problem

Test Scenario

Collecting Information

Tracing MAC Addresses on the FIs

Determining What Server is Hosting the Virtual Machine

Gathering Information about the Upstream Switches

Summary

Defining the Traffic Flow

Testing only the UCSNetworking

MAC Addresses Not Learned on the Fabric Interconnects

Related Information

Introduction

This document describes how to troubleshoot basic networking connectivity issues on virtual machines.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

- Unified Computing System Manager Domain (UCSM)

- Cisco Unified Computing System Manager (UCSM) Command Line Interface (CLI)

- Cisco UCS B-Series and C-Series servers

- Networking basic concepts

- ESXi

Components Used

The information in this document is based on these software versions:

- Cisco UCS Manager version 2.x and later

- Cisco UCS 6200, 6300, 6400, and 6500 Series Fabric Interconnect

- Cisco UCS 2200, 2300, and 2400 Series Fabric extender I/O Module

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

A common scenario for infrastructure administrators deploying network or configuration changes, is to lose networking connectivity on their virtual machines. This document aims to provide guidance on the troubleshooting process to identify the most usual problems.

Understanding the Problem

The most common issue is to lose ping between virtual machines. To get the full picture, we can start asking:

- Are both virtual machines hosted in UCS servers?

- Are both virtual machines on the same UCSM Domain?

- Are the virtual machines trying to communicate on the same VLAN?

- What kind of network configuration we are using on the hypervisor side? (ESXi distributed switch, NIC teaming, and so on.)

- What is the model of the upstream switches?

Test Scenario

Two new virtual machines were provisioned and configured to use VLAN 70, however, they cannot ping each other or their default gateway.

Virtual Machines

- IMM-Transition-4.0.1

- Alma Linux 9

Hypervisor

- VMware ESXi, 7.0.3, 20842708

Collecting Information

MAC an IP addresses of both virtual machines:

- IMM-Transition-4.0.1

- MAC: 00:50:56:ba:28:53

- IP: 192.168.70.23

- Host IP: 10.31.123.38

- Alma Linux 9

- MAC: 00:50:56:ba:46:96

- IP: 192.168.70.24

- Host IP: 10.31.123.40

Tracing MAC Addresses on the FIs

FI-A # connect nxos

FI-A(nxos)# show mac address-table vlan 70

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link

VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------

* 70 0050.56ba.4696 dynamic 30 F F Veth725 ------------------------------------------->>> VM Alma Linux 9 is learned on FI-A veth725 and VLAN 70 as expected

FI-B # connect nxos

FI-B(nxos)# show mac address-table vlan 70

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link

VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------

* 70 0050.56ba.2853 dynamic 10 F F Veth688 ---------------------------------------->>> VM IMM-Transition-4.0.1 is learned on FI-B veth688 and VLAN 70 as expected

Determining What Server is Hosting the Virtual Machine

FI-A(nxos)# show running-config interface vethernet725

!Command: show running-config interface Vethernet725

!Time: Thu Feb 1 11:59:39 2024

version 5.0(3)N2(4.13k)

interface Vethernet725

description server 1/3, VNIC vnic_a1 ---------------------------------------->>> VM Alma Linux 9 is hosted on Server 1/3 and the vnic name is vnic_a1

switchport mode trunk

no lldp transmit

no lldp receive

no pinning server sticky

pinning server pinning-failure link-down

no cdp enable

switchport trunk allowed vlan 69-70,72,470

bind interface port-channel1287 channel 725

service-policy type queuing input org-root/ep-qos-BestEffort

no shutdown

FI-B(nxos)# show running-config interface vethernet 688

!Command: show running-config interface Vethernet688

!Time: Thu Feb 1 12:06:44 2024

version 5.0(3)N2(4.13k)

interface Vethernet688

description server 1/5, VNIC vnic_b1 ---------------------------------------->>> VM IMM-Transition-4.0.1 is hosted on Server 1/5 and the vnic name is vnic_b1

switchport mode trunk

switchport trunk allowed vlan 69-70,72,470

no lldp transmit

no lldp receive

no pinning server sticky

pinning server pinning-failure link-down

no cdp enable

service-policy type queuing input org-root/ep-qos-BestEffort

bind interface port-channel1282 channel 688

no shutdown

Gathering Information  about the Upstream Switches

FI-A(nxos)# show cdp neighbors

Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge

S - Switch, H - Host, I - IGMP, r - Repeater,

V - VoIP-Phone, D - Remotely-Managed-Device,

s - Supports-STP-Dispute

Device-ID                    Local Intrfce Hldtme   Capability   Platform   Port ID

MGMT-SWITCH

mgmt0        140       R S I    WS-C3650-12X4  Gig1/0/35

Nexus-1

Eth1/1       158       R S I s  N5K-C5672UP-1  Eth1/3

Nexus-2

Eth1/2       133       R S I s  N5K-C5672UP-1  Eth1/3

FI-A(nxos)# show cdp neighbors

Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge

S - Switch, H - Host, I - IGMP, r - Repeater,

V - VoIP-Phone, D - Remotely-Managed-Device,

s - Supports-STP-Dispute

Device-ID                    Local Intrfce   Hldtme   Capability   Platform   Port ID

MGMT-SWITCH

mgmt0           139        R S I    WS-C3650-12X4 Gig1/0/36

Nexus-1

Eth1/1           167        R S I s  N5K-C5672UP-1 Eth1/4

Nexus-2

Eth1/2           132        R S I s  N5K-C5672UP-1 Eth1/4

Summary

- Virtual machine MAC addresses are learned on FI-A and FI-B respectively and VLAN 70.

- Virtual machines are hosted in different UCS servers but in the same UCSM Domain.

- Upstream switches are N5K-C5672UP-1 and connect to interfaces ethernet1-2 in both fabric interconnects.

Defining the Traffic Flow

- If source and destination are on the same subnet or VLAN, the traffic is forwarded on the same broadcast domain.

- If source and destination are on a different subnet or vlan, the traffic is forwarded into another broadcast domain.

- If source and destination are learned in the same Fabric Interconnect, the traffic is switched locally by the Fabric Interconnect.

- If source and destination are learned in a different Fabric Interconnect, the traffic is forwarded upstream.

For this particular scenario:

- Source and destination are on the same broadcast domain, but learned on different fabric interconnects, so the traffic is sent to the upstream network.

Testing only the UCS Networking

To test the local switching of the fabric interconnect, therefore, not involving the upstream network on the traffic flow, a failover can be forced for both virtual machines to be learned in the same fabric interconnect. On this example, VM IMM-Transition-4.0.1 is going to be moved to FI-A.

- From previous troubleshooting:

- VM Alma Linux 9 is hosted on Server 1/3, learned on FI-A and using veth725, which is vnic_a1.

- VM IMM-Transition-4.0.1 is hosted on Server 1/5, learned on FI-B and using veth688, which is vnic_b1.

- On UCSM:

- Server 1/5 has 2 vNICs on FI-A and 2 on FI-B

- To force the repinning to FI-A, disable the vNICs on the B side, starting with the vNIC used by the VM, for this scenario vnic_b0 and vnic_b1 were disabled.

- With all vNICs on FI-B disabled, VM IMM-Transition-4.0.1 is now learned on FI-A, along with VM Alma Linux 9.

FI-A(nxos)# show mac address-table vlan 70

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link

VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------

* 70 0050.56ba.2853 dynamic 0 F F Veth686 ------------------------------------->>> VM Alma Linux 9

* 70 0050.56ba.4696 dynamic 10 F F Veth725 ------------------------------------>>> VM IMM-Transition-4.0.1

- With everything else on the UCS side configured as expected, the ping works now as the traffic is being switched locally by FI-A. Hence, the investigation needs to continue on the upstream network.

MAC Addresses Not Learned on the Fabric Interconnects

- Verify if the VLAN is correctly configured on the vNICs.

- Verify if the VLAN is correctly configured on the uplinks.

FI-A(nxos)# show running-config interface port-channel 1

!Command: show running-config interface port-channel1

!Time: Fri Feb 2 13:05:59 2024

version 5.0(3)N2(4.13k)

interface port-channel1

description U: Uplink

switchport mode trunk

pinning border

switchport trunk allowed vlan 1,69-70,72,470

speed 1000

- Verify if the VLAN is correctly configured on ESXi.

- Validate the vmnic used by the virtual machine on the ESXi host. Use the esxtop with option n  to get the binding.

- Virtual machines are using vmnic1 and vmnic2 on host 1/3.

- Mapping MAC addresses from ESXi vmnics to UCS vNICs

[root@esx38:~] esxcfg-nics -l

Name PCI Driver Link Speed Duplex MAC Address MTU Description

vmnic0 0000:06:00.0 nenic Up 20000Mbps Full 00:25:b5:04:38:a0 9000 Cisco Systems Inc Cisco VIC Ethernet NIC

vmnic1 0000:07:00.0 nenic Up 20000Mbps Full 00:25:b5:04:38:a1 9000 Cisco Systems Inc Cisco VIC Ethernet NIC

vmnic2 0000:08:00.0 nenic Up 20000Mbps Full 00:25:b5:04:38:b0 9000 Cisco Systems Inc Cisco VIC Ethernet NIC

vmnic3 0000:09:00.0 nenic Up 20000Mbps Full 00:25:b5:04:38:b1 9000 Cisco Systems Inc Cisco VIC Ethernet NIC

- Is the OS forwarding the frame? (Confirm with a packet capture.)

- VIC adapter

- IOM (HIFs and NIFs)

Related Information

- Cisco Technical Support & Downloads