Troubleshoot Basic Networking Issues on Virtual Machines

Introduction

This document describes how to troubleshoot basic networking connectivity issues on virtual machines.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Unified Computing System Manager Domain (UCSM)
• Cisco Unified Computing System Manager (UCSM) Command Line Interface (CLI)
• Cisco UCS B-Series and C-Series servers
• Networking basic concepts
• ESXi

Components Used

The information in this document is based on these software versions:

• Cisco UCS Manager version 2.x and later
• Cisco UCS 6200, 6300, 6400, and 6500 Series Fabric Interconnect
• Cisco UCS 2200, 2300, and 2400 Series Fabric extender I/O Module

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

A common scenario for infrastructure administrators deploying network or configuration changes, is to lose networking connectivity on their virtual machines. This document aims to provide guidance on the troubleshooting process to identify the most usual problems.

Understanding the Problem

The most common issue is to lose ping between virtual machines. To get the full picture, we can start asking:

• Are both virtual machines hosted in UCS servers?
• Are both virtual machines on the same UCSM Domain?
• Are the virtual machines trying to communicate on the same VLAN?
• What kind of network configuration we are using on the hypervisor side? (ESXi distributed switch, NIC teaming, and so on.)
• What is the model of the upstream switches?

Test Scenario

Two new virtual machines were provisioned and configured to use VLAN 70, however, they cannot ping each other or their default gateway.

Virtual Machines

• IMM-Transition-4.0.1
• Alma Linux 9

Hypervisor

• VMware ESXi, 7.0.3, 20842708

Collecting Information

MAC an IP addresses of both virtual machines:

• IMM-Transition-4.0.1
  • MAC: 00:50:56:ba:28:53
  • IP: 192.168.70.23
  • Host IP: 10.31.123.38

• Alma Linux 9
  • MAC: 00:50:56:ba:46:96
  • IP: 192.168.70.24
  • Host IP: 10.31.123.40

Tracing MAC Addresses on the FIs

FI-A # connect nxos
FI-A(nxos)# show mac address-table vlan 70
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 70 0050.56ba.4696 dynamic 30 F F Veth725 ------------------------------------------->>> VM Alma Linux 9 is learned on FI-A veth725 and VLAN 70 as expected

FI-B # connect nxos
FI-B(nxos)# show mac address-table vlan 70
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 70 0050.56ba.2853 dynamic 10 F F Veth688 ---------------------------------------->>> VM IMM-Transition-4.0.1 is learned on FI-B veth688 and VLAN 70 as expected

Determining What Server is Hosting the Virtual Machine

FI-A(nxos)# show running-config interface vethernet725

!Command: show running-config interface Vethernet725
!Time: Thu Feb 1 11:59:39 2024

version 5.0(3)N2(4.13k)

interface Vethernet725
description server 1/3, VNIC vnic_a1 ---------------------------------------->>> VM Alma Linux 9 is hosted on Server 1/3 and the vnic name is vnic_a1
switchport mode trunk
no lldp transmit
no lldp receive
no pinning server sticky
pinning server pinning-failure link-down
no cdp enable
switchport trunk allowed vlan 69-70,72,470
bind interface port-channel1287 channel 725
service-policy type queuing input org-root/ep-qos-BestEffort
no shutdown

FI-B(nxos)# show running-config interface vethernet 688

!Command: show running-config interface Vethernet688
!Time: Thu Feb 1 12:06:44 2024

version 5.0(3)N2(4.13k)

interface Vethernet688
description server 1/5, VNIC vnic_b1 ---------------------------------------->>> VM IMM-Transition-4.0.1 is hosted on Server 1/5 and the vnic name is vnic_b1
switchport mode trunk
switchport trunk allowed vlan 69-70,72,470
no lldp transmit
no lldp receive
no pinning server sticky
pinning server pinning-failure link-down
no cdp enable
service-policy type queuing input org-root/ep-qos-BestEffort
bind interface port-channel1282 channel 688
no shutdown

Gathering Information  about the Upstream Switches

FI-A(nxos)# show cdp neighbors
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
S - Switch, H - Host, I - IGMP, r - Repeater,
V - VoIP-Phone, D - Remotely-Managed-Device,
s - Supports-STP-Dispute


Device-ID                    Local Intrfce Hldtme   Capability   Platform   Port ID
MGMT-SWITCH
                             mgmt0        140       R S I    WS-C3650-12X4  Gig1/0/35
Nexus-1
                             Eth1/1       158       R S I s  N5K-C5672UP-1  Eth1/3
Nexus-2
                             Eth1/2       133       R S I s  N5K-C5672UP-1  Eth1/3

FI-A(nxos)# show cdp neighbors
Capability Codes: R - Router, T - Trans-Bridge, B - Source-Route-Bridge
S - Switch, H - Host, I - IGMP, r - Repeater,
V - VoIP-Phone, D - Remotely-Managed-Device,
s - Supports-STP-Dispute

Device-ID                    Local Intrfce   Hldtme   Capability   Platform   Port ID
MGMT-SWITCH
                            mgmt0           139        R S I    WS-C3650-12X4 Gig1/0/36
Nexus-1
                           Eth1/1           167        R S I s  N5K-C5672UP-1 Eth1/4
Nexus-2
                           Eth1/2           132        R S I s  N5K-C5672UP-1 Eth1/4

Summary

• Virtual machine MAC addresses are learned on FI-A and FI-B respectively and VLAN 70.
• Virtual machines are hosted in different UCS servers but in the same UCSM Domain.
• Upstream switches are N5K-C5672UP-1 and connect to interfaces ethernet1-2 in both fabric interconnects.

Defining the Traffic Flow

• If source and destination are on the same subnet or VLAN, the traffic is forwarded on the same broadcast domain.
• If source and destination are on a different subnet or vlan, the traffic is forwarded into another broadcast domain.
• If source and destination are learned in the same Fabric Interconnect, the traffic is switched locally by the Fabric Interconnect.
• If source and destination are learned in a different Fabric Interconnect, the traffic is forwarded upstream.

For this particular scenario:

• Source and destination are on the same broadcast domain, but learned on different fabric interconnects, so the traffic is sent to the upstream network.

Testing only the UCS Networking

To test the local switching of the fabric interconnect, therefore, not involving the upstream network on the traffic flow, a failover can be forced for both virtual machines to be learned in the same fabric interconnect. On this example, VM IMM-Transition-4.0.1 is going to be moved to FI-A.

• From previous troubleshooting:
  • VM Alma Linux 9 is hosted on Server 1/3, learned on FI-A and using veth725, which is vnic_a1.
  • VM IMM-Transition-4.0.1 is hosted on Server 1/5, learned on FI-B and using veth688, which is vnic_b1.
• On UCSM:

• Server 1/5 has 2 vNICs on FI-A and 2 on FI-B
• To force the repinning to FI-A, disable the vNICs on the B side, starting with the vNIC used by the VM, for this scenario vnic_b0 and vnic_b1 were disabled.

• With all vNICs on FI-B disabled, VM IMM-Transition-4.0.1 is now learned on FI-A, along with VM Alma Linux 9.

FI-A(nxos)# show mac address-table vlan 70
Legend:
* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC
age - seconds since last seen,+ - primary entry using vPC Peer-Link
VLAN MAC Address Type age Secure NTFY Ports/SWID.SSID.LID
---------+-----------------+--------+---------+------+----+------------------
* 70 0050.56ba.2853 dynamic 0 F F Veth686 ------------------------------------->>> VM Alma Linux 9
* 70 0050.56ba.4696 dynamic 10 F F Veth725 ------------------------------------>>> VM IMM-Transition-4.0.1

• With everything else on the UCS side configured as expected, the ping works now as the traffic is being switched locally by FI-A. Hence, the investigation needs to continue on the upstream network.

MAC Addresses Not Learned on the Fabric Interconnects

• Verify if the VLAN is correctly configured on the vNICs.

• Verify if the VLAN is correctly configured on the uplinks.

FI-A(nxos)# show running-config interface port-channel 1

!Command: show running-config interface port-channel1
!Time: Fri Feb 2 13:05:59 2024

version 5.0(3)N2(4.13k)

interface port-channel1
description U: Uplink
switchport mode trunk
pinning border
switchport trunk allowed vlan 1,69-70,72,470
speed 1000

• Verify if the VLAN is correctly configured on ESXi.

• Validate the vmnic used by the virtual machine on the ESXi host. Use the esxtop with option n  to get the binding.

• Virtual machines are using vmnic1 and vmnic2 on host 1/3.
• Mapping MAC addresses from ESXi vmnics to UCS vNICs

[root@esx38:~] esxcfg-nics -l
Name PCI Driver Link Speed Duplex MAC Address MTU Description
vmnic0 0000:06:00.0 nenic Up 20000Mbps Full 00:25:b5:04:38:a0 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic1 0000:07:00.0 nenic Up 20000Mbps Full 00:25:b5:04:38:a1 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic2 0000:08:00.0 nenic Up 20000Mbps Full 00:25:b5:04:38:b0 9000 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic3 0000:09:00.0 nenic Up 20000Mbps Full 00:25:b5:04:38:b1 9000 Cisco Systems Inc Cisco VIC Ethernet NIC

• Is the OS forwarding the frame? (Confirm with a packet capture.)
• VIC adapter
• IOM (HIFs and NIFs)

Related Information

• Cisco Technical Support & Downloads