Using LIBfc with VMware and Cisco VIC to troubleshoot initiator/target communication

Introduction

This document describes how to use the hidden libfc debugs to gain low level visibility into the port login (PLOGI) process used in Fibre Channel (FC) communication within ESXi.  By enabling debug_logging we are able to see the Converged Network Adapter's (CNA) information about the Extended Link Service (ELS) frames such as Fabric Login (FLOGI), Port Login (PLOGI), that we normally would not be able to see.  This can be useful if there isn't a Finisar handy or a SPAN and you want to ensure what the host is/isn't completing in the FC stack.

Contributed by Brian Hopkins, Cisco TAC Engineer.

Supported Configuration

Currently this is only supported on ESX with a Cisco Virtual Interface Card (VIC), other adapters as far as I know do not support this feature.

Identifying Current Setting

You can use the following command on the ESXi host to ensure this value isn't already set:

From the CLI of ESXi:
esxcli system module parameters list -m libfc_92

esxcli system module parameters list -m libfcoe_92

The output should look like the following, notice how the value isn't configured for debug_logging, which is the value we will be changing in the next steps.

Change LIBfc debug_logging Setting

In order to get the additional information to show up in the /var/log/vmkernel.log file on ESXi we need to enable the debug_logging and will have to restart the host:

esxcli system module parameters set -p debug_logging=0xf -m libfc_92

esxcli system module parameters set -p debug_logging=0xf -m libfcoe_92

After you enter this commands you can check again to ensure the value is now set to 0xf:

We are still not finished, you will not see the new logs show up until you restart the ESXi host.  After you have rebooted the ESXi host you can verify you see this new updated data in the vmkernel.log file by running the following command:

cat /var/log/vmkernel.log | grep "<6>"

Since all the commands have this <6> header it makes them easy to find, I have included a snip below of this new useful information showing the FLOGI and PLOGI states:

Change LIBfc debug_logging back to original setting:

You can change this back to the default by inserting the 2 commands below and restarting the ESXi host.  We are basically just zeroing out the change from before to set this back to the default:

esxcli system module parameters set -p debug_logging= -m libfc_92

esxcli system module parameters set -p debug_logging= -m libfcoe_92

You can run the same commands again to ensure the change is successful:

From the CLI of ESXi:
esxcli system module parameters list -m libfc_92

esxcli system module parameters list -m libfcoe_92

They should both look like the following:

After rebooting the ESX host you can ensure the debugging is gone in the log by checking with this command:

tail /var/log/vmkernel.log | grep "<6>"