Configuring the Queue Depth of the nfnic driver on ESXi 6.7 for use with VMWare VVOL

Introduction

This document describes the process of configuring the maximum Queue Depth and the Outstanding Input/Output (IO) on a native fiberchannel network interface card (nfnic) driver. In the VMware ESXi 6.7 hypervisor, the fiberchannel network interface card (fnic) driver was replaced with the nfnic driver for all Cisco adapters.

The default queue depth of the nfnic driver is set to 32 and on all initial releases of the nfnic driver there is no way to adjust the nfnic queue depth. This limits all Maximum Device Queue Depths and Disk Schedule Number Requests Outstanding to 32. It has also caused issues while using vSphere Virtual Volumes since the recommended queue depth is 128. The effects of this limit can also be seen on any VMs that experience a higher workload and require a larger queue depth in general.

Contributed to by Michael Baba, Josh Good and Alejandro Marino; Cisco TAC Engineers.

Background Information

Enhancement created to add ability to configure queue depth parameter: https://bst.cloudapps.cisco.com/bugsearch/bug/CSCvo09082

Starting with version 4.0.0.35 of the nfnic driver, you can adjust the "lun_queue_depth_per_path" via the ESXi Command Line Interface (CLI). This driver version can be manually installed to the ESXi host if it is not already on it. 

The nfnic driver 4.0.0.35 can be found in the UCS Blade Firmware bundle 4.0.4 and can also be downloaded separately from VMware. You should refer to the UCS Hardware and Software Compatibility page to get the latest recommended driver for your specific hardware and software combination. 

Checking and Updating the nfnic Driver

Checking installed driver

To check the currently installed version of the nfnic driver, run: 

esxcli software vib list | grep nfnic

You should see something like: 

[root@localhost:~] esxcli software vib list | grep nfnic
nfnic                          4.0.0.14-1OEM.670.1.28.10302608         Cisco        VMwareCertified   2019-08-24
[root@localhost:~]

If you do not see any output, then you currently do not have the nfnic driver installed. Please refer to the UCS Hardware and Software Compatibility page to check if your configuration should be using the nfnic or fnic driver. 

Upgrading the nfnic driver

Detailed instructions to install the latest drivers are beyond the scope of this guide. Please refer to UCS Driver Installation for Common Operating Systems or VMware's documentation for step-by-step instructions to upgrade the driver. Once the driver is upgraded you can use the same commands above to verify the version. 

Configuring the nfnic Driver

Configuring the Queue Depth Parameter

Once the correct driver is installed we can check that the module parameters are available to configure with:

esxcli system module parameters list -m nfnic

We can see in this output that the default value is set to 32, however, we can configure any value from 1-1024. If using vSphere Virtual Volumes, it is recommended to set this value to 128. We would recommend to reach out to VMware and your storage vendor for any other specific recommendations. 

Sample Output: 

[root@localhost:~] esxcli system module parameters list -m nfnic
Name                      Type   Value  Description
------------------------  -----  -----  --------------------------------------------------------------
lun_queue_depth_per_path  ulong         nfnic lun queue depth per path: Default = 32. Range [1 - 1024]
[root@localhost:~]

To change the Queue Depth parameter, the command is below. In the below example we are changing it to 128, but your value may be different depending on your environment. 

esxcli system module parameters set -m nfnic -p lun_queue_depth_per_path=128

Using the same command as above we can config the change has been made: 

[root@localhost:~] esxcli system module parameters list -m nfnic
Name                      Type   Value  Description
------------------------  -----  -----  --------------------------------------------------------------
lun_queue_depth_per_path  ulong  128    nfnic lun queue depth per path: Default = 32. Range [1 - 1024]
[root@localhost:~]

Configure the outstanding IO on the Protocol Endpoint

We can now configure the Outstanding IOs on the Protocol Endpoint to match the Queue Depth above (in our example, 128) and then check to make sure both values have changed to 128.

NOTE: You may need to reboot the host before this configuration change can be made.

To change the Queue Depth for a specific device:

esxcli storage core device set -O 128 -d naa.xxxxxxxxx

To find the device ID you can use the below command:

esxcli storage core device list

To confirm the changes for a specific device: 

esxcli storage core device list -d naa.xxxxxxxxxx

An example with output. We can see that the "Device Max Queue Depth:" and "No of outstanding IOs with competing worlds:" are both still 32.

[root@localhost:~] esxcli storage core device list -d naa.600a09803830462d803f4c6e68664e2d                                                                                        
naa.600a09803830462d803f4c6e68664e2d                                                                                                                                                
   Display Name: VMWare_SAS_STG_01                                                                                                                                                  
   Has Settable Display Name: true                                                                                                                                                  
   Size: 2097152                                                                                                                                                                    
   Device Type: Direct-Access                                                                                                                                                       
   Multipath Plugin: NMP                                                                                                                                                            
   Devfs Path: /vmfs/devices/disks/naa.600a09803830462d803f4c6e68664e2d                                                                                                             
   Vendor: NETAPP                                                                                                                                                                   
...snip for length...                                                                                            
   Is Boot Device: false                                                                                                                                                            
   Device Max Queue Depth: 32                                                                                                                                                       
   No of outstanding IOs with competing worlds: 32                                                                                                                                  
   Drive Type: unknown                                                                                                                                                              
   RAID Level: unknown                                                                                                                                                              
   Number of Physical Drives: unknown                                                                                                                                               
   Protection Enabled: false                                                                                                                                                        
   PI Activated: false                                                                                                                                                              
   PI Type: 0                                                                                                                                                                       
   PI Protection Mask: NO PROTECTION
   Supported Guard Types: NO GUARD SUPPORT
   DIX Enabled: false
   DIX Guard Type: NO GUARD SUPPORT
   Emulated DIX/DIF Enabled: false

Now we change it to 128 for this device

esxcli storage core device set -O 128 -d naa.600a09803830462d803f4c6e68664e2d

And when checking the same output we can see "Device Max Queue Depth:" and "No of outstanding IOs with competing worlds:" are both now 128. If the changes are not immediately reflected then a reboot of the ESXi host may be needed. 

[root@localhost:~] esxcli storage core device list -d naa.600a09803830462d803f4c6e68664e2d
naa.600a09803830462d803f4c6e68664e2d
   Display Name: VMWare_SAS_STG_01
   Has Settable Display Name: true
   Size: 2097152
   Device Type: Direct-Access 
   Multipath Plugin: NMP
   Devfs Path: /vmfs/devices/disks/naa.600a09803830462d803f4c6e68664e2d
   Vendor: NETAPP  
...snip for length... 
   Is Boot Device: false
   Device Max Queue Depth: 128
   No of outstanding IOs with competing worlds: 128
   Drive Type: unknown
   RAID Level: unknown
   Number of Physical Drives: unknown
   Protection Enabled: false
   PI Activated: false
   PI Type: 0
   PI Protection Mask: NO PROTECTION
   Supported Guard Types: NO GUARD SUPPORT
   DIX Enabled: false
   DIX Guard Type: NO GUARD SUPPORT
   Emulated DIX/DIF Enabled: false