Field Notice: FN - 72155 - NFNIC Driver Returns Target Aborted IO - Software Upgrade Recommended

Notice

THIS FIELD NOTICE IS PROVIDED ON AN "AS IS" BASIS AND DOES NOT IMPLY ANY KIND OF GUARANTEE OR WARRANTY, INCLUDING THE WARRANTY OF MERCHANTABILITY. YOUR USE OF THE INFORMATION ON THE FIELD NOTICE OR MATERIALS LINKED FROM THE FIELD NOTICE IS AT YOUR OWN RISK. CISCO RESERVES THE RIGHT TO CHANGE OR UPDATE THIS FIELD NOTICE AT ANY TIME.

Revision History

Revision	Publish Date	Comments
1.0	13-Apr-21	Initial Release

Products Affected

Affected OS Type	Affected Software Product	Affected Release	Affected Release Number	Comments
NON-IOS	Unified Computing System (UCS) Drivers	OS Based Driver ISO	esxi	VMWare ESXi 6.7 VMWare ESXi 7.0

Defect Information

Defect ID	Headline
CSCvx64370	NFNIC driver returns target aborted IO as completed successfully

Problem Description

In certain conditions, the Native FNIC (NFNIC) driver might incorrectly mark an aborted IO operation as successful. The driver will not attempt to retry the IOs and as a result data will be missing or corrupted on the storage target.

This is applicable to target aborted IOs only. For example, IOs which are aborted by a storage array. If an IO times out or is aborted by the initiator, the NFNIC driver will correctly handle the operation. Any Unified Computing System (UCS) servers which utilize the NFNIC driver (both rack and blade servers) are impacted.

Background

The NFNIC driver is responsible for handling Fibre-Channel (FC) IOs (frames), which includes error handling and reporting. This driver was introduced in the VMware ESXi 6.7 release and is different from the “legacy” FNIC driver. Only the NFNIC driver handles this condition incorrectly.

Problem Symptom

This issue can result in data integrity issues on storage targets. Data might be missing, incomplete, or corrupted with no warning prior to loss of data integrity.

Target initiated aborts will mostly occur when Fibre-Channel (FC) frames are dropped or received out of order by the target. This might be triggered by the addition or deletion of FC port-channel members, which can initiate a frame drop. You might see the VMKernel log message "hdr status: FCPIO_ABORTED IO failure!".

An example of a VMKernel log message is shown here:

WARNING: nfnic: <2>: fnic_fcpio_icmnd_cmpl_handler: 1850: sc: 0x459a92c02ac0 tag: 0x5e5 hdr status: FCPIO_ABORTED IO failure!

Workaround/Solution

This issue requires a software update to the NFNIC driver. The best solution is to upgrade the NFNIC driver to the fixed 4.0.0.65 release or later. Review the HCL Tool for the appropriate driver to run for your hardware/software/firmware configuration. The NFNIC 4.0.0.65 driver for ESXi 6.7 and 7.0 can be downloaded from VMware Customer Connect.

For certain storage arrays, there might be hardened storage array code which can avoid this situation. See the defect notes of Cisco bug ID CSCvx64370 for more details. Cisco still recommends to upgrade to a fixed NFNIC release even if you run hardened storage code.

In order to update your Cisco NFNIC driver, enter this command on your ESXi host:

#esxcli software vib install -v /path/to/driver/

Here is an example:

[root@esxhost1:~] esxcli software vib install -v /tmp/nfnic-4.0.0.65-1OEM.670.0.0.8169922.x86_64.vib

How To Identify Affected Products

This command can be used on the ESXi CLI shell in order to confirm the version of NFNIC driver on your ESXi host. Note that if the NFNIC driver is not listed, then the NFNIC driver is not in use.

#esxcli software vib list | grep nfnic

An example is shown here:

[root@esxhost1:~] esxcli software vib list | grep nfnic
nfnic                          4.0.0.40-1OEM.670.0.0.8169922         Cisco        VMwareCertified   2020-04-04

In this example, the NFNIC is Version 4.0.0.40. It will need to be upgraded. The appropriate NFNIC driver that users should run can be determined with the HCL Tool.

Additional Information

This issue might also (rarely) be triggered as a result of a write queue (WQ) error due to bad hardware. The symptoms of this failure will be the same, and an upgrade of the NFNIC driver will prevent any data loss or corruption. If customers continue to see messages such as the “FCPIO_ABORTED” message shown after an NFNIC driver upgrade, contact the Cisco Technical Assistance Center (TAC).

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

• Open a service request on Cisco.com
• By email or telephone

Receive Email Notification For New Field Notices

My Notifications—Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.