Guidelines for using NVMe over Fabrics (NVMeoF) with RoCEv2 on Linux
General Guidelines and Limitations
-
Cisco recommends that you check UCS Hardware and Software Compatibility specific to your Cisco IMC release to determine support for NVMeoF. NVMeoF is supported on Cisco UCS C-Series M5 and later servers.
-
NVMeoF with RoCEv2 is supported only with the Cisco UCS VIC 14xx series adapters. NVMeoF is not supported on Cisco UCS VIC 12xx or 13xx series adapters.
-
When creating RoCEv2 interfaces, use Cisco IMC provided Linux-NVMe-RoCE adapter policy.
-
Only two RoCEv2 enabled vNICs per adapter are supported.
-
Booting from an NVMeoF namespace is not supported.
-
Layer 3 routing is not supported.
-
RoCEv2 does not support bonding.
-
Saving a crashdump to an NVMeoF namespace during a system crash is not supported.
-
NVMeoF cannot be used with usNIC, VMFEX, VxLAN, VMQ, VMMQ, NVGRE, Geneve offload and DPDK features.
-
Netflow monitoring is not supported on RoCEv2 interfaces.
-
In the Linux-NVMe-RoCE policy, do not change values of Queue Pairs, Memory Regions, Resource Groups, and Priority settings other than to Cisco provided default values. NVMeoF functionality may not be guaranteed with different settings for Queue Pairs, Memory Regions, Resource Groups, and Priority.
-
The QoSno drop class configuration must be properly configured on upstream switches such as Cisco Nexus 9000 series switches. QoS configurations vary between different upstream switches.
-
Set MTU size correctly on the VLANs and QoS policy on upstream switches.
-
Spanning Tree Protocol (STP) may cause temporary loss of network connectivity when a failover or failback event occurs. To prevent this issue from occurring, disable STP on uplink switches.
Interrupts
-
Linux RoCEv2 interface supports only MSIx interrupt mode. Cisco recommends that you avoid changing interrupt mode when the interface is configured with RoCEv2 properties.
-
The minimum interrupt count for using RoCEv2 with Linux is 8.
Downgrade Limitations
Cisco recommends that you remove the RoCEv2 configuration before downgrading to any non-supported RoCEv2 release.