Guidelines for using NVMe over Fabrics (NVMeoF) with RoCEv2 on Linux
General Guidelines and Limitations
-
Cisco recommends that you check UCS Hardware and Software Compatibility specific to your Cisco IMC release to determine support for NVMeoF. NVMeoF is supported on Cisco UCS C-Series M5 and later servers.
-
NVMeoF with RoCEv2 is supported only with the Cisco UCS VIC 14xx series adapters. NVMeoF is not supported on Cisco UCS VIC 12xx or 13xx series adapters.
-
When creating RoCEv2 interfaces, use Cisco IMC provided Linux-NVMe-RoCE adapter policy.
-
When configuring RoCEv2 interfaces, use both the enic and enic_rdma binary drivers downloaded from cisco.com and install the matched set of enic and enic_rdma drivers. Attempting to use the binary enic_rdma driver downloaded from cisco.com with an inbox enic driver does not work.
-
Only two RoCEv2 enabled vNICs per adapter are supported.
-
Booting from an NVMeoF namespace is not supported.
-
Layer 3 routing is not supported.
-
RoCEv2 does not support bonding.
-
Saving a crashdump to an NVMeoF namespace during a system crash is not supported.
-
NVMeoF cannot be used with usNIC, VxLAN, VMQ, VMMQ, NVGRE, and DPDK features.
-
The QoSno drop class configuration must be properly configured on upstream switches such as Cisco Nexus 9000 series switches. QoS configurations vary between different upstream switches.
-
Set MTU size correctly on the VLANs and QoS policy on upstream switches.
-
Spanning Tree Protocol (STP) may cause temporary loss of network connectivity when a failover or failback event occurs. To prevent this issue from occurring, disable STP on uplink switches.
Interrupts
-
Linux RoCEv2 interface supports only MSIx interrupt mode. Cisco recommends that you avoid changing interrupt mode when the interface is configured with RoCEv2 properties.
-
The minimum interrupt count for using RoCEv2 with Linux is 8.
Downgrade Limitations
Cisco recommends that you remove the RoCEv2 configuration before downgrading to any non-supported RoCEv2 release.