Guidelines for using NVMe over Fabrics (NVMeoF) with RoCEv2 on Linux
General Guidelines and Limitations:
-
Cisco recommends you check UCS Hardware and Software Compatibility specific to your UCS Manager release to determine support for NVMeoF. NVMeoF is supported on UCS M5 and later B-Series and C- Series servers.
-
NVMe over RDMA with RoCEv2 is supported with the fourth generation Cisco UCS VIC 1400 Series and UCS VIC 15000 Series adapters. NVMe over RDMA is not supported on UCS 6324 Fabric Interconnects or on UCS VIC 1200 Series and 1300 Series adapters.
-
When creating RoCEv2 interfaces, use Cisco UCS Manager provided Linux-NVMe-RoCE adapter policy.
Note
Do not use the default Linux Adapter policy with RoCEv2; RoCEv2 interfaces will not be created in the OS.
-
When configuring RoCEv2 interfaces, use both the enic and enic_rdma binary drivers downloaded from Cisco.com and install the matched set of enic and enic_rdma drivers. Attempting to use the binary enic_rdma driver downloaded from Cisco.com with an inbox enic driver will not work.
-
RoCEv2 supports maximum two RoCEv2 enabled interfaces per adapter.
-
Booting from an NVMeoF namespace is not supported.
-
Layer 3 routing is not supported.
-
RoCEv2 does not support bonding.
-
Saving a crashdump to an NVMeoF namespace during a system crash is not supported.
-
NVMeoF cannot be used with usNIC, VMFEX, VxLAN, VMQ, VMMQ, NVGRE, GENEVE Offload, and DPDK features.
-
Netflow monitoring is not supported on RoCEv2 interfaces.
-
In the Linux-NVMe-RoCE policy, do not change values of Queue Pairs, Memory Regions, Resource Groups, and Priority settings other than to Cisco provided default values. NVMeoF functionality may not be guaranteed with different settings for Queue Pairs, Memory Regions, Resource Groups, and Priority.
-
The QoS no drop class configuration must be properly configured on upstream switches such as Cisco Nexus 9000 series switches. QoS configurations will vary between different upstream switches.
-
Set MTU size correctly on the VLANs and QoS policy on upstream switches.
-
Spanning Tree Protocol (STP) may cause temporary loss of network connectivity when a failover or failback event occurs. To prevent this issue from occurring, disable STP on uplink switches.
-
UCS Manager does not support fabric failover for vNICs with RoCEv2 enabled.
Interrupts
-
Linux RoCEv2 interface supports only MSIx interrupt mode. Cisco recommends avoiding changing interrupt mode when the interface is configured with RoCEv2 properties.
-
The minimum interrupt count for using RoCEv2 with Linux is 8.
Downgrade Limitations:
-
Cisco recommends you remove the RoCEv2 configuration before downgrading to any non-supported RoCEv2 release.