Step 1

Navigate to CONFIGURE > Policies. Click Create Policy, select UCS Domain platform type, search or choose System QoS, and click Start.

Step 2

In the General page, enter the policy name and click Next, and then in the Policy Details page, configure the property setting for System QoS policy as follows:

• For Priority, choose Platinum

• For Allow Packet Drops, uncheck the check box.

  Note	For more information on MTU field, see MTU Properties in Guidelines for Using SMB Direct support on Windows using RDMA over converged Ethernet (RoCE) v2

Step 3

Click Create

Step 4

Associate the System QoS policy to the Domain Profile and deploy.

Step 1

In the Windows host, go to the Device Manager and select the appropriate Cisco VIC Internet Interface.

Step 2

Go to Tools > Computer Management > Device Manager > Network Adapter > click on VIC Network Adapter > Properties > Advanced > Network Direct Functionality. Perform this operation for both the smb-server and smb-client vNICs.

Step 3

Verify that RoCE is enabled on the host operating system using PowerShell.

PS C:\Users\Administrator> Get-NetOffloadGlobalSetting
 
ReceiveSideScaling           : Enabled
ReceiveSegmentCoalescing     : Enabled
Chimney                      : Disabled
TaskOffload                  : Enabled
NetworkDirect                : Enabled
NetworkDirectAcrossIPSubnets : Blocked
PacketCoalescingFilter       : Disabled

Step 4

Bring up Powershell and enter the command:

get-SmbClientNetworkInterface

Step 5

Enter enable - netadapterrdma [-name] ["Ethernetname"]

Step 6

Verify the overall RoCE v2 Mode 1 configuration at the Host as follows:

1. Use the Powershell command netstat -xan to verify the listeners in both the smb-client and smb-server Windows host; listeners will be shown in the command output.

   

2. Go to the smb-client server fileshare and start an I/O operation.

3. Go to the performance monitor and check that it displays the RDMA activity.

   

Step 7

In the Powershell command window, check the connection entries with the netstat -xan output command to make sure they are displayed. You can also run netstat -xan from the command prompt. If the connection entry shows up in netstat-xan output, the RoCE v2 mode1 connections are correctly established between client and server.

Note	IP values are representative only.

Step 8

By default, Microsoft's SMB Direct establishes two RDMA connections per RDMA interface. You can change the number of RDMA connections per RDMA interface to one or any number of connections.

PS C:\Users\Administrator> Set-ItemProperty -Path ` "HKLM:\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters" ConnectionCountPerRdmaNetworkInterface -Type DWORD -Value 4 –Force

Step 1

Navigate to CONFIGURE > Policies. Click Create Policy, select UCS Server platform type, search or choose LAN Connectivity policy, and click Start.

Step 2

In the policy General page, enter the policy name, select the Target Platform as UCS Server (Standalone) or UCS Server (FI-Attached), and click Next.

Step 3

In the Policy Details page, click Add vNIC to create a new vNIC.

Step 4

In the Add vNIC page, follow the configuration parameters to enable the RoCE vNIC settings:

1. In the General section, provide a name for virtual ethernet interface.

2. In the Consistent Device Naming (CDN) section of the Standalone server or the Failover section of FI-attached server, do the following:

   • Click Select Policy link below the Ethernet QoS. Use the Create New button to create a new Ethernet QoS policy with the following property settings:

     • MTU—The Maximum Transmission Unit (MTU) or packet size that the virtual interface accepts. For MTU, choose or enter 1500, 4096, or 9000

     • Rate Limit, Mbps—The value in Mbps (0-10G/40G/100G depending on Adapter Model) to use for limiting the data rate on the virtual interface.

     • Class of Service—Class of Service to be associated to the traffic on the virtual interface.

     • Burst—The burst traffic, in bytes, allowed on the vNIC.

     • For Priority, choose or enter Best-effort

     • Enable Trust Host CoS, slide to enable

       

   • Click Select Policy link below the Ethernet Adapter. Use Create New button to create a new Ethernet Adapter policy with the following property settings:

     • For Enable RDMA over Converged Ethernet, slide to enable.

     • For Queue Pairs, select or enter 256

     • For Memory Regions, select or enter 65536

     • For Resource Groups, select or enter 2

     • For Version, choose Version 2

     • For Class of Service, choose or enter 5

       

   • In the Connection section, use the following property setting for VMQ Connection and to create VMMQ Adapter policy:

     • For connection, select VMQ.

     • Enable Virtual Machine Multi-Queue using slider button.

     • For Number of Sub vNICs, select or enter 4

     • For VMMQ Adapter Policy, click Select Policy link below the VMMQ Adapter Policy and do the following:

       • Click Create New to create a new policy. In the General page, enter the name of the policy and click Select Default Configuration to search and select MQ-SMBd, the pre-defined VMMQ Adapter default configuration.

         Attention

         Do not modify the pre-defined parameters under Policy Details page, retain the default settings.

       • Click Next and then Create.

       

   • Click Add to add and save the new vNIC settings.

   Note	All the fields with * are mandatory and ensure it is filled out or selected with appropriate policies.

Step 5

Click Create to complete the LAN Connectivity policy with RoCE v2 property settings.

Step 6

Associate the LAN Connectivity policy to the server profile.

Step 1

Go the Hyper-V switch manager.

Step 2

Create a new Virtual Network Switch (vswitch) for the RoCE v2-enabled Ethernet interface.

1. Choose External Network and select VIC Ethernet Interface 2 and Allow management operating system to share this network adapter.

2. Click OK to create the create the virtual switch.

Step 3

Configure the non-default vport and enable RDMA with the following Powershell commands:

add-vmNetworkAdapter -switchname vswitch -name vp1 -managementOS

enable-netAdapterRdma -name "vEthernet (vp1)"

1. Configure set-switch using the following Powershell command.

   new-vmswitch -name setswitch -netAdapterName “Ethernet x” -enableEmbeddedTeam $true

   This creates the switch. Use the following to display the interfaces:

   get-netadapterrdma

   add-vmNetworkAdapter -switchname setswtch -name svp1

   You will see the new vport when you again enter

   get-netadapterrdma

2. Add a vport.

   add-vmNetworkAdapter -switchname setswtch -name svp1

   You will see the new vport when you again enter

   get-netadapterrdma

3. Enable the RDMA on the vport:

   enable-netAdapterRdma -name “vEthernet (svp1)”

Step 4

Configure the IPV4 addresses on the RDMA enabled vport in both servers.

Step 5

Create a share in smb-server and map the share in the smb-client.

1. For smb-client and smb-server in the host system, configure the RoCE v2-enabled vNIC as described above.

2. Configure the IPV4 addresses of the primary fabric and sub-vNICs in both servers, using the same IP subnet and same unique vlan for both.

3. Create a share in smb-server and map the share in the smb-client.

Step 6

Verify the Mode 2 configuration.

1. Use the Powershell command netstat -xan to display listeners and their associated IP addresses.

   

2. Start any RDMA I/O in the file share in smb-client.

   

3. Issue the netstat -xan command again and check for the connection entries to verify they are displayed.

   

Step 1

Navigate to CONFIGURE > Policies. In the Add Filter field, select Type: LAN Connectivity.

Step 2

Select the appropriate LAN Connectivity policy created for RoCE V2 configuration and use the delete icon on the top or bottom of the policy list.

Step 3

Click Delete to delete the policy.

Step 4

Upon deleting the RoCE v2 configuration, re-deploy the server profile and reboot the server.

Step 1

Navigate to CONFIGURE > Policies. Click Create Policy, select UCS Domain platform type, search or choose System QoS, and click Start.

Step 2

In the General page, enter the policy name and click Next, and then in the Policy Details page, configure the property setting for System QoS policy as follows:

Step 3

Click Create.

Step 4

Associate the System QoS policy to the Domain Profile.

Step 1

Open the /etc/default/grub file for editing.

Step 2

Add intel_iommu=on to the end of GRUB_CMDLINE_LINUX.

sample /etc/default/grub configuration file after adding intel_iommu=on:
# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap biosdevname=1 rhgb quiet intel_iommu=on
GRUB_DISABLE_RECOVERY="true"

Step 3

After saving the file, generate a new grub.cfg file.

# grub2-mkconfig -o /boot/grub2/grub.cfg

# grub2-mkconfig -o /boot/grub2/efi/EFI/redhat/grub.cfg

Step 4

Reboot the server. You must reboot your server for the changes to take after enabling IOMMU.

Step 5

Verify the server is booted with intel_iommu=on option.

cat /proc/cmdline | grep iommu

Note its inclusion at the end of the output.

Step 1

Install the device-mapper-multipath package if it is not installed already

Step 2

Enable and start multipathd:

# mpathconf --enable --with_multipathd y

Step 3

Edit the etc/multipath.conf file to use the following values :

Step 4

Flush with the updated multipath device maps.

# multipath -F

Step 5

Restart multipath service:

# systemctl restart multipathd.service

Step 6

Rescan multipath devices:

# multipath -v2

Step 7

Check the multipath status:

# multipath -ll

Step 1

Navigate to CONFIGURE > Policies. In the Add Filter field, select Type: LAN Connectivity.

Step 2

Select the appropriate LAN Connectivity policy created for RoCE V2 configuration and use the delete icon on the top or bottom of the policy list.

Step 3

Click Delete to delete the policy.

Step 4

Upon deleting the RoCE v2 configuration, re-deploy the server profile and reboot the server.

Step 1

Navigate to CONFIGURE > Policies. Click Create Policy, select UCS Domain platform type, search or choose System QoS, and click Start.

Step 2

In the General page, enter the policy name and click Next, and then in the Policy Details page, configure the property setting for System QoS policy as follows:

• For Priority, choose Platinum

• For Allow Packet Drops, uncheck the check box.

• For MTU, set the value as 9216.

Step 3

Click Create.

Step 4

Associate the System QoS policy to the Domain Profile.

Step 1

Copy the eNIC vSphere Installation Bundle (VIB) or offline bundle to the ESXi server.

Step 2

Use the command to install nenic driver:

esxcli software vib install -v {VIBFILE}
or
esxcli software vib install -d {OFFLINE_BUNDLE}

esxcli software vib install -v /tmp/nenic-2.0.4.0-1OEM.700.1.0.15843807.x86_64.vib

esxcli software acceptance set --level=<level>

Step 1

In the vSphere Client, navigate to the host.

Step 2

On the Configure tab, expand Networking and select Virtual Switches.

Step 3

Click on Add Networking.

Step 4

Select connection type Vmkernel Network Adapter.

Step 5

Select New Standard Switch and click Next.

Step 6

Add physical adapters to the new standard switch.

1. Under Assigned Adapters, select New Adapters.

2. Select one or more adapters from the list and click OK. To promote higher throughput and create redundancy, add two or more physical network adapters to the Active list.

3. (Optional) Use the up and down arrow keys to change the position of the adapter in the Assigned Adapters list.

4. Click Next.

Step 7

For the new standard switch you just created for the VMadapter or a port group, enter the connection settings for the adapter or port group.

1. Enter a label that represents the traffic type for the VMkernel adapter.

2. Set a VLAN ID to identify the VLAN the VMkernel uses for routing network traffic.

3. Select IPV4 or IPV6 or both.

4. Select an MTU size from the drop-down menu. Select Custom if you wish to enter a specific MTU size. The maximum MTU size is 9000 bytes.

   Note	You can enable Jumbo Frames by setting an MTU greater than 1500.

5. After setting the TCP/IP stack for the VMkernel adapter, select a TCP/IP stack.

   To use the default TCP/IP stack, select it from the available services.

   Note	Be aware that the TCP/IP stack for the VMkernel adapter cannot be changed later.

6. Configure IPV4 and/or IPV6 settings.

Step 8

On the Ready to Complete page, click Finish.

Step 9

Check the VMkernel ports for the VM Adapters or port groups with NVMe RDMA in the vSphere client, as shown in the Results below.

Step 1

Go to vCenter where your ESXi host is connected.

Step 2

Click on Host>Configure>Storage adapters.

Step 3

Click +Add Software Adapter. The following dialog box will appear.

Step 4

Select Add software NVMe over RDMA adapter and the vmrdma port you want to use.

Step 5

Click OK

Step 1

Use ssh to access the host system.

Step 2

Enter esxcfg-nics -l to list the vmnics on ESXi.

Step 3

Use esxcli rdma device list to list the vmrdma devices. When the enic driver registers with ESXi the RDMA device for a RDMA capable VNIC, ESXi creates a vmrdma device and links it to the corresponding vmnic.

Step 4

Use esxcli rdma device protocol list to check the protocols supported by the vmrdma interface.

Step 5

Use esxcli nvme adapter list to list the NVMe adapters and the vmrdma and vmnic interfaces it is configured on.

Step 6

All vmhbas in the system can be listed using esxcli storage core adapter list. The vmhba configured over RDMA.

Step 1

Check and enable Nonvolatile Memory Express (NVMe) on the vmrdma device. If NVMe is enabled, the system returns the following message.

esxcli nvme fabrics enable -p RDMA -d vmrdma0

Step 2

Discover the NVMe on the array by entering the following command.

esxcli nvme fabrics discover -a vmhba64 -l transport_address figure with esxcli nvme fabrics discover -a vmhba64 -l 50.2.84.100

Step 3

Perform NVMe fabric interconnect.

esxcli nvme fabrics discover -a vmhba64 -l transport_address p Transport Service
ID -s Subsystem NQN

Step 4

The NVMe controller displays a list of the controllers connected to NVMe. The NVMe namespace list should shows all the NVMe drivers discovered.

esxcli nvme fabrics discover -a vmhba64 -l transport_address p Transport Service
ID -s Subsystem NQN

[root@ESXiUCSA:~] esxcli nvme fabrics enable -p RDMA -d vmrdma0
NVMe already enabled on vmrdma0 [root@ESXiUCSA:~] esxcli nvme fabrics discover -a vmhba64 -l 50.2.84.100
Transport Type Address Family Subsystem Type Controller ID Admin Queue Max Size Transport
Address Transport Service ID Subsystem NQN
-------------- -------------- -------------- ------------- --------------------
----------------- -------------------- -------------
RDMA IPV4 NVM 65535 31 50.2.84.100
4420 nq.210-06.com.purestorage:flasharray:2dp1239anjkl484
[root@ESXiUCSA:~] esxcli nvme fabrics discover -a vmhba64 -l 50.2.84.100 p 4420 -s
nq.210-06.com.purestorage:flasharray:2dp1239anjkl484
Controller already connected

Step 1

Navigate to CONFIGURE > Policies. In the Add Filter field, select Type: LAN Connectivity.

Step 2

Select the appropriate LAN Connectivity policy created for RoCE V2 configuration and use the delete icon on the top or bottom of the policy list.

Step 3

Click Delete to delete the policy.

Step 4

Upon deleting the RoCE v2 configuration, re-deploy the server profile and reboot the server.