Step 1

In the Navigation pane, click Servers.

Step 2

Expand Servers > Policies.

Step 3

Expand the node for the organization where you want to create the policy.

Step 4

Expand Adapter Policies and choose the existing adapter policy for Win-HPN-SMBd.

1. On the General tab, scroll down to RoCE and click the Enabled radio button.

2. In the RoCE Properties field, under Version 1, click the Disabled radio button. For Version 2, click the Enabled radio button.

3. For Queue Pairs, enter 256.

4. For Memory Regions, enter 131072.

5. For Resource Groups, enter 2.

6. For Priority, choose Platinum No-Drop COS. from the dropdown.

7. Click Save Changes.

Step 5

Next, create an Ethernet Adapter Policy. In the Navigation pane, click LAN.

Step 6

Expand LAN > Policies.

Step 7

Right-click the vNIC Templates node and choose Create vNIC Template.

Step 8

Go to vNIC Properties under the General tab and modify the vNIC policy settings as follows:

1. Set MTU to 1500 or 4096 or 9158.

2. For the Adapter Policy, select Win-HPN-SMBd

3. For the QoS policy, specify Platinum.

Step 9

Click Save Changes.

Step 10

After you save the changes, Cisco UCS Manager will prompt you to reboot. Reboot the system.

Step 1

In the Windows host, go to the Device Manager and select the appropriate Cisco VIC Internet Interface.

Step 2

Select the Advanced tab and verify that the Network Direct Functionality property is Enabled. If not, enable it and click OK.

Step 3

Go to Tools > Computer Management > Device Manager > Network Adapter > click VIC Network Adapter > Properties > Advanced > Network Direct Functionality. Perform this operation for both the smb-server and smb-client vNICs.

Step 4

Verify that RoCE is enabled on the host operating system using PowerShell.

PS C:\Users\Administrator> Get-NetOffloadGlobalSetting
 
ReceiveSideScaling           : Enabled
ReceiveSegmentCoalescing     : Enabled
Chimney                      : Disabled
TaskOffload                  : Enabled
NetworkDirect                : Enabled
NetworkDirectAcrossIPSubnets : Blocked
PacketCoalescingFilter       : Disabled

Step 5

Bring up the Powershell and execute the get -SmbClientNetworkInterface command.

PS C:\Users\Administrator>
PS C:\Users\Administrator> Get-SmbClientNetworklnterface
Interface Index    RSS Capable    RDKA Capable    Speed    IpAddresses    Friendly Name
---------------    ------------   ------------   -------   -----------  ---------------
14                True            False          40 Gbps    {10.37.60.162}    vEthernet (vswitch)
26                True            True           40 Gbps    {10.37.60.158}    vEthernet (vpl)
9                 True            True           40 Gbps    {50.37.61.23}     Ethernet 2
5                 False           False          40 Gbps    {169.254.10.S}    Ethernet (Kernel Debugger)
8                 True            False          40 Gbps    {169.254.4.26}    Ethernet 3
PS C:\Users\Administrator>

Step 6

Enter enable - netadapterrdma [-name] ["Ethernetname"]

Step 7

Verify the overall RoCEv2 Mode 1 configuration at the host:

1. Use the Powershell command netstat -xan to verify the listeners in both the smb-client and smb-server Windows host; listeners will be shown in the command output.

   PS C:\Users\Administrator>
   PS C:\Users\Administrator> netstat -xan
   Active NetworkDirect Connections, Listeners, SharedEndpoints
   Mode    Iflndex    Type    Local Address    Foreign Address    PID
   Kernel    9        Listener  50.37.61.23:445    NA             0
   Kernel    26       Listener  10.37.60.158:445   NA             0
   PS C:\Users\Administrator>

2. Go to the smb-client server fileshare and start an I/O operation.

3. Go to the performance monitor and check that it displays the RDMA activity.

Step 8

In the Powershell command window, check the connection entries with the netstat -xan output command to make sure they are displayed. You can also run netstat -xan from the command prompt. If the connection entry shows up in netstat-xan output, the RoCEv2 mode1 connections are correctly established between client and server.

PS C:\Users\Administrator> nctstat -xan
Active NetworkDirect Connections, Listeners, SharedEndpoints
Mode    IfIndex    Type    Local Address        Foreign Address    PID
Kernel   4    Connection    50.37.61.22:445    50.37.61.71:2240    0
Kernel   4    Connection    50.37.61.22:445    50.37.61.71:2496    0
Kernel   11   Connection    50.37.61.122:445   50.37.61.71:2752    0
Kernel   11   Connection    50.37.61.122:445   50.37.61.71:3008    0
Kernel   32   Connection    10.37.60.155:445   50.37.60.61:49092   0
Kernel   32   Connection    10.37.60.155:445   50.37.60.61:49348   0
Kernel   26   Connection    50.37.60.32:445    50.37.60.61:48580   0
Kernel   26   Connection    50.37.60.32:445    50.37.60.61:48836   0
Kernel   4    Listener      50.37.61.22:445    NA                  0
Kernel   11   Listener      50.37.61.122:445   NA                  0
Kernel   32   Listener      10.37.60.155:445   NA                  0
Kernel   26   Listener      50.37.60.32:445    NA                  0

Step 9

By default, Microsoft's SMB Direct establishes two RDMA connections per RDMA interface. You can change the number of RDMA connections per RDMA interface to one or any number of connections.

PS C:\Users\Administrator> Set-ItemProperty -Path ` "HKLM:\SYSTEM\CurrentControlSet\Services
\LanmanWorkstation\Parameters" ConnectionCountPerRdmaNetworkInterface -Type DWORD -Value 4 –Force

Step 1

In the Navigation pane, click Servers.

Step 2

Expand Servers > Service Profiles.

Step 3

Expand Service Profiles > vNICs and choose the VMQ Connection policy profile to configure.

Step 4

Go to vNIC Properties under the General tab and scroll down to the Policies area. Modify the vNIC policy settings as follows:

1. For the Adapter Policy, make sure it uses Win-HPN-SMBd or the adapter policy configured earlier for Mode 1.

2. For the QoS policy, select best-effort.

Step 5

Click Save Changes.

Step 6

In the Navigation pane, click LAN.

Step 7

Expand LAN > Policies > QoS Policy Best Effort.

Step 8

Set Host Control to Full.

Step 9

Click Save Changes.

Step 10

After you save the changes, Cisco UCS Manager will prompt you to reboot. Reboot the interface.

Step 1

Go to the Hyper-V switch manager.

Step 2

Create a new Virtual Network Switch (vswitch) for theRoCEv2-enabled Ethernet interface.

1. Choose External Network and select VIC Ethernet Interface 2 and Allow management operating system to share this network adapter.

2. Click OK to create the virtual switch.

Step 3

Configure the non-default vport and enable RDMA with the following Powershell commands:

add-vmNetworkAdapter -switchname vswitch -name vp1 -managementOS

enable-netAdapterRdma -name "vEthernet (vp1"

PS C:\Users\Administrator>
PS C:\Users\Administrator> add - vmNet workAdapter -switchName vswitch -name vpl -managementOS 
PS C:\Users\Administrator> enable-netAdapterRdma -name "vEthernet (vpl)"
PS C:\Users\Administrator>

1. Configure the set-switch using the following Powershell command.

   new-vmswitch -name setswitch -netAdapterName “Ethernet x” -enableEmbeddedTeam $true

   This creates the switch. Use the following to display the interfaces:

   get-netadapterrdma

   add-vmNetworkAdapter -switchname setswtch -name svp1

   You will see the new vport when you again enter

   get-netadapterrdma

2. Add a vport:

   add-vmNetworkAdapter -switchname setswtch -name svp1

   You see the new vport when you again enter:

   get-netadapterrdma

3. Enable the RDMA on the vport:

   enable-netAdapterRdma -name “vEthernet (svp1)”

Step 4

Configure the IPv4 addresses on the RDMA enabled vport in both servers.

Step 5

Create a share in smb-server and map the share in the smb-client.

1. For smb-client and smb-server in the host system, configure the RoCEv2-enabled vNIC as described above.

2. Configure the IPv4 addresses of the primary fabric and sub-vNICs in both servers, using the same IP subnet and same unique VLAN for both.

3. Create a share in smb-server and map the share in the smb-client.

Step 6

Finally, verify the Mode 2 configuration.

1. Use the Powershell command netstat -xan to display listeners and their associated IP addresses.

   PS C:\Users\Administrator>
   PS C:\Users\Administrator> netstat -xan
   Active    NetworkDirect Connections, Listeners, SharedEndpoints
   Mode    Iflndex    Type    Local Address    Foreign Address    PID
   Kernel    9 Listener    50.37.61.23:445    NA    0
   Kernel    26 Listener    10.37.60.158:445    NA    0
   PS C:\Users\Administrator>

2. Start any RDMA I/O in the file share in smb-client.

3. Issue the netstat -xan command again and check for the connection entries to verify they are displayed.

   PS C:\Users\Administrator>
   PS C:\Users\Administrator> netstat -xan
   Active NetworkDirect Connections, Listeners, SharedEndpoints
   Mode    IfIndex    Type    Local Address    Foreign Address    PID
   Kernel    9    Connection    50.37.61.23:192    50.37.61.184:445    0
   Kernel    9    Connection    50.37.61.23:448    50.37.61.184:445    0
   Kernel    9    Connection    50.37.61.23:704    50.37.61.214:445    0
   Kernel    9    Connection    50.37.61.23:960    50.37.61.214:445    0
   Kernel    9    Connection    50.37.61.23:1216    50.37.61.224:44    05
   Kernel    9    Connection    50.37.61.23:1472    50.37.61.224:445    0
   Kernel    9    Connection    50.37.61.23:1728    50.37.61.234:445    0
   Kernel    9    Connection    50.37.61.23:1984    50.37.61.234:445    0
   Kernel    9    Listener    50.37.61.23:445    NA
   Kernel    26    Listener    10.37.60.158:445    NA
   PS C:\Users\Administrator>

Step 1

In the Navigation pane, click Servers.

Step 2

Expand Servers > Service Profiles.

Step 3

Expand the node for the organization where you want to create the policy.

Step 4

Click on vNICs and go to the Network tab in the work area.

1. On the Network tab, scroll down to the desired vNIC and click on it, then click Modify.

2. A popup dialog box will appear. Scroll down to the Adapter Performance Profile area, and click on the Adapter Policy drop-down. Choose Linux-NVMe-RoCE from the drop-down list.

3. Click OK.

Step 5

Click Save Changes.

Step 1

Open the /etc/default/grub file for editing.

Step 2

Add intel_iommu=on to the end of the line for GRUB_CMDLINE_LINUXas shown in the sample file below.

sample /etc/default/grub configuration file after adding intel_iommu=on:
# cat /etc/default/grub
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap biosdevname=1 rhgb quiet intel_iommu=on
GRUB_DISABLE_RECOVERY="true"

Step 3

Save the file.

Step 4

After saving the file, run the following command to generate a new grub.cfg file:

Step 5

Reboot the server. You must reboot your server for the changes to take after enabling IOMMU.

Step 6

Verify that the server booted with the intel_iommu=on option by checking the output file.

cat /proc/cmdline | grep iommu

[root@localhost basic-setup]# cat /proc/cmdline | grep iommu
BOOT_IMAGE=/vmlinuz-3.10.0-957.27.2.el7.x86_64 root=/dev/mapper/rhel-
root ro crashkernel=auto rd.lvm.lv=rhel/root rd.lvm.lv=rhel/swap rhgb 
quiet intel_iommu=on LANG=en_US.UTF-8

Step 1

Install the device-mapper-multipath package if it is not installed already

Step 2

Enable and start multipathd:

# mpathconf --enable --with_multipathd y

Step 3

Edit the etc/multipath.conf file to use the following values :

defaults {
polling_interval    10
path_selector    "queue-length 0"
path_grouping_policy    multibus
fast_io_fail_tmo    10
no_path_retry    0
features    0
dev_loss_tmo    60
user_friendly_names    yes
}

Step 4

Flush with the updated multipath device maps.

# multipath -F

Step 5

Restart multipath service:

# systemctl restart multipathd.service

Step 6

Rescan multipath devices:

# multipath -v2

Step 7

Check the multipath status:

# multipath -ll

Step 1

In the Navigation pane, click Servers.

Step 2

Expand Servers > Service Profiles.

Step 3

Expand the node for the organization where you want to create the policy.

Step 4

Modify the vNIC policy, according to the steps below.

1. On the Network tab, scroll down to the desired vNIC and click on it, then click Modify.

2. A popup dialog box will be displayed. Scroll down to the Policies area, and choose Linux from the Adapter Policy drop-down list.

3. Click OK.

Step 5

Click Save Changes.

Step 1

In the Navigation pane, click Servers.

Step 2

Expand Servers > Service Profiles.

Step 3

Expand the node for the organization where you want to create the policy.

Step 4

Click on a RDMA service profile you created and expand the service profile.

Step 5

Right-click on vNICs and choose Create vNIC to create a new vNIC.

1. Name the new VNIC.

2. On the MAC address drop-down, select the option from Manual using OUI or Domain Pools in the drop-down.

3. Select which VLAN you want use from the list.

4. In the Adapter Performance Profile, select the default adapter policy named VMWareNVMeRoCEv2.

5. Click OK. The interface is now configured for one port.

Step 6

Click Save Changes.

Step 1

Copy the eNIC VIB or offline bundle to the ESX server. The example below uses the Linux scp utility to copy the file from a local system to an ESX server located at 10.10.10.10: and uses the location /tmp.

scp nenic-2.0.4.0-1OEM.700.1.0.15843807.x86_64.vib root@
10.10.10.10:/tmp

Step 2

Specifying the full path, issue the command shown below.

esxcli software vib install -v {VIBFILE}

esxcli software vib install -d {OFFLINE_BUNDLE}

esxcli software vib install -v /tmp/nenic-2.0.4.0-1OEM.
700.1.0.15843807.x86_64.vib

Step 1

In the vSphere Client, navigate to the host.

Step 2

On the Configure tab, expand Networking and select Virtual Switches.

Step 3

Click on Add Networking.

Step 4

Select connection type Vmkernel Network Adapter.

Step 5

Select New Standard Switch and click Next.

Step 6

Add physical adapters to the new standard switch.

1. Under Assigned Adapters, select New Adapters.

2. Select one or more adapters from the list and click OK. To promote higher throughput and create redundancy, add two or more physical network adapters to the Active list.

3. (Optional) Use the up and down arrow keys to change the position of the adapter in the Assigned Adapters list.

4. Click Next.

Step 7

For the new standard switch you just created for the VMadapter or a port group, enter the connection settings for the adapter or port group.

1. Enter a label that represents the traffic type for the VMkernel adapter.

2. Set a VLAN ID to identify the VLAN the VMkernel uses for routing network traffic.

3. Select IPV4 or IPV6 or both.

4. Select an MTU size from the drop-down menu. Select Custom if you wish to enter a specific MTU size. The maximum MTU size is 9000 bytes.

   Note	You can enable Jumbo Frames by setting an MTU greater than 1500.

5. After setting the TCP/IP stack for the VMkernel adapter, select a TCP/IP stack.

   To use the default TCP/IP stack, select it from the available services.

   Note	Be aware that the TCP/IP stack for the VMkernel adapter cannot be changed later.

6. Configure IPV4 and/or IPV6 settings.

Step 8

On the Ready to Complete page, click Finish.

Step 9

Check the VMkernel ports for the VM Adapters or port groups with NVMe RDMA in the vSphere client, as shown in the Results below.

Step 1

Go to vCenter where your ESXi host is connected.

Step 2

Click on Host>Configure>Storage adapters.

Step 3

Click +Add Software Adapter.

Step 4

Select Add software NVMe over RDMA adapter and the vmrdma port you want to use.

Step 5

Click OK

Step 1

Use ssh to access the host system.

Step 2

Enter esxcfg-nics -l to list the vmnics on ESXi.

Name   PCI          Driver  Link  Speed     Duplex  MAC Address       MTU  Description
vmnico 0000:3b:00.0 ixgben  Down  0Mbps     Half    2c:f8:9b:a1:4c:e6 1500 Intel(R) Ethernet Controller X550
vmnic1 0000:36:00.1 ixgben  Up    1000Mbps  Full    2c:f8:9b:a1:4c:e7 1500 Intel(R) Ethernet Controller X550
vmnic2 0000:1d:00.0 nenic   Up    50000Mbps Full    2c:f8:9b:79:8d:bc 1500 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic3 0000:1d:00.1 nenic   Up    50000Mbps Full    2c:f8:9b:79:8d:bd 1500 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic4 0000:63:00.0 nenic   Down  0Mbps     Half    2c:f8:9b:51:b3:3a 1500 Cisco Systems Inc Cisco VIC Ethernet NIC
Venic5 0000:63:00.1 nenic   Down  0Mbps     Half    2c:f8:9b:51:b3:3b 1500 Cisco Systems Inc Cisco VIC Ethernet NIC

Name   PCI          Driver  Admin Status Link Status Speed Duplex MAC Address       MTU  Description
vmnico 0000:3b:00.0 ixgben  Up           Down        0     Half   2c:f8:9b:a1:4c:e6 1500 Intel(R) Ethernet Controller X550
vmnic1 0000:36:00.1 ixgben  Up           Up          1000  Full   2c:f8:9b:a1:4c:e7 1500 Intel(R) Ethernet Controller X550
vmnic2 0000:1d:00.0 nenic   Up           Up          50000 Full   2c:f8:9b:79:8d:bc 1500 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic3 0000:1d:00.1 nenic   Up           Up          50000 Full   2c:f8:9b:79:8d:bd 1500 Cisco Systems Inc Cisco VIC Ethernet NIC
vmnic4 0000:63:00.0 nenic   Up           Down        0     Half   2c:f8:9b:51:b3:3a 1500 Cisco Systems Inc Cisco VIC Ethernet NIC
Venic5 0000:63:00.1 nenic   Up           Down        0     Half   2c:f8:9b:51:b3:3b 1500 Cisco Systems Inc Cisco VIC Ethernet NIC

Step 3

Use esxcli rdma device list to list the vmrdma devices.


[root@RackServer:~] esxcli rdma device list 
Name    Driver State  MTU  Speed   Paired Uplink Description
-----   ------ -----  ---  -----   ------------- -----
vmrdma0 nenic  Active 4096 50 Gbps vmnic1        Cisco UCS VIC 15XXX (A0)
vmrdmal nenic  Active 4096 50 Gbps vmnic2        Cisco UCS VIC 15XXX (A0)
[root@StockholmRackServer:~] esxcli rdma device vmknic list
Device  Vmknic NetStack
------- ------ --------
vmrdma0 vmk1   defaultTcpipStack
vmrdma1 vmk2   defaultTcpipStack

Step 4

Use esxcli rdma device list to check the protocols supported by the vmrdma interface.

Step 5

Use esxcli rdma device protocol list to check the protocols supported by the vmrdma interface.


[root@RackServer:~] esxcli rdma protocol list 
Device  RoCE v1 RoCE v2 iWARP
-----   ------- ------- -----
vmrdma0 false   true    false
vmrdmal false   true    false

Step 6

Use esxcli nvme adapter list to list the NVMe adapters and the vmrdma and vmnic interfaces it is configured on.


[root@RackServer:~] esxcli nvme adapter list 
Adapter Adapter Qualified Name          Transport Type Driver   Associated Devices 
------- ----------------------          -------------- ------   ------------------
vmhba64 aqn: nvmerdma:2c-f8-9b-79-8d-bc RDMA           nvmerdma vmrdmaR, vmnic2 
vmhba65 aqn: nvmerdma:2c-f8-9b-79-8d-bd RDMA           nvmerdma vmrdma1, vmnic3

Step 7

All vmhbas in the system can be listed using esxcli storage core adapter list.

[root@RackServer:~] esxcli storage core adapter list
HBA Name Driver   Link State UID                                  Capabilities        Description
-------- ------   ---------- ------------------------------------ ------------------- -------------------------------------
vmhbao   nfnic    link-down  fc.10002cf89b798dbe:20002cf89b798dbe Second Level Lun ID (0000:1d:00.2) Cisco Corporation Cisco 
                                                                                        UCS VIC Fnic Controller
vmhba1   vmw_ahci link-n/a   sata.vmhba1                                              (0000:00:11.5) Intel Corporation Lewisburg 
                                                                                        SATA AHCI Controller
vmhba2   nfnic    link-down  fc.10002cf89b798dbf:20002cf89b798dbf Second Level Lun ID (0000:1d:00.3) Cisco Corporation Cisco 
                                                                                        UCS VIC Fnic Controller
vmhba3   nfnic    link-down  fc.10002cf89b51b33c:20002cf89b51b33c Second Level Lun ID (0000:63:00.2) Cisco Corporation Cisco 
                                                                                        UCS VIC Fnic Controller 
vmhba4   nfnic    link-down  fc.10002cf89b51b33d:20002cf89b51b33d Second Level Lun ID (0000:63:00.3) Cisco Corporation Cisco 
                                                                                        UCS VIC Fnic Controller 
vmhba5   lsi_mr3  link-n/a   sas.5cc167e9732f9b00                                     (0000:3c:00.0) Broadcom Cisco 126 Modular 
                                                                                        Raid Controller with 2GB cache
vmhba64  nvmerdma link-n/a   rdma.vmnic2:2c: f8:9b:79:8d:bc                         VMware NVMe over RDMA Storage Adapter on vmrdma0
vmhba65  nvmerdma link-n/a   rdma.vmnic3:2c:f8:9b:79:8d:bd                          VMware NVMe over RDMA Storage Adapter on vmrdma1

Step 1

Check and enable NVMe on the vmrdma device.

Step 2

Discover the NVMe fabric on the array by entering the following command:

Step 3

Perform NVMe fabric interconnect.

Step 4

Repeat steps 1 through 4 to configure the second adapter.

Step 5

Display the controller list to verify the NVMe controller is present and operating.

[root@RackServer:~] esxcli nvme controller list
Name                                      Controller Number Adapter  Transport Type Is Online
----------------------------------------  ----------------- -------- -------------- ---------
nqn.2010-06.com.purestorage: flasharray.  258               vmhba64  RDMA           true
5ab274df5b161455#vmhba64#50.2.84.100:4420 
nqn.2010-06.com.purestorage: flasharray.  259               vmhba65  RDMA           true
Sab274df5b161455#vmhba65#50.2.83.100:4420 
[root@RackServer:~] esxcli nvme namespace list
Name                                 Controller Number Namespace ID Block Size Capacity in MB
------------------------------------ ----------------  ------------ ---------- --------------
eui.00e6d65b65a8f34824a9374e00011745 258               71493        512        102400
eui.00e6d65b65a8f34024a9374e00011745 259               71493        512        102400

Step 1

In the Navigation pane, click Servers.

Step 2

Expand Servers > Service Profiles.

Step 3

Expand the node for the profile to delete.

Step 4

Click on vNICs and select the desired interface. Right click and select Delete from the dropdown.

Step 5

Click Save Changes.