Configuring Single Root I/O Virtualization

Configuring BIOS and Cisco IMC Parameters

Enabling SR-IOV BIOS Parameters

Procedure

Step 1

In the Navigation pane, click the Compute menu.

Step 2

In the BIOS tab, click the Configure BIOS tab.

Step 3

Select the I/O tab.

Step 4

Ensure that the Intel VT for directed IO option is enabled.

Note

 

Intel VT for directed IO option is enabled, by default, on Cisco UCS 220 and C240 M8 servers.

Step 5

Select the Processor tab.

Step 6

Ensure that the Intel Virtualization Technology option is enabled.

Step 7

Click Save and reboot the server.

Note

 

For the server with AMD CPU, SR-IOV Support option needs to be enabled at I/O tab and SVM Mode needs to be enabled at Processor tab.

Enabling SR-IOV VFs using Cisco IMC GUI

Before you begin

Ensure that the required BIOS options are enabled before performing this procedure.

Procedure

Step 1

In the Navigation pane, click the Networking menu.

Step 2

In the Networking pane, select the Adapter Card.

Step 3

In the Adapter Card pane, click the vNICs tab.

Step 4

In the vNICs pane, click eth0 or eth1.

Step 5

In the SR-IOV Properties area, edit the information in the following fields:

Note

 

SR-IOV Properties is available under Features tab for Cisco UCS C-Series M7 and later servers.

Name

Description

No. of VFs field

Enter an integer between 1 and 64.

Note

 

Other SR-IOV properties are enabled only when you enter an integer between 1 and 64
Receive Queue Count Per VF field

The number of receive queue resources to allocate.

Enter an integer between 1 and 8.

Transmit Queue Count Per VF field

The number of transmit queue resources to allocate.

Enter an integer between 1 and 8.

Completion Queue Count Per VF field

The number of completion queue resources to allocate. In general, the number of completion queue resources you should allocate is equal to the number of transmit queue resources plus the number of receive queue resources.

Enter an integer between 1 and 16.

Interrupt Count field

The number of interrupt resources to allocate. In general, this value should be equal to the number of completion queue resources.

Enter an integer between 1 and 16.

Step 6

Save and reboot the server.

Disabling SR-IOV VFs using Cisco IMC GUI

Procedure

Step 1

In the Navigation pane, click the Networking menu.

Step 2

In the Networking pane, select the Adapter Card.

Step 3

In the adapter Card pane, click the vNICs tab.

Step 4

In the vNICs pane, click eth0 or eth1.

Step 5

In the SR-IOV Properties area, set the No. of VFs to 0.

SR-IOV Properties is available under Feature for Cisco UCS C-Series M7 and later servers.

Step 6

Save and reboot the server.

Enabling SR-IOV VFs using Cisco IMC CLI

SUMMARY STEPS

  1. server# scope chassis
  2. server /chassis # scope adapter index
  3. server /chassis/adapter# scope host-eth-if {eth0 | eth1 | name}
  4. server /chassis/adapter/host-eth-if# set vf-count Count
  5. server /chassis/adapter/host-eth-if# set vf-wq-count Count
  6. server /chassis/adapter/host-eth-if# set vf-rq-count Count
  7. server /chassis/adapter/host-eth-if# set vf-cq-count Count
  8. server /chassis/adapter/host-eth-if# vf-intr-count Count
  9. server /chassis/adapter/host-eth-if# commit

DETAILED STEPS

  Command or Action Purpose

Step 1

server# scope chassis

Enters the chassis command mode.

Step 2

server /chassis # scope adapter index

Enters the command mode for the adapter card at the PCI slot number specified by index.

Note

 

The server must be powered on before you can view or change adapter settings.

Step 3

server /chassis/adapter# scope host-eth-if {eth0 | eth1 | name}

Enters the host Ethernet interface command mode for the specified vNIC.

Step 4

server /chassis/adapter/host-eth-if# set vf-count Count

Specify number of VFs for each PF.

Enter an integer between 1 and 64; the default is zero.

Step 5

server /chassis/adapter/host-eth-if# set vf-wq-count Count

Specify number of Transmit Queues for each VF.

Enter an integer between 1 and 8.

Step 6

server /chassis/adapter/host-eth-if# set vf-rq-count Count

Specify number of Receive queues for each VF.

Enter an integer between 1 and 8.

Step 7

server /chassis/adapter/host-eth-if# set vf-cq-count Count

Specify number of Completion Queues for each VF.

Enter an integer between 1 and 16; the default is zero.

The value is sum of wq and rq.

Step 8

server /chassis/adapter/host-eth-if# vf-intr-count Count

Specify number of interrupts for each VF.

Enter an integer between 1 and 16

Step 9

server /chassis/adapter/host-eth-if# commit

Commits the transaction to the system configuration.

Note

 

Committed host-eth-if settings takes effect upon the next host power cycle.

Example

server# scope chassis
server /chassis # scope adapter 1
server /chassis/adapter # scope host-eth-if eth0
server /chassis/adapter/host-eth-if # set vf-count 10
server /chassis/adapter/host-eth-if *# set vf-wq-count 1
server /chassis/adapter/host-eth-if *# set vf-rq-count 4
server /chassis/adapter/host-eth-if *# set vf-cq-count 5
server /chassis/adapter/host-eth-if *# set vf-intr-count 8
server /chassis/adapter/host-eth-if *# commit

Disabling SR-IOV VFs using Cisco IMC CLI

SUMMARY STEPS

  1. server# scope chassis
  2. server /chassis # scope adapter index
  3. server /chassis/adapter# scope host-eth-if {eth0 | eth1 | name}
  4. server /chassis/adapter/host-eth-if# set vf-count 0
  5. server /chassis/adapter/host-eth-if# commit

DETAILED STEPS

  Command or Action Purpose

Step 1

server# scope chassis

Enters the chassis command mode.

Step 2

server /chassis # scope adapter index

Enters the command mode for the adapter card at the PCI slot number specified by index.

Note

 

The server must be powered on before you can view or change adapter settings.

Step 3

server /chassis/adapter# scope host-eth-if {eth0 | eth1 | name}

Enters the host Ethernet interface command mode for the specified vNIC.

Step 4

server /chassis/adapter/host-eth-if# set vf-count 0

Configuring vf-count to zero would disable SR-IOV and ignore vf-intr-count, vf-cq-count, vf-rq-count and vf-wq-count values.

Step 5

server /chassis/adapter/host-eth-if# commit

Commits the transaction to the system configuration.

Note

 

Committed host-eth-if settings takes effect upon the next host power cycle.

Example

server# scope chassis
server /chassis # scope adapter 1
server /chassis/adapter # scope host-eth-if eth0
server /chassis/adapter/host-eth-if # set vf-count 0
server /chassis/adapter/host-eth-if *# commit

Configuring SR-IOV VFs on the ESXi Host Server

Installing Cisco eNIC Driver

Before you begin

Ensure that the required BIOS parameters and SR-IOV VFs configurations are completed.

Procedure

Step 1

Install the enic driver on the host.

The following example shows the installation of eNIC driver on ESXi:
[root@localhost:/vmfs/volumes/645c8bdd-c655e553-8ba0-e8d32272f6c0] esxcli software vib
install -v /vmfs/volumes/C240M7-Standalone/nenic-2.0.10.0-1OEM.800.1.0.20143090.x86_64.vib
--no-sig-check
Installation Result
Message: The update completed successfully, but the system needs to be rebooted for the
changes to be effective.
VIBs Installed: CIS_bootbank_nenic_2.0.10.0-1OEM.800.1.0.20143090
VIBs Removed: CIS_bootbank_nenic_2.0.11.0-1OEM.800.1.0.20143090
VIBs Skipped:
Reboot Required: true
DPU Results:
[root@localhost:/vmfs/volumes/645c8bdd-c655e553-8ba0-e8d32272f6c0]

Step 2

Reboot the server to load the enic driver into the running kernel.

Step 3

After reboot, execute the command esxcli software vib list | grep nenic to check the driver version.

Example

Following example shows installation for ESXi 7.0:

[root@host:/vmfs/volumes/64763c96-0098e171-f241-ecf40c0dc446] esxcli software vib install -v 
/vmfs/volumes/c240M6-F1/nenic-2.0.9.0-1OEM.700.1.0.15843807.x86_64.vib --no-sig-check
Installation Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: CIS_bootbank_nenic_2.0.9.0-1OEM.700.1.0.15843807
   VIBs Removed: CIS_bootbank_nenic_2.0.10.0-1OEM.700.1.0.15843807
   VIBs Skipped:

Verifying the Total Number of SR-IOV VFs per Port on the Host

Procedure

Step 1

Login to your VMware ESXi Host Client.

Alternatively, you can also access your host from vSphere vCenter Client and browse to Configure > Networking > Physical adapters.

Step 2

Go to Host > Manage and select the Hardware tab.

Step 3

Select PCI Devices from the list.

Step 4

From the drop-down list, select SR-IOV Capable.

The list shows all the SR-IOV capable devices.

Creating SR-IOV VFs on the Host

Procedure

Step 1

Login to your VMware ESXi Host Client.

Alternatively, you can also access your host from vSphere vCenter Client and browse to Configure > Networking > Physical adapters.

Step 2

Go to Host > Manage and select the Hardware tab.

Step 3

Select PCI Devices from the list.

Step 4

From the drop-down list, select SR-IOV Capable.

The list shows all the SR-IOV capable devices.

Step 5

Select the vNIC for which you wish to create the VFs

Step 6

Click Configure SR-IOV.

Configure SR-IOV for Cisco VIC Ethernet NIC window is displayed.

Step 7

Perform the following:

Field

Description

Enabled radio button

Select Yes to enable the configuration.

Virtual functions field

Enter an integer between 1 and 64.

Step 8

Click Save and then reboot the host server.

Configuring the Switch

This procedure is require to add a VF to VM.

Before you begin

Ensure that the SR-IOV VFs are configured.

Procedure

Step 1

Login to your VMware ESXi Host Client.

Step 2

Go to Host > Networking and select the Virtual switches tab.

Step 3

Click Add standard virtual switch.

The Add standard virtual switch window is displayed

Step 4

Complete the following:

Field

Description

vSwitch Name field

Enter a suitable name for the virtual switch.

MTU field

Default value: 1500

Maximum value: 9000

Uplink 1 drop-down list

From the drop-down list, select the PCIe devices for which you created the SR-IOVs.

Link Discovery drop-down list

From the drop-down list, select the Mode and the Protocol.

Note

 

These fields remain as default.

Security

Choose from the following options:

  • Promiscuous mode—Accept, Reject, or Inherit from vSwitch.

  • MAC address changes—Accept, Reject, or Inherit from vSwitch.

  • Forged trasmits—Accept, Reject, or Inherit from vSwitch.

NIC teaming

Choose from the following:

  • Load balancing—From the drop-down list choose the Load balancing. Values are: Inherit from vSwitch,

  • Network failover detection—From the drop-down list choose the network failover detection. Values are: Inherit from vSwitch,

  • Notify switches—Choose the notify switches. Values are Yes, No, Inherit from vSwitch.

  • Fallback—Choose the fallback. Values are Yes, No, Inherit from vSwitch.

  • Override failover order—From the drop-down list choose the override failover order. Values are Yes or No,

  • Failover order—Choose the failover order.

Traffic Shaping

Perform the following:

  • Status—Choose the status. Values are Enabled, Disabled, Inherit from vSwitch.

  • Average bandwidth—Enter the average bandwidth.

  • Peek bandwidth—Enter the peek bandwidth.

  • Burst size—Enter the burst size.

Note

 

Traffic shaping policy is applied to the traffic of each virtual network adapter attached to the virtual switch.

What to do next

Create a Virtual Port.

Creating a Virtual Port

Before you begin

Ensure that the SR-IOV VFs are configured.

Procedure

Step 1

Login to your VMware ESXi Host Client.

Step 2

Go to Host > Networking and select the Port Groups tab.

Step 3

Click Add port group.

Add port group-New port group window is displayed.

Step 4

Complete the following:

Field

Description

Name field

Enter a suitable name for the virtual port.

VLAN ID field

Enter the VLAN ID.

Virtual Switch drop-down list

From the drop-down list, select the virtual switch.

Security

Choose from the following options:

  • Promiscuous mode—Accept, Reject, or Inherit from vSwitch.

  • MAC address changes—Accept, Reject, or Inherit from vSwitch.

    Forged trasmits—Accept, Reject, or Inherit from vSwitch.

Creating a New Virtual Machine (VM)

Before you begin

  • Host with Desktop Environment

  • sudo user with admin rights

  • Virtualization packages are installed

  • OS ISO image is copied to the host server datastore

Procedure

Refer ESXi documentation to install VMs.

Adding SR-IOV VF on the VM

Procedure

Step 1

Go to vSphere Client and right-click the host server.

Step 2

Browse to VM Policies > Edit Settings.

Edit Settings window is displayed.

Step 3

Under the Virtual Hardware tab and click ADD NEW DEVICE.

Step 4

From the drop-down list, select Network Adapter.

Step 5

New Network is added in the Virtual Hardware tab.

Step 6

Expand this New Network option and complete the following:

Field

Description

Status checkbox

Check Connect At Power On.

Adapter Type drop-down list

Select PCI Device passthrough from the list.

Device field

Displays the selected device.

CHANGE DEVICE button

Click CHANGE DEVICE to open the Device Selection window.

Select the desired SR-IOV device from the list and click SELECT.

Access Type field

Displays the type of path selected.

Step 7

Save and reboot the server.

Installing OS on Guest VM on ESXi

Procedure

Step 1

Right-click the host node and navigate to vCenter > New Virtual machine.

Step 2

Select Creation Type > Create New Virtual Machine and click Next.

Step 3

Enter a name and folder and click Next.

Step 4

Select a compute resource, choose a node and click Next.

Step 5

Select Storage and check the datastore radio-button, and click Next.

Step 6

Select the compatability ESXi 8.0 or later and click Next.

Step 7

Select a guest OS as RHEL Linux9 (64-bit), and click Next.

Step 8

Customize the hardware set CPU to 2, and Memory values to 4 GB.

Step 9

Expand the Memory tab and check Reserve all guest memory (All locket) checkbox.

Step 10

Select New CD/DVD Drive (Datastore ISO file), and check Connect At Power On checkbox.

Step 11

Under CD/DVD Media, browse and select the Linux ISO image and click Next.

Step 12

Click Finish.

Step 13

Install the Linux drivers.

Following example shows the Linux driver installation:

[root@host drivers]# rpm -ivh kmod-enic-4.4.0.1-930.10.rhel8u7.x86_64.rpm
Verifying...###############################################################[100%]
Preparing...###############################################################[100%]
Updating / installing...
     1:kmod-enic-4.4.0.1-930.10.rhel8u7####################################[100%]

Configuring SR-IOV VFs on the Linux Host Server

Installing Cisco eNIC Driver

Before you begin

Ensure that the SR-IOV BIOS parameters and SR-IOV VFs configurations are completed.

Procedure

Step 1

Install the enic driver on the host.

Following example shows the installation of eNIC driver on RHEL:
[root@rack-134 drivers] #
[root@rack-134 drivers]# rpm -ivh kmod-enic-4.6.0.0-977.3.rhel9u2_5.14.0_284.11.1.x86_64.rpm 
Verifying...###############################################################[100%]
Preparing...###############################################################[100%]
Updating / installing...
    1:kmod-enic-4.6.0.0-977.3.rhel9u2_5####################################[100%]
[root@rack-134 drivers] #
[root@rack-134 drivers] #

Step 2

Reboot the server to load enic driver into the running kernel.

Step 3

Execute modinfo enic to check enic driver is loaded.

Following example shows the output of modinfo enic:

root@rack-134 ~]# modinfo enic
filename:     /lib/modules/5.14.0-284.11.1.e19_2.x86_64/extra/enic/enic.ko
version:      4.6.0.0-977.3
retpoline:    Y
license:      GPL v2
author:       Scott Feldman <scofeldm@cisco.com>
description:  Cisco VIC Ethernet NIC Driver
rhelversion:  9.2
srcversion:   72313628E52D2242A9BA7BE
alias:        pci:v00001137d000002B7sv*sd*bc*sc*i*
alias:        pci:v00001137d00000071sv*sd*bc*sc*i*
alias:        pci:v00001137d00000044sv*sd*bc*sc*i*
alias:        pci:v00001137d00000043sv*sd*bc*sc*i*
depends:
retpoline:    Y
name:         enic
vermagic:     5.14.0-284.11.1.e19_2.x86_64 SMP preempt mod_unload modversions
sig_id:       PKCS#7
signer:       Cisco UCS Driver Signing REL Cert
sig_key:      DO:54:9A:88:88:DD: 0E:7A
sig_hashalgo: sha256
signature:    A9:85:85:8C:C3:A6:13:C3:64:70:52:72:39:4B: 87:77: B1:72:A1:65:
              E9:DD:1D: BB: 3A: C4:8E:9E:5D:39:D8:2A:F7: 8D: A0:7B:30:F4:05:5A: 
              5D: FB:8B:16:94:17:2E:F0:9D: C3:B9:8F:73: CA:12:2F: B6:6F:25:04: 
              7A: 6C: 9A: 62:63: EF: 42: D7:3D: BF:8E:F8:A2:1B:B9: FF: A0:A1:20:10: 
              FO:AA:CC:87:D2: CD: 82:9C:59:36:0D:C6:10:88: AE:2C: BB:4E:27:1B: 
              3E:A6:03:2A:37: D2:26:8C:41:1A:48:40:7A: F9:C8:A7: DF:37:8D:48: 
              57:E9:EA:16:EA:15:97: B6:0D: CB:27:15:13:E8:A7:F3: FC:23:98:A8: 
              FO:BD:DB:ED:71:ED: 9F:DB:11:0F:3E:59:0B:69:15:AA: D1:C6:20:D1: 
              8D:C9:6C:47:92: FA:68:08:15:0D:D8:5A: 84:15:D6:71:23:32:ED: 8D: 
              OB:28:31:6F:2F: 8A:EC: B1:44:2D: DE:2F:26: DA: 77:0A:B6:37:90:A1:
              94:8D:5F:53:E8:53:73:13:97:5D: 91:73:02:2B:27:81:3D:C6:9F:4E: 
              1F:40:99:13: FA:5D:C6:AB:52: B5:6E:6B: D6:B6:C3:11:32:E8:29:74: 
              E6:98: E1:7C:C5:66: F7:51:17: DB: A1: B9:5E:64:17:A0
parm:         rxcopybreak: Maximum size of packet that is copied to a new buffer 
on receive (uint)
[root@rack-134~]#

Verifying the Total number of SR-IOV VFs per Port on the Host

Before you begin

Ensure that Cisco eNIC driver is installed.

Procedure

Log into the host server and run the following command and replace interface_name with actual interface name on the host. 

# cat /sys/class/net/interface_name/device/sriov_totalvfs

Example

Following example shows the total number for SR-IOV VFs created from Cisco IMC on p1p1 interface:

[root@rack-134 ~]# cat /sys/class/net/p1p1/device/sriov_totalvfs 
32
[root@rack-134 ~]#

Creating SR-IOV VFs on the Host

Enabling SR-IOV VFs from Cisco IMC does not create SR-IOV VFs on the host by default. To create SR-IOV VFs on the host, use the following procedure:

Procedure

Step 1

Execute the following command to create SR-IOV VFs on the host:

echo number_of_sriov_devices > /sys/class/net/sriov interface_name/device/sriov_numvfs

Example:

Following example shows the creation of 6 SR-IOV VFs on p1p1 interface.
[root@rack-134 ~]# echo 6 > /sys/class/net/p1p1/device/sriov_numvfs 
[root@rack-134 ~]#

Step 2

Execute the following command to verify the SR-IOV VFs created:

# cat /sys/class/net/interface_name/device/sriov_numvfs

Example:

Following example shows the verification of SR-IOV VFs on p1p1 interface.
[root@rack-134 ~]# cat /sys/class/net/p1p1/device/sriov_numvfs
6
[root@rack-134 ~]#

Step 3

(Optional) Alternatively, IP link command shows created SR-IOV VFs.

# ip link show interface_name

Example:

Following example shows created 6 SR-IOV VFs on p1p1 interface.
[root@rack-134 ~]# ip link show pipl
2: plpl:  <BROADCAST, MULTICAST, UP, LOWER_UP>mtu 9000 qdisc mq state UP mode 
DEFAULT group default qlen 10 00
link/ether 98: a2:c0:66:32:80 brd ff:ff:ff:ff:ff:ff 
vf 0 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, 
link-state auto, trust off, query_rss off
vf 1 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, 
link-state auto, trust off, query_rss off
vf 2 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, 
link-state auto, trust off, query_rss off
vf 3 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, 
link-state auto, trust off, query_rss off
vf 4 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, 
link-state auto, trust off, query_rss off
vf 5 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff, spoof checking off, 
link-state auto, trust off, query_rss off
altname enp9s0
altname eno5
[root@rack-134 ~]#

Note

 

After the host server reboots, the created SR-IOV VFs get removed from the host.

By adding the command from Step 1 to rc.local file, the same number of SR-IOV VFs can be created each time the host server boots up.

What to do next

You can create a new virtual machine.

Creating a New Virtual Machine (VM)

Before you begin

  • Host with Desktop Environment

  • sudo user with admin rights

  • Virtualization packages are installed

  • OS ISO image is copied to the host server

Procedure

Step 1

Verify the virtualization is enabled on the host server by using this command:

# lscpu | grep Virtualization

Example:

This example shows the Intel's virtualization technology VT-x is enabled.
[user@rack-111 ~]$ lscpu | grep Virtualization
Virtualization: VT-x
[user@rack-111 ~]$

Step 2

Verify the KVM modules are loaded by using this command:

# lsmod | grep kvm

Example:

This example shows KVM modules are loaded in the host server.
[user@rack-111 ~]$ lsmod | grep kvm
kvm_intel    409600    8
kvm         1134592    1 kvm_intel 
irqbypass      6384    290 vfio_pci_core, kvm
[user@rack-111 ~]$

Step 3

Type virt-manager command at the terminal to launch Virtual Machine Manager GUI.

Step 4

At the Virtual Machine Manager, click File > New Virtual Machine to create a new virtual machine.

Step 5

At New VM window, select Local install media (ISO image or CDROM) option and click Forward.

Step 6

At Choose ISO or CDROM install media, click Browse.

Step 7

At Locate ISO media volume window, click Browser Local.

Step 8

Go to the folder that has ISO image. Select ISO image and click Open.

Step 9

Click Forward.

Step 10

Select the desire Memory and CPU settings for the VM and click Forward.

Step 11

Choose the VM's disk image size and click Forward.

Step 12

Enter a name for the VM in the Name field and click Finish.

You may monitor the OS installation progress.

Adding SR-IOV VF on the VM

Before you begin

VM is powered off.

Procedure

Step 1

At the Virtual Machine Manager, right click on the VM and select Open.

Step 2

Click Show virtual hardware detail icon next to the Monitor icon.

Step 3

Click Add Hardware.

Step 4

At Add New Virtual Hardware window, select PCI Host Device.

Under PCI Device Details tab, assign a created SR-IOV VF to the VM.

Step 5

Click Finish.

Step 6

Power on the VM.

What to do next

You can now log into the VM, install Cisco eNIC driver 4.6.0.0-977.3 or later, reboot the VM, and then use the ip link command to verify the added SR-IOV VF.