vWAAS with SR-IOV
This section has the following topics:
Single-Root I/O Virtualization (SR-IOV) is a standard developed by the Peripheral Component Interconnect Special Interest Group (PCI SIG) to improve virtualization of PCI devices.
SR-IOV enables the VMs to share the I/O device in a virtualized environment. SR-IOV achieves this by bypassing the hypervisor’s involvement in data movement:
- SR-IOV provides independent memory space, interrupts, and DMA streams for each virtual machine.
- The SR-IOV architecture allows a device to support multiple virtual functions, and therefore minimizes the hardware cost of each additional function.
- SR-IOV-enabled Ethernet controllers support direct assignment of part of the port resources to guest operating systems that use the SR-IOV standard. This capability enhances the performance of the guest VMs.
Table 2-6 shows the two types of functions used with SR-IOV.
Table 2-6 SR-IOV Physical Functions and Virtual Functions
- A full PCI Express (PCIe) function that includes the SR-IOV extended capability, which is used to configure and manage the SR-IOV functionality.
- Physical Functions are discovered, managed, and configured as normal PCIe devices. Physical Functions configure and manage the SR-IOV functionality by assigning Virtual Functions.
- A lightweight PCIe function that contains all the resources necessary for data movement, but has a carefully minimized set of configuration resources.
- Each Virtual Function is derived from a Physical Function. The number of Virtual Functions an Ethernet controller can have is limited by the device hardware.
Interoperability and Platforms Supported for vWAAS with SR-IOV
This section contains the following topics:
WAAS Central Manager and vWAAS with SR-IOV
Devices with SR-IOV are registered to the Central Manager in the same manner as other vWAAS devices, and you can use the cms deregister EXEC command to deregister these devices as you would for other vWAAS devices.
The following list shows how vWAAS devices with SR-IOV are displayed on the Central Manager:
- vWAAS with SR-IOV on KVM (RHEL, CentOS or NFVIS) is displayed as OE-VWAAS-KVM.
- vWAAS with SR-IOV on ESXi is displayed as OE-VWAAS-ESX.
Platforms Supported for vWAAS with SR-IOV
Table 2-7 shows the WAAS version and platforms supported for vWAAS with SR-IOV.
Note Although Intel X710 is capable of 10 Gbps speed, vWAAS with SR-IOV using Intel X710 on NFVIS is supported for 1 Gbps speed, as part of vBranch solution.
Note The supported firmware version for Intel X710 NIC is 5.05
Table 2-7 WAAS Version and Platforms Supported for vWAAS with SR-IOV
vWAAS-150, 200, 750, 1300, 2500, 6000
vWAAS-150, 200, 750, 1300, 2500, 6000
vWAAS -12000, 50000, 150000
Upgrade/Downgrade Considerations for vWAAS with SR-IOV
Consider the following when you upgrade or downgrade a vWAAS instance with SR-IOV:
– The upgrade procedure for vWAAS instances with SRIOV is the same as for any other vWAAS devices.
– Before a downgrade from Version 6.4.1x or 6.4.3 to an earlier version, from the host, remove SR-IOV interfaces from the devices that will not support this functionality when operating in an earlier WAAS version. Downgrade of vWAAS instances with SR-IOV is blocked for unsupported WAAS versions. Table 2-7 displays minimum WAAS versions supported for SR-IOV.
– At the device level, if you downgrade a vWAAS instance with SR-IOV to a version earlier than 6.4.1x or 6.4.3 (depending on your WAAS configuration), a warning message is displayed at the start of the downgrade process. This warning message is displayed if the device supports SR-IOV functionality, even if the device does not use the SR-IOV interface, because downgrade of vWAAS instances with SR-IOV is blocked for unsupported WAAS versions.
– At the device group level, if you downgrade a device group that contains at least one device that supports SR-IOV functionality, a warning message is displayed at the start of the downgrade process, because downgrade of vWAAS instances with SR-IOV is blocked for unsupported WAAS versions.
For more information on the upgrade or downgrade process, see the Release Note for Cisco Wide Area Application Services.
Deploying vWAAS with SR-IOV
This section contains the following topics:
Deploying vWAAS with SR-IOV on KVM
This section contains the following topics:
Configuring Host Settings for vWAAS on KVM (CentOS or RHEL) with SR-IOV for UCS C-Series
One-time host settings are required to use the SR-IOV functionality on KVM Hypervisor for UCS C-Series.
To configure the required host settings for deploying vWAAS on KVM with SR-IOV, follow these steps:
Step 1 Enable Intel Virtualization Technology for Directed I/O (VT-d) in the host BIOS.
Use the command cat /proc/cpuinfo | grep -E ‘vmx|svm’ | wc -l to verify that you have enabled VT-d.
The command value should be greater than 0.
Step 2 Enable I/O MMU:
a. In the file /etc/default/grub, add intel_iommu=on to GRUB_CMDLINE_LINUX.
b. After you make changes to GRUB_CMDLINE_LINUX, the following will be displayed:
GRUB_CMDLINE_LINUX="crashkernel=auto rd.lvm.lv=centos/root rd.lvm.lv=centos/swap rhgb quiet intel_iommu=on"
c. For the changes to take effect, compile: grub2-mkconfig -o /boot/grub2/grub.cfg.
d. Reboot the host.
Step 3 Enable SR-IOV Virtual Functions (for more information on Virtual Functions, see About SR-IOV).
Enable SR-IOV VFs:
a. Verify the maximum number of Virtual Functions allowed for the specified interface.
For example, if the SR-IOV-supported interface is enpls0f0:
- Verify the value of /sys/class/net/enp1s0f0/device/sriov_totalvfs.
b. Set the desired number of Virtual Functions at /sys/class/net/enp1s0f0/device/sriov_numvfs.
- On enpls0f0:
echo 7 > /sys/class/net/enp1s0f0/device/sriov_numvfs
Step 4 Remove SR-IOV configuration:
If you need to remove SR-IOV configuration for a specific interface, for example, enp1s0f0, use the command echo 0 at /sys/class/net/enp1s0f0/device/sriov_numvfs, and also remove the lines with enp1s0f0 interface name present in /etc/rc.d/rc.local.
Deploying vWAAS with SR-IOV on KVM (CentOS or RHEL) Using Deployment Script for UCS C-Series
vWAAS on KVM for SR-IOV is deployed using launch.sh script file on UCS C-Series.
To deploy vWAAS on KVM with SR-IOV functionality using the deployment script, follow these steps (from the launch.sh script file output):
Step 1 To check the pre-requisite host configuration, run the following command:
Step 2 To launch VM with BRIDGE or MACVTAP interfaces, run the following command:
./launch.sh <VM_NAME> <INTF_TYPE> <INTF1_NAME> <INTF2_NAME>
- where INTF_TYPE can be either BRIDGE or MACVTAP.
- where INTF1_NAME and INTF2_NAME are the desired names based on the selected INTF_TYPE.
Step 3 To launch vWAAS(not vCM) with SRIOV interface(s), run the following command:
./launch.sh <VM_NAME> <INTF_TYPE> <INTF1_NAME> <INTF_TYPE> <INTF2_NAME>
- where first INTF_TYPE option can be BRIDGE or MACVTAP or SRIOV.
- where second INTF_TYPE option should be SRIOV.
- INTF1_NAME and INTF2_NAME are the desired names based on the selected INTF_TYPE.
Deploying vWAAS with SR-IOV on KVM Using NFVIS Portal for ENCS-W Series
To deploy vWAAS on KVM with SR-IOV using the NFVIS portal for the ENCS-W Series, follow these steps:
Step 1 At the Cisco Enterprise NFV Solution, navigate to the VM Deployment tab.
Step 2 The VM Deployment screen displays a navigation row, shown in Figure 2-1, to highlight where you are in the VM deployment process.
Figure 2-1 VM Deployment Process Navigation Row
5 Review & Deploy
Before you enter information to begin the VM deployment process, the VM Deployment navigation row shows 1 Images highlighted.
Note You must specify all parameters for the VM during VM deployment. After the VM is deployed, you cannot make changes to the VM. If you need to change any parameter for a deployed VM, you must delete that VM and deploy a new VM.
Step 3 To register the VM image, at the VN Name field, enter the name of the VM.
Step 4 From the List of Images on the Device table listing, select an image for the VM that will be deployed, or click Upload to upload an image.
Step 5 Click Next.
Step 6 The VM Deployment navigation row shows 2 Profiles highlighted.
Step 7 The Profiles screen is displayed, showing the Select Profiles table listing, which has columns for profile name, CPUs, memory (in MB), and disk size (in MB).
Step 8 From the Select Profiles table listing, click the radio button next to the profile you want to use, or click “+” to add a new profile.
a. If you click “+” to create a new profile, a new, empty row is displayed for you to enter information.
b. Click Save to create the new profile.
Step 9 Click Next.
Step 10 The VM Deployment navigation row shows 3 Networks highlighted.
Step 11 The Select Network Interface screen is displayed, showing the Select Network Interface table listing, which has columns for VNIC number and network name.
Step 12 From the Select Network Interface table listing, check the check box next to one or more NVIC numbers that you want to attached to the VM you selected/created in Steps 1-5, or click “+” to add a new VNIC for the specified VM.
a. If you click “+” to create a new VNIC, a new empty row is displayed for you to enter information.
b. Click Save to create the new VNIC.
Step 13 The VM Deployment navigation row still shows 3 Networks highlighted.
The Networks and Bridges table listing is displayed, which you use to add or delete networks and associated bridges.
Consider the following as you use the Networks and Bridges table listing:
- The table listing displays columns for network name, VLAN (if applicable), bridge, and port (if applicable).
- The table listing shows the available networks and bridges on the NFVIS server. Initially, the table listing shows the default networks: lan-net and wan-net and associated bridges.
- The top right corner of the table toolbar shows the selected row and the total number of rows, for example, “Selected 2 / Total 4”.
- To associate multiple VLANs with a network, you must separate the VLAN numbers with a comma and no space, for example, “100,200”.
- To associate multiple ports with a network, you must separate the port numbers with a comma and no space, for example, “1,2”.
- A network and bridge operate as one entity. To delete a network and bridge, click the radio button for that network and bridge row. Click Delete. The page automatically refreshes (there is no confirmation question). You can delete one network and bridge at a time.
Step 14 Click Next.
Step 15 The VM Deployment navigation row shows 4 Configuration highlighted.
The Port Forwarding (Optional) screen is displayed.
Step 16 At the Port Number field, enter the number of the port for port forwarding.
Step 17 At the External Port Number field, enter the number of the external port. The external port is accessible from the WAN bridge only.
Step 18 Click Next.
Step 19 The VM Deployment navigation row shows 5 Review & Deploy highlighted.
The following message is displayed: Starting VM deployment. Redirecting to Status Page.
Step 20 Click OK.
Step 21 The page refreshes and the Status Page is displayed, showing the VM Status table listing, with columns for VM name, profile name, status, and VNC console.
As the VM is being deployed, the status shows VM in Transient State. After deployment is complete, the status shows VM is running.
Step 22 After deployment is complete, use the Management tab to manage the VM with tasks including power off, power on, reboot, and delete.
Deploying vWAAS with SR-IOV on ESXi
This section contains the following topics:
Configuring Host Settings for vWAAS with SR-IOV on ESXi for UCS C-Series
Before you begin, note the ESXi host requirements, as shown in Table 2-8:
Table 2-8 ESXi Host Requirements for vWAAS with SR-IOV for UCS C-Series
Intel X710 NIC Specification
Tested Driver Version
Tested Firmware Version
Note Without compatible drivers, the Intel X710 will not be detected.
To create a VF in ESXi, follow these steps:
Step 1 Enable and login to the ESXi shell.
Step 2 Execute the lspci | grep -i intel | grep -i 'ethernet\|network' command. Note the port order of this command.
Step 3 Use the following command to create VFs:
# esxcli system module parameters set -m i40e -p max_vfs=Y,Z
- Y,Z represents the number of VF's to be created respectively for each port.
max_vfs=5,0 represents 5 VFs on adapter 1 port 1
max_vfs=0,5 represents 5 VFs on adapter 1 port 2.
[root@localhost:~] lspci | grep -i intel | grep -i ‘ethernet\|network’
000:01:00.0 Network controller: Intel Coporation I350 Gigabit Network Connection vmnic2]
Step 4 To verify the value of the VFs to be created, use the esxcli system module parameters list -m i40e command:
Step 5 To create the VFs, reboot the host.
Step 6 After the reboot is complete, you can verify the VFs by using:
- the lspci command or
- the vSphere client DirectPath I/O Configuration screen (Figure 2-2)
Navigate to Host > Configuration > Hardware > Advanced Settings.
Figure 2-2 vSphere Client DirectPath I/O Configuration Screen
Configuring SR-IOV Interfaces for vWAAS on ESXi for UCS-C Series
To configure SR-IOV interfaces for vWAAS on ESXi for UCS-C Series, follow these steps:
Step 1 After deploying vWAAS, power down the vWAAS.
Step 2 Right-click and choose Edit Settings.
Step 3 Navigate to Virtual Machine Properties > Resources tab.
Step 4 At the listing, choose Memory.
The Memory Resource Allocation screen is displayed (Figure 2-3).
Figure 2-3 vWAAS Memory Resource Allocation Screen
Step 5 Click Reserve all guest memory.
Step 6 Click OK.
Step 7 Navigate to Virtual Machine Properties > Hardware tab.
Step 8 Click Add.
The Device Type screen is displayed (Figure 2-4).
Figure 2-4 vWAAS Add Hardware > Device Type Screen
Step 9 For device type, select PCI Device.
Step 10 Click Next.
The Choose PCI Device screen is displayed (Figure 2-5).
Figure 2-5 vWAAS Add Hardware > Choose PCI Device Screen
Step 11 Choose the VF you want to connect to.
Step 12 Click Next.
Step 13 Click Finish.
Step 14 To begin using the VF, start the VM.
vWAAS Upgrade and Downgrade Considerations
This section has the following upgrade and downgrade topics for vWAAS and vCM models.
For full information on the upgrade or downgrade process for WAAS and vWAAS devices, see the Release Note for Cisco Wide Area Application Services.
vWAAS Upgrade and vWAAS Nodes
- When upgrading vWAAS, do not upgrade more than five vWAAS nodes at the same time on a single UCS box. Upgrading more than five vWAAS nodes at the same time may cause the vWAAS devices to go offline and to diskless mode.
- vWAAS for WAAS 6.4.1 requires additional resources before upgrading from WAAS 6.2.3d to WAAS 6.4.1.
– Upgrading from the WAAS Central Manager: If you initiate and complete the upgrade from the WAAS Central Manager without increasing resources for vWAAS, alarms (CPU & RAM) to indicate insufficient resource allocation will be displayed on the WAAS Central Manager after the upgrade process is completed. No alarms are displayed at the beginning of the upgrade process.
– Upgrading from the WAAS CLI: If you initiate an upgrade to WAAS 6.4.1 with the CLI, a warning on insufficient resources is displayed at the start of the upgrade process.
vWAAS Upgrade and SCSI Controller Type
If the virtual host was created using an OVA file of vWAAS for WAAS Version 5.0 or earlier, and you have upgraded vWAAS within WAAS, you must verify that the SCSI Controller Type is set to VMware Paravirtual. Otherwise, vWAAS will boot with no disk available and will fail to load the specified configuration.
If needed, change the SCSI controller type to VMware Paravirtual by following these steps:
Step 1 Power down the vWAAS.
Step 2 From the VMware vCenter, navigate to vSphere Client > Edit Settings > Hardware.
Step 3 Choose SCSI controller 0.
Step 4 From the Change Type drop-down list, verify that the SCSI Controller Type is set to VMware Paravirtual. If this is not the case, choose VMware Paravirtual.
Step 5 Click OK.
Step 6 Power up the vWAAS, with WAAS Version 6.2.3, or WAAS 6.1.x or later. WAAS Version 6.1.x is the minimum version used.
vWAAS Upgrade and vCM-100 with RHEL KVM or KVM on CentOS
If you upgrade to WAAS Version 6.2.3, or downgrade from WAAS Version 6.2.3 to an earlier version, and use a vCM-100 model with the following parameters, the vCM-100 may not come up due to GUID Partition Table (GPT) boot order errors.
- vCM-100 has default memory size of 2 GB
- vCM-100 uses the RHEL KVM or KVM on CentOS hypervisor
- You use the restore factory-default command or the restore factory-default preserve basic-config command
If you are upgrading a vCM-100 model from an earlier WAAS version to WAAS Version 6.2.3, the upgrade process on this type of configuration will automatically clear system and data partition.
If you upgrade the vCM device to WAAS Version 6.2.3 via the console, a warning message similar to the following will be displayed:
WARNING: Upgrade of vCM device to 6.2.0 (or) higher version with ‘/sw’ and ‘/swstore’ size less than 2GB will clear system and data partition.
If you upgrade the vCM device to WAAS Version 6.2.3 via the GUI, a warning message is not displayed.
The restore factory-default command erases user-specified information stored in the flash image, including the starting configuration of the device, and also removes data from the disk, user-defined partitions, and the entire Central Manager database.
To resolve this situation, follow these steps:
Step 1 Power down the vWAAS using the virsh destroy vmname command or the virt manager.
Step 2 Power up the vWAAS using the virsh start vmname command or the virt manager.
Note This upgrade/downgrade scenario does not occur for vCM-100 models whose memory size is upgraded to 4 GB.
Migrating a Physical Appliance Being Used as a WAAS CM to a vCM
To migrate a physical appliance being used as a primary WAAS Central Manager to a vCM, follow these steps:
Step 1 Introduce vCM as the Standby Central Manager by registering it to the Primary Central Manager.
Step 2 Configure both device and device-group settings through Primary CM and ensure that devices are getting updates. Wait for two to three data feed poll rate so that the Standby CM gets configuration sync from the Primary CM.
Step 3 Ensure that the Primary CM and Standby CM updates are working.
Step 4 Switch over CM roles so that vCM works as Primary CM. For additional details please refer to “ Converting a Standby Central Manager to a Primary Central Manager ” section of the WAAS Configuration Guide.
vWAAS Downgrade Considerations
Consider the following when you downgrade vWAAS to an earlier WAAS version:
- vWAAS models vCM-500N and vCM-1000N, introduced in WAAS v5.5.1, cannot be downgraded to a version less than v5.5.1.
- On the UCS E-Series Server Module running vWAAS, downgrading to a version earlier than 5.1.1 is not supported. On other vWAAS devices you cannot downgrade to a version earlier than 4.3.1.
Note If the vWAAS device is downgraded in the following scenarios:
—from vWAAS for WAAS Version 6.4.1a to WAAS Version 6.2.3x, or
—from vWAAS for WAAS Version 6.x to 5.x
the WAAS alarm filesystem_size_mismatch is displayed; it indicates that the partition was not created as expected. To clear the alarm, use the disk delete-data-partitions command to re-create the DRE partitions.