Install Cisco Catalyst 8000V in KVM Environment

Red Hat Enterprise Linux (RHEL) is based on Kernel-based Virtual Machine (KVM), an open source, full virtualization solution for Linux on x86 hardware that contains virtualization extensions.

This chapter provides information on how to install Cisco Catalyst 8000V in KVM environments.

You can install the Cisco Catalyst 8000V virtual router as a virtual machine on Red Hat KVM virtualization. First, manually create the VM. Then, install the software using either an .iso file or a qcow2 file. Install Cisco Catalyst 8000V in a KVM environment by using the:

  • GUI Tool: Download and install the virt-manager RPM package on the KVM server. Virt-manager is a desktop user interface for managing virtual machines. Installation by using the GUI is the recommended installation method.

  • Command Line Interface (CLI): Use the CLI to install the Cisco Catalyst 8000V VM.

Installation requirements

This section lists the supported host Linux operating system versions on which Cisco Catalyst 8000V has been validated. All other deployment combinations not mentioned in this section will fall under Cisco Catalyst 8000V non-certified support model. Ask your local Cisco Sales representative for more details.

Supported RHEL versions (Linux KVM based on Red Hat Enterprise Linux)

This table lists the supported RHEL versions for Cisco Catalyst 8000V.

Cisco IOS XE release

Red Hat Enterprise Linux version

Cisco IOS XE 17.18.x releases

Red Hat Enterprise Linux versions 9.2 and 8.10

Cisco IOS XE 17.16.x releases

Red Hat Enterprise Linux versions 9.2 and 8.10

Cisco IOS XE 17.15.x releases

Red Hat Enterprise Linux versions 9.2, 8.10, and 8.4

Cisco IOS XE 17.14.x releases

Red Hat Enterprise Linux versions 9.2, 8.4, and 7.7

Cisco IOS XE 17.13.x releases

Red Hat Enterprise Linux versions 9.2, 8.4, and 7.7

Cisco IOS XE 17.12.x releases

Red Hat Enterprise Linux versions 9.2, 8.4, and 7.7

Cisco IOS XE 17.11.x releases

Red Hat Enterprise Linux versions 8.4 and 7.7

Cisco IOS XE 17.10.x releases

Red Hat Enterprise Linux versions 8.4 and 7.7

Cisco IOS XE 17.9.x releases

Red Hat Enterprise Linux versions 8.4 and 7.7

Cisco IOS XE 17.8.x releases

Red Hat Enterprise Linux versions 8.4 and 7.7

Cisco IOS XE 17.7.x releases

Red Hat Enterprise Linux versions 8.4 and 7.7

Cisco IOS XE 17.6.x releases

Red Hat Enterprise Linux versions 7.7 and 7.5

Cisco IOS XE 17.5.x releases

Red Hat Enterprise Linux versions 7.7 and 7.5

Cisco IOS XE 17.4.x releases

Red Hat Enterprise Linux versions 7.7 and 7.5

From Cisco IOS XE 17.12.1 release, Cisco Catalyst 8000V is also supported on Intel Atom® C3000 processor (Denverton) CPU-based servers with Intel x550 NIC on RHEL 8.4 KVM hypervisor. You can operate Cisco Catalyst 8000V on other x86 CPUs with various hypervisor operating system versions. However, support is provided only for the versions listed in the Supported RHEL versions table.

Supported SUSE Linux® Enterprise Server (SLES) versions

This table lists the supported SLES versions for Cisco Catalyst 8000V.

Cisco IOS XE release

SUSE Linux® Enterprise Server version

Cisco IOS XE 17.18.x releases

SUSE Linux Enterprise Server version 15 SP5

Cisco IOS XE 17.16.x releases

SUSE Linux Enterprise Server version 15 SP5

Cisco IOS XE 17.15.x releases

SUSE Linux Enterprise Server version 15 SP5

Cisco IOS XE 17.14.x releases

SUSE Linux Enterprise Server version 15 SP3

Cisco IOS XE 17.13.x releases

SUSE Linux Enterprise Server version 15 SP3

Cisco IOS XE 17.12.x releases

SUSE Linux Enterprise Server version 15 SP3

Cisco IOS XE 17.11.x releases

SUSE Linux Enterprise Server version 15 SP3

Cisco IOS XE 17.10.x releases

SUSE Linux Enterprise Server version 15 SP3

Cisco IOS XE 17.9.x releases

SUSE Linux Enterprise Server version 15 SP3

Cisco IOS XE 17.6.3 releases

SUSE Linux Enterprise Server version 15 SP3

Supported Ubuntu versions

This table lists the supported Ubuntu versions for Cisco Catalyst 8000V.

Supported IOS XE release

Ubuntu version

Cisco IOS XE 17.18.1a and later

Ubuntu 22.04 LTS

Supported vNICs

This table lists the supported vNIC versions for Cisco Catalyst 8000V.

Supported IOS XE release

Supported VNIC

Cisco IOS XE Release 17.18.1a and later

ConnectX-6

Cisco IOS XE Release 17.10.1 and later

Ixgbe

Cisco IOS XE Release 17.9.1 and later

iavf

Cisco IOS XE Release 17.9.1 and later

ConnectX-5VF

Cisco IOS XE Release 17.4.1 to Cisco IOS XE 17.8.x releases

i40evf

Cisco IOS XE Release 17.4.1 and later

Virtio

Cisco IOS XE Release 17.4.1 and later

ixgbevf

Supported vCPU configurations

This table lists the supported vCPU configurations for installing Cisco Catalyst 8000V in KVM environments.

Supported IOS XE release

Supported vCPU configuration

Cisco IOS XE 17.4.1 and later

1 vCPU: requires minimum 4 GB RAM allocation

Cisco IOS XE 17.4.1 and later

2 vCPUs: requires minimum 4 GB RAM allocation

Cisco IOS XE 17.4.1 and later

4 vCPUs: requires minimum 4 GB RAM allocation

Cisco IOS XE 17.4.1 and later

8 vCPUs: requires minimum 8 GB RAM allocation

Cisco IOS XE 17.11.1a and later

16 vCPUs: requires minimum 8 GB RAM allocation

Creating the VM using the GUI tool

You can install the Cisco Catalyst 8000V virtual router as a virtual machine on Red Hat KVM virtualization by using the GUI tool. You can use the CLI to perform this installation. However, Cisco recommends using the GUI tool.

Summary

Download these key components that you require for installing Cisco Catalyst 8000V in KVM environments by using the GUI tool.

  • Download and install the virt-manager RPM package on the KVM server.

  • Download either the .qcow2 image or the .iso image from the Cisco Software Download page. Copy the file onto a local device or a network device.

Workflow

Perform these tasks to install Cisco Catalyst 8000V in KVM environments.

  1. Create the virtual machine and configure the virtual machine settings.
  2. Add a serial console.
  3. Add a host USB device.
  4. Customize the virtual machine configuration. See the respective procedures for detailed steps.

Create the VM

Perform these steps if you want to create a new VM in a KVM environment by using the GUI tool.

Procedure

Step 1

Launch the virt-manager GUI.

Step 2

Click Create a New Virtual Machine.

Step 3

Select the Cisco Catalyst 8000V qcow2 or iso file location after performing one of these two actions:

  • If you have downloaded the .qcow2 file, select Import Existing Disk Image.
  • If you have downloaded the .iso file, select Local Install Media (ISO Image or CDROM).

Step 4

Configure the memory and the CPU parameters.

Step 5

Configure the virtual machine storage settings.

Step 6

If you want to add additional hardware before creating the VM, click Customize configuration before install. The system displays the Add Hardware button. Click this button to add various hardware options, such as additional disks or a serial port interface.

Step 7

Add a serial console.

This is an optional step. To view the detailed steps to add a serial console, see Add a serial console.

Step 8

Customize your virtual machine configuration.

This is an optional step. To view the detailed steps to customize your configuration, see Customize the VM configuration.

Step 9

Click Finish.

What to do next

Access the Cisco Catalyst 8000V console by performing one of these actions:

Add a serial console

Perform these steps to add a serial console to the hardware and configure the kernel console.

Procedure

Step 1

In the virt-manager GUI, click Add Hardware.

Step 2

From the menu, choose the Serial option.

Step 3

From the Device Type drop-down field, select TCP net console (tcp).

Step 4

Specify the port number, and select the Use Telnet checkbox.

Step 5

Click Finish.

Step 6

After adding all necessary hardware, click Begin Installation.

Add a host USB device

To add the USB host device using virt-manager, complete these steps.

If you use a Cisco IOS XE 17.9.x version, a 17.12.x version, or a later version, Cisco Catalyst 8000V on KVM supports USB device pass-through which allows a USB device connected to the host machine to be passed directly to your VM. To add a USB host device, you must modify the VM configuration by using tools like virt-manager, virsh, or by directly editing the XML configuration of the VM.

Procedure

Step 1

In the virt-manager GUI, click Add Hardware.

Step 2

Select the USB host device.

Step 3

From the USB Device drop-down field, choose the USB device that you want to attach to the Cisco Catalyst 8000V.

Step 4

Click Finish.

Note

 

Cisco Catalyst 8000V supports USB disk hot-plug and allows up to two USB disk hot-plug devices to be attached.. When you add the devices, ensure you add one device at a time, by using the virt-manager.

Customize the VM configuration

Before you begin

Perform the Create the VM workflow by using a .qcow2 or an .iso image. If you want to customize your configuration before creating the instance, follow this procedure. Ensure you perform this task before you apply the Day 0 configuration to your device.

Procedure

Step 1

In the creating VM workflow, when you choose the Customize configuration before install option, the Add Hardware button is displayed. Click this button.

Step 2

Choose the Storage option.

Step 3

Select the Select Managed Or Other Existing Storage check box.

Step 4

Click Browse and navigate to the location where you have the c8000v_config.iso file.

This step is applicable only when you want to add a Day 0 or bootstrap configuration.

Step 5

From the Device-type drop-down field, choose IDE CDROM.

Step 6

Click Finish.

Step 7

To adjust the VM boot options in the virtual manager, including the boot device order, select IDE CDROM or the c8000v_config.isooption. Use the Up Arrow to move the device to the first position in the boot order.

Step 8

Click Apply to apply the changes you made to the boot order.

The image disk device option is pushed down as the second option in the boot device order.

Step 9

After adding all the necessary hardware, click Begin Installation to start the new VM deployment.

What to do next

Perform the Day 0 configuration for your device. To learn how to do this, see Day 0 Configuration.

Create the VM using CLI

Before you begin

  • Download and install the virt-install RPM package on the KVM server.

  • Download the .qcow2 image from the Cisco Catalyst 8000V software installation image package. Then copy the file to a local device or network device.

Procedure

Step 1

To create the VM for a .qcow2 image, use the virt-install command.

Creates a new VM instance and starts the instance.

Example:


                        virt-install                    \
                        --connect=qemu:///system   \
                        --name=my_c8kv_vm           \
                        --os-type=linux            \
                        --os-variant=rhel4         \
                        --arch=x86_64              \
                        --cpu host                 \
                        --vcpus=1,sockets=1,cores=1,threads=1   \
                        --hvm                      \
                        --ram=4096                 \
                        --import                   \
                        --disk path=<path_to_c8000v_qcow2>,bus=ide,format=qcow2   \
                        --network bridge=virbr0,model=virtio                        \
                        --noreboot

Step 2

To create the VM, for a .iso image, perform these steps:

  1. Run the qemu-img command.

    Creates an 8G disk image in the .qcow2 format.

    Example:

    
                                qemu-img create -f qcow2 c8000v_disk.qcow2 8G

  2. Run the virt-install command.

    Creates a new VM instance. Cisco Catalyst 8000V installs the image onto the specified disk file after this command is executed. This command requires the correct permissions for the VM to be created.

    Example:

    
                                virt-install               \
                                --connect=qemu:///system  \
                                --name=my_c8000v_vm          \
                                --description "Test VM"   \
                                --os-type=linux           \
                                --os-variant=rhel4        \
                                --arch=x86_64             \
                                --cpu host                \
                                --vcpus=1,sockets=1,cores=1,threads=1    \
                                --hvm                     \
                                --ram=4096                \
                                --cdrom=<path_to_c8000v_iso>           \
                                --disk path=c8000v_disk.qcow2,bus=virtio,size=8,sparse=false,cache=none,format=qcow2   \
                                --network bridge=virbr0,model=virtio     \
                                --noreboot

    This sample configuration creates a 1 vCPU VM with 4G of RAM having one network interface and one serial port.

Step 3

To provide the Day 0 configuration through the c8000v_config.iso disk image, add an additional parameter to the virt-install command.

Example:

virt-install --disk path=/my/path/c8000v_config.iso,device=cdrom,bus=ide

For more information, see Day 0 Configuration.

Step 4

After the installation is complete, the Cisco Catalyst 8000V VM shuts down. Run the virshstart command to start the VM.

Warning

 

Due to this issue specific to RHEL, when you launch Cisco Catalyst 8000V in a RHEL environment by using the virt-install command, you must set the host mode as follows:

  • For Red Hat Enterprise Linux 6, use --cpu host .

  • For Red Hat Enterprise Linux 7, use --cpu host-model .

Performance enhancements in KVM environment

Increase the performance of a Cisco Catalyst 8000V instance running in a KVM environment by modifying certain settings on the KVM host. These settings are independent of the IOS XE configuration settings on the Cisco Catalyst 8000V instance.

  • These settings might affect the number of VMs you can instantiate on a server.

  • Tuning steps are most impactful for a small number of VMs that you instantiate on a host.

To improve the KVM configuration performance, Cisco recommends that you do the actions listed in this table.

Action

Sample configuration (if applicable)

Enable vCPU pinning

 
 <cputune>
    <vcpupin vcpu='0' cpuset='3'/>
    <emulatorpin cpuset='3'/>
  </cputune>

Enable emulator pinning

 
 <cputune>
    <vcpupin vcpu='0' cpuset='3'/>
    <emulatorpin cpuset='3'/>
  </cputune>

Enable numa tuning.

Ensure that all vCPUs are pinned to the physical cores on the same socket.

 
  <numatune>
    <memory mode='strict' nodeset='0'/>
  </numatune>

Set hugepage memory backing

 
 <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <hugepages>
      <page size='1048576' unit='KiB' nodeset='0'/>
    </hugepages>
    <nosharepages/>
  </memoryBacking>

Use virtio instead of IDE

 
<devices>
    <emulator>/usr/libexec/qemu-kvm</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='qcow2'/>
      <source file='/var/lib/libvirt/images/rhel7.0.qcow2'/>
      <backingStore/>
      <target dev='vda' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>

Use graphics VNC instead of SPICE.

 
<graphics type='vnc' port='5900' autoport='yes' listen='127.0.0.1' keymap='en-us'>
      <listen type='address' address='127.0.0.1'/>
    </graphics>

Remove unused devices such as USB, tablet devices

Disable memballoon

 
<memballon model='none'>

CPU pinning

Improve the performance of KVM environments by using the KVM CPU Affinity option to assign a virtual machine to a specific processor. To use this option, configure CPU pinning on the KVM host.

In the KVM host environment, use these commands:

Command

Action
virsh nodeinfo

Verifies the host topology to find out how many vCPUs are available for pinning by using the following command.

virsh capabilities

Verifies the available vCPU numbers.

virsh vcpupin<vmname> <vcpu#> <host core#>

Pins the virtual CPUs to sets of processor cores.

Execute this KVM command for each vCPU on your Cisco Catalyst 8000V instance.

  • This example pins the virtual CPU 1 to host core 3.

    virsh vcpupin c8000v 1 3

  • This example shows the KVM commands needed if you have a Cisco Catalyst 8000V configuration with four vCPUs and the host has eight cores:

    virsh vcpupin c8000v 0 2

    virsh vcpupin c8000v 1 3

    virsh vcpupin c8000v 2 4

    virsh vcpupin c8000v 3 5

    The host core number can be any value from 0 to 7. For more information, refer to the official KVM documentation.


Note
When you configure CPU pinning, consider the CPU topology of the host server. If you are using a Cisco Catalyst 8000V instance with multiple cores, do not configure CPU pinning across multiple sockets.

BIOS settings

Optimize the performance of the KVM configuration by applying the recommended BIOS settings listed in this table.

Configuration

Recommended setting

Intel Hyper-Threading Technology

Disabled

Number of Enable Cores

ALL

Execute Disable

Enabled

Intel VT

Enabled

Intel VT-D

Enabled

Intel VT-D coherency support

Enabled

Intel VT-D ATS support

Enabled

CPU Performance

High throughput

Hardware Prefetcher

Disabled

Adjacent Cache Line Prefetcher

Disabled

DCU Streamer Prefetch

Disable

Power Technology

Custom

Enhanced Intel Speedstep Technology

Disabled

Intel Turbo Boost Technology

Enabled

Processor Power State C6

Disabled

Processor Power State C1 Enhanced

Disabled

Frequency Poor Override

Enabled

P-State Coordination

HW_ALL

Energy Performance

Performance

Host OS settings

Cisco recommends that you use hugepages and enable emulator pinning on the host. These actions are recommended on the host.

  • Enable IOMMU=pt

  • Enable intel_iommu=on

  • Enable hugepages

  • Use SR-IOV if your system supports it for higher networking performance. Check the SR-IOV limitations your system might have.

In addition to enabling hugepages and emulator pinning, configure: nmi_watchdog=0 elevator=cfq transparent_hugepage=never


Note

If you use Virtio VHOST USER with VPP or OVS-DPDK, you can increase the buffer size to 1024 (rx_queue_size='1024' ) provided the version of your QEMU supports it.

IO Settings

You can use SR-IOV to enhance performance. However, this may introduce limitations such as the number of virtual functions (VF), OpenStack limitations for SR-IOV like QoS support, live migration, and security group support.

If you use a modern vSwitch such as fd.io VPP or OVS-DPDK, reserve at least two cores for the VPP worker threads or the OVS-DPDK PMD threads.

Configure these parameters to run the VPP through command line.

Parameter

Description and notes

-cpu host

This parameter causes the VM to inherit the host OS flags. You need libvirt version 0.9.11 or later for this feature to be available in the XML configuration.

-m 8192

Requires 8GB RAM for optimal zero packet drop rates.

rombar=0

Disables PXE boot delays. Set rombar=0 to the end of each device option list or add "<rom bar=off />" to the device xml configuration.

KVM halt polling

halt_poll_ns is a KVM module parameter that allows you to alter the behaviour of how idle KVM guest virtual CPUs (vcpus) are handled.

When a virtual CPU in a KVM guest has no threads to run, QEMU traditionally halts the idle CPU. By default, this setting specifies a period of 400 nanoseconds during which a virtual CPU waits and polls before entering a CPU Idle state.

When new work arrives during the polling period, before the vCPU is halted, the vCPU is immediately ready to execute the work. If the vCPU has already entered the idle state when new work arrives, it must be brought out of the idle state before starting the new task. The transition from idle to running state causes additional latency, which can negatively affect latency-sensitive workloads.

With the default kernel parameters, the guest Cisco Catalyst 8000V router CPU consumes all the available resources on the host CPU.

Configure KVM halt polling

Configure the halt_poll_ns parameter in KVM to improve performance.

You can configure halt_poll_ns in two ways:

  • Large halt_poll_ns: n this case, more CPU time is spent busy-spinning for events that wake the virtual CPU, and fewer ACPI deep sleep cycles occur. This means more power is consumed. However, there are fewer wakeups from deep sleep states. Depending on the configured state, this can cause issues such as cache misses.

  • Small halt_poll_ns: In this case, less CPU time is spent busy-spinning for events that wake the CPU, and more ACPI deep sleep cycles occur. This results in lower power consumption, but more wakeups from deep sleep states are required. Increased wakeups can lead to many deep sleep instances. Depending on the configuration, this can cause many cache misses and longer wakeup times.

Procedure

Step 1

At run time, run echo 0 > /sys/module/kvm/parameters/halt_poll_ns.

Step 2

To load the module, perform this configuration.

 # rmmod kvm_intel
 # rmmod kvm
 # modprobe kvm halt_poll_ns=0
 # mpdprobe kvm_intel

Step 3

When you boot the device, add kvm.halt_poll_ns=<specify value> in the parameters section of grub2.

Troubleshoot KVM installation issues

See this section to troubleshoot common issues while installing Cisco Catalyst 8000V in KVM environments.

Issue

In a KVM environment, when you clone a Cisco Catalyst 8000V virtual machine using the virt-manager virtual machine manager, it results in a Cisco Catalyst 8000V virtual machine that you might not be able to boot. The issue occurs because virt-manager creates a cloned image that is larger than the original Cisco Catalyst 8000V VM image. The additional bytes, typically in the kilobyte range, prevent the virtual machine from booting.

Workaround

Use one of these workarounds to resolve this issue.

  • Use the virt-clone command to clone the Cisco Catalyst 8000V VM image.

  • For a cloned Cisco Catalyst 8000V VM image created by virt-manager during the bootup, select the GOLDEN image to boot instead of packages.conf.

  • In the Create a new virtual machine window, deselect Allocate Entire Disk Now before creating the new Cisco Catalyst 8000V VM. This action allows the cloned Cisco Catalyst 8000V VM image to boot. Do not use this workaround for nested cloning. Apply it only to the first cloned Cisco Catalyst 8000V VM image.