Upgrading Cisco VTS

Upgrading VTS involves:

The upgrade script enables you to upgrade the VTC and VTF. You can also roll back an upgrade, using the script.

The following table provides details about the options available in the upgrade script. Ensure that you review this table before you proceed with the upgrade.

Option

Usage, and Description, Suboptions

upgrade vtc 

python upgrade.py upgrade vtc -b BACKUP_DIR -ip VTC_IP

This option upgrades the VTC, host agents, and plugins.

This option has the following suboptions:

  • -b BACKUP_DIR or --backup-dir BACKUP_DIR—To specify the backup directory.

  • -ip VTC_IP or --vtc-ip VTC_IP—To specify the VTC IP address.

  • -gwip or --gateway-ip—The private gateway IP address.

Note   

In an HA deployment, VTC_IP has to be the VIP when you upgrade the Active. For the standby, it has to be the management IP address of the VTC VM.

For a standalone deployment, it has to be the management IP address of the VM. You must ensure that you specify the correct IP address during the upgrade. Specifying an incorrect IP address leads to problems after you upgrade. You will need to use the rollback option (more details below) to roll back the upgrade, in this scenario.

upgrade vtf 

python upgrade.py upgrade vtf  -ip VTC_IP
Note   

The script assumes that VTFs were installed using the VTS GUI Install Capabilities feature.

This option upgrades the VTFs.

This option has the following suboption:

-ip VTC_IP or --vtc-ip VTC_IP—To specify the VTC IP address.

Note   

The IP you specify while upgrading VTF has to be the VTC underlay IP address. For an HA deployment, this is the underlay VIP IP address.

rollback 

python upgrade.py rollback  -b BACKUP_DIR -ip VTC_IP

This option has the following suboptions:

  • -b BACKUP_DIR or --backup-dir BACKUP_DIR—To specify the backup directory.

  • -ip VTC_IP or --vtc-ip VTC_IP—To specify the VTC IP address. This should be the management IP address of the VTC.

Note   

If you upgrade from an earlier version of Cisco VTS to this version of Cisco VTS, and then, at a later point in time, roll back to the earlier version, you will lose the changes you have made to the system after the upgrade, and the system will become unusable. We recommend that you do not roll back to the earlier version, after you upgrade the system to this version and provision and manage overlays.

-h or --help

This option provides help, when used with any of the above options. For example: 

python upgrade.py upgrade vtf -h

The following sections provide details on upgrading Cisco VTS using the upgrade script.

Upgrading VTC

Before you run the upgrade script:

  • Destroy the existing IOS XRv VM.

  • Confirm that you are currently running the supported VTS software version. Run the version_info command to verify this. For example: 

    root@VTS1# version_info
vts_version= 
build_number=

  • Sync all the devices in your Datacenter from the VTS GUI so that the VTS knows about the current state/configuration of all the devices. If any of the devices are out-of-sync, perform “sync-from” operation. See the Synchronizing Configuration section in the Cisco VTS User Guide for more information on this feature.

  • Run show_tech_support -a on VTS. This is to capture the VTC running configuration. Error logs are also captured when you run this command.

  • Set up a shared drive which is accessible to the VTC VM to use as a backup drive. We recommend that you use an external drive for backup, and not the the VTC VM. Backing up on the VTC VM will increase the VM disc size.

  • Ensure that you do not have a /home/cisco folder on the VTC VM. The upgrade script uses /home/cisco folder. If you have a /home/cisco folder already, rename the folder before you upgrade. You may revert the name after you complete and verify the upgrade.

For High Availability (HA) deployments:

  • Check the HA information on the Active and Standby VMs to avoid issues and decide if proactive analysis needs to be done before the VTS upgrade. For example: 
    root@VTS1:/opt/cisco/package/vtc/bin# crm status
root@VTS2:/opt/cisco/package/vtc/bin# crm status

    On both Active and Standby, ensure that VTC1 and VTC2 are online and VTC1 is set as Master while VTC2 is set as Slave. See also Verifying the VTC High Availability

  • Ensure that there is no mismatch between the GUI passwords for Active and Standby VTC. To verify, make sure that the credentials file on both VTCs are the same. For example: 
    root@VTS1:/opt/cisco/package/vtc/bin# cat /var/vts/openstack/credentials
[ncs]
username = admin
password = YLdKnf3qSsKA2JWQT9a0Sg==
root@VTS2-Upgrade:/opt/cisco/package/vtc/bin# cat /var/vts/openstack/credentials
[ncs]
username = admin
password = YLdKnf3qSsKA2JWQT9a0Sg==
  • Make sure the VTS GUI is reachable using the VIP IP address for VTC, and provisioning on VTS works via the same.

Important Notes:

  • If you are upgrading an HA installation, you must run the following steps first on the Active and then on the Standby.

  • The upgrade script takes a back up of the existing installation to the back up directory you specify while running the script. We recommend that you keep the back up safe until you complete and verify the upgrade, and ensure everything is working. The upgrade script uses the back up if you need to roll back the upgrade, in case you encounter errors upon upgrade.

  • We recommend that you review the table above, which describes the upgrade script options and suboptions.

  • You must run the upgrade script as a root user.

    Step 1   Copy the upgrade ISO image to the VTC VM. For example: 
    root@VTS:/home/admin#
    Step 2   Mount the VTS upgrade ISO image located in the untared folder. For example: 
    root@VTS:/home/admin# mount -o loop VTS-upgrade-2.3.iso /mnt
    Step 3   Mount external location for backup. for example: 
    mkdir /home/admin/extdrive
    Step 4   Run the upgrade script which is available at /mnt/upgrades/python. For example: 
    python upgrade.py upgrade vtc -b BACKUP_DIR -ip VTC_IP
    Note   

    VTC_IP is the management IP address of VIP when VTC is in HA mode, and management address of the VTC when VTC in non-HA mode

    After a successful upgrade, the following message is displayed: 

    {   'Host Agent': 'Upgraded',
    'JDK and Apache ant': 'No Upgrade Required',
    'NCS': 'No Upgrade Required',
    'Third Party Packages': 'Upgraded',
    'VMM Plugins': 'Upgraded',
    'VTC': 'Upgraded',
    'VTS GUI': 'Upgraded'}

    If the script fails to upgrade any of the host agents, a message is displayed on the screen. For example: 

    "'Host Agent': 'Upgrade Failed’,” 
"status": “Error” 
"status_msg: “<error message>"

    The details are available in the following log file: 

    /var/log/vts-upgrade/upgrade.log

    If the host agent upgrade fails on any of the hosts, you must manually upgrade the host agents on these hosts. From the Cisco VTS GUI, go to Inventory > Host Inventory. Select the ToR on which the host is connected, and use Install Capabilities to install host agent. See Installing OpenStack Host Agent.

    Note   

    In the case of an HA deployment, you must run the above steps first on the Active, and then on the Standby. HA is re-established only after the Standby is upgraded to the current version of VTS.

    The VTC VM automatically reboots after the upgrade script is run successfully.

    Step 5   In the case on an HA deployment, run the following command on the Standby node: 
    crm node online

    Upgrading IOS XRv

    To upgrade IOS XRv VM, do the following:

    Note

    In case of an HA deployment, you need to do these steps on both the VMs.

      Step 1   Bring up the IOS XRv VM with new image. See Creating an IOS XRv VM and Creating an ISO for IOS XRv.
      Step 2   After the deployment is complete, edit the VM settings. Add a DC/DVD Drive selecting the Datastore ISO file and point to the XRNC.iso file which is the same and was used earlier with the XRv running the earlier VTS image, generated using the same system.cfg and same IP addresses.
      Step 3   Power on the VM.
      Note   

      With both earlier version of VTS and with this version, XRv registers to the VTS using the XR-DL’s management IP address.

      Step 4   (Only for HA deployments) In order to run in HA mode again, run the setup script by providing the underlay IP address of the other IOS XRv DL. Run this on both the IOS XRv DL. See Enabling IOS XRv High Availability.

      Step 5   Run the “check sync" operation from the VTS-GUI to verify the device is in sync with VTS. If not run the “sync to” operation.
      Step 6   Repeat the steps above to Upgrade the second XRv.
      Step 7   Verify whether both IOS XRv VMs are online. For example: 
      isco@XRVR-DL1:~$ sudo crm status

      Upgrading VTF

      Before you run the upgrade script:

      • Verify whether all the VTFs co-exist on the same network as the VTC. If not, then you need to add static route on the VTC so that VTF can register with VTC.

      • Verify if all the VTFs co-exist on the same network as IOS XRv. If not, you need to add static route on the XR-DL so that VTF can reach XR-DL.

      To upgrade VTF, run the upgrade script from the VTC, with the following option.

      Note

      In an HA installation, run the upgrade script from the Active VTC. 

      cd /mnt/vts_upgrade/upgrades/python
Python upgrade.py upgrade vtf -ip VTC_IP

      This is the VTC underlay IP. When running in HA mode, this is the underlay VIP IP address.

      After a successful upgrade, the following message is displayed: For example: 

      'VTF': 'Upgraded’

      In case VTF upgrade fails on a host, a message is displayed on the screen. For example: 

      “'VTF': 'Upgrade Failed’,” 
"status": “Error” 
"status_msg:  “<error message>"

      The details are also available in the following log file: 

      /var/log/vts-upgrade/upgrade.log

      If VTF upgrade fails on any of the hosts, you must manually upgrade VTF on the host. First delete the VTF VM from the host on which upgrade failed. Then, from the Cisco VTS GUI, go to Inventory > Host Inventory. Select the TOR on which the host is connected, and use Install Capabilities feature to install VTF. See Installing VTF on OpenStack.

      Verifying the Upgrade

      To verify whether the upgrade was successful, do the following:
        Step 1   Verify the version information. For example: 
        /mnt/vts_upgrade# version_info
        vts_version=2.3.0
build_number=<build number>
job_name=<job name>
build_date=<build date>
git_revision=<git revision>
git_branch=<git branch>
        Step 2   Verify that the upgraded VTC is online and also HA Active/Standby is established.

        Performing a Rollback

        In case there is a failure during an upgrade, the script automatically does a rollback of the upgrade.

        While you upgrade an HA setup, if the upgrade fails on the Standby after a successful upgrade of the Active, you can run the script on the Active with the rollback option to roll back the upgrade to ensure that both the Standby and Active are running the same (earlier)version of VTS. You must:
        1. Verify whether the Standby is automatically rolled back to previous version.
        2. Run the below script with rollback option on the Active. To do this:

          1. Mount the upgrade ISO again. For example: 

            mount -o loop VTS-upgrade-2.3.0-13.iso /mnt

          2. Run the script with rollback option. For example: 

            root@VTS2:/home/admin# cd /mnt/upgrades/python
root@VTS2:/mnt/upgrades/python# upgrade.py rollback -b /home/admin/extdrive -ip 192.1.7.92
            where /home/admin/extdrive is backup directory, and -ip and 192.1.7.92 is the VTC_IP
        If you want to roll back the upgrade after a successful upgrade on both Active and Standby, you must run the script in the following sequence:

        1. Run the script with the rollback option on the Standby,

        2. Run the script with the rollback option on the Active.

        Another use case is when you encounter errors after you specify an incorrect VTC IP and complete the upgrade of the VTC and VTFs. In this case, you can do the following:

        1. Rollback the VTC upgrade

        2. Upgrade VTC with the correct IP

        3. Upgrade VTF with the correct IP