Report Problems Effectively to TAC Series: NFVIS Upgrade

Introduction

This document describes general troubleshooting tips for properly reporting NFVIS upgrades to TAC for faster case resolution.

Prerequisites

Requirements

Cisco recommends you have basic knowledge of these topics:

• NFVIS software

Components Used

This document is not restricted to specific software and hardware versions, however, all the data was collected from an Enterprise Network Compute System (ENCS) running a 4.x version.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

In this document, you find the commands TAC needs for the initial triage of a Network Function Virtualization Infrastructure Software (NFVIS) upgrade problem in order to have a great TAC experience from the very first contact.

Also, you find general troubleshooting tips covering the most common scenarios for this problem so you can find a solution prior opening a TAC case.

The purpose of this document is not to explain any troubleshooting procedures extensively, if available, references to more in deep troubleshooting guides are provided.

Symptoms of NFVIS Upgrade Problems

• If an upgrade does not complete successfully, NFVIS performs a rollback to the original version.

You can validate the upgrade status with the show system upgrade apply-image command.

encs#show system upgrade apply-image
                                         UPGRADE
NAME                        STATUS       FROM       UPGRADE TO
----------------------------------------------------------------
Cisco_NFVIS-4.12.3-FC4.iso  ROLLED_BACK  4.9.5-FC1  4.12.3-FC4
Cisco_NFVIS-4.9.1-FC3.iso   SUCCESS      4.8.1-FC1  4.9.1-FC3
Cisco_NFVIS-4.9.4-FC2.iso   SUCCESS      4.9.1-FC3  4.9.4-FC2
Cisco_NFVIS-4.9.5-FC1.iso   SUCCESS      4.9.4-FC2  4.9.5-FC1

Show logging helps in understanding if the upgrade failed:

(output modified for brevity)

encs%SYS-6-UPGRADE_APPLY: In-progress
encs%SYS-3-UPGRADE_APPLY: Upgrade failed due to exception
encs%SYS-6-UPGRADE_ROLLBACK: System rolled back
encs%SYS-6-UPGRADE_ROLLBACK: Rebooting system

encs%SYS-3-UPGRADE_APPLY: Upgrade failed due to exception
encs%SYS-3-UPGRADE_APPLY: Upgrade failed due to exception
encs%SYS-3-UPGRADE_APPLY: Upgrade failed due to exception

• "Pane is dead" error can be seen during the booting process. 

Upgrade Guide can be found at Chapter: Upgrade Cisco NFVIS

Information TAC Needs for Initial Triage

• Admin tech from NFVIS and CIMC: Refer to Cisco Enterprise Network Functions Virtualization Tech-Support Generation
• Console logs
• Desired upgrade path: Specify source and target release.
• If the error is seen inside the virtual machine:
  • If it is Cisco related (such as a CAT8000v or ISRv) include the show tech-support as well.
  • For third party virtual machines, additional details of the implementation, logs, and engagement of the 3rd party can be needed.
• If "Pane is dead" error is seen:
  • Validate the storage being used. Either it is empty, or cannot be properly recognized.
• Explain the upgrade method used (CLI or GUI), or if this is a fresh NFVIS install.

Note: Validate upgrade steps, and if the upgrade path is supported in Cisco Network Function Virtualization Infrastructure Software Getting Started Guide and the supported image type prior opening a TAC case.

NFVIS Upgrade General Tips

Large version leap upgrades are not encouraged, instead, a fresh install and the availability of a proper, operating external backup of virtual machines is desirable for disaster recovery scenarios. 

Please refer to the supported paths at Cisco Network Function Virtualization Infrastructure Software Getting Started Guide.

Use the show system upgrade command that gives historical context about which upgrades were done in the past.

For upgrades failing via GUI, you can discard a problem in the GUI itself and try via CLI, and if failing, having NFVIS console logs captured is helpful.

Note: For new installation failures via Kernel-based Virtual Machine (KVM), a common step is to use the host image mapping option described at Host Image Mapping.

General Guidelines for Additional Problems Found

If, after the upgrade procedure, additional problems are found such as with the NFV software, a screenshot of the  virtual network topology diagram is encouraged, in order to understand visually the service chaining and the relevant ENCS/Cisco/3rd party devices involved.

It is highly recommended to have an understanding about Open Virtual Switch (OVS) and the virtual networking aspects in which NFVIS is based on. Refer to Understand NFVIS Virtual Networks: OVS, DPDK and SR-IOV.