Troubleshooting

Collecting Troubleshooting Information

Information collected using the commands listed below can be sent to Cisco Technical Support for troubleshooting purposes.

SUMMARY STEPS

    1.    show system sysmgr service name vman

    2.    show mgmt-infra trace message vman_trace

    3.    virtual-service move name virtual-services-name [core | log] to destination-url

    4.    show mgmt-infra trace settings vman_trace

    5.    set trace control vman_trace buffer-size buffer-size

    6.    set trace control vman_trace clear [location active]

    7.    set trace vman_trace level {debug | default | err | info | warning} [location active]


DETAILED STEPS
     Command or ActionPurpose
    Step 1 show system sysmgr service name vman


    Example: 
    Switch# show system sysmgr service name vman
Service "virtManager" ("vman", 40):
UUID = 702, PID = 6482, no SAP
State: SRV_STATE_STARTED  (entered at tim                                                                                                                    e Thu Nov 19 18:27:01 2015).
Restart count: 1
Time of last restart: Thu Nov 19 18:27:01                                                                                                                     2015.
The service never crashed since the last                                                                                                                     reboot.
Tag = N/A
Plugin ID: 0
     

    This command shows the health of the virtualization manager (VMAN) process.

     
    Step 2 show mgmt-infra trace message vman_trace
     

    This command contains information logged by the VMAN process.

     
    Step 3virtual-service move name virtual-services-name [core | log] to destination-url


    Example: 
    Switch# virtual-service move name openflow_agent core to flash:
     

    Moves application log or core files to a specified destination location. This command can be used when the application running in the container has an issue (but the container is running as expected).

     
    Step 4 show mgmt-infra trace settings vman_trace


    Example: 
    Switch# show mgmt-infra trace settings vman_trace
One shot Trace Settings:

Buffer Name: vman_trace
Default Size: 262144
Current Size: 262144
Traces Dropped due to internal error: Yes
Total Entries Written: 4653
One shot mode: No
One shot and full: No
Disabled: False
     

    This command displays trace settings of a trace buffer.

     
    Step 5 set trace control vman_trace buffer-size buffer-size
     

    This command sets the trace buffer size.

     
    Step 6 set trace control vman_trace clear [location active]
     

    This command clears the trace buffer.

     
    Step 7 set trace vman_trace level {debug | default | err | info | warning} [location active]
     

    This command sets the trace level.

     

    Troubleshooting: Installing Applications in a Virtual Services Container

    This topic describes the possible reasons why installation of an application in a virtual services container may not have been successful, and the corresponding solutions.

    Problem    Installation of an application in a virtual services container is not successful.
    Possible Cause    Installation of the application may still be ongoing.
    Check the installation status, by using the show virtual-service list command. The following sample output shows an application that has status Installed. 
    Switch# show virtual-service list
Virtual Service List:


Name                    Status             Package Name
------------------------------------------------------------------------------
openflow_agent          Installed         ofa-1.1.64142n-cat3000-SSA-k9.ova
    Possible Cause    An application with the same name has already been installed.
    Solution    Ensure that an application with the same name has not already been installed, by using the show virtual-service list command. You can verify this by referencing the Name field.
    Possible Cause    The target media has not been installed.
    Solution    Target media for the switch — flash. Ensure that the target media is installed, by using the show version command. 
    Switch# show version
Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.07.03.E3.369 EARLY DEPLOYMENT PROD BUILD ENGINEERING NOVA_WEEKLY BUILD, synced to  FLO_DSGS7_BENI_MR2_SYNC
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2015 by Cisco Systems, Inc.
Compiled Thu 12-Nov-15 05:27 by gereddy



Cisco IOS-XE software, Copyright (c) 2005-2015 by cisco Systems, Inc.
All rights reserved.  Certain components of Cisco IOS-XE software are
licensed under the GNU General Public License ("GPL") Version 2.0.  The
software code licensed under GPL Version 2.0 is free software that comes
with ABSOLUTELY NO WARRANTY.  You can redistribute and/or modify such
GPL code under the terms of GPL Version 2.0.
(http://www.gnu.org/licenses/gpl-2.0.html) For more details, see the
documentation or "License Notice" file accompanying the IOS-XE software,
or the applicable URL provided on the flyer accompanying the IOS-XE
software.



ROM: IOS-XE ROMMON
BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 1.2, RELEASE SOFTWARE (P)

ott-of-c3k-118 uptime is 36 minutes
Uptime for this control processor is 38 minutes
System returned to ROM by Power Failure
System image file is "flash:cat3k_caa-universalk9.SSA.03.07.03.E3.369.152-3.3.69.E3.bin"
Last reload reason: Power Failure



This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.

A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html

If you require further assistance please contact us by sending email to
export@cisco.com.

License Level: Ipservices
License Type: Permanent
Next reload license Level: Ipservices

cisco WS-C3650-24TS (MIPS) processor with 4194304K bytes of physical memory.
Processor board ID FDO1909E0XX
1 Virtual Ethernet interface
56 Gigabit Ethernet interfaces
2048K bytes of non-volatile configuration memory.
4194304K bytes of physical memory.
257008K bytes of Crash Files at crashinfo:.
1550272K bytes of Flash at flash:.
0K bytes of Dummy USB Flash at usbflash0:.
0K bytes of  at webui:.

Base Ethernet MAC Address          : 84:b8:02:55:82:00
Motherboard Assembly Number        : 73-15127-05
Motherboard Serial Number          : FDO19091CMW
Model Revision Number              : D0
Motherboard Revision Number        : A0
Model Number                       : WS-C3650-24TS
System Serial Number               : FDO1909E0XX


Switch Ports Model              SW Version        SW Image              Mode
------ ----- -----              ----------        ----------            ----
*    1 28    WS-C3650-24TS      03.07.03.E3.369   cat3k_caa-universalk9 BUNDLE


Configuration register is 0x102
    Possible Cause    There is insufficient space to install an application.
    Solution    Check the amount of space that is available, by using the dir command. 
    Switch# dir flash:
Directory of flash:/

16168  -rw-     2097152  Nov 17 2015 19:50:45 +00:00  nvram_config
16185  -rw-    88145920  Nov 17 2015 06:33:32 +00:00  ofa-1.1.64144n-cat3000-SSA-k9.ova
16169  -rw-    88145920  Nov 16 2015 20:04:16 +00:00  ofa-1.1.64148n-cat3000-SSA-k9.ova
16190  -rw-       17134  Oct 20 2015 20:21:25 +00:00  eci_config.out
16184  -rw-       58202   Oct 9 2015 01:24:55 +00:00  vman_ofa.log.6568.20151009012455
16201  -rw-       10188  Oct 28 2015 20:28:12 +00:00  iosdMemLeaks.txt
16191  -rw-         345  Oct 20 2015 20:21:25 +00:00  eci_show.out
16199  -rw-         954  Nov 17 2015 20:25:24 +00:00  virtual-instance.conf
24245  -rw-    53754776   Oct 9 2015 14:06:37 +00:00  cat3k_caa-infra.SSA.0.DEV-0.pkg
16167  -rw-         556  Oct 14 2015 02:41:30 +00:00  vlan.dat
16174  drwx        4096  Nov 17 2015 19:49:57 +00:00  dc_profile_dir
16176  drwx        4096  Jul 27 2015 20:03:59 +00:00  wnweb_store
24244  -rw-     5661364   Oct 9 2015 14:06:48 +00:00  cat3k_caa-drivers.SSA.0.DEV-0.pkg
16193  -rw-         356  Oct 20 2015 20:21:25 +00:00  eci_debug.out
16171  drwx        4096  Jul 27 2015 20:43:41 +00:00  onep
16186  drwx        4096  Nov 17 2015 20:00:23 +00:00  temp_amd
80801  drwx        4096   Nov 2 2015 14:22:41 +00:00  tracelogs
16192  -rw-       11792  Oct 20 2015 20:21:25 +00:00  eci_show_content.out
16194  -rw-        9543  Oct 20 2015 20:21:25 +00:00  eci_debug_content.out
16195  -rw-         356  Oct 20 2015 20:21:25 +00:00  eci_clear.out
16196  -rw-        1752  Oct 20 2015 20:21:25 +00:00  eci_clear_content.out
24246  -rw-    46494884   Oct 9 2015 14:06:41 +00:00  cat3k_caa-iosd-universalk9.SSA.0.DEV-0.pkg
24247  -rw-    30839508   Oct 9 2015 14:06:38 +00:00  cat3k_caa-platform.SSA.0.DEV-0.pkg
24248  -rw-   126614208   Oct 9 2015 14:06:48 +00:00  cat3k_caa-wcm.SSA.0.DEV-0.pkg
16212  -rw-      478495  Oct 28 2015 20:36:55 +00:00  iosdProcMemDetail.txt
16175  -rw-   328263356  Nov 16 2015 15:35:23 +00:00  cat3k_caa-universalk9.SSA.03.07.03.E3.369.152-3.3.69.E3.bin
    Possible Cause    Disk quota for container is insufficient.
    Solution    Ensure that sufficient disk quota is allotted to the virtual services container, by using the show virtual-service global command. 
    Switch# show virtual-service global
Virtual Service Global State and Virtualization Limits:

Infrastructure version : 1.5
Total virtual services installed : 1
Total virtual services activated : 1

Machine types supported   : LXC
Machine types disabled    : KVM

Maximum VCPUs per virtual service : 1
Resource virtualization limits:
Name                         Quota     Committed     Available
--------------------------------------------------------------
system CPU (%)                   6             1             5
memory (MB)                    256           256             0
flash (MB)                     256           219            37
    Possible Cause    An invalid OVA package has been used for installation (Invalid package/Parsing error/Invalid machine specification error).
    Solution    Ensure that the OVA package copied to the device matches in size with the OVA package on the FTP server. Refer to the release for details or Contact Cisco Technical Support to ensure that the OVA file provided is compatible with the device operating system and not corrupted.
    Possible Cause    The virtual services container does not install properly due to unknown reasons.
    Solution    Uninstall the virtual services container. If the problem persists, collect general troubleshooting information and contact Cisco Technical Support.
    Related Information
    Collecting Troubleshooting Information

    Troubleshooting: Activating Applications in a Virtual Services Container

    This topic describes the possible reasons why the activation of an application in a virtual services container may not have been successful, and the corresponding solutions.

    Problem    Activation of an application in a virtual services container is not successful.
    Possible Cause    Activation of the application may still be ongoing.
    Solution    Check the activation status of the application, by using the show virtual-service list command. The following sample output shows an application that status Activated. 
    Switch# show virtual-service list

Virtual Service List:


Name                    Status             Package Name
------------------------------------------------------------------------------
openflow_agent          Activated          ofa-1.1.64148n-cat3000-SSA-k9.ova
    Possible Cause    The virtual services container does not have sufficient resources for activation of the application.
    Solution    Check if the device has sufficient resources for virtualization, including—memory, disk space, and CPU utilization. You can display the resource requirement for virtualization, by using the show virtual-service command. 
    Switch# show virtual-service

Virtual Service Global State and Virtualization Limits:

Infrastructure version : 1.5
Total virtual services installed : 1
Total virtual services activated : 1

Machine types supported   : LXC
Machine types disabled    : KVM

Maximum VCPUs per virtual service : 1
Resource virtualization limits:
Name                         Quota     Committed     Available
--------------------------------------------------------------
system CPU (%)                   6             1             5
memory (MB)                    256           256             0
    Possible Cause    The application does not activate properly due to unknown reasons.
    Solution    Deactivate and uninstall the application. If the problem persists, collect general troubleshooting information and contact Cisco Technical Support.
    Related Information
    Collecting Troubleshooting Information

    Troubleshooting: Uninstalling Applications in a Virtual Services Container

    This topic describes the possible reasons why you may not have been successful with the process of uninstalling an application in a virtual services container, and the corresponding solutions.

    Problem    Uninstallation of an application from the virtual services container is not successful.
    Possible Cause    The application being uninstalled is not completely deactivated.
    Solution    Check the activation status of the application, by using the show virtual-service list command. The following sample output shows an application in the Deactivated status, and can be uninstalled. 
    Switch# show virtual-service list

Virtual Service List:


Name                    Status             Package Name
------------------------------------------------------------------------------
openflow_agent          Deactivated        ofa-1.1.64148n-cat3000-SSA-k9.ova

ott-of-c3k-118#delete flash:virtual-instance.conf  
ott-of-c3k-118#reload
    Possible Cause    The application does not uninstall due to unknown reasons.
    Solution    As a last resort, delete the virtual-instance.conf, by using the delete command, and then reload the device. 
    Switch# delete bootflash:virtual-instance.conf
Switch#  reload
    If the problem persists, collect general troubleshooting information and contact Cisco Technical Support.

    Troubleshooting: Deactivating Applications in a Virtual Services Container

    This topic describes the possible reasons why you may not have been successful with the process of deactivating an application in a virtual services container, and the corresponding solutions.

    Problem    Deactivation of an application is not successful.
    Possible Cause    The application being deactivated is not activated.
    Solution    Check the activation status of the application, by using the show virtual-service list command. The following sample output shows an application that is in the Activated state, and can be deactivated. 
    Switch# show virtual-service list

Virtual Service List:


Name                    Status             Package Name
------------------------------------------------------------------------------
openflow_agent          Activated        ofa-1.1.64148n-cat3000-SSA-k9.ova
    Possible Cause    Deactivation takes a long time (5 minutes).
    Solution    Check if application directories are in use. Ensure that there are no shells open in the application file system directories on the device.
    Possible Cause    The application does not deactivate gracefully due to unknown reasons.
    Solution    As a last resort, uninstall the application (if you have not already done this) and delete the virtual-instance.conf configuration file, by entering the delete command, and then reload the device. This step deletes all applications installed in the virtual services container. 
    Switch# delete flash:virtual-instance.conf
Switch# reload
    Solution    If the problem persists, collect general troubleshooting information and contact Cisco Technical support.