Guest

Cisco Virtual Application Cloud Segmentation Services Installation and Upgrade Guide, Release 5.4STV3.0

Book Contents
Find Matches in This Book
Available Languages
Download Options

Book Title

Cisco Virtual Application Cloud Segmentation Services Installation and Upgrade Guide, Release 5.4STV3.0

Chapter Title

Troubleshooting Installation Issues

Results

Updated:
June 16, 2016

Chapter: Troubleshooting Installation Issues

Troubleshooting Installation Issues

This chapter contains the following sections.

Cleaning the Cisco VACS Environment

Use the following order to manually clean the Cisco VACS environment.

    Step 1   Ensure that all the containers are deleted. However, if the deletion or rollback fails, you must manually remove the VMs from the VC, and run the VC inventory in the Cisco UCS Director.
    Step 2   Remove all the hosts that were added to Nexus1000V DVS

    For information on how to manually remove hosts, see Removing the Hosts Manually.

    Step 3   Remove Cisco Nexus 1000V.

    For information on how to manually remove the Cisco Nexus 1000V VSM, see Removing the Cisco Nexus 1000V VSM Manually.

    Step 4   Remove Cisco VSUM.

    For information on how to manually remove Cisco VSUM, see Removing a Cisco Virtual Switch Update Manager Installation Manually.

    Step 5   Remove Cisco PNSC.

    For information on how to manually remove Cisco PNSC, see Removing a PNSC Installation Manually.

    Problems with Installing Cisco VACS

    This section includes symptoms, possible causes, and solutions for issue (s) encountered while applying the Cisco VACS patch.

    Symptom:When you apply the Cisco VACS patch to the Cisco UCS Director using the Hypertext Transfer Protocol (HTTP) from an IIS based HTTP Server, the patch installation fails with an error message. During this installation failure, there is no network connectivity issues between the HTTP server and the Cisco UCS Director.

    Error Message: 
    (TestAdapter.java:699) [TestAdapter] downloadPackageUrl - caught an exception java.net.
SocketException: Connection reset

    Possible Causes: This occurs when you use a Windows machine as the Web/HTTP Server (IIS).

    Verification and Solution: To resolve this problem, configure a website that is dedicated to download large files by disabling the HTTP Keep-Alives Enabled option for a website that the ISS hosts. To disable this option, do the following:

    • For ISS version 6:

      1. Right click the configured website and click Properties. The Websites Properties dialog box appears.

      2. Choose the Connections tab and uncheck Enable HTTP keep-alive.

    • For ISS version 7 and 7.5:

      1. Select the configured website and double click the HTTP Response headers in the Features view..

      2. In the Actions panel, click Set Common Headers. Alternatively, right click in the HTTP Response Headers Feature screen and click Set Common Headers.

      3. Uncheck Enable HTTP keep-alive.

    Problems with Upgrading Cisco VACS

    This section includes symptoms, possible causes, and solutions for problems that you could encounter while upgrading Cisco VACS.

    1. Symptom: The Cisco VACS upgrade process fails with an error.

      Error Message: 
      tar: /opt /infra/install/<VACS_PATCH>/patch.tar: Not found in archive 
localhost logger: tar: Error exit delayed from previous errors

      Possible Causes: This occurs when the installation of the previous upgrade patch was stopped abruptly, possible because Cisco UCS Director re-booted by itself.

      Verification and Solution: To resolve this problem, do the following:

      1. Delete all Cisco VACS related files from /opt/infra/install by executing the following comamnd:

        rm -rf /opt/infra/install/VACS*

      2. Reapply the Cisco upgrade patch.

    2. Symptom:The Cisco VSUM upgrade process fails with an error.

      Error Message: 
      Cannot update the Extension info at vCenter.
update_extension.sh failed with an exception: Exception in thread 
"main" com.vmware.ciscon1kvim25.InvalidArgument
      Possible Causes:

      1. Cisco VSUM extension in VMware vSphere does not have the companyURL field as unset.

      2. VMware vSphere has already been upgraded to 6.0.

      Verification and Solution: To resolve this problem, perform the following steps to recover the Cisco VSUM, re-register the extension to VMware vSphere, and to re-manage the VSM (to Cisco VSUM):
      • For vCenter on Windows, do the following:

        1. Go to https://VCIP/mob and log in with the vCenter credentials.

        2. In a web browser, choose Content > Extension Manager.

        3. Click UnregisterExtension, enter com.cisco.n1kv and then click Invoke method.

        4. Log into the Windows VM hosting the vCenter, choose Start, and then run services.msc.
        5. Right-click the VMware vSphere Web Client and click Stop.

        6. Go to C:\ProgramData\VMware\vSphere and in the Web Client\ vc-packages\vsphere-client-serenity directory, delete the entire com.cisco.n1kv folder.

        7. Choose Start and then run services.msc.

        8. Right-click the VMware vSphere Web Client and then click Start.

      • For vCenter on Linux, do the following:

        1. Go to https://VCIP/mob and log in with the default credentials.

        2. In a web browser, choose Content > Extension Manager.

        3. Click UnregisterExtension, enter com.cisco.n1kv, and then click Invoke method.

        4. Go to the /var/lib/vmware/ vsphere-client/vc-packages/vsphere- client-serenity/ directory in the vCenter VM and delete the entire com.cisco.n1kv folder using the following command:

          rm -rf /var/lib/vmware/vsphere-client/ vc-packages/ vsphere- client-serenity/com.cisco.n1kv-x-x

        5. Restart VMware vSphere Web Client service using the following command:

          /etc/init.d/vsphere-client restart

        6. Check if the web client UI is working after starting the service and logging out.

      • To re-register Cisco VSUM, do the following:

        1. Log in to Cisco VSUM using the default credentials (root/cisco).

        2. Navigate to /etc/cisco/app_install.

        3. Copy the app.cfg template file to app.cfg using the following command:

          cp app.cfg.template app.cfg

          Note

          The app.cfg template file is a hidden file in the /etc/cisco/app_install folder.

        4. Open the app.cfg file by using the following command:

          vi app.cfg

        5. Update the IP address and change the vCenterUsernameFormat and vCenterPasswordFormat values from hex to plain.

        6. Enter the administrator credentials in the vCenterUsername and vCenterPassword fields and save this configuration.

        7. Navigate to /etc/cisco/app_install and run the following command to re-register Cisco VSUM:

          service tomcat stop; bash config_app.sh -rf; service tomcat start

      • To re-manage the VSM, do the following:

        1. Log in to Cisco UCS Director using the root credentials.

        2. Navigate to /opt/infra/inframgr/resources.

        3. Execute the managevsm.sh script using the following command:

          ./managevsm.sh

        4. Once the VSMs are re-managed, re-run the vsum upgrade python script (Upgrade_VSUM_1.3_1.5.6.py) located at /opt/infra/inframgr/resources to upgrade Cisco VSUM to 1.5.6.headless.

          Note

          If you see problems while you add hosts or perform any other VSM related operations, you can flap the svs connection on the vCenter and redo the operation.

    Troubleshooting Cisco PNSC Installation Issues

    Removing a PNSC Installation Manually

    You can manually remove an unsuccessful or a failed PNSC installation using the following steps.

      Step 1   Delete the database entry for the <pnsc-vm-name> you want to delete, from the UCSD table STINGRAY_AJAX_DATA.

      To delete the database entry from the UCS Director database table, see Deleting a Database Entry From UCS Director Database Table.

      Step 2   From the UCS Director UI, delete the entry for the respective <pnsc-name> from Administration > Physical Account > Multi Domain Managers.
      Step 3   Select the particular PNSC entry that you want to delete from the table and click Delete.
      Step 4   Power off the PNSC-VM from the vSphere Client and delete the VM.

      Problems with Installing Cisco PNSC

      This section includes symptoms, possible causes, and solutions for issues encountered while installing Cisco PNSC.

      Symptom:The Cisco PNSC installation workflow fails with an error message.

      Error Message: 
      java.net.UnknownHostException:< hostname_of_the_server >,
selectedContext=<None>

      Possible Causes:This occurs when the hosts are added to the VC via the host names and the DNS settings on the UCS Director are incorrect. The reachability to the host and the PNSC OVA deployment fails when either the DNS server is wrong, or when the DNS server is not listed as the first one in the UCS Director.

      Verification and Solution: You must verify that the DNS server IP address is valid and the DNS server is listed as the first. If not, change the order of the DNS server in such a way, that the preferred DNS server is always listed as the first in the list. To change the order or view the DNS IP address, do the following:

      1. Log into UCS Director.

      2. Navigate to Administration > Guided Setup > Initial System Configuration > Launch.

      3. Skip all the steps until the DNS Server appears. Edit the order of the DNS server list or add the right DNS servers.

      4. Skip the remaining steps and click Submit. You can relaunch the Cisco PNSC installation for a successful deployment.

      Troubleshooting Cisco Virtual Switch Update Manager Installation Issues

      Removing a Cisco Virtual Switch Update Manager Installation Manually

      Use the following procedure to manually remove Cisco Virtual Switch Update Manager while installing the Cisco Nexus 1000V for Cisco VACS.

        Step 1   Delete the database entry for the respective <vsum-vm-name> from the UCSD (Cisco UCS Director) table ‘STINGRAY_AJAX_DATA’.

        Delete the <vsum-vm>. For detailed instructions about deleting a database entry, see Deleting a Database Entry From UCS Director Database Table.

        Step 2   Delete the extension - 'com.cisco.n1kv.headless' from the vCenter Server MOB from https://<vcenter-server-ip>/mob and click Content > Extension Manager.
        Step 3   Click UnregisterExtension and add the Extension key Value = com.cisco.n1kv.headless.
        Step 4   Power off the <vsum-VM> from the vSphere Client and delete it.

        Troubleshooting Cisco Nexus 1000V VSM Installation Issues

        Removing the Cisco Nexus 1000V VSM Manually

        Use the following procedure to manually remove a Cisco Nexus 1000V VSM.

          Step 1   From the Cisco Nexus 1000V console, execute the following commands: 
          n1k-dvs-name# configure terminal
n1k-dvs-name(config)#svs connection vCenter
n1k-dvs-name(config-svs-conn)#no vmware dvs
This will remove the DVS from the vCenter Server and any associated port-groups. Do you really want to proceed(yes/no)? [yes] yes
n1k-dvs-name(config-svs-conn)#no connect
n1k-dvs-name(config-svs-conn)#end
n1k-dvs-name#copy running-config startup-config (optional)
          Step 2   Power off the primary and secondary VSM VMs from the vSphere Client and delete both the VMs.
          Step 3   Delete the database entry for the respective <Nexus1000V-dvs-name> from the Cisco UCSD table STINGRAY_AJAX_DATA.

          To delete the database entry, see Deleting a Database Entry From UCS Director Database Table.

          Step 4   Select the entry for the respective <Nexus1000V dvs> from Cisco UCS Director UI from Administration > Physical Account > Manage Network Elements and click Delete Nework Element.
          Step 5   Delete the lock entry for a Cisco Nexus 1000V from the STINGRAY_LOCK_DATA table using the DELETE FROM STINGRAY_LOCK_DATA WHERE LOCKID=’<VSM-IP>’ ; command.

          Problems with Installing Cisco Nexus 1000V

          This section includes symptoms, possible causes, and solutions for issues encountered while installing Cisco Nexus 1000V.

          1. Symptom:The Cisco Nexus 1000V installation fails.

            Possible Causes:The ESX host, on which the VSM VM is being deployed, will be unreachable to the VSUM. The error message in the SR states the following error message: 
            Handler failed with error - No route to host

            Verification and Solution:Check if the ESX host is added to VCenter via the hostname or the IP address. Ping that hostname or IP address from the VSUM console and see if the ping is successful or not. If it is not successful, rectify the network/DNS settings and retry.

          2. Symptom:The Cisco Nexus1000V Installation fails with nsc-pa-agent install failure. (PNSC-N1kv connection).

            Possible Causes:Time sync issues between Cisco PNSC and Cisco Nexus 1000V.

            Verification and Solution:To resolve this issue, do the following:

            1. Choose Administration > System > System Task to perform an inventory collection for the virtual account, Cisco PNSC, and Cisco Nexus 1000V:

              1. VMware Inventory collector Virtual Account Name

              2. Infra Network Inventory collection task.

              3. PNSC Inventory collector PNSC Name.

                Proceed to the next step if the problem is not resolved.

            2. Verify that the Cisco Nexus 1000V status is in a running mode in the Cisco PNSC. To verify this, do the following:

              1. Enter https://server-ip-address, where server-ip-address is the IP address of Cisco PNSC.

              2. In the Prime Network Services Controller window, enter the appropriate administration credentials.

              3. Choose Resource Management > Resources > VSM and verify that the VSM is running. If not, proceed to step 3.

            3. Check the connectivity between Cisco Nexus 1000V and Cisco PNSC. If the connectivity is fine, proceed to step 4.

            4. Ensure that the NTP server is reachable from Cisco PNSC, and the time synchronization has happened properly. To verify this, log into Cisco PNSC using SSH and verify the time stamp using the show clock command. If the NTP server is not reachable from Cisco PNSC and if you want to change the NTP server, do the following:

              1. In the Prime Network Services Controller window, enter the appropriate administration credentials.

              2. Set the time zone.

                1. Choose Administration > System Profile > root > system profile > default.

                2. Select the time zone under the General tab, and click Save.

              3. Add an external NTP server as the time source.

                1. Choose Administration > System Profile > root > system profile > default.

                2. Select Add NTP Server under the Policy tab.

                3. Enter the NTP server host name or the IP address, click OK, and then click Save.


                  Caution

                  We recommend that you do not set the time zone after you add the NTP server.

            5. Verify the connectivity between Cisco Nexus 1000V and the NTP server.

            6. Check the time stamp using the and the show clock and the show ntp peer status commands.

            7. If the NTP server is not reachable and if you want to configure a new NTP server, use the ntp server < NTP server IP address > command.

            8. Verify the time and the NTP synchronization status using the show clock and the show ntp peer-status commands.

            9. Log into Cisco Nexus 1000V using the SSH application and unconfigure the PA agent configuration using the following commands: 
              nsc-policy-agent
             no policy-agent-image
             no shared-secret
             no registration-ip
            10. Reconfigure the PA agent using the following commands: 
              nsc-policy-agent
     registration-ip <PNSC Ip address>
     shared-secret <shared Secret used during PNSC deployment >
     policy-agent-image bootflash:/vsmcpa.3.2.2b.bin
            11. Check the PA agent status using the following command: 
              show nsc-pa status

              If the status shows an error message, then unconfigure the NTP server on Cisco 1000V using the no ntp server < NTP server IP address > command and configure the system clock using the clock set command, where the time should be 30 seconds ahead of the PNSC time stamp. Once configured, repeat step 9 and step 10.

            12. After the installation of the PA is successful, perform step 1 and create a template.

          Troubleshooting Adding Hosts Issues

          Removing the Hosts Manually

          Use the following procedure to manually remove the hosts added to the Cisco Nexus 1000V distributed virtual switch (DVS).

            Step 1   If not already added, add a physical NIC or VMNIC to the vSphere Standard Switch and navigate to vSphere Standard Switch.
            Attention:

            This step is applicable only if one physical NIC or VMNIC is migrated to the N1KV DVS. If more than a physical NIC or VMNIC is migrated to the Cisco Nexus 1000V DVS, then skip to Step 3.

            Step 2   Select the Physical NIC or VMNIC from Properties > Network Adapters > Add.
            Step 3   Migrate the VMKNIC back to the vSwitch.
            Note   

            You must select the appropriate VLAN ID that facilitates management connectivity.

            Step 4   In the vSphere Distributed Switch, navigate to Manage Virtual Adapters and select the management VMKNICs and click Migrate. Choose the appropriate vSphere Standard Switch and the port-group that facilitates management connectivity, and proceed with the migration.

            Repeat this step for all relevant and necessary VMKNICs.

            Step 5   To remove newly created Layer 3 control (Nexus 1000V control) VMKNIC and the VTEP VMKNICs, navigate to the Manage Virtual Adapters in the vSphere Distributed switch and select the appropriate virtual adapter and click Remove.
            Step 6   To move all virtual machines from Cisco Nexus 1000V to vSwitch, select the VM and navigate to Edit Settings and change the network adapter's port group mapping to a port-group on the VMware Standard vSwitch.
            Step 7   To remove the host from the distributed virtual switch, perform the following steps:
            1. In the Networking sub menu, select the Nexus 1000V DVS from the left panel.
            2. From the Hosts tab on the right panel, select the particular host to be deleted.
            3. Right click and select Remove from the vSphere Distributed Switch.
            Note   

            These steps ensure that all the PNICs associated with the DVS are removed and are made available.

            Step 8   To remove the VIB from the host, move the host to maintenance mode and SSH to the host and execute the following command : esxcli software vib remove -n cisco-vem-v170-esx
            Step 9   To clean up the VTEP entries for the host from the UCS Director database, perform the following steps:
            1. Log on to Cisco UCS Director as root user and enter mysql --user=admin --password=<ucsd db password> <db-name> to access the UCS Director database.
            2. Use the following syntax to remove the VTEP entries for the host that was removed: 
              DELETE from STINGRAY_VTEPS_PER_HOST where HOSTIP="<VEM-HOSTIP>";
e.g. 
DELETE from STINGRAY_VTEPS_PER_HOST where HOSTIP="10.10.10.1";

            Problems with Adding Hosts

            This section includes symptoms, possible causes, and solutions for issues encountered while adding hosts.

            1. Symptom:The Port-profile screen does not show any port-profiles and manually adding one, throws an error stating that the vSwitch migration spec is missing.

              Verification and Solution: To resolve this issues, verify that there is atleast one port group on the vSwitch. If the default port groups "VM Network" and the "Management VM" are not available, do the following:

              1. Navigate to vCenter and create a port group on the vSwitch, of type, virtual machine.

              2. In the UCS Director, navigate to Virtual > Compute > Your VC Account > Polling > Request Inventory Collection.

                After the inventory collection is completed, open the Add Host wizard and navigate to the Port Profile screen. The list of port profile suggestions is displayed.

            2. Symptom:The add host operation fails when you add a host that has a VMWare DVS installed on it.

              Verification and Solution:Either remove the VMWare DVS from that host or use another host that does not have the VMWare DVS installed on it.

            3. Symptom:The add host operation could fail when you add a host which has a previous Cisco Nexus 1000V vib version.

              Verification and Solution:Remove the Cisco Nexus 1000V vib version and add it again.

            4. Symptom:The add host screen does not display the list of available hosts.

              Verification and Solution:Reboot the VSUM.

            5. Symptom: The add host operation fails with an error message.

              Error Message: 
              INTERNAL_ERROR, VSUM Error Message : null

              Possible Causes: The vSwitch of the PNIC that was used for the add host operation does not have any port groups.

              Verification and Solution:To resolve this issue, do the following:

              1. Navigate to vCenter and create a VM network port group on the vSwitch, of type, virtual machine.

              2. In the UCS Director, navigate to Virtual > Compute > Your VC Account > Polling > Request Inventory Collection.

                After the inventory collection is completed, open the Add Host wizard and complete the installation process.

            6. Symptom: vm_kernel_IP_in_use VSUM Error Message : null

              Verification and Solution:Change the VMKNIC IP (L3 or vteps) and submit the Add-host operation.

            7. Symptom:virtualNIC_dendency_on_PNIC, VSUM Error Message : null

              Verification and Solution:There was a VM template using VM network pg on that vSwitch. When you try to migrate the pnic mapped to vswitch to dvs, VSUM(Nexus1000V installer) throws an error.

            8. Symptom: Vlans_not_backed, VSUM Error Message : null

              Verification and Solution: If VLANs of existing vmknics/new L3 control vmknic/Vteps/VMs are not present in the uplink port-profile chosen for the pnics.

            9. Symptom: VSM_operation error, VSUM Error Message : null

              Verification and Solution: Some of the configuration commands have failed on the Nexus 1000V. Log in to Nexus 1000V and check the accounting log. The show accounting log will show some commands as 'FAILURE'.

            10. Symptom:The add host operation completes without VSUM throwing up an exception, but the workflow times out.

              Verification and Solution: Reboot the VSUM

            Generic Troubleshooting Issues

            This section includes symptoms, possible causes, and solutions for issue(s) encountered while creating a new template or editing an existing template.

            Symptom:Creating a new template or editing an existing template fails with an error message.

            Error Message: 
            No Registered PNSC Account found for the selected virtual account < Account Name > and switch 
< switch Name >

            Possible Causes:Communication issues between Cisco PNSC and Cisco Nexus 1000V.

            Verification and Solution:To resolve this issue, do the following:

            1. Choose Administration > System > System Task to perform an inventory collection for the virtual account, Cisco PNSC, and Cisco Nexus 1000V:

              1. VMware Inventory collector Virtual Account Name

              2. Infra Network Inventory collection task.

              3. PNSC Inventory collector PNSC Name.

                Proceed to the next step if the problem is not resolved.

            2. Verify that the Cisco Nexus 1000V status is in a running mode in the Cisco PNSC. To verify this, do the following:

              1. Enter https://server-ip-address, where server-ip-address is the IP address of Cisco PNSC.

              2. In the Prime Network Services Controller window, enter the appropriate administration credentials.

              3. Choose Resource Management > Resources > VSM and verify that the VSM is running. If not, proceed to step 3.

            3. Check the connectivity between Cisco Nexus 1000V and Cisco PNSC. If the connectivity is fine, proceed to step 4.

            4. Ensure that the NTP server is reachable from Cisco PNSC, and the time synchronization has happened properly. To verify this, log into Cisco PNSC using SSH and verify the time stamp using the show clock command. If the NTP server is not reachable from Cisco PNSC and if you want to change the NTP server, do the following:

              1. In the Prime Network Services Controller window, enter the appropriate administration credentials.

              2. Set the time zone.

                1. Choose Administration > System Profile > root > system profile > default.

                2. Select the time zone under the General tab, and click Save.

              3. Add an external NTP server as the time source.

                1. Choose Administration > System Profile > root > system profile > default.

                2. Select Add NTP Server under the Policy tab.

                3. Enter the NTP server host name or the IP address, click OK, and then click Save.


                  Caution

                  We recommend that you do not set the time zone after you add the NTP server.

            5. Verify the connectivity between Cisco Nexus 1000V and the NTP server.

            6. Check the time stamp using the and the show clock and the show ntp peer status commands.

            7. If the NTP server is not reachable and if you want to configure a new NTP server, use the ntp server < NTP server IP address > command.

            8. Verify the time and the NTP synchronization status using the show clock and the show ntp peer-status commands.

            9. Log into Cisco Nexus 1000V using the SSH application and unconfigure the PA agent configuration using the following commands: 
              nsc-policy-agent
             no policy-agent-image
             no shared-secret
             no registration-ip
            10. Reconfigure the PA agent using the following commands: 
              nsc-policy-agent
     registration-ip <PNSC Ip address>
     shared-secret <shared Secret used during PNSC deployment >
     policy-agent-image bootflash:/vsmcpa.3.2.2b.bin
            11. Check the PA agent status using the following command: 
              show nsc-pa status

              If the status shows an error message, then unconfigure the NTP server on Cisco 1000V using the no ntp server < NTP server IP address > command and configure the system clock using the clock set command, where the time should be 30 seconds ahead of the PNSC time stamp. Once configured, repeat step 9 and step 10.

            12. After the installation of the PA is successful, perform step 1 and create a template.

            Deleting a Database Entry From UCS Director Database Table

              Step 1   Log on to Cisco UCS Director as root user and enter mysql --user=admin --password=<ucsd db password> <db-name> to access the UCS Director database. 
              example:  
mysql --user=admin --password=cloupia db_private_admin
              Step 2   Enter the following SQL query to delete a database entry from the UCS Director database table: 
              DELETE FROM <table_name>
WHERE <some_column>=<some_value>;
example: 
DELETE FROM STINGRAY_AJAX_DATA
WHERE VMNAME = ’vsum_vm’;
              Step 3   Enter the following SQL query to examine the entries in the table before or after the deletion: 
              SELECT ALL <comumn_name> FROM <table_name>;
Example:
SELECT ALL VMNAME FROM STINGRAY_AJAX_DATA;

              Was this Document Helpful?

              Feedback

              Let Us Help