Chapter 1. Overview

About Cisco Intercloud Fabric

Overview of the Troubleshooting Process

Overview of Best Practices

Contacting Cisco

Chapter 2. Troubleshooting Cisco Intercloud Fabric Installation and Initial Setup

Issues with the Infrastructure Setup

Chapter 3. Troubleshooting Issues with Upgrading Intercloud Fabric

Addressing the Intercloud Fabric VSM Active Upgrade Failed Error

Addressing the Intercloud Fabric VSM Upgrade Failed Error

Chapter 4. Troubleshooting the Intercloud Fabric cloud Setup

Issues with the Cloud Setup

Deleting the Intercloud Fabric cloud Setup

Chapter 5. Troubleshooting VM Lifecycle Management

Issues with Template Creation

Issues with VM Migration

Issues with Linux Cloud VMs

Issues with Windows Cloud VMs

Issues with Monitoring VMs

Chapter 6. Troubleshooting Intercloud Fabric Components

Issues with the Intercloud Fabric Virtual Supervisor Module

Issues with Prime Network Services Controller

Issues with the Intercloud Fabric Switch

Issues with the Intercloud Fabric Router

Chapter 7. Using Intercloud Fabric Component Commands and Logs

Using Intercloud Fabric Director Logs

Using Intercloud Fabric Director Service Request Logs and Details

Using Cisco Intercloud Fabric Troubleshooting Commands

Using Prime Network Services Controller Logs

Using Prime Network Services Controller Troubleshooting Commands

Using Intercloud Fabric Firewall Logs

Using Windows Cloud VM Logs

Using Windows Cloud VM Troubleshooting Commands

Chapter 8. Obtaining Help from Cisco

Cisco Intercloud Fabric Support Community

Contacting Cisco Customer Support

Obtaining Logs for Cisco Customer Support

Copyright Page

Chapter 1. Overview

- About Cisco Intercloud Fabric

- Overview of the Troubleshooting Process

- Overview of Best Practices

- Contacting Cisco

About Cisco Intercloud Fabric

Cisco Intercloud Fabric provides a faster and flexible response to business needs and addresses the potential challenges with hybrid clouds. A hybrid cloud is an interaction between private and provider clouds where private clouds extend to provider clouds and use provider cloud resources in a secure and scalable way. Cisco Intercloud Fabric enables you to place workloads across heterogeneous environments in multiple provider clouds. Cisco Intercloud Fabric provides the architectural foundation for secure hybrid clouds, which allows enterprises to easily and securely connect the private clouds to the provider cloud as needed and on demand. With a hybrid cloud, enterprises can combine the benefits of private and provider clouds. Cisco Intercloud Fabric provides the following benefits:

- Provides a single point of management and control for virtual workloads across multiple provider clouds.

- Provides a choice of cloud providers, such as Amazon Web Services, Microsoft Azure, and multiple Intercloud Fabric provider-based clouds.

- Provides highly secure, scalable connectivity to extend private clouds to service provider clouds.

- Enforces consistent network and workload policies throughout the hybrid cloud.

- Enables workload mobility to and from service provider clouds for virtual workloads.

Cisco Intercloud Fabric

Overview of the Troubleshooting Process

Follow these steps to troubleshoot your network:

1. Gather information that defines the specific symptoms.

2. Identify all potential causes for the symptoms.

3. Systematically eliminate each potential cause (from most likely to least likely) until the symptoms disappear.

Overview of Best Practices

Best practices are the recommended steps you should take to ensure the proper operation of your appliance. We recommend the following best practices for most networks:

- Maintain a consistent Cisco Intercloud Fabric release across all network devices.

- Refer to the release notes for your Cisco Intercloud Fabric release for the latest features, limitations, and caveats.

- Document private data center details, including the vNIC configuration used by virtual machines (VMs) and the corresponding vSwitch configuration to ensure that they are consistent with each other.

- Verify and troubleshoot any new configuration changes after implementing a change.

Contacting Cisco

If you cannot resolve a problem by using the information in this guide, contact Cisco Customer Support for help. For more information, see Contacting Cisco Customer Support.

Chapter 2. Troubleshooting Cisco Intercloud Fabric Installation and Initial Setup

- Issues with the Infrastructure Setup

Issues with the Infrastructure Setup

This section includes symptoms, possible causes, and solutions for issues encountered while setting up the Intercloud Fabric infrastructure.

Symptom: After installing Cisco Intercloud Fabric, the GUI does not launch and is stuck on the loading page (with a picture of clouds).

Possible Cause: One of the services did not start correctly or took longer than expected to start.

Verification and Solution:

1. Log in to Intercloud Fabric VM as shelladmin.

2. Select option 3 to stop services.

3. Select option 4 to start services and wait three to five minutes while services restart.

4. Select option 2 to display the service status.

After all services have restarted, the Intercloud Fabric launches.

Note

After restarting services in Step 3, you can repeatedly select option 2 to display the service status.

Symptom: There is an error when attempting to upload a Cisco Intercloud Fabric image tar file.

Possible Cause: This issue occurs if you use a browser other than Chrome. If you are using Chrome, it is possible that the image tar file did not completely upload.

Verification and Solution:

1. If you are not using the Chrome browser, use Chrome and try again.

2. If you encounter the error again, enable the root account by using the shelladmin credentials. The password for the shelladmin account is the same password that was entered when deploying the Intercloud Fabric appliance.

3. Use the scp command to copy manually the tar file into the /opt/infra/uploads folder of Intercloud Fabric Director.

4. On the Images page of the Infrastructure Setup dialog, click the Browse button and select the file from the local desktop.

5. Click Next.

Note

Do not click the Upload button.

6. Proceed with the remainder of the setup.

Symptom: The Intercloud image fails to successfully import into Cisco Prime Network Services Controller.

Error Message:

Handler failed with error - Image Upload Failed: End point timed out.
Check for IP, password, space, or access related issues.

Possible Cause: Cisco Prime Network Services Controller (Prime Network Services Controller) cannot connect to Cisco Intercloud Fabric.

Verification and Solution: If the import operation does not complete within a reasonable amount of time, check the Recent Jobs list in Prime Network Services Controller for the status and error message. The Task line item lists the last error encountered, if any, while importing and the number of retries attempted.

Symptom: After restarting Prime Network Services Controller services, the job for importing an Intercloud image shows as running, but the job does not progress.

Possible Cause: If Prime Network Services Controller services are restarted while the import operation is running, the task will not resume.

Verification and Solution: Cancel the job in Intercloud Fabric Director and import the image again.

Symptom: The Prime Network Services Controller server cannot resolve hostnames to IP addresses.

Error Message:

PNSC can't reach Public IP address

Possible Cause: The DNS server IP address is not entered correctly in the device profile. The DNS server resolves the hostname to an IP address. If ESX hosts were added to the VMware vCenter client using hostnames instead of IP addresses, those hostname-to-IP address mappings must exist in the DNS server.

Verification and Solution: Ensure that the DNS server IP address is entered correctly in the device profile, as follows:

1. In the Policies tab, choose Intercloud Infrastructure Policies.

2. If the DNS server IP address is incorrect, click Edit and enter the correct IP address.

For more information about configuring the DNS server IP address, see the Cisco Prime Network Services Controller User Guide.

Symptom: The Intercloud Fabric Director has an error regarding the shared secret.

Possible Cause: A shared secret that exceeds 11 characters was entered in the Intercloud Fabric Director.

Verification and Solution: Ensure that the shared secret is less than 12 characters.

Symptom: While deploying an Intercloud Fabric Director OVF template, VMware vSphere displays the error "The OVF package is invalid and cannot be deployed. The provided network mapping between OVF networks and the system network is not supported by any host."

Possible Cause: The VMware distributed virtual switch (DVS) is out of synchronization.

Verification and Solution: Ensure that the VMware DVS is properly synchronized with all hosts. For information about synchronizing the VMware DVS, see the VMware vSphere documentation.

Symptom: If the setting "Net.ReversePathFwdCheckPromisc=1" in the ESX host where the Intercloud Fabric Extender is hosted is not enabled, you might experience traffic loss or duplicate packages between enterprise and cloud VM traffic or Intercloud Fabric Switch module flap at the Intercloud Fabric VSM. This requirement applies only if you are using VMware virtual switch to host the Intercloud Fabric Extender; it does not apply if you are using a Cisco Nexus 1000V switch.

Possible Cause: If Intercloud Fabric Extender is hosted on a VMware vSwitch or distributed switch (VDS) and if the vSwitch or distributed switch is connected to multiple physical NICs, you must enable the setting Net.ReversePathFwdCheckPromisc=1 in the ESX host where the Intercloud Fabric Extender is hosted.

Verification and Solution: Enable the setting Net.ReversePathFwdCheckPromisc=1 in the ESX host where the Intercloud Fabric Extender is hosted. The setting is found under Host > Configuration > Advanced Settings > Net in the VMware vSphere UI.

Note

If the value of the Net.ReversePathFwdCheckPromisc configuration option is changed while the ESXi host is running, you must toggle (disable then re-enable) the Promiscuous Mode check box in the Intercloud Fabric Extender trunk port group security settings for the change to take effect.

- For VMware virtual switch, you must set the trunk port group to allow All VLAN IDs in the VMware vSphere GUI.

- For a security policy for the trunk port group on the VMware virtual switch, set the Promiscuous Mode, MAC Address Changes and Forged Transmits to Accept in the VMware vSphere GUI. This requirement applies only if you are using a VMware virtual switch. It does not apply if you are using a Cisco Nexus 1000v switch.

Chapter 3. Troubleshooting Issues with Upgrading Intercloud Fabric

- Addressing the Intercloud Fabric VSM Active Upgrade Failed Error

- Addressing the Intercloud Fabric VSM Upgrade Failed Error

Addressing the Intercloud Fabric VSM Active Upgrade Failed Error

If upgrading an Intercloud Fabric VSM fails with the error "Active Upgrade Failed," you can address by using the CLI.

Procedure

Step 1

Log in to the CLI of the Intercloud Fabric VSM that failed to upgrade successfully.

Step 2

At the prompt, enter the show module command.

Step 3

In the output, locate the module that did not upgrade to the newer version.

Step 4

Enter the following command:

reload module module-number

where module-number is the number of the module that did not upgrade.

Addressing the Intercloud Fabric VSM Upgrade Failed Error

If Intercloud Fabric VSM fails to upgrade with the error "Intercloud Fabric VSM Upgrade Failed," you can upgrade the Intercloud Fabric VSM manually.

Before You Begin

Confirm that you can access the upgrade software bundle from Cisco.com.

Procedure

Step 1

If you have not already done so, download the upgrade software bundle (icfb-k9-upgrade-2.3.1.pkg.zip) from Cisco.com.

Step 2

Unzip the icfb-k9-upgrade-2.3.1.zip file. This provides the upgrade-icfb-infra.2.3.1.tar file.

Step 3

Untar the upgrade-icfb-infra.2.2.1.tar file. This provides the Intercloud Fabric VSM ISO image.

Step 4

Log in to the Intercloud Fabric VSM CLI. If the Intercloud Fabric VSM is part of a high availability (HA) pair, log in to the CLI of the active Intercloud Fabric VSM.

Step 5

Ensure that sufficient space is available for the image file to be copied to the Intercloud Fabric VSM as follows:

1. For a standalone Intercloud Fabric VSM or the active Intercloud Fabric VSM in an HA pair, enter the dir bootflash: command.

2. For the standby Intercloud Fabric VSM in an HA pair, enter the dir bootflash://sup-standby/ command.

3. If needed, delete any unnecessary files on the standby Intercloud Fabric VSM to make space available.

Step 6

Copy the ISO image to bootflash by entering the following command:

switch# copy scp://filepath/iso-filename bootflash:iso-filename

where filepath is the image location and iso-filename is the name of the ISO image file.

Step 7

Install the ISO image by entering the following command:

switch# install all iso bootflash:///iso-filename

In an HA setup, as part of the upgrade process, the standby Intercloud Fabric VSM is loaded with the new image. After the Intercloud Fabric VSM becomes the HA standby again, the upgrade process initiates a switchover. The upgrade then continues from the new active Intercloud Fabric VSM.

Step 8

After the installation operation is complete, log in to the CLI of the standalone or active Intercloud Fabric VSM and verify that the switch is running the required software version by entering the following command:

switch# show version

Step 9

Adjust the startup-config size by copying the running configuration to the startup configuration as follows:

switch# copy running-config startup-config

Step 10

Display the log for the installation by entering the following command:

switch# show install all status

Step 11

To display the log for the installation on the standby Intercloud Fabric VSM in an HA pair, enter the following commands:

switch# attach module module-number
switch# show install all status

where module-number is the number of the standby Intercloud Fabric VSM.

Chapter 4. Troubleshooting the Intercloud Fabric cloud Setup

- Issues with the Cloud Setup

- Deleting the Intercloud Fabric cloud Setup

Issues with the Cloud Setup

This section includes symptoms, possible causes, and solutions for issues encountered while setting up an Intercloud Fabric cloud.

Symptom: Creating an Intercloud Fabric cloud fails.

Possible Cause: The network does not have enough available IP addresses.

Verification and Solution: Modify the IP address pools that are associated with the specific Intercloud Fabric cloud and recreate the Intercloud Fabric cloud.

Symptom: Credentials validation fails.

Possible Cause: The cloud provider credentials were entered incorrectly.

Verification and Solution: Re-enter the cloud provider credentials.

Symptom: From the VMware vCenter client, you cannot determine which VSM virtual machine (VM) configuration is active (primary) and which configuration is standby (secondary).

Possible Cause: While deploying the Intercloud Fabric Switch in high availability mode, you created the VMs with names similar to the Intercloud Fabric VSM high availability VM names.

Verification and Solution: As a good practice, prefix active VM names with "pri" (primary) and standby VM names with "sec" (secondary).

Symptom: Intercloud Fabric cloud creation fails with a message stating that SCP is not present on the VM.

Error Message:

Jan 23 2015 20:32:15 GMT Error occurred while Template Creation: PNSC Task State: Failed. Task Message: OS not compatible:scp not present on VM
Jan 23, 2015 20:33:49 GMT Handler failed with error - PNSC Task State: failed. Task Message: OS not compatible:scp not present on VM, selectedContext=<None>
Jan 23, 2015 20:33:49 GMT Task #1 (InterCloud Create Template (Create Template in Cloud)) failed after 456 seconds

Possible Causes:

- SCP is not installed or is installed in the wrong location.

- SSH is not fully installed.

Verification and Solution:

1. Confirm that SCP is installed and in the correct location by entering the command whereis -b scp.

2. Confirm that SSH is fully installed. If it is not, you can use the yum command to update the SSH package with the command yum update package-name.

Symptom: Creating an Intercloud Fabric cloud fails with the message, "Failed to create instance."

Error Message:

Jan 23, 2015 07:28:08 GMT [Deploy IcSwitchInst : icldCiscoITDD3-iclink-ics-1] [Step 1 of 4 : Instantiate IcSwitch][Create instance failed for instance: Failed to create instance: 00050a12-fab0-43d6-005-a12fab043d6Error not populated][Retry Count:5]
Jan 23, 2015 07:28:08 GMT (Deploy IcSwitchInst : icldCiscoITDD3-iclink-ics-1 failed: Create instance failed for instance: Failed to create instance: 00050a12-fab0-43d6-0005-0a12fab043d6Error not populated)
Jan 23, 2015 07:28:08 GMT 'icldCiscoITDD3' is the last IcfCloud for Provider Account 'CiscoITDDwedd'.

Possible Cause: Multiple situations can cause this problem.

Verification: The most likely cause is that the networking infrastructure on the VMware vCenter client that controls how the VMs communicate prevents the Intercloud Fabric VSM from establishing a connection with the Intercloud Fabric Switch instance that is being brought up on the cloud provider.

Symptom: After rebooting the host on which Cisco Intercloud Fabric Director, Prime Network Services Controller, Intercloud Fabric Extender, and Intercloud Fabric VSM are deployed and powering on all VMs, there is no communication between private VMs and cloud VMs.

Possible Cause: The Intercloud Fabric Extender host was rebooted out-of-band, resulting in the loss of communication and modules being unable to register with the Intercloud Fabric VSM.

Verification and Solution:

1. Check module status by logging into the Intercloud Fabric VSM console and entering the show module command. The results should indicate that modules are not registered and that communication has been lost.

2. In the Cisco Intercloud Fabric Director GUI, reboot the Intercloud Fabric Extender and the Intercloud Fabric Switch. Because this reboot occurs in-band, the Intercloud Fabric VSM can register the modules and communication is restored.

Symptom: Intercloud Fabric cloud (IcfCloud) is deployed but the tunnel is not operational.

Possible Cause: The control-channel port, and/or the data-tunnel port, is not open in the firewall.

Verification and Solution:

1. Verify that the protocol/port uses the following command in ICX:

icx-1# show intercloud tunnel config
ICS-Profile:
uuid                     2B59D348-9E48-F950-92B2-5258A9E2999F
public-ip                54.151.47.163
ICE-uuid                 855FD5BF-2BD0-7864-57DB-954DA517D887
Site-to-Site-tunnel-profile:
Heartbeat-interval       1
Heartbeat-retries        120
Tunnel-Protocol          UDP
Tunnel-Port              6644
Site-to-site-key-profile [Configured]
Encryption Algorithm     AES-128-CBC
Hash Algorithm           SHA1
Key-Profile-Checksum     B722147B-276704CF-FBA8ACB6-13371C99-ECE116BF
ReKey-State              Not-In-Progress
Epoch                    0

2. Do one of the following:

- If the desired protocol/port is not shown in the above command, change the protocol for site-to-site tunnel in tunnel-profile in Intercloud Fabric:

- If the default tunnel-profile is used in IcfCloud, it cannot be changed. In that case, create/clone a new IcfCloud with a new tunnel-profile with the desired protocol.

- If the non-default tunnel-profile is used in IcfCloud, edit the tunnel-profile with the correct protocol in the site-to-site tunnel section. To enforce the modified tunnel-profile, you must reboot/switchover IcfCloud. If IcfCloud is in HA mode, you must reboot/switchover IcfCloud twice.

- If the correct protocol/port is shown in the above command, use the following command to test that ports are open in the firewall so to establish site-to-site tunnel between ICX and ICS:

icx-1# intercloud test ics-reachability
PORT       STATE                     SERVICE         REASON
6644/udp   open                      data-tunnel     success
6646/tcp   closed                    data-tunnel     Connection refused
22/tcp     open                      ssh             success
443/tcp    open                      https           success
icx-1#

- For a TCP HTTPS-based tunnel, make sure tcp/22 and tcp/443 are open.

- For a TCP-based tunnel, make sure tcp/22, tcp/443 and tcp/6646 are open.

- For a UDP-based tunnel, make sure tcp/22, tcp/443 and udp/6644 are open.

If the data-tunnel in HTTPS Mode, 6646/tcp, is unused, then you can ignore its status.

Symptom: Creation of Intercloud Fabric cloud to AWS fails.

Error Message:

Unable to create tempVM

Possible Cause: No subnet, or default subnet, attached on VPC.

Verification and Solution: On the AWS console, verify if VPC has a subnet attached and whether the attached subnet ID is a default subnet. If default subnet does not exist, AWS support must be contacted to recreate. If default subnet exists, associate to VPC.

Deleting the Intercloud Fabric cloud Setup

Use this procedure to delete the Intercloud Fabric cloud setup.

Procedure

Step 1

Delete all the services such as ICF Firewall or ICF Router associated with the Intercloud Fabric cloud.

1. Log in to the Intercloud Fabric.

2. Choose Intercloud > Infrastructure.

3. In the Infrastructure tab, click the Launch PNSC button.
The PNSC GUI appears.

4. Choose Resource Management > Manage Resources > tenant > Network Services.

5. Select the service such as ICF Firewall or ICF Router and click Delete.
Ensure that the services are deleted by navigating to Resource Management > Manage Resources > tenant > Network Services before you proceed to the next step.

Step 2

Terminate all the virtual machines associated with the Intercloud Fabric cloud.

1. Log in to the Intercloud Fabric.

2. Choose Intercloud > Compute.

3. Select the Intercloud Fabric cloud and click the VM tab.

4. Select the virtual machine and click Terminate.
You can delete only one virtual machine at a time. Repeat this step to delete the other virtual machines.

Step 3

Delete the VDC associated with the Intercloud Fabric cloud and the provider account.

1. Log in to the Intercloud Fabric.

2. Choose Policies > Virtual/Hypervisor Policies > Virtual Data Centers.

3. In the All User Groups pane, choose the VDC group that you want to delete.

4. Click the VDC tab.

5. Select the VDC and click Delete.
You can delete only one VDC at a time. Repeat this step to delete the other VDCs.

Step 4

Delete the Intercloud Fabric cloud and the templates in the cloud.

1. Log in to the Intercloud Fabric.

2. Choose Intercloud > IcfCloud.

3. Select the Intercloud Fabric cloud and click Delete.
The completion of the service request indicates that the Intercloud Fabric cloud is deleted.

Note:
When you delete the last Intercloud Fabric cloud associated with a virtual cloud account, all associated templates are deleted.

4. For the Amazon Web Services (AWS) provider account, delete any snapshots or volumes associated with the Intercloud Fabric cloud from the AWS console.

Step 5

Delete the Intercloud Fabric VSM.

1. Log in to the Intercloud Fabric.

2. Choose Intercloud > Network > All Clouds.

3. Click the IcfVSM tab.

4. Select the IcfVSM and click Terminate.
To check the status of the task, log in to the VMware vSphere Client and navigate to Recent Tasks.

Step 6

Power off and delete the virtual machine.

1. Log in to the VMware vSphere Client.

2. Select the Intercloud Fabric virtual machine and click Power off.

3. Select the Intercloud Fabric virtual machine and click Delete from Disk.

Chapter 5. Troubleshooting VM Lifecycle Management

- Issues with Template Creation

- Issues with VM Migration

- Issues with Linux Cloud VMs

- Issues with Windows Cloud VMs

- Issues with Monitoring VMs

Issues with Template Creation

This section includes symptoms, possible causes, and solutions for issues associated with creating a template.

Symptom: Template creation fails with the message "OS not compatible:VMware tools not present on VM."

Error Message:

Jan 24, 2015 03:26:47 GMT Template Creation in Progress: 96 %
Jan 24, 2015 03:26:47 GMT Creating Template: OS not compatible:VMware tools not present on VM
Jan 24, 2015 03:26:47 GMT Retrying: 2

Verification and Solution: Confirm the following:

- The image operating system (OS) is supported.

- VMware tools are installed in the VMware vCenter client.

Symptom: Template creation fails with the message "Cannot find appropriate ICA image."

Error Message:

Jan 23, 2015 23:35:15 GMT Handler failed with error - Cannot find appropriate ICA image for the VmImage org-root/vm-img-CiscoITRH64smallscp, selectedContext=<None>
Jan 23, 2015 23:35:15 GMT Task #1 (InterCloud Create Template (Create Template in Cloud)) failed after 0 seconds

Verification and Solution: In the VMware vCenter client, confirm that the image is based on a supported guest OS version.

Symptom: Intercloud Fabric 2.3.1 only supports Master Boot Record (MBR-based) partition tables. As a result, disks with a GUID Partition Table (GPT) fails and displays an error message.

Error Message:

Unsupported operating system.

Possible Cause: Intercloud Fabric does not support the GPT standard for disk partition tables for both Windows and Linux based operating systems. Possible conditions include:

- An on-board VM is moved back to enterprise.

- The VM is moved from enterprise to cloud.

- the user uploads an OVA on Prime Network Services Controller.

Verification and Solution: Check the partition style on the VM.

For Linux, run the command:

fdisk -lu

For Windows:

1. Select Start > Control Panel.

2. In the Control Panel, choose System and Security > Administrative Tools > Computer Management.

3. Select Disk Management.

4. Right-click on Disk 0 and select Properties.

5. Select the Volumes tab and check the Partition style for the disk.

Issues with VM Migration

This section includes symptoms, possible causes, and solutions for issues associated with migrating a VM.

Symptom: While creating a virtual data center, the user group or policy drop-down lists are blank.

Possible Cause: The user group or the policies are not created.

Verification and Solution: Confirm that the user group and all required policies are created.

Symptom: The VM IP address is not shown in the VM report for private VMs.

Possible Cause: When a VM is migrated from the cloud to the private data center, the VM IP address does not appear in the VM report.

Verification and Solution: Use either of the following options to retrieve the VM IP address:

- To use the Intercloud Fabric GUI:

1. In the Intercloud Fabric GUI, choose Organization, and click the Reservations tab.

2. In the list of service requests, locate the service request ID that was used to create the VM.

The IP address associated with the service request ID is the required VM IP address.

- To use the end-user portal, confirm that VMware Tools has been installed on the VM.

Symptom: While creating a compute policy, the Select Resource drop-down list displays a blank list.

Possible Cause: The VMware vCenter IP address might be entered incorrectly or there is no connectivity to the VMware vCenter client.

Verification and Solution: Check the VMware vCenter IP address and connectivity.

Symptom: An end-user does not see the VM in the list or sees a blank list.

Possible Cause: The end-user VMs are assigned to a virtual data center that has a different user group than the user.

Verification and Solution: Log in and determine whether or not the VMs are assigned to a virtual data center that has the same user group as the user. If the user group is different, change it to the same user group as the user.

Symptom: The virtual data center list in the migration dialog is blank.

Possible Cause: The virtual data center list shown in the migration dialog box is the list of target-side virtual data centers (on the cloud when migrating to the cloud, or in the private data center when migrating to the private data center).

Verification and Solution: Determine whether or not the virtual data centers are created and have the same user group.

Symptom: The resource (host) list in the migration-back dialog is blank.

Possible Cause: The resource list of the compute policy that is associated with the target virtual data center is not configured.

Verification and Solution: Make sure that resource list is configured.

Symptom: The migration fails with the message, "VM does not have interfaces configured …."

Possible Cause: The network policy for the destination virtual data center does not have at least one vNIC configured.

Verification and Solution: Make sure that the network policy has at least one vNIC configured.

Symptom: The migration fails with the message, "Failed to refresh …."

Possible Cause: The connectivity from Prime Network Services Controller to the VMware vCenter client has failed.

Verification and Solution: Check the connectivity with the VMware vCenter client.

Symptom: The migration fails with the message, "Port group profile mis-match."

Possible Causes:

- The VM is missing one or more NICs in the virtual data center network policy.

- The VM is missing one or more mandatory NICs in the network policy.

Verification and Solution: Add any missing NICs to the virtual data center network policy.

Symptom: The migration fails with the message, "Not able to connect to PNSC…."

Possible Cause: There is no connectivity to Prime Network Services Controller.

Verification and Solution: See Issues with Prime Network Services Controller.

Symptom: Migration fails with some other error after showing a progress message of greater than 0%.

Possible Cause: The possible causes vary.

Verification and Solution: See Issues with Prime Network Services Controller.

If the issue is still unresolved, send the Prime Network Services Controller log files and Cisco Intercloud Fabric Director (ICFD) log files to the next level of support. For more information, see Obtaining Logs for Cisco Customer Support.

Symptom: Converting an image fails due to the VM having a partitioned disk.

Possible Cause: Cisco Intercloud Fabric Director does not support a VM with multiple disks or a partitioned disk.

Verification and Solution: Ensure that the VM does not have a partitioned disk.

Symptom: You choose multiple VMs to migrate, but only one VM is migrated.

Possible Cause: Cisco Intercloud Fabric Director does not support migrating multiple VMs concurrently.

Verification and Solution: Choose only one VM at a time to migrate.

Symptom: After choosing to migrate a VM, Cisco Intercloud Fabric Director has two or more service requests to migrate that VM.

Possible Cause: While a VM is being migrated, you attempt to migrate that VM again.

Verification and Solution: Do not migrate a VM while it is being migrated.

Symptom: You cannot cancel a VM task for a VM that is rebooting.

Possible Cause: The process is stuck due to the VM rebooting.

Verification and Solution: Restart the CPM process.

Symptom: VM Move to Cloud failed with error: These port-groups in the VM can not be found in network-profile: [vlan-34-test].

Possible Cause: The Port Profiles configured on the VM is not included the Intercloud VDC Network Policy.

Verification and Solution: Add the Port Profile to InterCloud Network Policy. The VM Move to Cloud is successful.

Symptom: The VM does not boot in the cloud provider.

Possible Cause: The VM fails to boot up on the cloud provider.

Verification and Solution: You can access the cloud providers portal to check the boot status by using one of the following options:

- In Amazon, select the VM on the Amazon console and choose Actions > Instance Settings > Get System Log to review the instance system log. You can use the VM boot log to determine if the reboot was successful or failed. A successful reboot will pass both status checks.

- In Azure, select the VM instance and click the Monitor tab to review the activity. Both the disk and the network should show activity.

- In Cisco Intercloud Services, you can use the instance console to verify the cVM has rebooted.

- In CloudStack, you can use the instance console to verify the cVM has rebooted.

- In Cisco Intercloud Services – V, open a support case.

Symptom: Migration of VM fails with error "Failed to shutdown VM Guest OS. Error: vmware-tools not present. Please shutdown the VM's Guest OS manually and reinitiate VM move".

Possible Cause: ICFs attempt to shutdown VM's Guest OS fails due to the absense of vmware-tools on the VM.

Verification and Solution: Manually shutdown the VM's Guest OS by logging into the VM and reinitiating the VM migration.

Symptom: If an enterprise Virtual Machine is created using the vCenter Guest Customization Wizard, while either cloning a VM or creating one from a template, the migration request will fail due to the Customization Wizard adding the following lines at the end of the ifcfg-ethxxx files in the cloned or created VM:

check_link_down(){
return 1;
}

Possible Cause: The Intercloud Fabric code that attempts to migrate this Virtual Machine to the cloud does not have support for parsing function in Intercloud Fabric 2.3.1.

Verification and Solution: Cancel the service request, manually remove the lines that were added to the end of the ifcfg-ethxxx file, and retry the migration request.

Symptom: Intercloud Fabric 2.3.1 only supports Master Boot Record (MBR-based) partition tables. As a result, disks with a GUID Partition Table (GPT) fails and displays an error message.

Error Message:

Unsupported operating system.

Possible Cause: Intercloud Fabric does not support the GPT standard for disk partition tables for both Windows and Linux based operating systems. Possible conditions include:

- An on-board VM is moved back to enterprise.

- The VM is moved from enterprise to cloud.

- the user uploads an OVA on Prime Network Services Controller.

Verification and Solution: Check the partition style on the VM.

For Linux, run the command:

fdisk -lu

For Windows:

1. Select Start > Control Panel.

2. In the Control Panel, choose System and Security > Administrative Tools > Computer Management.

3. Select Disk Management.

4. Right-click on Disk 0 and select Properties.

5. Select the Volumes tab and check the Partition style for the disk.

Issues with Linux Cloud VMs

This section includes symptoms, possible causes, and solutions for issues associated with Linux VMs on public clouds.

Symptom: RedHat/CentOS style cloud VMs deployed in Azure with Intercloud Fabric may cause traffic to be unpredictably routed.

Possible Cause: There may be two default routes in the IP routing table. For example, in the RedHat 6.1 CVM below, after ICF access tunnel is up, you will see:

- The Azure provisioned cloud interface, csc0 in subnet 10.200.0.0/16 with default route configured through 10.200.0.1.

- One overlay interface, eth0, provisioned by Intercloud Fabric in subnet 10.2.0.0/24 with default route configured through 10.2.0.75.

[root@rhel61-lnic ~]# ip route list
168.63.129.16 via 10.200.0.1 dev csc0
168.63.129.16 via 10.200.0.1 dev csc0  proto static
10.2.0.0/24 dev eth0  proto kernel  scope link  src 10.2.0.11
169.254.0.0/16 dev csc0  scope link  metric 1002
169.254.0.0/16 dev eth0  scope link  metric 1003
10.200.0.0/16 dev csc0  proto kernel  scope link  src 10.200.0.5
default via 10.2.0.75 dev eth0 <--- default route over overlay interface 'eth0' towards the enterprise
default via 10.200.0.1 dev csc0  proto static <--- default route over provider interface 'csc0'
[root@rhel61-lnic ~]#

Verification and Solution: Delete the default route that directs traffic over the provider interface 'csc0'. For example, delete the default route over the provider interface for the following RedHat 6.1 VM:

[root@rhel61-lnic ~]# route delete default gw 10.200.0.1
[root@rhel61-lnic ~]# ip route list
168.63.129.16 via 10.200.0.1 dev csc0
168.63.129.16 via 10.200.0.1 dev csc0  proto static
10.2.0.0/24 dev eth0  proto kernel  scope link  src 10.2.0.11
169.254.0.0/16 dev csc0  scope link  metric 1002
169.254.0.0/16 dev eth0  scope link  metric 1003
10.200.0.0/16 dev csc0  proto kernel  scope link  src 10.200.0.5
default via 10.2.0.75 dev eth0 <---- Now there is only one default route via the overlay interface 'eth0'
[root@rhel61-lnic ~]#

Issues with Windows Cloud VMs

This section includes symptoms, possible causes, and solutions for issues associated with Windows VMs on public clouds.

Symptom: There is no connectivity to the Windows cloud VM.

Possible Cause: The cause can be due to memory leaks, the necessary ports not being open, or a variety of other errors.

Verification and Solution:

1. Check the Windows cloud VM reachability by using SSH and a public IP address, or by using RDP or ping if these have not been disabled in the cloud VM.

1. Check the cloud VM status in the EC2 console.

2. If Amazon Elastic Compute Cloud (EC2) status shows "2/2 checks passed" and "running," reboot the cloud VM from the EC2 console and check connectivity.

3. If connectivity is restored, check for any relevant errors in system event logs by using eventvwr.msc.

4. If the system event logs do not contain any obvious errors, a memory leak might exist. Use perfmon.exe to isolate any excessive non-paged pool usage during upcoming boot sessions.

2. Check the EC2 firewall settings for the Windows cloud VM and ensure that the necessary ports are open: TCP and UDP port 6644, and TCP port 22.

Symptom: There is an issue related to the Windows cloud VM subagent.

Possible Cause: The subagent process is not running or the subagent process is not listening to the correct TCP port.

Verification and Solution: Try any of the following solutions:

- Confirm that the sub_agent.exe file is present in the C:\Program Files\Cisco\ICA folder. If the file is not present, copy the file to the folder.

- In a command window, determine whether or not the subagent process is running.

c:\>tasklist /FI "imagename eq sub_agent.exe"
Image Name                     PID Session Name        Session#    Mem Usage
========================= ======== ================ =========== ============
sub_agent.exe                 2616 Services                   0      4,508 K

If the agent is not running, start it.

- Confirm that the subagent process is listening on TCP port 6644.

c:\>netstat -anB

Active Connections
...
TCP    0.0.0.0:6644           0.0.0.0:0              LISTENING
[sub_agent.exe]
...

Symptom: There is no connectivity to the SSH server.

Possible Causes:

- The vm_trust.properties file does not exist.

- The SSH server is not running.

- The SSH server is not listening to the correct TCP port.

- The SSH public key for the root user cannot be retrieved.

Verification and Solution: Try any of the following solutions:

- If the subagent process is running, but the process is not listening on TCP port 6644, determine whether or not the C:\Program Files (x86)\Cisco\vm_trust.properties file exists.

If vm_trust.properties file is absent, determine whether or not the SSH server is running:

c:\>sc query freesshdservice
SERVICE_NAME: freesshdservice
TYPE               : 110  WIN32_OWN_PROCESS  (interactive)
STATE              : 4  RUNNING
(STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)
WIN32_EXIT_CODE    : 0  (0x0)
SERVICE_EXIT_CODE  : 0  (0x0)
CHECKPOINT         : 0x0
WAIT_HINT          : 0x0

If the SSH server is not running, confirm that the SSH server directory is populated with the necessary files:

c:\>dir "C:\Program Files (x86)\Cisco"\
Volume in drive C has no label.
Volume Serial Number is 7C21-C2FC
Directory of C:\Program Files (x86)\Cisco
07/16/2013  05:23 AM    <DIR>          .
07/16/2013  05:23 AM    <DIR>          ..
07/16/2013  02:28 AM               672 DSAKey.cfg
07/16/2013  02:28 AM               256 freesshd.log
03/29/2013  07:33 PM         1,360,896 FreeSSHDService.exe
07/16/2013  02:28 AM             1,096 FreeSSHDService.ini
07/16/2013  02:27 AM               386 root

- Determine whether nor not the SSH server is listening on TCP port 22 and SSH connections can be established:

c:\>netstat –anB

Active Connections
...
TCP    0.0.0.0:22             0.0.0.0:0              LISTENING
[FreeSSHDService.exe]
...

- Determine whether or not the SSH public key for the root user can be retrieved successfully using the wget.exe command from the EC2 VM user data:

C:\Program Files\Cisco\ICA\wget.exe -t 1 --bind-address [peth IP] -O C:\Windows\Temp\ec2pubkey http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key

Symptom: There is an issue related to the Intercloud Agent Service (ICASVC).

Possible Cause:

- The ICASVC is not installed.

- The ICASVC is not running.

- The ICASVC is not set to the AUTO_START startup type.

Verification and Solution: Try any of the following solutions:

- If the ICASVC is not installed, determine whether or not the following registry entry is present:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\ICASvc]
"Type"=dword:00000010
"Start"=dword:00000002
"ErrorControl"=dword:00000001
"ImagePath"=hex(2):22,00,43,00,3a,00,5c,00,50,00,72,00,6f,00,67,00,72,00,61,00,\
6d,00,20,00,46,00,69,00,6c,00,65,00,73,00,5c,00,43,00,69,00,73,00,63,00,6f,\
00,5c,00,49,00,43,00,41,00,5c,00,49,00,63,00,61,00,53,00,76,00,63,00,2e,00,\
65,00,78,00,65,00,22,00,00,00
"DisplayName"="Cisco InterCloud Agent Service"
"ObjectName"="LocalSystem"
"Description"="Cisco InterCloud Agent Service"

If the registry entry is absent, import the ICASVC key into HKLM\System\CCS\Services.

- Determine whether or not the ICASVC is set to the AUTO_START startup type:

c:\>sc qc icasvc
[SC] QueryServiceConfig SUCCESS
SERVICE_NAME: icasvc
TYPE               : 10  WIN32_OWN_PROCESS
START_TYPE         : 2   AUTO_START
ERROR_CONTROL      : 1   NORMAL
BINARY_PATH_NAME   : "C:\Program Files\Cisco\ICA\IcaSvc.exe"
LOAD_ORDER_GROUP   :
TAG                : 0
DISPLAY_NAME       : Cisco InterCloud Agent Service
DEPENDENCIES       :
SERVICE_START_NAME : LocalSystem

Symptom: The ICASVC startup scripts are not running.

Possible Cause: The startup scripts are not present in the correct folder.

Verification and Solution: Confirm that the StartIca.vbs, StartSubagent.vbs, and start_subagent.bat files are present under the C:\Program Files\Cisco\ICA folder.

Symptom: The cloud VM has lost connectivity with the private data center.

Possible Cause: The certificate is not installed or is not correct.

Verification and Solution:

1. Confirm that the ctdrv.sys and ctmp.sys files are present in the C:\Program Files\Cisco\ICA folder.

2. Confirm that the Windows ICA driver certificate is installed:

C:\Program Files\Cisco\ICA>CertMgr.exe /s TrustedPublisher
==============Certificate # N ==========
Subject::
[0,0] 2.5.4.6 (C) US
[1,0] 2.5.4.8 (S) California
[2,0] 2.5.4.7 (L) San Jose
[3,0] 2.5.4.10 (O) Cisco Systems, Inc
[4,0] 2.5.4.11 (OU) Digital ID Class 3 - Microsoft Software Validation v2
[5,0] 2.5.4.3 (CN) Cisco Systems, Inc
…

Symptom: Encrypted/encapped packet injection is not working.

Possible Cause: Weakhost sends and receives are enabled.

Verification and Solution: For the encrypted/encapped packet injection to work, the provider adapter must have weakhost sends and receives disabled, which is the default setting.

1. Determine whether weakhost sends and receives are enabled or disabled:

> netsh interface ipv4 show interfaces level=verbose
Interface Local Area Connection 2 Parameters
...
Weak Host Sends                    : disabled
Weak Host Receives                 : disabled
...

2. If weakhost sends and receives are enabled, disable them:

> netsh interface ipv4 set interface [InterfaceNameOrIndex] weakhostsend=disabled
> netsh interface ipv4 set interface [InterfaceNameOrIndex] weakhostreceive=disabled

Symptom: The access tunnel with the Windows VM is broken.

Possible Cause: Third-party Windows Filtering Platform (WFP) filters are blocking datagrams on TCP port 6644 or the UDP port over which the subagent has initiated the Datagram Transport Layer Security (DTLS) connection.

Verification and Solution: Enter the following command and examine the output:

> netstat -anB

- To determine whether or not the UDP port is causing the problem, look for the subagent.exe port.

- To determine whether or not third-party WFP filters are causing the problem, look at the sysinfo section of the output for WFP filters for Windows VMs.

To resolve the issue, restart the services.

Symptom: There is an issue with the data path between the overlay miniport and the cloud provider miniport.

Possible Cause: Third-party Network Driver Interface Specification (NDIS) intermediate drivers or additional NDIS miniports are affecting the data path between the overlay miniport and the cloud provider miniport.

Verification and Solution: Enter the following command and examine the output:

> netstat -anB

All NDIS drivers are enumerated in the sysinfo section of the output for Windows VMs.

Symptom: cVM deployment fails for one of the two interfaces.

Possible Cause: By default, port profiles are named "icfCloud" if a name is not provided during the Intercloud Fabric tenant configuration.

Verification and Solution: When deploying cVM, make sure you have a network policy defined with port profiles for each vNIC that is on the same tenant. When modifying the port profiles after cVM deployment, ensure to configure the appropriate tenant on the port profiles that does not violate the deployment solution above.

Symptom: Migration of a Windows 2012 VM or import of a Windows 2012 VM OVA fails with error "VM Guest OS was not shutdown gracefully. Please resume and shutdown Windows fully prior to initiating import or move actions on this VM image".

Possible Cause: ICF detected that the VM was not shutdown gracefully prior to the user initiating a VM migration or exporting it as an OVA from vCenter. Not shutting down the Guest OS gracefully can lead to corruption of the file system.

Verification and Solution: Power on your VM on vCenter and gracefully shutdown the Guest OS. Reinitiate your VM migration or export of OVA from vCenter.

Issues with Monitoring VMs

This section includes symptoms, possible causes, and solutions for issues associated with monitoring VMs.

Symptom: The VM statistics in the end-user portal show VMs that do not exist.

Possible Cause: The VMs were deleted from the VMware vCenter client but, because the periodic refresh has not occurred, the VMs still appear.

Verification and Solution: Manually refresh the VM list.

Chapter 6. Troubleshooting Intercloud Fabric Components

- Issues with the Intercloud Fabric Virtual Supervisor Module

- Issues with Prime Network Services Controller

- Issues with the Intercloud Fabric Switch

- Issues with the Intercloud Fabric Router

Issues with the Intercloud Fabric Virtual Supervisor Module

This section includes symptoms, possible causes, and solutions for issues associated with the Cisco Intercloud Fabric virtual supervisor module (VSM).

Symptom: In the output of the show module command, the Intercloud Fabric Switch modules are not shown as online.

Possible Cause: The Intercloud Fabric Switch is not registered with the Cisco Intercloud Fabric VSM due to one of the following reasons:

- The Intercloud Fabric Switch management VLAN is missing from the IcfCloud Trunk port profiles. the port profiles are named IcfCloud_Name_ICS_Trunk_Tunnel, IcfCloud_Name_ICX_Trunk_Tunnel, IcfCloud_Name_ICX_Trunk_Tunnel_Enterprise.

- The Intercloud Fabric Switch management VLAN is NOT identified as the "system vlan" for the IcfCloud Trunk port profiles. The port profiles are named IcfCloud_Name_ICS_Trunk_Tunnel, IcfCloud_Name_ICX_Trunk_Tunnel, IcfCloud_Name_ICX_Trunk_Tunnel_Enterprise.

Verification and Solution: Manually add the VLANs configuration interface in PNSC.

Symptom: There is no traffic from the private VM to the cloud VM.

Possible Cause:

- The cloud VM VLAN is missing from the IcfCloud Trunk port profiles. The port profiles are named IcfCloud_Name_ICS_Trunk_Tunnel, IcfCloud_Name_ICX_Trunk_Tunnel, IcfCloud_Name_ICX_Trunk_Tunnel_Enterprise.

Verification and Solution: Manually add the VLANs via the VSM configuration interface in PNSC.

Issues with Prime Network Services Controller

This section includes symptoms, possible causes, and solutions for issues associated with Prime Network Services Controller.

Symptom: The UI task status displays a failure message.

Possible Cause: The possible causes vary.

Verification and Solution: Check the resource manager logs that are located under /var/log/resource-mgr. The svc_res_dme.log log file is for UI-to-Resource Manager communication. The svc_res_cloudproviderAG.log log file is for AG-to-CPM communication.

Symptom: The access tunnel has a failure.

Possible Cause: The possible causes vary.

Verification and Solution: Check the logs on the Intercloud Fabric Switch for the cause.

Symptom: The Intercloud image fails to import into Prime Network Services Controller.

Possible Cause: Prime Network Services Controller encountered an error.

Verification and Solution: In the Prime Network Services Controller GUI, check the Recent Jobs list for the status and an error message if the import operation does not complete within a reasonable amount of time. The Task line item lists the last error encountered, if any, while importing and the number of retries attempted.

Symptom: After restarting Prime Network Services Controller services, the job for importing an Intercloud image is displayed as running, but the job does not progress.

Possible Cause: If Prime Network Services Controller services are restarted while an import operation is running, the task will not resume.

Verification and Solution: Cancel the job in the Intercloud Fabric Director and import the image again.

Symptom: The Intercloud Fabric Director cannot validate Amazon Web Services (AWS) credentials.

Possible Cause:

- Not all of the required ports are open.

- The DNS server is not reachable from Prime Network Services Controller.

- The NTP server is not reachable from Prime Network Services Controller.

Verification and Solution:

1. Make sure that all required ports are open:

- TCP ports—22, 443, 3389, 6644, and 6646

- UDP ports—6644 and 6646

2. Make sure that the DNS service is configured and that the DNS server is reachable from Prime Network Services Controller.

3. Make sure that the NTP service is configured and that the NTP server is reachable from Prime Network Services Controller.

Symptom: Integrated Gateway is unreachable from Prime Network Services Controller.

Possible Cause: After restarting Prime Network Services Controller, or restarting Intercloud Fabric Virtual Machine, the Integrated Gateway is unreachable.
Verification and Solution:

1. SSH to Prime Network Services Controller and enter the following details:

PNSC# connect local-mgmt
PNSC(local-mgmt)# service restart

2. Log in to Prime Network Services Controller and check if the Integrated Gateway state changes to reachable/running.

Note

If the Integrated Gateway state does not change to reachable/running, change the Integrated Gateway configuration parameters, such as the description. This will change the Integrated Gateway state to reachable/running by attempting a new connection.

Symptom: IcLink is deployed but the Configuration status remains "Applying" and the Tunnel status shows "not operational". Likewise, when ICS is successfully deployed and gets a Private IP (with the Configuration status remaining "Applying"), it implies that ICS is not reachable from Prime Network Services Controller.

Possible Cause: Prime Network Services Controller should be behind NAT (for Prime Network Services Controller to ICS communication).

Verification and Solution: The route on ICS needs to be modified or deleted (Prime Network Services Controller should be able to reach ICS via Provider Private Gateway).

Issues with the Intercloud Fabric Switch

This section includes symptoms, possible causes, and solutions for issues associated with the Intercloud Fabric Switch (ICS).

Symptom: The ICS fails to power up and Amazon Web Services (AWS) displays a "401 Unauthorized" error.

Possible Cause: The credentials were changed or the access key was deleted on AWS.

Verification and Solution: Enter the correct credentials for the provider account. For information about entering the credentials, see the Amazon Web Services documentation.

Symptom: ICS does not appear under "show module" for the registered cVSM.

Possible Cause: VSM reachability from ICS over Management Interface not working.

Verification and Solution: When using different VLANs for ICS, either add a static route for that subnet or make sure there is a reachability to that subnet through the default gateway. Configuration from the ICFD UI: Intercloud > Network > IcfVSM and highlight the VSM and choose "Configure Route".

Symptom: Packets arriving at the ICX from the enterprise are not sent toward the site-to-site tunnel to the ICS.

Possible Cause: The possible causes vary.

Verification and Solution: To debug packets lost, enter the following commands:

show intercloud tunnel brief
show intercloud tunnel config

vemcmd show port
vemcmd show port vlans
vemcmd show stats
vemcmd show port-drops ingress/egress
show system internal tunnel-manager dp
show system internal event-log tunnel manager

Symptom: Unknown unicast packets, with MACs that are not in the MAC Pool, are not forwarded over the site-to-site tunnel.

Possible Cause: On the ICX, unknown unicast packets forwarded from the enterprise to the cloud over the site-to-site tunnel will be dropped if their destination MAC address is not a part of the MAC Pool for the ICLink.

Verification and Solution: Use this command to display the feature status for the blocking of unknown unicast MACs that are not in the MAC Pool:

vemcmd show macpool veth1-0

This command is applicable on the ICX only. The feature is always on by default. Contact customer support if you need to disable this feature.

Issues with the Intercloud Fabric Router

This section includes symptoms, possible causes, and solutions for issues associated with the Intercloud Fabric Router.

Symptom: The Intercloud Fabric Router remains in a creating or unreachable state as displayed in Prime Network Services Controller.

Possible Cause:

- A process crashed or failed to run.

- The Intercloud Fabric Router policy agent is down.

- The tunnel status between the Intercloud Fabric Router and Intercloud Fabric Switch is down.

- The control channel between Intercloud Fabric Router and Intercloud Fabric Switch is down.

- The Intercloud Fabric Router management sub-interface is down.

- Prime Network Services Controller is not reachable from the Intercloud Fabric Router.

Verification and Solution: Use the Amazon Web Services (AWS) management console to determine whether or not the Intercloud Fabric Router is in a running state and confirm that two out of two checks passed. Reaching the two-for-two checks-passed state can take up to 20 minutes from the time that you checked the management console. If the Intercloud Fabric Router is functioning properly, the following message is displayed:

Waiting up to 1800 sec for sub_agent to add CSR to CGp Kumo sub_agent init ok, CSR is added to CGp.

If the Intercloud Fabric Router is not working properly, the following message is displayed:

Waiting up to 1800 sec for sub_agent to add CSR to CGp Cannot get sub_agent state file)

If the Intercloud Fabric Router is not in a running state in AWS, complete the following steps:

1. In the system console log, determine whether or not any process has crashed or failed to run, such as a uCode process fault.

2. Using SSH, connect to the Intercloud Fabric Router public IP address as provided in AWS and verify that the Intercloud Fabric Router policy agent (which communicates with Prime Network Services Controller) is up:

# show remote-management status

3. Log in to the Intercloud Fabric Switch and verify that the tunnel status between the Intercloud Fabric Router and Intercloud Fabric Switch is up:

# show intercloud tunnel brief

4. From the Intercloud Fabric Switch, verify that the control channel between Intercloud Fabric Router and Intercloud Fabric Switch is up:

# show intercloud ctrl-channel connections

5. Verify that the Intercloud Fabric Router management sub-interface that was configured by using Prime Network Services Controller is up and that the IP address is configured:

# show ip interface brief

6. Verify that Prime Network Services Controller is reachable from the Intercloud Fabric Router by using the ping command:

# ping pnsc-ip-address

Symptom: After a configuration change, the Intercloud Fabric Router displays a "Failed to Apply" error message in Prime Network Services Controller.

Possible Cause: The configuration contains an invalid value.

Verification and Solution:

To verify the problem:

1. In the Prime Network Services Controller GUI, choose the Intercloud Fabric Router and click Edit.

2. Click the Fault tab.

3. View the error message to identify the cause.

To solve the problem:

1. In the Prime Network Services Controller GUI, correct the configuration.

2. Confirm that the Intercloud Fabric Router can reach the "running" state.

Symptom: Connectivity between a private VM and a cloud VM in different VLANs fails intermittently.

Possible Causes:

- A Nexus 1000V is used in the private data center with Unknown-Unicast-Flooding-Block enabled.

- The private VM and cloud VM are in different VLANs, thereby causing inter-VLAN routing to fail.

Verification and Solution:

1. Confirm that packets are being dropped by entering the following command on the VEM:

vemcmd show port-drops

2. Determine whether Unknown-Unicast-Flooding-Block is enabled or disabled by entering the following command on the Nexus 1000V:

privateVSM# show run | include uufb
uufb enable

3. If Unknown-Unicast-Flooding-Block is enabled, disable it by entering the following command in config mode:

no uufb enable

If Unknown-Unicast-Flooding-Block is already disabled, see other topics in this section for more information about troubleshooting inter-VLAN routing issues.

Symptom: For cloud VMs whose default gateway points to the Intercloud Fabric Router that is in the cloud, inter-VLAN routing does not work, but communication within the same VLAN does work.

Possible Cause:

- The Intercloud Fabric Router sub-interface IP addresses are incorrect.

- The default gateway for the cloud VM does not point to the Intercloud Fabric Router sub-interface IP address.

- The Intercloud Fabric Router does not have the appropriate routes.

- The Intercloud Fabric Switch is routing a VLAN that is not in the list of allowed VLANs in the Nexus 1000V default trunk.

- The cloud VM cannot reach the default gateway.

Verification and Solution:

1. In the Prime Network Services Controller GUI, ensure that the Intercloud Fabric Router sub-interfaces are configured with the correct IP address.

2. Log in to the cloud VM and verify that its default gateway points to the Intercloud Fabric Router sub-interface IP address (from Step 1).

3. Log in to the Intercloud Fabric Router and verify that the appropriate routes are present.

4. Log in to the Intercloud Fabric Switch and verify that the VLAN being routed is in the list of allowed VLANs in the Nexus 1000V default trunk:

# vemcmd show port
# vemcmd show port vlans

5. Verify that the cloud VM can reach the default gateway, which is the Intercloud Fabric Router sub-interface.

Symptom: For cloud VMs whose default gateway points to the Intercloud Fabric Router that is in the private data center, inter-VLAN routing does not work, but communication within the same VLAN does work.

Possible Cause:

- The Intercloud Fabric Router sub-interface IP addresses are incorrect.

- The Intercloud Fabric Switch is routing a VLAN that is not in the list of allowed VLANs in the Nexus 1000V default trunk.

- The Intercloud Fabric Router does not have the appropriate routes.

- The default gateway for the cloud VM is an Intercloud Fabric Router that is not in the cloud.

- The cloud VM cannot reach the default gateway.

- Inter-VLAN routing is not working.

Verification and Solution:

1. In the Prime Network Services Controller GUI, confirm that the Extend Default Gateway option is checked for the Intercloud Fabric Router sub-interface and that the IP addresses are correct.

2. Log in to the Intercloud Fabric Switch and verify that the VLAN being routed is in the list of allowed VLANs in the Nexus 1000V default trunk:

# vemcmd show port
# vemcmd show port vlans

3. From the Intercloud Fabric Router, confirm that the IP address from Step 1 is displayed in the output of the following command:

# vemcmd show arp all

4. Log in to the Intercloud Fabric Router and verify that the appropriate routes are present.

5. In the Prime Network Services Controller GUI, uncheck the Extend Default Gateway box.

6. Log in to the cloud VM and change the default gateway to the Intercloud Fabric Router that is in the cloud.

7. Verify that the cloud VM can reach the default gateway, which is the Intercloud Fabric Router sub-interface.

8. Verify that inter-VLAN routing works.

Chapter 7. Using Intercloud Fabric Component Commands and Logs

- Using Intercloud Fabric Director Logs

- Using Intercloud Fabric Director Service Request Logs and Details

- Using Cisco Intercloud Fabric Troubleshooting Commands

- Using Prime Network Services Controller Logs

- Using Prime Network Services Controller Troubleshooting Commands

- Using Intercloud Fabric Firewall Logs

- Using Windows Cloud VM Logs

- Using Windows Cloud VM Troubleshooting Commands

Using Intercloud Fabric Director Logs

You can use Intercloud Fabric Director logs to assist in troubleshooting. The logs provide error details, which are required for debugging.

Procedure

Step 1

Log in to Cisco Intercloud Fabric.

Step 2

Choose Administration > Support Information.

Step 3

Complete the following fields:

Field Name

Description

Support Information drop-down list

Choose the information to view:

- System Information (Basic)—Displays the status of different components within Cisco Intercloud Fabric.

- System Information (Advanced)—Displays a more detailed status of different components within Cisco Intercloud Fabric.

- Show Log—Displays the last few lines of the selected Intercloud Fabric Director log file.

- Download All Logs—Downloads all of the Intercloud Fabric Director log files in a zip file.

- Debug Logging—Displays debug logging information.

Show Log drop-down list

This option appears if you choose Show Log in the Support Information drop-down list.

Choose the type of Intercloud Fabric Director log to view:

- Infra Manager—Infrastructure manager log. You will use this log more often than the other logs.

- Web Context Cloud Manager

- Tomcat Log

- Authenticator Log

- Mail Delivery Log

- Patch Log

Step 4

Click Submit, Show Logs, or Download, as appropriate.

Using Intercloud Fabric Director Service Request Logs and Details

You can use service request logs to assist in troubleshooting. The logs provide detailed descriptions of all current operations, including progress, percentage complete, and error messages in the case of a failure. You can also view input or output details for additional information.

Procedure

Step 1

Log in to Cisco Intercloud Fabric.

Step 2

Choose Organization > Service Requests.

Step 3

Choose the Service Requests tab.

Step 4

Choose a service request.

Step 5

Click View Details.

Step 6

Choose either or both of the following:

- Log tab—Displays the logs for the chosen service request.

- Input/Output tab—Displays all input and output for the chosen service request.

Step 7

(Optional)Click Refresh to view the latest information.

Using Cisco Intercloud Fabric Troubleshooting Commands

You can use the commands in the following table to troubleshoot issues associated with Cisco Intercloud Fabric.

Commands

Description

show module

Displays module information.

show port-profile brief

Displays the brief version of the port profile configuration.

show running-config port-profile

Displays the port profile configuration, including interface assignments.

show intercloud tunnel brief

Displays the brief version of the Cisco Intercloud Fabric tunnel configuration.

show intercloud tunnel port

Displays the Cisco Intercloud Fabric tunnel port configuration.

show intercloud vm vm_name

Displays the Cisco Intercloud Fabric VM configuration.

show system internal event-log interface-mgr

Displays the interface manager event log system information.

show platform software vnic-if interface-map

Displays the mapping between the logical interface on the router with the vNIC and the vNIC MAC address.

show internal {gi1|gi8} controller

Displays information about the Gi1 or Gi8 controller.

show remote-management status

Displays the Intercloud Fabric Router remote management settings.

show virtual-service

Displays an overview of resources used by virtual service containers.

tail -f /bootflash/intflib_log.txt

Displays the tail of the bootflash log.

vemcmd show arp all

Displays the ARP table on the Virtual Ethernet Module (VEM).

vemcmd show arp filter

Displays the ARP filters on the VEM.

vemcmd show port

Displays information about ports on the VEM to verify that the ports of the host added to the DVS are listed, and that the ports are correctly configured as access or trunk on the host.

Using Prime Network Services Controller Logs

You can use Prime Network Services Controller logs to assist in troubleshooting issues with Prime Network Services Controller.

Procedure

Step 1

In Intercloud Fabric, choose Intercloud > Infrastructure.

Step 2

Click Download PNSC Logs and confirm the operation.
The download starts and progress is displayed. The download can take a few minutes to complete.

Step 3

When prompted, choose a location for the tgz file.

Step 4

Use a desktop tool to unzip or untar the downloaded file.

Using Prime Network Services Controller Troubleshooting Commands

Prime Network Services Controller provides commands that can help you troubleshoot issues.

To enter Prime Network Services Controller troubleshooting commands, log in to the console as described in this procedure.

Procedure

Step 1

In the VMware vSphere client, select the VM on which is deployed.

Step 2

Right-click the VM and choose Open Console.

Step 3

When prompted, enter the admin username and password.

Step 4

Enter a command from the following table to obtain the information you need.

Command

Description

show intercloud

Displays Cisco Intercloud Fabric information, which includes Prime Network Services Controller information.

show tech-support

Creates a tgz archive file that contains the necessary logs to send to technical support to get assistance with troubleshooting.

Using Intercloud Fabric Firewall Logs

The Intercloud Fabric Firewall has several logs that you can access to troubleshoot various issues.

Component with an Issue

Log Location

UI for resource manager communication

/var/log/resource-mgr/svc_res_dme.log

AG-to-Cisco Policy Management (CPM) communication

/var/log/resource-mgr/svc_res_cloudproviderAG.log

Access tunnel

Intercloud Fabric Switch logs

Policy agent registration with Prime Network Services Controller

Intercloud Fabric Firewall logs

Policy configuration

/var/log/policy-mgr

Using Windows Cloud VM Logs

You can use Windows cloud VM logs to assist in troubleshooting issues with the StartIca and StartSubagent VBScript startup scripts.

Procedure

Step 1

Log in to the Windows cloud VM machine.

Step 2

Navigate to the C:\Windows\Temp folder.

Step 3

Check the icasvc.log log file for errors related to the scripts.

Step 4

Check the startica.log log file for additional errors related to the scripts. For the first boot in EC2, the startica.vbs script performs several tasks that are often the sources of issues.

1. Check the log for successful entries about the following tasks:

- Installation of the driver certificate

- Installation of ctdrv.sys

- Extraction of the SSH public key using the wget command

- Installation of the freeSSHd server

- Initialization of the ctdrv driver service

2. For subsequent boots, because the main script task is to initialize the ctdrv driver service, check the log for the successful initialization of the ctdrv driver service.

Step 5

Ensure that the SSH service is running:

c:\>netstat -na

Using Windows Cloud VM Troubleshooting Commands

You can use the commands in the following table to troubleshoot issues associated with Windows cloud VMs from the Intercloud Fabric Switch.

Commands

Description

show intercloud vm vm_name log

Connects to the specified VM and obtains the user-mode-related logs. You can access the logs at bootflash.

show intercloud vm vm_name kernel-log

Connects to the specified VM and obtains the tunnel driver logs. You can access the logs at bootflash.

show running-config port-profile

Displays the port profile configuration, including interface assignments.

Chapter 8. Obtaining Help from Cisco

- Cisco Intercloud Fabric Support Community

- Contacting Cisco Customer Support

- Obtaining Logs for Cisco Customer Support

Cisco Intercloud Fabric Support Community

For more information about troubleshooting issues with Cisco Intercloud Fabric, visit the Cisco Intercloud Fabric support community:

Click Here

Contacting Cisco Customer Support

If you cannot solve a problem after using the troubleshooting suggestions in this guide, contact a customer service representative for assistance and further instructions. Before contacting Cisco, collect the following information so that Cisco Customer Support can assist you as quickly as possible:

- Version of Cisco Intercloud Fabric software that you are running

- Contact phone number

- Brief description of the problem

- Brief explanation of the steps you have already taken to isolate and resolve the problem

After collecting this information, see Obtaining Documentation and Submitting a Service Request for information on how to submit a service request.

Obtaining Logs for Cisco Customer Support

If you contact Cisco Customer Support for assistance, you might need to provide logs for them to assist in determining the cause of the problem. This procedure describes how to obtain logs for Cisco Intercloud Fabric.

Procedure

To obtain Intercloud Fabric logs:

1. In Intercloud Fabric, choose Administration > Support Information.

2. From the Support Information drop-down list, choose Show Log.

3. From the Show Log drop-down list, choose Infra Manager and click Show Logs.

4. Save the information in a file that you can send to Cisco Customer Support.

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS" WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: Click Here . Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R)