Chapter 1. Overview

About Cisco Intercloud Fabric

Overview of the Troubleshooting Process

Overview of Best Practices

Contacting Cisco

Chapter 2. Troubleshooting the Cisco Intercloud Fabric Installation and Initial Setup

Issues with the Infrastructure Setup

Chapter 3. Troubleshooting the Cisco Intercloud Fabric Upgrade

Issues with the Intercloud Fabric Upgrade

Chapter 4. Troubleshooting the Cisco Intercloud Fabric Cloud Setup

Issues with the Cloud Setup

Deleting the Intercloud Fabric Cloud Setup

Issues with the Intercloud Fabric Link

Issues Creating the Intercloud Fabric Link

Issues with the Intercloud Fabric Link Status Operation

Issues Updating the Intercloud Fabric Link

Issues Starting the Intercloud Fabric Link

Issues Stopping the Intercloud Fabric Link

Issues Rebooting the Intercloud Fabric Link

Issues Deleting the Intercloud Fabric Link

Issues with the Intercloud Fabric Link Switchover Operation

Chapter 5. Troubleshooting VM Lifecycle Management

Issues with Image Creation

Issues with Linux Cloud VMs

Issues with Windows Cloud VMs

Issues with Monitoring VMs

Chapter 6. Troubleshooting the Cisco Intercloud Fabric Components

Issues with the Intercloud Fabric Virtual Supervisor Module

Issues with Intercloud Fabric Controller

Issues with the Intercloud Fabric Switch

Issues with the Intercloud Fabric Routing Service

Issues Enabling and Configuring Intercloud Fabric Routing Service

Issues Disabling Intercloud Fabric Routing Service

Issues Connecting Intercloud Fabric Routing Service to Networks

Issues Disconnecting Intercloud Fabric Routing Service from Networks

Issues with Intercloud Fabric Routing Policy

Issues with Intercloud Fabric Virtual Data Center

Issues with Intercloud Fabric Routing Service Internal Errors

Issues with the Intercloud Fabric Routing Service Interface

Issues Enabling and Configuring Intercloud Fabric Routing Service Interface

Issues Disabling Intercloud Fabric Routing Service Interface

Issues Connecting Intercloud Fabric Routing Service Interface to Networks

Issues Disconnecting Intercloud Fabric Routing Service Interface from Networks

Issues with Intercloud Fabric Routing Service Interface Internal Creation Errors

Issues with Intercloud Fabric Routing Service Interface Internal Deletion Errors

Issues with the Intercloud Fabric Advanced Routing Service

Issues Enabling and Configuring Intercloud Fabric Advanced Routing Service

Issues Disabling Intercloud Fabric Advanced Routing Service

Issues Connecting Intercloud Fabric Advanced Routing Service to Networks

Issues Configuring Intercloud Fabric Advanced Routing Service Static Route

Chapter 7. Obtaining Help from Cisco

Cisco Intercloud Fabric Support Community

Contacting Cisco Customer Support

Obtaining Logs for Cisco Customer Support

Chapter 8. Additional Information

Related Documentation

Documentation Feedback

Obtaining Documentation and Submitting a Service Request

Copyright Page

Chapter 1. Overview

- About Cisco Intercloud Fabric

- Overview of the Troubleshooting Process

- Overview of Best Practices

- Contacting Cisco

About Cisco Intercloud Fabric

Cisco Intercloud Fabric provides a faster and flexible response to business needs and addresses the potential challenges with hybrid clouds. A hybrid cloud is an interaction between private and provider clouds where private clouds extend to provider clouds and use provider cloud resources in a secure and scalable way. Intercloud Fabric enables you to place workloads across heterogeneous environments in multiple provider clouds. Intercloud Fabric provides the architectural foundation for secure hybrid clouds, which allows enterprises to easily and securely connect the private clouds to the provider cloud as needed and on demand. Intercloud Fabric provides the following benefits:

- Provides a single point of management and control for virtual workloads across multiple provider clouds.

- Provides a choice of cloud providers, such as Amazon Web Services, Microsoft Azure, and multiple Intercloud Fabric provider-based clouds.

- Provides highly secure, scalable connectivity to extend private clouds to provider clouds.

- Enforces consistent network and workload policies throughout the hybrid cloud.

Cisco Intercloud Fabric

Overview of the Troubleshooting Process

Follow these steps to troubleshoot your network:

1. Gather information that defines the specific symptoms.

2. Identify all potential causes for the symptoms.

3. Systematically eliminate each potential cause (from most likely to least likely) until the symptoms disappear.

Overview of Best Practices

Best practices are the recommended steps you should take to ensure the proper operation of your appliance. We recommend the following best practices for most networks:

- Maintain a consistent Cisco Intercloud Fabric release across all network devices.

- Refer to the release notes for your Cisco Intercloud Fabric release for the latest features, limitations, and bugs.

- Document private data center details, including the vNIC configuration used by virtual machines (VMs) and the corresponding vSwitch configuration to ensure that they are consistent with each other.

- Verify and troubleshoot any new configuration changes after implementing a change.

Contacting Cisco

If you cannot resolve a problem by using the information in this guide, contact Cisco Customer Support for help. For more information, see Contacting Cisco Customer Support.

Chapter 2. Troubleshooting the Cisco Intercloud Fabric Installation and Initial Setup

- Issues with the Infrastructure Setup

Issues with the Infrastructure Setup

This section includes symptoms, possible causes, and solutions for issues encountered while setting up the Intercloud Fabric infrastructure.

Symptom: An infrastructure update is not supported after infrastructure creation is aborted.

Error Message:

An infrastructure update is not supported after infrastructure creation is aborted.
Delete the existing infrastructure and recreate it. For details, see the Troubleshooting Guide.

Possible Cause: After infrastructure creation is aborted, an infrastructure update is not supported and will return an error if it is attempted.

Verification and Solution: Delete the existing infrastructure and recreate it. For details, see the Troubleshooting Guide.

Symptom: A service request remains stuck in a Not_Started or an In_Progress state.

Possible Cause: The symptom is caused by a known bug.

Verification and Solution:

If the service request remains in a Not_Started or an In_Progress state for an abnormal period of time, do the following:

1. SSH into the system console.

2. Restart services. This will result in moving the service request to a System_Aborted state.

After the service request is moved into a System_Aborted state, do the following:

1. Perform the operation again through the Intercloud Fabric UI, if allowed.

2. If the operation fails or is not permitted, you must first delete the object and then create it again.

Symptom: Logging on to Intercloud Fabric from the browser fails.

Possible Cause:

- The browser URL may be incorrect.

- Check for any services errors. In addition, log in as the administrator to the Intercloud Fabric VM CLI and check if all services are up on Intercloud Fabric by selecting option 3, Show Services Status.

Verification and Solution: Use the obtained information to resolve the issue.

Symptom: The Intercloud Fabric browser log in fails and the CLI Show Services Status (option 3) shows that services are not running.

Verification and Solution: To resolve, log in as an administrator to the Intercloud Fabric VM CLI and use option 4 to Start Services or option 5 to Stop Services.

Symptom: Intercloud Fabric OVA deployment fails.

Possible Cause: Perform the following steps to identify the cause:

1. In VMware vCenter, review the recent tasks.

2. Review the information highlighted in red for the possible cause of the failure.

3. Select both the Tasks and Events tab and review any event highlighted in red for the possible cause of the failure.

Verification and Solution: Based on the possible causes, debug the issue accordingly.

Symptom: The Intercloud Fabric VM fails to power on after deployment.

Possible Cause: The mandatory OVA parameter fields were not correctly entered.

Verification and Solution: Reenter the mandatory OVA parameter fields.

Symptom: The Intercloud Fabric UI, API, and CLI fails to launch and you are prompted to reenter OVA parameters in the console.

Possible Cause: The mandatory IP address fields in OVA contain a value of 0.0.0.0.

Verification and Solution: Update the mandatory IP address fields in OVA.

Symptom: An error occurs when attempting to upload a Cisco Intercloud Fabric image tar file.

Possible Cause: This issue occurs if you use a browser other than Chrome. If you are using Chrome, it is possible that the image tar file did not upload completely.

Verification and Solution:

1. If you are not using the Chrome browser, use Chrome and try again.

2. If you encounter the error again, enable the root account by using the shelladmin credentials. The password for the shelladmin account is the same password that was entered when deploying the Intercloud Fabric appliance.

3. Use the scp command to copy the tar file into the /opt/infra/uploads folder of Intercloud Fabric.

4. On the Images page of the Infrastructure Setup dialog, click Browse and select the file from the local desktop.

5. Click Next.

Note

Do not click the Upload button.

6. Proceed with the remainder of the setup.

Symptom: The Intercloud Fabric image fails to successfully import into Intercloud Fabric Controller (ICFC).

Error Message:

Handler failed with error - Image Upload Failed: End point timed out.
Check for IP, password, space, or access related issues.

Possible Cause: ICFC cannot connect to Cisco Intercloud Fabric.

Verification and Solution: If the import operation does not complete within a reasonable amount of time, check the Recent Jobs list in ICFC for the status and any error messages. The Task line item lists the last error encountered, if any, while importing and the number of retries attempted.

Symptom: After restarting ICFC services, the job for importing an Intercloud Fabric image shows as running, but the job does not progress.

Possible Cause: If ICFC services are restarted while the import operation is running, the task does not resume.

Verification and Solution: Cancel the job in Intercloud Fabric and import the image again.

Symptom: The ICFC server cannot resolve hostnames to IP addresses.

Error Message:

ICFC cannot reach Public IP address

Possible Cause: The DNS server IP address is not entered correctly in the device profile. The DNS server resolves the hostname to an IP address. If ESX hosts were added to the VMware vCenter client using hostnames instead of IP addresses, those hostname-to-IP address mappings must exist in the DNS server.

Verification and Solution: Ensure that the DNS server IP address is entered correctly in the device profile, as follows:

1. In the Policies tab, choose Intercloud Infrastructure Policies.

2. If the DNS server IP address is incorrect, click Edit and enter the correct IP address.

For more information about configuring the DNS server IP address, see the Cisco Intercloud Fabric Administrator Guide.

Symptom: Intercloud Fabric displays an error regarding the shared secret.

Possible Cause: A shared secret that exceeds 11 characters was entered in Intercloud Fabric.

Verification and Solution: Ensure that the shared secret does not exceed 11 characters.

Symptom: While deploying an Intercloud Fabric OVF image, VMware vSphere displays the error, "The OVF package is invalid and cannot be deployed. The provided network mapping between OVF networks and the system network is not supported by any host."

Possible Cause: The VMware distributed virtual switch (DVS) is out of synchronization.

Verification and Solution: Ensure that the VMware DVS is properly synchronized with all hosts. For information about synchronizing the VMware DVS, see the VMware vSphere documentation.

Symptom: When using a VMware virtual switch to host the Intercloud Fabric Extender, any of the following occur:

- Traffic is lost.

- Duplicate packages are sent in private cloud and cloud VM traffic.

- Flapping occurs between the Intercloud Fabric Switch module and the Intercloud Fabric VSM.

Possible Cause: If Intercloud Fabric Extender is hosted on a VMware vSwitch or distributed switch and if the vSwitch or distributed switch is connected to multiple physical NICs, you must enable the setting Net.ReversePathFwdCheckPromisc=1 in the ESX host where the Intercloud Fabric Extender is hosted. This situation can occur if the setting Net.ReversePathFwdCheckPromisc=1 is not enabled in the ESX host where the Intercloud Fabric Extender resides.

Verification and Solution: Enable the setting Net.ReversePathFwdCheckPromisc=1 in the ESX host where the Intercloud Fabric Extender is hosted. The setting is found under Host > Configuration > Advanced Settings > Net in the VMware vSphere GUI.

Note

If the value of the Net.ReversePathFwdCheckPromisc configuration option is changed while the ESXi host is running, toggle (disable then re-enable) the Promiscuous Mode check box in the Intercloud Fabric Extender trunk port group security settings for the change to take effect.

- For a VMware virtual switch, set the trunk port group to allow All VLAN IDs in the VMware vSphere GUI.

- For a security policy for the trunk port group on the VMware virtual switch, set the Promiscuous Mode, MAC Address Changes, and Forged Transmits to Accept in the VMware vSphere GUI. This requirement applies only if you are using a VMware virtual switch. It does not apply if you are using a Cisco Nexus 1000V switch.

Chapter 3. Troubleshooting the Cisco Intercloud Fabric Upgrade

- Issues with the Intercloud Fabric Upgrade

Issues with the Intercloud Fabric Upgrade

This section includes symptoms, possible causes, and solutions for issues encountered while upgrading Intercloud Fabric.

Symptom: The routing policy (system_default_routing_policy) fails while upgrading Intercloud Fabric.

Error Message:

Failed to <CREATE/DELETE/UPDATE> Routing Policy(system_default_routing_policy) in Routing Service on < STANDALONE/PRIMARY/SECONDARY > ICS on ICFCloud(<Name of IcfCloud>). Error: authentication failed, {“error-code”:1,”error-message”:”Forbidden”,”detail”:””}.

Possible Cause: Modifications to the routing policy (system_default_routing_policy) without upgrading Intercloud Fabric link results in a failure in the Intercloud Fabric cloud Routing Service.

Verification and Solution: Ensure that you upgrade the Intercloud Fabric link if any modifications are made to the routing policy (system_default_routing_policy).

Chapter 4. Troubleshooting the Cisco Intercloud Fabric Cloud Setup

- Issues with the Cloud Setup

- Deleting the Intercloud Fabric Cloud Setup

- Issues with the Intercloud Fabric Link

Issues with the Cloud Setup

This section includes symptoms, possible causes, and solutions for issues encountered while setting up an Intercloud Fabric cloud.

Symptom: Creating an Intercloud Fabric cloud failed.

Possible Cause: The network does not have enough available IP addresses.

Verification and Solution: Modify the IP address pools that are associated with the specific Intercloud Fabric cloud and recreate the Intercloud Fabric cloud.

Symptom: Resources required for this REST API operation are not available.

Possible Cause: Locks are not available for the required associated objects.

Verification and Solution:

1. From the REST API/UI, review the service request API page for log information and each lock acquired and released. Use the log details to identify the ongoing operation or any previous service requests that did not release the locks.

2. Review the MongoDB locks collection for details on unreleased locks.

3. If the operation, after waiting for some time, does not complete or the locks are not released, restart services from the administrator console. This will result in a release of the locks and move the stalled operation in a system aborted state.

Symptom: The REST API operation cannot be performed because the referenced object with OID {0} cannot be found.

Possible Cause: The referenced object may have been deleted by a concurrent operation running at approximately the same time and is no longer present.

Verification and Solution:

1. From the Rest API/UI, review the associated objects to verify if they are available.

2. Query MongoDB for the object and identify, from the input object or ‘additionalLockableEntities’, the object being referred.

3. To resolve, refer to an alternate object that is present or create the associated object.

Symptom: Credentials validation failed.

Possible Cause: The cloud provider credentials were entered incorrectly.

Verification and Solution: Reenter the cloud provider credentials.

Symptom: From the VMware vCenter client, you cannot determine which VSM virtual machine (VM) configuration is active (primary) and which configuration is standby (secondary).

Possible Cause: While deploying the Intercloud Fabric Switch in high availability mode, you created the VMs with names similar to the Intercloud Fabric VSM high availability VM names.

Verification and Solution: As a best practice, prefix active VM names with "pri" (primary) and standby VM names with "sec" (secondary).

Symptom: Intercloud Fabric cloud creation fails with a message stating that SCP is not present on the VM.

Error Message:

Jan 23 2015 20:32:15 GMT Error occurred while Template Creation: ICFC Task State: Failed. Task Message: OS not compatible:scp not present on VM
Jan 23, 2015 20:33:49 GMT Handler failed with error - ICFC Task State: failed. Task Message: OS not compatible:scp not present on VM, selectedContext=<None>
Jan 23, 2015 20:33:49 GMT Task #1 (InterCloud Create Template (Create Template in Cloud)) failed after 456 seconds

Possible Causes:

- SCP is not installed or is installed in the wrong location.

- SSH is not fully installed.

Verification and Solution:

1. Confirm that SCP is installed and in the correct location by entering the command whereis -b scp.

2. Confirm that SSH is fully installed. If it is not, use the yum command to update the SSH package by entering yum update package-name.

Symptom: Creating an Intercloud Fabric cloud fails with the error, "Failed to create instance".

Error Message:

Jan 23, 2015 07:28:08 GMT [Deploy IcSwitchInst : icldCiscoITDD3-iclink-ics-1] [Step 1 of 4 : Instantiate IcSwitch][Create instance failed for instance: Failed to create instance: 00050a12-fab0-43d6-005-a12fab043d6Error not populated][Retry Count:5]
Jan 23, 2015 07:28:08 GMT (Deploy IcSwitchInst : icldCiscoITDD3-iclink-ics-1 failed: Create instance failed for instance: Failed to create instance: 00050a12-fab0-43d6-0005-0a12fab043d6Error not populated)
Jan 23, 2015 07:28:08 GMT 'icldCiscoITDD3' is the last IcfCloud for Provider Account 'CiscoITDDwedd'.

Possible Cause: Multiple situations can cause this problem.

Verification: The most likely cause is that the networking infrastructure on the VMware vCenter client that controls how the VMs communicate prevents the Intercloud Fabric VSM from establishing a connection with the Intercloud Fabric Switch instance that is being brought up on the cloud provider.

Symptom: After rebooting the host on which Intercloud Fabric, ICFC, Intercloud Fabric Extender, and Intercloud Fabric VSM are deployed and powering on all VMs, there is no communication between private VMs and cloud VMs.

Possible Cause: The Intercloud Fabric Extender host was rebooted out-of-band, resulting in a loss of communication and modules being unable to register with the Intercloud Fabric VSM.

Verification and Solution:

1. Check module status by logging into the Intercloud Fabric VSM console and entering the show module command. The results should indicate that modules are not registered and that communication has been lost.

2. In the Intercloud Fabric UI, reboot the Intercloud Fabric Extender and the Intercloud Fabric Switch. Because this reboot occurs in-band, the Intercloud Fabric VSM can register the modules and communication is restored.

Symptom: The Intercloud Fabric cloud is deployed but the tunnel is not operational.

Possible Cause: The control channel port or data tunnel port is not open in the firewall.

Verification and Solution:

1. Confirm that the required protocol or port is configured for the Intercloud tunnel by entering the following command:

icx-1# show intercloud tunnel config
ICS-Profile:
uuid                     2B59D348-9E48-F950-92B2-5258A9E2999F
public-ip                54.151.47.163
ICE-uuid                 855FD5BF-2BD0-7864-57DB-954DA517D887
Site-to-Site-tunnel-profile:
Heartbeat-interval       1
Heartbeat-retries        120
Tunnel-Protocol          UDP
Tunnel-Port              6644
Site-to-site-key-profile [Configured]
Encryption Algorithm     AES-128-CBC
Hash Algorithm           SHA1
Key-Profile-Checksum     B722147B-276704CF-FBA8ACB6-13371C99-ECE116BF
ReKey-State              Not-In-Progress
Epoch                    0

2. Do one of the following:

- If the desired protocol or port is not shown in the preceding command, change the protocol for the site-to-site tunnel in tunnel-profile in Intercloud Fabric:

- If the default tunnel-profile is used in Intercloud Fabric cloud, it cannot be changed. In that case, create or clone a new Intercloud Fabric cloud with a new tunnel profile with the desired protocol.

- If the nondefault tunnel profile is used in the Intercloud Fabric cloud, edit the tunnel profile with the correct protocol in the site-to-site tunnel section. To enforce the modified tunnel profile, you must reboot or switchover the Intercloud Fabric cloud. If the Intercloud Fabric cloud is in high availability mode, you must reboot or switch it over twice.

- If the correct protocol or port is shown in the preceding command, use the following command to verify that ports are open in the firewall and establish a site-to-site tunnel between ICX and ICS:

icx-1# intercloud test ics-reachability
PORT       STATE                     SERVICE         REASON
6644/udp   open                      data-tunnel     success
6646/tcp   closed                    data-tunnel     Connection refused
22/tcp     open                      ssh             success
443/tcp    open                      https           success
icx-1#

- For a TCP HTTPS-based tunnel, make sure that 22/tcp, and 443/tcp are open.

- For a TCP-based tunnel, make sure that 22/tcp, 443/tcp, and 6646/tcp are open.

- For a UDP-based tunnel, make sure that 22/tcp, 443/tcp, and 6644/udp are open.

If the data tunnel in HTTPS mode (6646/tcp) is unused, you can ignore its status.

Symptom: Creating an Intercloud Fabric cloud in AWS fails.

Error Message:

Unable to create tempVM

Possible Cause: No subnet or default subnet is attached on the VPC.

Verification and Solution: On the AWS console, verify that the VPC has a subnet attached and whether or not the attached subnet ID is a default subnet. If a default subnet does not exist, contact AWS support to recreate it. If a default subnet exists, associate it with the VPC.

Symptom: Intercloud Fabric link creation fails during deployment.

Possible Cause: There are several causes for Intercloud Fabric link creation failure. Use the following steps to identify where the failure occurs.

Verification and Solution:

1. Check the Service Request (SR) event log in Intercloud Fabric for error messages detailing at which step the Intercloud Fabric link failed and why.

2. Check this guide for Intercloud Fabric log error messages.

If the issue does not match any of the documented log file error messages, use ICFC for further debugging:

1. Log in to Intercloud Fabric.

2. Choose InterCloud Management > Infrastructure > VPCs and select the Intercloud Fabric link from the VPC list you are debugging.

3. Click the Recent Jobs tab and look for any failure associated with ICX or ICS deployment.

4. To verify configuration information, go to (ICFC) Visore by choosing hcloudEp > hcloud > tab.

For additional debugging, do the following:

1. SSH to ICFC.

2. If the failure occurs at the private cloud data center, review the ICFC logs by entering the following information:

/var/log/resource-mgr

For example:

/var/log/resource-mgr/svc_res_dme.log

/var/log/resource-mgr/svc_res_enterpriseAG.log

3. If the failure occurs at the cloud provider, review the ICFC logs by entering the following information:

/var/log/cloud/cloudprovider-mgr

For example:

/var/log/cloud/cloudprovider-mgr/cpm.log

/var/log/resource-mgr/svc_res_cloudprovider.AG.log

Symptom: Unable to allocate IP addresses.

Possible Cause: Intercloud Fabric cannot allocate the correct number of IP addresses for the ICX/ICS interfaces.

Verification and Solution: Verify that there are enough available IP addresses. Additional IP addresses can be copied from the error message shown in the Service Request (SR) logs.

Symptom: Port profile creation fails on cloud VSM (cVSM).

Possible Cause: There are varying causes for port profile creation failure.

Verification and Solution: For each possible cause, first check the cVSM Fault tab to identify the possible root cause, such as:

- NTP configuration: ICFC cannot communicate with cVSM if the ICFC clock is set ahead of the cVSM clock.

- Limit on the number of port profiles: cVSM cannot create more than 128 port profiles using the same system vLAN. Determine whether or not this limit has been reached.

Symptom: Intercloud Fabric flow times out during the port profile operation.

Possible Cause: Port profile operations can time out for multiple reasons:

- NTP configuration: ICFC cannot communicate with cVSM if the ICFC clock is set ahead of the cVSM clock.

- REST connectivity to cVSM is not working.

- cVSM service is down.

- Port profile configuration errors.

Verification and Solution:

1. Compare the time set for ICFC and cVSM. If the cVSM time is ahead of the ICFC time, the certificate that ICFC uses to authenticate itself to cVSM is not yet valid. Configure NTP on cVSM and retry the failed configuration. Also ensure that ESXi hosts are configured for NTP .

2. From the ICFC debug shell, enter the following curl command to check connectivity to cVSM REST API:

$ curl --cert /opt/cisco/cert/CACertificate.pem --key
/opt/cisco/cert/privKey.pem -k https://cVSM_IP/nsc-api

3. Check the service status on ICFC to confirm that vsm-service is running. Vsm-service is the application that talks to the cVSM REST API:

- CLI command:

# service status

- Shell command:

$ /opt/cisco/bin/state.sh

- Send ICFC logs to Cisco.

4. Check for errors that occurred during port profile configuration. Choose Intercloud Management > cVSM > Edit > Faults and look for any active errors. From ICFC Visore, search for "vaswVsm", navigate to its children, and look for "faultInst" objects and any active errors.

Symptom: ICFC rejects ICF port profile requests.

Possible Cause: There are varying causes for rejected ICF port profile requests.

Verification and Solution:

1. Confirm that you did not create a duplicate port profile.

2. Check if one of the ICFC validations on port profile was violated. If so, the error message is visible in the SR or ICFC logs.

3. Send log information to Cisco for further debugging.

Symptom: An ICX VM creation on vCenter failure occurs during ICX deployment.

Possible Cause: ICFC cannot connect with vCenter, or the host or data store where ICFC is deploying the ICX VM is unreachable.

Verification and Solution:

1. Check the ICF Service Request (SR) logs.

2. Review the ICFC logs (such as /var/log/resource-mgr/svc_res_enterpriseAG.log) for ICX VM instantiation failures.

3. Debug connectivity between ICFC and vCenter.

4. Restore connectivity to the host or data store or use an alternate host or data store.

Symptom: The Intercloud Fabric Switch (ICS) VM was created in ICFC, but ICX cannot register with ICFC.

Possible Cause: Communication issue between ICFC and ICX.

Verification and Solution: Using the management IP address, confirm that the ICX and ICFC can communicate with each other. If they are both reachable, review the ICX management interface port group configuration and confirm that the port group that is configured is the correct one. In addition, confirm that the ICX VM settings in the vCenter client are correct.

Symptom: The ICS upload or register image fails.

Possible Cause: Image creation fails while uploading ICFC or registering the ICS image on the public cloud.

Verification and Solution: Review the Service Request (SR) event log in ICF for error message details. If there is not sufficient information, check the ICFC CPM logs (/var/log/cloudprovider-mgr/cpm.log).

Symptom: ICFC fails to generate SSH certificates and pushes them to ICS using ICS's public IP address.

Possible Cause: Either the ICS public IP address is not reachable or port 22 is not open on the cloud.

Verification and Solution: Check the gateway IP address configured in ICFC and confirm that the gateway is reachable by doing the following:

1. Log in to the public cloud portal and check the security group and ACL configuration.

2. Review the public and private provider IP address configurations.

3. If you use an AWS provider VPC, confirm that the internet gateway is configured and associated with a VPC so that it can add a route to the routing table. For more information on creating a VPC, see Click Here .

Symptom: ICS and ICX are registered with ICFC, but the site-to-site tunnel is not operational.

Possible Cause: ICFC fails to push tunnel and device profiles to ICS to form a site-to-site tunnel.

Verification and Solution: Do the following:

1. Review the ICFC and ICX tabs for any configuration errors.

2. SSH to the ICX console through its management IP address for further debugging. For example:

ssh admin@icx_mgmt_ip

3. Examine the ICX log files for further analysis of the Intercloud Fabric link failure.

Symptom: Intercloud Fabric link redundancy (high availability) fails.

Possible Cause: Intercloud Fabric link redundancy is not available if one of the two pairs belonging to ICX or ICS, named primary pair (IcLink-1) and secondary pair (IcLink-2), is not established and in operation.

Verification and Solution: Review the ICX log files and the internal system event-log redundancy-mgr for further analysis of the Intercloud Fabric link failure.

Symptom: Both primary and secondary ICF link come up in Active state instead of one in Active and the other in Standby.

Possible Cause: A Network connectivity problem between primary and secondary ICXs in ICF link can cause this issue.

Verification and Solution: Reboot secondary in ICF link to restore ICF link to normal Active/Standby state.

Symptom: ICF link becomes partial_operational or not_operational after out-of-band (from provider portal) reboot operation, or a stop operation followed by a start operation, of ICS instance(s).

Possible Cause: An out-of-band reboot operation, or a stop operation followed by a start operation, can cause this issue.

Verification and Solution: Using the Intercloud Fabric UI, reboot the ICF link pair for the tunnel or module whose status is not showing up or online.

Deleting the Intercloud Fabric Cloud Setup

Use this procedure to delete the Intercloud Fabric cloud setup.

Procedure

Step 1

Delete all services, such as ICF Firewall or ICF Router, that are associated with the Intercloud Fabric cloud:

1. Log in to Cisco Intercloud Fabric.

2. Choose Intercloud > Infrastructure.

3. Choose Resource Management > Manage Resources > tenant > Network Services.

4. Select the service, such as an ICF Firewall or an ICF Router, and click Delete.
Ensure that the services are deleted by navigating to Resource Management > Manage Resources > tenant > Network Services before continuing.

Step 2

Terminate all the virtual machines associated with the Intercloud Fabric cloud:

1. Choose Intercloud > Compute.

2. Select the Intercloud Fabric cloud and click the VM tab.

3. Select the virtual machine and click Terminate.
You can delete only one virtual machine at a time. Repeat this step to delete the other virtual machines.

Step 3

Delete the VDC associated with the Intercloud Fabric cloud and the provider account:

1. Choose Policies > Virtual/Hypervisor Policies > Virtual Data Centers.

2. In the All User Groups pane, choose the VDC group that you want to delete.

3. Click the VDC tab.

4. Select the VDC and click Delete.
You can delete only one VDC at a time. Repeat this step to delete the other VDCs.

Step 4

Delete the Intercloud Fabric cloud and the images in the cloud:

1. Choose Intercloud > IcfCloud.

2. Select the Intercloud Fabric cloud and click Delete.
The completion of the service request indicates that the Intercloud Fabric cloud is deleted.

Note:
When you delete the last Intercloud Fabric cloud associated with a virtual cloud account, all associated images are deleted.

3. For the Amazon Web Services (AWS) provider account, use the AWS console to delete any snapshots or volumes associated with the Intercloud Fabric cloud.

Step 5

Delete the Intercloud Fabric VSM:

1. Choose Intercloud > Network > All Clouds.

2. Click the IcfVSM tab.

3. Select the IcfVSM and click Terminate.

Step 6

Power off and delete the virtual machine:

1. Log in to the VMware vSphere client.

2. Navigate to Recent Tasks and confirm that the Intercloud Fabric VSM has been deleted.

3. Select the Intercloud Fabric virtual machine and click Power off.

4. Select the Intercloud Fabric virtual machine and click Delete from Disk.

Issues with the Intercloud Fabric Link

This section includes symptoms, possible causes, and solutions for issues encountered while setting up an Intercloud Fabric link.

- Issues Creating the Intercloud Fabric Link

- Issues with the Intercloud Fabric Link Status Operation

- Issues Updating the Intercloud Fabric Link

- Issues Starting the Intercloud Fabric Link

- Issues Stopping the Intercloud Fabric Link

- Issues Rebooting the Intercloud Fabric Link

- Issues Deleting the Intercloud Fabric Link

- Issues with the Intercloud Fabric Link Switchover Operation

Issues Creating the Intercloud Fabric Link

Symptom: The network IP pool has insufficient number of IP addresses.

ICX mgmt IP address allocation failed. There are not enough IP addresses in IP pool {0}. Requested [{1}], Available [{2}].
ICS mgmt IP address allocation failed. There are not enough IP addresses in IP pool {0}. Requested [{1}], Available [{2}].

Possible Cause: The standalone Intercloud Fabric link network must have at least two IP addresses and at least four IP addresses available for HA.

Verification and Solution:

1. Add the IP pool or IP pool range.

2. Delete the failed Intercloud Fabric link.

3. Submit an Intercloud Fabric link create request using the newly created IP pool.

Note

REST API users can also perform an update operation on the failed Intercloud Fabric link with an updated payload.

Symptom: Configuring port profiles on cVSM fails.

Port profiles creation failed. <ICFC exception message>
Polling timed out. Waited 600 seconds for creation of ICF link port profiles.

Possible Cause:

- Unable to communicate with ICFC, or any HTTP or REST client error, while communicating with ICFC.

- The cVSM is down. Infrastructure status will show as not operational.

Verification and Solution:

1. Review services status on Intercloud Fabric and ICFC to determine if it is operational. If all services are operational, retry the configuration after some time.

2. Ensure that cVSM instance is operational on the vCenter. If cVSM went down during an out of band operation, power it on out of band.

3. Ensure that Intercloud Fabric and ICFC can connect to cVSM (use ping or ssh for reachability). Restore the connection if it remains unreachable.

Symptom: Configuring Intercloud Fabric cloud constructs fails.

Failed to create ICF cloud resources. <ICFC exception message>
Error creating ICF cloud resources on ICFC. <ICFC exception message>

The subscription policy limit for resource type 'Network Security Group count' was exceeded. The limit for resource type 'Network Security Group count' is {} per subscription, the current count is {}, and the requested increment is 1.

Possible Cause:

- Unable to communicate with ICFC or any HTTP or REST client error while communicating with ICFC.

- The security group or ACL limit was exceeded on the provider account.

Verification and Solution:

1. Retry the operation after some time.

2. Clean up stale or unused security groups from the provider account.

3. Delete the failed Intercloud Fabric link.

4. Submit the create Intercloud Fabric link request again.

Note

REST API users can also perform an update operation on failed Intercloud Fabric link with an updated payload.

Symptom: For ICX, one or more deployment tasks failed on ICFC.

Note

There are multiple possible causes for this issue.

Possible Cause: The host, or datastore, is not reachable or does not have enough resources. The curl error will be viewable in the SR log.

Failed to create/deploy ICX on vCenter.

Verification and Solution:

1. Ensure that the host and datastore is reachable from ICFC or resolve the failed connectivity.

2. Ensure that the host and datastore has sufficient resources to launch the ICX instance.

Possible Cause: ICX cannot reach ICFC.

ICX FSM registration exhausted.

Verification and Solution: Ensure that the correct management port profile is used for the ICX management port.

Possible Cause: The ICFC clock and the ICX clock differ by more than three minutes.

Unable to communicate with ICX. The time is out of sync on the ESX host where ICF VM or ICX VM is deployed. The clocks differ by {} minutes.

Verification and Solution:

1. Ensure that the NTP server configuration is properly set.

2. Ensure that the ESXi clock is correctly set when Intercloud Fabric and the ICX VM is brought up. If not, resolve the clock setting on the ESXi host.

Issues with the Intercloud Fabric Link Status Operation

The following Intercloud Fabric link status symptom has three different scenarios for possible cause and verification and solution.

Symptom: Scenario 1: The Intercloud Fabric link status fails.

Some of IcfLink deployment tasks failed were not complete, exiting workflow
ICF link deployment tasks failed
Deploy IcExtenderInst : link3-icx-1 FAILED. ICX failed to register with ICFC.
Deploy IcExtenderInst : link3-icx-2 FAILED. ICX failed to register with ICFC.

Possible Cause: The VLAN ID on the management port-profile for ICX and ICS is incorrectly configured.

Verification and Solution: Ensure that the correct VLAN is used for the management port-profile in Cisco Nexus 1000V or VMware vSwitch on the enterprise data center.

Symptom: Scenario 2: The Intercloud Fabric link status fails.

Enabling routing has failed:link4-ics-2::CreateRoutingServiceWorkflow::1 (SR oid XXXX-XXXX-XXXX-XXXX-XXXX)
Enabling Routing Service failed with error: Routing Service failed to create since link link4 is not-operational Check SR bad-link4-ics-2::CreateRoutingServiceWorkflow::1 (SR oid XXXX-XXXX-XXXX-XXXX-XXXX) for more details.

Possible Cause:

- The VLAN ID for management is missing in the enterprise trunk port-profile.

- Intercloud Fabric module(s) will not register to cVSM.

Verification and Solution:

1. Ensure that the management VLAN ID is added in the enterprise trunk port-profile prior the Intercloud Fabric creation.

2. Add the missing management VLAN ID in the enterprise trunk profile and again deploy the Intercloud Fabric link.

Symptom: Scenario 3: The Intercloud Fabric link status fails.

Deploy IcExtenderInst : link1-icx-1 FAILED. ICX failed to register with ICFC
ICF link deployment tasks failed
Some of IcfLink deployment tasks failed were not complete, exiting workflow

Possible Cause: The network’s gateway cannot be reached.

Verification and Solution: Ensure that the network’s gateway is valid for the Intercloud Fabric link creation.

Issues Updating the Intercloud Fabric Link

Symptom: The Intercloud Fabric link update fails.

Failed to connect to ICFC service
ICFC is not responding. Contact support for further assistance

Possible Cause: ICFC cannot be reached during the Intercloud Fabric link update operation.

Verification and Solution:

1. Ensure ICFC services are operational.

2. Ensure ICFC can be reached by Intercloud Fabric.

Symptom: The Intercloud Fabric link update fails due to insufficient IP addresses.

Allocation of IPs for ICX and ICS components of ICF link failed
ICX mgmt IP address allocation failed.9048://There are not enough usable IP addresses available in the IP pool v36-p2 (OID:0856be64-4cfd-408d-82c0-2302f299916c). The IPs in the IP pool are all in use by ICF or other systems. Requested [1], Available [0].

Possible Cause: There are an insufficient number of IP addresses in the IP pool.

Verification and Solution: Ensure that there are at least three IP addresses for standalone Intercloud Fabric link deployment and six IP addresses for HA Intercloud Fabric deployment.

Issues Starting the Intercloud Fabric Link

Symptom: The Intercloud Fabric link fails to start.

IcfLink operation start failed
672664192334953970113101849022577607556 FAILED. virtual machine does not exist

Possible Cause: The ICX has been terminated out of band on VMware vCenter. Review the VM (ICX) in VMware vCenter.

Verification and Solution: Delete the Intercloud Fabric link and redeploy the new Intercloud Fabric link.

Issues Stopping the Intercloud Fabric Link

Symptom: The Intercloud Fabric link fails to stop.

ICFC Error Message:VM Cycle property value validation failed.
The lifecycle operation stop on ICS failed.

Possible Cause: ICS is being terminated out of band. Check the instance on AWS.

Verification and Solution: Delete the Intercloud Fabric link and redeploy the new Intercloud Fabric link.

Issues Rebooting the Intercloud Fabric Link

Symptom: The HA Intercloud Fabric link reboot fails.

Contacting to reboot ICF link link2-sa
Reboot IcSwitchInst :link2-sa-ics-1 FAILED. Reboot Server failed for instance: i-f08af96c request POST https://ec2.us-east-1.amazonaws.com/ HTTP/1.1 failed with code 400, error: AWSError{requestId='278abe93-9684-4d9e-a0c0-fb586f18ca71', requestToken='null', code='IncorrectState', message='Cannot reboot instance i-f08af96c that is currently in stopped state.', context='{Response=, Errors=}'}

Possible Cause: The ICS instance is in the powered off state.

Verification and Solution:

1. Ensure ICS is running prior to the Intercloud Fabric reboot occurring.

2. Perform a lifecycle operation on ICS if it is powered off.

Issues Deleting the Intercloud Fabric Link

Symptom: The Intercloud Fabric link fails to delete.

Failed to undeploy Routing Service from Icflink link1-ha (Failed to connect to ICFC service)

Possible Cause: ICFC service is down during the Intercloud Fabric link delete operation.

Verification and Solution: Restart the ICFC service if it is not operational.

Issues with the Intercloud Fabric Link Switchover Operation

Symptom: The HA Intercloud Fabric link fails to switchover.

Standby ICS is not found to perform switchover of IcLink
The ICF link switchover failed. Standby ICS cannot be found and cannot perform the switchover of ICF link.

Possible Cause: The secondary Intercloud Fabric link is offline due to an issue on ICX and ICS and may have been powered off.

Verification and Solution:

1. Ensure ICX and ICS instances are running.

2. Perform a lifecycle operation on ICX and ICS if they are powered off.

Chapter 5. Troubleshooting VM Lifecycle Management

- Issues with Image Creation

- Issues with Linux Cloud VMs

- Issues with Windows Cloud VMs

- Issues with Monitoring VMs

Issues with Image Creation

This section includes symptoms, possible causes, and solutions for issues associated with creating an image.

Symptom: CentOS 7.X stops working after binary updates.

Possible Cause: When CentOS systems are manually or automatically updated using yum, the systems are upgraded to the latest release of CentOS 7.x. In result, CentOS 7.x stops working because the upgraded version may not be supported by Intercloud Fabric.

Note

Historically, CentOS releases corresponded in a sequential manner to the minor or major releases of RHEL (for example, CentOS 6.1 release corresponded with RHEL 6.1 release). With CentOS 7.x, the model changed and versions are not upgraded in a sequential manner, but to the newest version available.

Verification and Solution: To limit kernel updates on CentOS to supported levels of CentOS 7.x and prevent automatic kernel updates, do the following:

1. Edit /etc/yum.conf.

2. Add the following to the main package repository:

exclude=kernel* redhat-release* centos-release*

Symptom: Uploading a RHEL/CentOS 7.X OVA image during catalog creation and image upload fails with error message:

Error Message:

The configuration file for interfaces <interface name> cannot be found.
Fix the configuration in the Guest OS and retry importing the image.
For further details, refer to the Troubleshooting Guide.

Possible Cause: RHEL/CentOS systems name network devices in a predictable manner based on BIOS/PCI physical hotplug slot information. When a VM running on ESX is exported as an OVA image, sometimes VMware vSphere changes the PCI topology on the exported OVA causing a mismatch between the interface configuration file inside the OVA image and the OVA description of the NIC PCI hotplug slot. This inconsistency between what is described in the OVA compared to what is actually configured and present in the OVA image causes Intercloud Fabric to reject the image.

Verification and Solution: Deploy the OVA in the VMware vSphere environment and ensure the NIC names, as seen by the OS and device names, match. Fix any mismatched names and retry the upload.

Symptom: Image creation fails with the error "OS not compatible:VMware tools not present on VM".

Error Message:

Jan 24, 2015 03:26:47 GMT Image Creation in Progress: 96 %
Jan 24, 2015 03:26:47 GMT Creating Image: OS not compatible:VMware tools not present on VM
Jan 24, 2015 03:26:47 GMT Retrying: 2

Verification and Solution: Confirm the following:

- The image operating system (OS) is supported.

- VMware tools are installed in the VMware vCenter client.

Symptom: Image creation fails with the error "Cannot find appropriate ICA image".

Error Message:

Jan 23, 2015 23:35:15 GMT Handler failed with error - Cannot find appropriate ICA image for the VmImage org-root/vm-img-CiscoITRH64smallscp, selectedContext=<None>
Jan 23, 2015 23:35:15 GMT Task #1 (InterCloud Create Image (Create Image in Cloud)) failed after 0 seconds

Verification and Solution: In the VMware vCenter client, confirm that the image is based on a supported guest OS version.

Symptom: Intercloud Fabric 2.3.1 only supports Master Boot Record (MBR)-based partition tables. As a result, disks with a GUID Partition Table (GPT) fail and display an error message.

Error Message:

Unsupported operating system.

Possible Cause: Intercloud Fabric does not support the GPT standard for disk partition tables for both Windows and Linux operating systems. Possible conditions include:

- An on-board VM is moved back to the private cloud.

- The VM is moved from the private cloud to the cloud.

- The user uploads an OVA on ICFC.

This problem might also occur during VM migration.

Verification and Solution: Check the partition style on the VM.

For Linux, run the command:

fdisk -lu

For Windows:

1. Choose Start > Control Panel.

2. In the Control Panel, choose System and Security > Administrative Tools > Computer Management.

3. Select Disk Management.

4. Right-click Disk 0 and choose Properties.

5. Click the Volumes tab and check the Partition style for the disk.

Issues with Linux Cloud VMs

This section includes symptoms, possible causes, and solutions for issues associated with Linux VMs on public clouds.

Symptom: Red Hat/CentOS-style cloud VMs deployed in Azure with Intercloud Fabric might route traffic incorrectly.

Possible Cause: There might be two default routes in the IP routing table. For example, in the Red Hat 6.1 CVM and earlier, after ICF access tunnel comes up:

- Azure provisions one cloud interface (csc0) in subnet 10.200.0.0/16 with a default route configured through 10.200.0.1.

- Intercloud Fabric provisions one overlay interface (eth0) in subnet 10.2.0.0/24 with a default route configured through 10.2.0.75.

[root@rhel61-lnic ~]# ip route list
168.63.129.16 via 10.200.0.1 dev csc0
168.63.129.16 via 10.200.0.1 dev csc0  proto static
10.2.0.0/24 dev eth0  proto kernel  scope link  src 10.2.0.11
169.254.0.0/16 dev csc0  scope link  metric 1002
169.254.0.0/16 dev eth0  scope link  metric 1003
10.200.0.0/16 dev csc0  proto kernel  scope link  src 10.200.0.5
default via 10.2.0.75 dev eth0 <--- default route over overlay interface 'eth0' towards the enterprise
default via 10.200.0.1 dev csc0  proto static <--- default route over provider interface 'csc0'
[root@rhel61-lnic ~]#

Verification and Solution: Delete the default route that directs traffic over the csc0 provider interface. For example, delete the default route over the provider interface for the following Red Hat 6.1 VM:

[root@rhel61-lnic ~]# route delete default gw 10.200.0.1
[root@rhel61-lnic ~]# ip route list
168.63.129.16 via 10.200.0.1 dev csc0
168.63.129.16 via 10.200.0.1 dev csc0  proto static
10.2.0.0/24 dev eth0  proto kernel  scope link  src 10.2.0.11
169.254.0.0/16 dev csc0  scope link  metric 1002
169.254.0.0/16 dev eth0  scope link  metric 1003
10.200.0.0/16 dev csc0  proto kernel  scope link  src 10.200.0.5
default via 10.2.0.75 dev eth0 <---- Now there is only one default route via the overlay interface 'eth0'
[root@rhel61-lnic ~]#

Issues with Windows Cloud VMs

This section includes symptoms, possible causes, and solutions for issues associated with Windows VMs on public clouds.

Symptom: Image creation and VM migration fails for Windows 2008 R2 with the error "Missing SHA-256 signatures Verification and processing Support".

Error Message:

Windows 2008R2: Missing SHA-256 signatures Verification and processing Support. Please install updates:
https://technet.microsoft.com/en-us/library/security/3033929https://support.microsoft.com/en-us/kb/2921916

Verification and Solution: Install the following updates on the TemplGuest OS:

- Patch: Click Here

- Hotfix KB: Click Here

This step is only required for Windows Server 2008 R2.

Symptom: There is no connectivity to the Windows cloud VM.

Possible Cause: There might be a memory leak, an unopened port, or a variety of other problems.

Verification and Solution:

1. Verify Windows cloud VM reachability by using SSH and a public IP address, or by using RDP or ping (if available):

1. Check the cloud VM status in the Amazon Elastic Compute Cloud (EC2) console.

2. If the EC2 status shows "2/2 checks passed" and "running," reboot the cloud VM from the EC2 console and check connectivity.

3. After connectivity is restored, review the system event logs for any relevant errors by using eventvwr.msc.

4. If the system event logs do not contain any obvious errors, a memory leak might exist. Use perfmon.exe to isolate any excessive, nonpaged pool usage during upcoming boot sessions.

2. Review the EC2 firewall settings for the Windows cloud VM and ensure that the necessary ports are open:

- TCP and UDP port 6644

- TCP port 22

Symptom: There is an issue related to the Windows cloud VM subagent.

Possible Cause: The subagent process is not running or is not listening on the correct TCP port.

Verification and Solution: Try any of the following solutions:

- Confirm that the sub_agent.exe file is present in the C:\Program Files\Cisco\ICA folder. If the file is not present, copy it to the folder.

- In a command window, determine whether the subagent process is running.

c:\>tasklist /FI "imagename eq sub_agent.exe"
Image Name                     PID Session Name        Session#    Mem Usage
========================= ======== ================ =========== ============
sub_agent.exe                 2616 Services                   0      4,508 K

If the agent is not running, start it.

- Confirm that the subagent process is listening on TCP port 6644.

c:\>netstat -anB

Active Connections
...
TCP    0.0.0.0:6644           0.0.0.0:0              LISTENING
[sub_agent.exe]
...

Symptom: There is no connectivity to the SSH server.

Possible Causes:

- The vm_trust.properties file does not exist.

- The SSH server is not running.

- The SSH server is not listening on the correct TCP port.

- The SSH public key for the root user cannot be retrieved.

Verification and Solution: Try any of the following solutions:

- If the subagent process is running, but the process is not listening on TCP port 6644, determine whether or not the C:\Program Files (x86)\Cisco\vm_trust.properties file exists.

If the vm_trust.properties file is absent, determine whether or not the SSH server is running:

c:\>sc query freesshdservice
SERVICE_NAME: freesshdservice
TYPE               : 110  WIN32_OWN_PROCESS  (interactive)
STATE              : 4  RUNNING
(STOPPABLE, NOT_PAUSABLE, IGNORES_SHUTDOWN)
WIN32_EXIT_CODE    : 0  (0x0)
SERVICE_EXIT_CODE  : 0  (0x0)
CHECKPOINT         : 0x0
WAIT_HINT          : 0x0

If the SSH server is not running, confirm that the SSH server directory is populated with the necessary files:

c:\>dir "C:\Program Files (x86)\Cisco"\
Volume in drive C has no label.
Volume Serial Number is 7C21-C2FC
Directory of C:\Program Files (x86)\Cisco
07/16/2013  05:23 AM    <DIR>          .
07/16/2013  05:23 AM    <DIR>          ..
07/16/2013  02:28 AM               672 DSAKey.cfg
07/16/2013  02:28 AM               256 freesshd.log
03/29/2013  07:33 PM         1,360,896 FreeSSHDService.exe
07/16/2013  02:28 AM             1,096 FreeSSHDService.ini
07/16/2013  02:27 AM               386 root

- Determine whether or not the SSH server is listening on TCP port 22 and SSH connections can be established:

c:\>netstat –anB

Active Connections
...
TCP    0.0.0.0:22             0.0.0.0:0              LISTENING
[FreeSSHDService.exe]
...

- Determine whether or not the SSH public key for the root user can be retrieved successfully by using the wget.exe command from the EC2 VM user data:

C:\Program Files\Cisco\ICA\wget.exe -t 1 --bind-address [peth IP] -O
C:\Windows\Temp\ec2pubkey http://169.254.169.254/latest/meta-data/public-keys/0/openssh-key

Symptom: There is an issue related to the Intercloud Agent Service (ICASVC).

Possible Cause:

- The ICASVC is not installed.

- The ICASVC is not running.

- The ICASVC is not set to the AUTO_START startup type.

Verification and Solution: Try any of the following solutions:

- If the ICASVC is not installed, determine whether or not the following registry entry is present:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\ICASvc]
"Type"=dword:00000010
"Start"=dword:00000002
"ErrorControl"=dword:00000001
"ImagePath"=hex(2):22,00,43,00,3a,00,5c,00,50,00,72,00,6f,00,67,00,72,00,61,00,\
6d,00,20,00,46,00,69,00,6c,00,65,00,73,00,5c,00,43,00,69,00,73,00,63,00,6f,\
00,5c,00,49,00,43,00,41,00,5c,00,49,00,63,00,61,00,53,00,76,00,63,00,2e,00,\
65,00,78,00,65,00,22,00,00,00
"DisplayName"="Cisco InterCloud Agent Service"
"ObjectName"="LocalSystem"
"Description"="Cisco InterCloud Agent Service"

If the registry entry is absent, import the ICASVC key into HKLM\System\CCS\Services.

- Determine whether or not the ICASVC is set to the AUTO_START startup type:

c:\>sc qc icasvc
[SC] QueryServiceConfig SUCCESS
SERVICE_NAME: icasvc
TYPE               : 10  WIN32_OWN_PROCESS
START_TYPE         : 2   AUTO_START
ERROR_CONTROL      : 1   NORMAL
BINARY_PATH_NAME   : "C:\Program Files\Cisco\ICA\IcaSvc.exe"
LOAD_ORDER_GROUP   :
TAG                : 0
DISPLAY_NAME       : Cisco InterCloud Agent Service
DEPENDENCIES       :
SERVICE_START_NAME : LocalSystem

Symptom: The ICASVC startup scripts are not running.

Possible Cause: The startup scripts are not present in the correct folder.

Verification and Solution: Confirm that the StartIca.vbs, StartSubagent.vbs, and start_subagent.bat files reside in the C:\Program Files\Cisco\ICA folder.

Symptom: The cloud VM loses connectivity with the private data center.

Possible Cause: The certificate is not installed or is not correct.

Verification and Solution:

1. Confirm that the ctdrv.sys and ctmp.sys files are present in the C:\Program Files\Cisco\ICA folder.

2. Confirm that the Windows ICA driver certificate is installed:

C:\Program Files\Cisco\ICA>CertMgr.exe /s TrustedPublisher
==============Certificate # N ==========
Subject::
[0,0] 2.5.4.6 (C) US
[1,0] 2.5.4.8 (S) California
[2,0] 2.5.4.7 (L) San Jose
[3,0] 2.5.4.10 (O) Cisco Systems, Inc
[4,0] 2.5.4.11 (OU) Digital ID Class 3 - Microsoft Software Validation v2
[5,0] 2.5.4.3 (CN) Cisco Systems, Inc
…

Symptom: Encrypted or encapsulated packet injection is not working.

Possible Cause: Weak host sends and receives are enabled.

Verification and Solution: For encrypted or encapsulated packet injection to work, the provider adapter must have weak host sends and receives disabled, which is the default setting.

1. Determine whether weak host sends and receives are enabled or disabled:

> netsh interface ipv4 show interfaces level=verbose
Interface Local Area Connection 2 Parameters
...
Weak Host Sends                    : disabled
Weak Host Receives                 : disabled
...

2. If weak host sends and receives are enabled, disable them:

> netsh interface ipv4 set interface [InterfaceNameOrIndex] weakhostsend=disabled
> netsh interface ipv4 set interface [InterfaceNameOrIndex] weakhostreceive=disabled

Symptom: The access tunnel with the Windows VM is broken.

Possible Cause: Third-party Windows Filtering Platform (WFP) filters are blocking datagrams on TCP port 6644 or the UDP port over which the subagent has initiated the Datagram Transport Layer Security (DTLS) connection.

Verification and Solution: Enter the following command and examine the output:

> netstat -anB

- To determine whether or not the UDP port is causing the problem, look for the subagent.exe port.

- To determine whether or not third-party WFP filters are causing the problem, look at the sysinfo section of the output for WFP filters for Windows VMs.

To resolve the issue, restart the services.

Symptom: There is an issue with the data path between the overlay miniport and the cloud provider miniport.

Possible Cause: Third-party Network Driver Interface Specification (NDIS) intermediate drivers or additional NDIS miniports are affecting the data path between the overlay miniport and the cloud provider miniport.

Verification and Solution: Enter the following command and examine the output:

> netstat -anB

All NDIS drivers are enumerated in the sysinfo section of the output for Windows VMs.

Symptom: Cloud VM deployment fails for one of the two interfaces.

Possible Cause: By default, port profiles are named "icfCloud" if a name is not provided during Intercloud Fabric tenant configuration.

Verification and Solution: When deploying cloud VM, make sure you have a network policy defined with port profiles for each vNIC that is on the same tenant. When modifying the port profiles after deploying the cloud VM, configure the appropriate tenant on the port profile that does not violate the deployment solution.

Symptom: Migrating a Windows 2012 VM or importing a Windows 2012 VM OVA fails with the error, "VM Guest OS was not shutdown gracefully. Please resume and shutdown Windows fully prior to initiating import or move actions on this VM image".

Possible Cause: ICF detected that the VM did not shut down gracefully prior to the user initiating a VM migration or exporting it as an OVA from the VMware vCenter client. Not shutting down the guest OS gracefully can corrupt the file system.

Verification and Solution: Power on the VM in the VMware vCenter client and shut down the guest OS. After the guest OS has shut down, reinitiate the request to migrate the VM or export the OVA from the VMware vCenter client.

Issues with Monitoring VMs

This section includes symptoms, possible causes, and solutions for issues associated with monitoring VMs.

Symptom: The VM statistics in the end-user portal show VMs that do not exist.

Possible Cause: The VMs were deleted from the VMware vCenter client; however, they still appear because the periodic refresh did not occur.

Verification and Solution: Manually refresh the VM list.

Chapter 6. Troubleshooting the Cisco Intercloud Fabric Components

- Issues with the Intercloud Fabric Virtual Supervisor Module

- Issues with Intercloud Fabric Controller

- Issues with the Intercloud Fabric Switch

- Issues with the Intercloud Fabric Routing Service

- Issues with the Intercloud Fabric Routing Service Interface

- Issues with the Intercloud Fabric Advanced Routing Service

Issues with the Intercloud Fabric Virtual Supervisor Module

This section includes symptoms, possible causes, and solutions for issues associated with the Cisco Intercloud Fabric virtual supervisor module (VSM).

Symptom: In the output of the show module command, the Intercloud Fabric Switch modules are not shown online.

Possible Cause: The Intercloud Fabric Switch is not registered with the Cisco Intercloud Fabric VSM due to one of the following reasons:

- The Intercloud Fabric Switch management VLAN is missing from the Intercloud Fabric cloud trunk port profiles. The port profiles are named IcfCloud_Name_ICS_Trunk_Tunnel, IcfCloud_Name_ICX_Trunk_Tunnel, and IcfCloud_Name_ICX_Trunk_Tunnel_Enterprise.

- The Intercloud Fabric Switch management VLAN is not identified as the system VLAN for the Intercloud Fabric cloud trunk port profiles. The port profiles are named IcfCloud_Name_ICS_Trunk_Tunnel, IcfCloud_Name_ICX_Trunk_Tunnel, and IcfCloud_Name_ICX_Trunk_Tunnel_Enterprise.

Verification and Solution: Manually add the VLAN configuration interface in ICFC.

Symptom: There is no traffic from the private VM to the cloud VM.

Possible Cause: The cloud VM VLAN is missing from the Intercloud Fabric cloud trunk port profiles. The port profiles are named IcfCloud_Name_ICS_Trunk_Tunnel, IcfCloud_Name_ICX_Trunk_Tunnel, and IcfCloud_Name_ICX_Trunk_Tunnel_Enterprise.

Verification and Solution: Manually add the VLANs via the VSM configuration interface in ICFC.

Issues with Intercloud Fabric Controller

This section includes symptoms, possible causes, and solutions for issues associated with the Intercloud Fabric Controller (ICFC).

Symptom: The UI task status displays a failure message.

Possible Cause: The possible causes vary.

Verification and Solution: Review the resource manager logs in the /var/log/resource-mgr folder. The svc_res_dme.log log file is for UI-to-Resource Manager communication. The svc_res_cloudproviderAG.log log file is for AG-to-CPM communication.

Symptom: The access tunnel fails.

Possible Cause: The possible causes vary.

Verification and Solution: Review the logs on the Intercloud Fabric Switch.

Symptom: The Intercloud image fails to import into ICFC.

Possible Cause: ICFC encountered an error.

Verification and Solution: In ICFC, review the Recent Jobs list for the status and an error message if the import operation does not complete within a reasonable amount of time. The Task line item lists the last error encountered, if any, and the number of retries attempted.

Symptom: After restarting ICFC services, the job for importing an Intercloud image is displayed as running, but the job does not progress.

Possible Cause: If ICFC services are restarted while an import operation is running, the task does not resume.

Verification and Solution: Cancel the job in Intercloud Fabric and import the image again.

Symptom: Intercloud Fabric cannot validate Amazon Web Services (AWS) credentials.

Possible Cause:

- Not all of the required ports are open.

- The DNS server is not reachable from ICFC.

- The NTP server is not reachable from ICFC.

Verification and Solution:

1. Make sure that all required ports are open:

- TCP ports—22, 443, 3389, 6644, and 6646

- UDP ports—6644 and 6646

2. Make sure that the DNS service is configured and the DNS server is reachable from ICFC.

3. Make sure that the NTP service is configured and the NTP server is reachable from ICFC.

Symptom: Integrated Gateway is unreachable from ICFC.

Possible Cause: After restarting ICFC or the Intercloud Fabric VM, the Integrated Gateway is unreachable.
Verification and Solution:

1. SSH to ICFC and enter the following details:

ICFC# connect local-mgmt
ICFC(local-mgmt)# service restart

2. In ICFC, check if the Integrated Gateway state changes to reachable/running.

Note

If the Integrated Gateway state does not change to reachable/running, change an Integrated Gateway configuration parameter, such as the description. This changes the Integrated Gateway state to reachable/running by attempting a new connection.

Symptom: An Intercloud Fabric link is deployed but the configuration status remains "Applying" and the tunnel status shows "not operational". Similarly, when ICS is successfully deployed and obtains a private IP address (with a configuration status "Applying"), the configuration status indicates that ICS is not reachable from ICFC.

Possible Cause: ICFC must be behind NAT for ICFC-to-ICS communication.

Verification and Solution: Modify or delete the route on ICS. ICFC should be able to reach ICS via provider private gateway.

Symptom: ICFC fails to respond or come up.

Possible Cause: Multiple situations can cause this problem.

Verification and Solution:

1. Using SSH, connect to the Intercloud Fabric VM CLI as an administrator.

2. Enter option 11 to display the container details for ICFC information and current status.

3. Enter option 15 to launch the ICFC console for further detailed ICFC information.

Note

For more information about Intercloud Fabric VM CLI console, see the Cisco Intercloud Fabric Administrator Guide, section "Using Cisco Intercloud Fabric VM CLI Commands."

Symptom: The ICFC container status shows that it is running but fails to respond to queries.

Possible Cause: Multiple situations can cause this problem.

Verification and Solution:

1. Using SSH, connect to the Intercloud Fabric VM CLI as an administrator.

2. Enter option 15 to launch the ICFC console.

3. Run the 'connect local-mgmt' command, followed by the 'service status' command, on the ICFC CLI to determine the ICFC service status.

Note

For more information about Intercloud Fabric VM CLI console, see the Cisco Intercloud Fabric Administrator Guide, section "Using Cisco Intercloud Fabric VM CLI Commands."

Issues with the Intercloud Fabric Switch

This section includes symptoms, possible causes, and solutions for issues associated with the Intercloud Fabric Switch (ICS). The symptoms include the following:

Symptom: A site-to-site tunnel does not come up.

Possible Cause: For TCP and UDP tunnel type:

- Port 6644 on ICX is not open. Both UDP and TCP tunnel types need port 6644 open for the tunnel to be established.

Verification and Solution: To open port 6644 behind the firewall, complete the following steps:

1. Verify that the tunnel received the configuration parameters (which describe the site-to-site tunnel type) and the tunnel state by entering the following command:

show intercloud tunnel config

2. Verify that the tunnel state is up by entering the following command on ICX and ICS:

show intercloud tunnel brief

For HTTPS tunnel on ICX, continue with the following steps.

3. Display the counter related to the tunnel establishment failure by entering the following commands:

show intercloud tunnel statistics

show systm internal tunnel-manager dp

4. To collect the log information, enter the following command:

show tech-support

Note

An intermittent network issue may cause the site-to-site tunnel from coming up. A reboot of the site-to-site link from the Intercloud Fabric UI may fix this issue. In result, a high availability failover may be triggered if high availability is configured. If there is no high availability, it will result in all cloud VMs to lose connectivity with the cloud provider for the duration of the site-to-site reboot operation.

5. Repeat step 3 on ICS and collect the statistics and log and forward to Cisco Customer Support.

Symptom: Access tunnel does not come up.

Possible Cause:

Verification and Solution:

1. Verify the tunnel has received the configuration parameters and the tunnel state status by entering the following command:

show intercloud tunnel config

2. Verify that the tunnel state is up by entering the following command on ICX and ICS:

show intercloud tunnel brief

If the tunnel has not received tunnel configuration parameters, the first command will return blank entries for tunnel parameter for access configuration.

3. Reboot the site-to-site link from the Intercloud Fabric UI to force ICS to receive access tunnel management parameters toward the cloud VM.

4. If ICS has received the access tunnel configuration parameters, review if any ICS cores are present by entering the following command:

show cores > bootflash/cores.log

5. Collect the ICS logs by entering the following command:

show tech-support > bootflash/logs

6. Copy any cores and logs and forward to Cisco Customer Support.

7. Reboot the cloud VM.

Note

A reboot of the cloud VM from the Intercloud Fabric UI will result in a cleaned cloud VM. ICS will then resume connecting with the cloud VM and open an access tunnel after the reboot.

8. If the issue remains, collect any remaining cores and logs and contact Cisco Customer Support for further troubleshooting.

Symptom: A cloud VM cannot reach another cloud VM on the same VLAN.

Possible Cause:

Verification and Solution:

1. Verify the tunnel has received the configuration parameters by entering the following command:

show intercloud tunnel config

If the tunnel has not received tunnel configuration parameters, this command will return blank entries for tunnel parameters for access configuration.

2. Reboot the site-to-site link from the Intercloud Fabric UI to force ICS to receive access tunnel management parameters toward the cloud VM.

3. If ICS has received the access tunnel configuration parameters, review if any cores are present by entering the following command:

show cores > bootflash/cores.log

4. Collect the ICS logs by entering the following command:

show tech-support > bootflash/logs

5. Copy any cores and logs and forward to Cisco Customer Support.

6. Reboot both cloud VMs.

Note

A reboot of the cloud VM from the Intercloud Fabric UI will result in a cleaned cloud VM. ICS will then resume connecting with the cloud VM and open an access tunnel after the reboot.

7. Review the tunnel access state for both cloud VMs. If the tunnels are up, connectivity should be established between the two cloud VMs by entering the following command:

show intercloud tunnel brief

8. If the issue remains, contact Cisco Customer Support after collecting any cores and logs by entering the following command:

show tech-support > bootflash/logs

Symptom: Packets from the cloud provider cannot reach the cloud VM on the same VLAN.

Possible Cause: One of the following may be dropping packets:

- ICX

- ICS

- End VM

Verification and Solution: To debug the packets lost, complete the following steps for either ICX, ICS (for HTTPS tunnel), or ICS (in case of UDP tunnel):

- ICX

1. Verify the tunnel has received the configuration parameters and the tunnel state status by entering the following command:

show intercloud tunnel config

This command verifies the type of site-to-site tunnel operation.

2. Verify the tunnel is up by entering the following command:

show intercloud tunnel brief

If the tunnel is up, proceed to step 7.

3. Verify the VEM state for the presence of cloud port and VLAN membership for the port by entering the following commands:

vemcmd show port

vemcmd show port vlans

4. Review the VEM packet counters on ICX by entering the following commands:

vemcmd show stats

vemcmd show port-drops ingress/egress

If the drop count for the port associated with the cloud facing port is going up, packets may be getting dropped. This may be due to the port, not being configured for VLAN membership, getting extended into the cloud.

5. Determine if the cloud facing port is configured properly for VLAN membership by entering the following command:

vemcmd show port vlans

6. For HTTP site-to-site tunnel, review the ICX tunnel counters and verify if the drop counter for the tunneled packets is going up by entering the following commands:

show intercloud tunnel statistics

show system internal tunnel-manager dp

If the drop counter for the tunneled packets is going up, this may be due to an installation error. Contact Cisco Customer Support.

7. If packets are successfully moving out of ICX, check the following:

- For ICS (for HTTPS tunnel):

show system internal tunnel-manager dp

- For ICS (in case of UDP tunnel):

show intercloud tunnel statistics

Note

Review the packet drop counters. If the cloud VM port counters for data transmitted and received are moving up, the packets are reaching the cloud VM. In this case, log in to the cloud VM and review the packet flow on the cloud VM.

Symptom: Packets from the cloud VM cannot reach the cloud provider.

Note

This issue assumes that the cloud VM access tunnel is up and working. If the cloud VM and the cloud provider VM is not the same VLAN, you can use one to verify the packet path from cloud VM to ICS. The Routing Service is responsible for packet switching from one VLAN to another VLAN. Refer to Issues with the Routing Service section for Routing Service troubleshooting operations.

Possible Cause: The possible causes include the following:

- The cloud VM is not configured with routes pointing toward the cloud provider.

- ICS is dropping packets.

- ICX is dropping packets.

- End VM is dropping packets.

Verification and Solution: Debug the packets lost by performing the following steps for ICS and ICX:

- For ICS, verify the route table entries on the cloud VM point toward the cloud provider VM:

1. Review the route table from the cloud VM by entering the following command:

show intercloud vm-name <Name of the VM> system-log

If the route table entries are accurate, review the packet counters on ICS to verify the packets are reaching ICS.

2. Review the interface counters by entering the following command:

show intercloud tunnel statistics

If the end VM is not on the same VLAN, but the packet counter for the access tunnel are function properly, use the Routing Service debug operations to verify that the packets get the VLAN switched. If the end VM is on the same VLAN as the cloud VLAN, continue to step 3.

3. Review the site-to-site tunnel type by entering the following command:

show intercloud tunnel config

4. Verify the site-to-site tunnel is up by entering the following command:

show intercloud tunnel brief

5. Review the HTTP site-to-site tunnel counters by entering the following commands:

show intercloud tunnel statistics

show system internal tunnel-manager dp

Note

If the drop counter for the tunneled packets is going up, this may be due to an installation error. Contact Cisco Customer Support.

If packets are successfully moving out of ICS, proceed to the ICS procedure.

6. Review ICS (for HTTPS tunnel) by entering the following command:

show intercloud tunnel statistics

Review the packet drop counters. If the site-to-stie port counters for data transmitted and received are moving up, the packets are reaching ICX. In this case, log in to ICX and review the packet flow.

7. If packets are successfully moving out of ICX, check the following:

- For ICS (for HTTPS tunnel):

show system internal tunnel-manager dp

- For ICS (in case of UDP tunnel):

show intercloud tunnel statistics

Note

Review the packet drop counters. If the cloud VM port counters for data transmitted and received are moving up, the packets are reaching the cloud VM. In this case, log in to the cloud VM and review the packet flow on the cloud VM.

- Perform the following steps for ICX:

1. Verify the site-to-site tunnel is up by entering the following command:

show intercloud tunnel brief

2. Review the HTTP site-to-site tunnel counters on ICX by entering the following commands:

show intercloud tunnel statistics

show system internal tunnel-manager dp

Note

If the drop counter for the tunneled packets is going up, this may be due to an installation error. Contact Cisco Customer Support.

If packets are successfully moving out of ICX, review the end VM.

3. Review ICX (in case of UDP tunnel) by entering the following command:

show intercloud tunnel statistics

Note

Review the packet drop counters. If the cloud VM port counters for data transmitted and received are moving up, the packets are reaching the cloud provider VM. In this case, log in to the cloud provider VM and review the packet flow.

Symptom: ICS fails to power up and AWS displays a "401 Unauthorized" error.

Possible Cause: The credentials were changed or the access key was deleted on AWS.

Verification and Solution: Enter the correct credentials for the provider account. For information about entering the credentials, see the Amazon Web Services documentation.

Symptom: ICS does not appear under "show module" for the registered cVSM.

Possible Cause:

- VSM reachability from ICS over the management interface is not working.

- One of the ports on ICX is not set to trunk mode.

- The host port is not set to promiscuous mode. As a result, the ICX host does not receive all packets from ICS.

Verification and Solution:

1. When using different VLANs for ICS, either add a static route for that subnet or make sure the subnet can be reached through the default gateway. To configure this, choose Intercloud > Network > IcfVSM, highlight the VSM, and click Configure Route.

2. Log in to the VMware vCenter client. Right-click the ICX VM and choose Edit Settings. Three network adapters appear. Verify that the correct NIC (the NIC facing the cloud side) is set to "trunk mode".

3. Select the ICX host and click the Configuration tab. Select Networking, choose the vSwitch responsible for the port (physical adapter) serving the ICX VM, and click Properties. In the Ports tab, click Edit and navigate to Security. Set the following to "accept mode:"

- Promiscuous mode

- MAC address change

- Forged transmits

Symptom: Packets arriving at the ICX from the private cloud are not sent to the site-to-site tunnel to the ICS.

Possible Cause: The possible causes vary.

Verification and Solution: To debug packets lost, enter the following commands:

show intercloud tunnel brief
show intercloud tunnel config

vemcmd show port
vemcmd show port vlans
vemcmd show stats
vemcmd show port-drops ingress/egress
show system internal tunnel-manager dp
show system internal event-log tunnel manager

Symptom: Unknown unicast packets, with MAC addresses that are not in the MAC address pool, are not forwarded over the site-to-site tunnel.

Possible Cause: On the ICX, unknown unicast packets forwarded from the private cloud to the cloud over the site-to-site tunnel are dropped if their destination MAC address is not a part of the MAC address pool for the Intercloud Fabric link.

Verification and Solution: Use the following command to display the feature status for the blocking of unknown unicast MAC addresses that are not in the MAC address pool:

vemcmd show macpool veth1-0

This command applies only to the ICX. The feature is on by default. Contact customer support if you need to disable this feature.

Symptom: The overlay address on the cloud VM is not configured.

Possible Cause: The subagent process did not receive or push the IP address configuration from the ICS to the cloud VM, but the control channel between the ICS and the cloud VM is verified as up and working.

Verification and Solution: Open a session to the ICS and enter the following command:

show system internal event-log tunnel-mgr

Issues with the Intercloud Fabric Routing Service

This section includes symptoms, possible causes, and solutions for issues associated with the Routing Service.

- Issues Enabling and Configuring Intercloud Fabric Routing Service

- Issues Disabling Intercloud Fabric Routing Service

- Issues Connecting Intercloud Fabric Routing Service to Networks

- Issues Disconnecting Intercloud Fabric Routing Service from Networks

- Issues with Intercloud Fabric Routing Policy

- Issues with Intercloud Fabric Virtual Data Center

- Issues with Intercloud Fabric Routing Service Internal Errors

Issues Enabling and Configuring Intercloud Fabric Routing Service

Symptom: Creating the Routing Service fails.

Possible Cause:

- The transport network does not exist.

- There are not enough IP addresses in the transport network.

- There are not enough IP addresses for the management IP addresses.

- The ICFC tunnel is down.

- The Intercloud Fabric Switch cannot be reached from ICFC.

- There are not enough MAC addresses in ICFC.

- ICFC cannot be reached.

Verification and Solution:

1. Create a transport network, redeploy the link, and review the service request logs for warnings:

Routing Service creation aborted. No transport network found.

2. Allocate more IP addresses to the transport network, redeploy the link, and review the service request logs for warnings:

Routing Service creation aborted. No ips in transport netowrk <network name>.

3. Allocate more IP addresses to the management network and review the service request logs for warnings:

Routing Service management IP allocation failed pool ID <oid>.

Use this procedure if ICFC tunnel is down:

1. Log in to cVSM and make sure the tunnel endpoints are registered.

2. Log in to the Intercloud Fabric Switch and confirm that the tunnel is operational.

3. From the Intercloud Fabric Switch, confirm that the Routing Service service is running.

4. From the Intercloud Fabric Switch, confirm that the data is correct for the port configuration of the Routing Service trunk.

5. Confirm that the port profile configurations for the cVSM are correct.

Review the service request logs for warnings:

Failed to create the Routing Service instance <Routing Service IP address> for icfCloud.
This is primarily due to a network connectivity issue between ICFC and ICS.

Use this procedure if the Intercloud Fabric Switch cannot be reached:

1. Ping the Intercloud Fabric Switch IP address and confirm that the configurations are correct.

2. Log in to cVSM and confirm that the tunnel endpoints are registered.

3. Confirm that the port profile configuration for cVSM are correct.

Review the service request logs for warnings:

Failed to create the Routing Service instance <Routing Service IP address> for icfCloud.
This is primarily due to a network connectivity issue between ICFC and ICS.

If there are not enough MAC pool addresses, review the service request logs for warnings:

If ICFC cannot be reached, make sure the information provided during the OVA installation is accurate.
If ICFC cannot be reached due to invalid information during installation, reinstall the OVA with the correct information.

Symptom: Configuring the Routing Service fails.

Possible Cause: The Routing Service login fails.

Verification and Solution: Review the service request logs for warnings.

Symptom: Creating a Routing Service fails because no transport Network is present in Intercloud Fabric.

Routing Service creation aborted. No transport network found.

Possible Cause: The presence of a transport Network is required to create a Routing Service.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Create the transport Network in Intercloud Fabric.

3. Disable the Routing Service on the Intercloud Fabric cloud.

4. Once successfully disabled, reenable the Routing Service on the Intercloud Fabric cloud.

Symptom: Creating a Routing Service fails because the transport Network does not have a free IP address.

Routing Service creation aborted. No ips in transport network <network name>.

Possible Cause: Routing Service creation fails when the transport Network is different from the management Network.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Allocate at least one more IP address to the transport Network.

3. Disable the Routing Service on the Intercloud Fabric cloud.

4. Once successfully disabled, reenable the Routing Service on the Intercloud Fabric cloud.

Symptom: Creating a Routing Service fails because the management Network does not have one (or two, if in high availability mode) free IP address(es).

Routing Service management IP address allocation from <network name> failed.

Possible Cause: Routing Service creation fails when the availability of one (or two) free IP address(es) on the management Network is not present.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Allocate at least one (or two, if in high availability mode) more IP address(es) to the management Network.

3. Disable the Routing Service on the Intercloud Fabric cloud.

4. Once successfully disabled, reenable the Routing Service on the Intercloud Fabric cloud.

Issues Disabling Intercloud Fabric Routing Service

Symptom: Removing the Routing Service IP address fails if the Routing Service is removed while a Cisco Intercloud Fabric link is being deleted.

Possible Cause:

- The management IP address removal fails.

- The interface IP address removal fails.

Verification and Solution:

1. Delete the Routing Service-related object so that the Cisco Intercloud Fabric link can be deleted. Review the service request logs for warnings:

Failed to release Mgmt IP <ips> for IG Instance <Routing Service name> (error message).

2. If deleting the Cisco Intercloud Fabric link does not succeed, review the service request logs for warnings:

Failed to release Ips <ipAddresses> allocated to Routing Service networkInterface <icRouterInterface. name>.

Symptom: Deleting the Routing Service fails.

Possible Cause:

- ICFC displays an error.

- ICFC times out.

Verification and Solution:

1. Confirm that the ICFC IP address can be reached and review the service request logs for warnings.

2. Confirm that the Intercloud Fabric Switch container exists and review the service request logs for warnings.

Symptom: The Routing Service interface IP address release fails.

Possible Cause: The allocation failure occurs because of one, or more, of the following interface types:

- Management Network interface

- Transport Network interface

- Data only Network interface

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

Failed to release Mgmt IP <ips> for IG Instance <Routing Service name> (error message).

2. Review the workflow error logs for further debugging.

Failed to release Mgmt IP <ips> for IG Instance <Routing Service name> (error message).

Issues Connecting Intercloud Fabric Routing Service to Networks

Symptom: Creating an interface fails when connecting the Routing Service to networks.

Possible Cause:

- There are insufficient IP addresses for one or more networks.

- Creating an ICFC network interface fails.

- ICFC times out.

- The Routing Service does not appear in the system or is in a failed state.

- The Intercloud Fabric Switch or the integrated gateway cannot be reached.

Verification and Solution:

1. Review the service request logs for warnings.

2. If ICFC times out, confirm that the Routing Service interface is properly configured and that the ICFC IP address can be reached. In addtion, review the service request logs for warnings.

Symptom: No interfaces are created on the Routing Service when connecting to networks.

Possible Cause:

- No data Layer 3 network (other than management and transport) is associated with the VDC.

- The interface already exists.

- The Routing Service is not in the system or is in a failed state.

Verification and Solution: Review the service request logs for warnings and check the Routing Service.

Symptom: The Intercloud Fabric link operational status goes DOWN while the Routing Service create request is in progress.

Routing Service <name> failed to create since link <name> is not operational.
Failed to create the Routing Service instance <management interface IP address> for ICF cloud.
This is primarily due to network connectivity issue between ICF and ICS.

Possible Cause: The operational status of the Intercloud Fabric link for the Intercloud Fabric cloud on which the Routing Service is to be created is not UP when the Routing Service create request is in progress. Once the Routing Service create request starts, Intercloud Fabric waits a maximum of 6 minutes for the Intercloud Fabric link operational status to go back UP.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Review ICFC. Verify that the 'proxyManagedEndpointStatus' MO, corresponding to the Routing Service instance, is not present in ICFC.

3. Review the ICS instance. Review the Routing Service Status and verify the associated ICS instance, when reachable, to confirm that the Routing Service container has not been created.

4. Fix the Intercloud Fabric link operational DOWN status.

5. After the Intercloud Fabric link is UP, disable the Routing Service on the Intercloud Fabric cloud.

6. Once successfully disabled, reenable the Routing Service on the Intercloud Fabric cloud.

Symptom: The Routing Service creation request from ICFC, to the associated ICS instance, fails.

Failed to create the Routing Service instance <management interface IP address> for ICF cloud.
This is primarily due to network connectivity issue between ICF and ICS.

Possible Cause: The Intercloud Fabric link operation is UP but the Routing Service instance creation request from ICFC, to the associated ICS instance, fails due to one of the following:

- Intermittent Network connectivity issue between ICFC and that ICS instance.

- Internal error within that ICS instance because it failed to successfully process the Routing Service instance creation request from ICFC.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Review ICFC. Verify that the 'proxyManagedEndpointStatus' MO, corresponding to the Routing Service instance, is present in ICFC with the value BOOTSTRAP_FAILED.

3. Review ICS. Review the Routing Service Status and verify the associated ICS instance, when reachable, to confirm that the Routing Service container has not been created.

4. If the Intercloud Fabric link is operationally UP, confirm the operationally UP status.

5. If the Intercloud Fabric link is UP, confirm that the issue is not due to intermittent connectivity issues by attempting to ping the ICS provider public IP address.

6. If there are no intermittent Network connectivity issues, determine whether or not ICS has any internal errors when attempting to process the Routing Service create request.

7. After fixing the root cause, disable the Routing Service on the Intercloud Fabric cloud.

8. Once successfully disabled, reenable the Routing Service on the Intercloud Fabric cloud.

Symptom: ICFC fails to log in to the successfully created Routing Service instance.

Failed to create the Routing Service instance <management interface IP address> for ICF cloud.
This is primarily due to network connectivity issue between ICF and ICS.

Possible Cause: The Routing Service instance creation request from ICFC, to the associated ICS instance, succeeds. However, ICFC is unable to successfully complete the initial handshake with that Routing Service instance due to one of the following:

- Intermittent Network connectivity issue between ICFC and that Routing Service instance.

- Internal error within the Routing Service instance because it failed to successfully process the initial handshake from ICFC.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Review ICFC. Verify that the 'proxyManagedEndpointStatus' MO, corresponding to the Routing Service instance, is present in ICFC with the value BOOTSTRAP_FAILED.

3. Review ICS. Review the Routing Service Status and verify the associated ICS instance to confirm that the Routing Service container is present.

4. If the Intercloud Fabric link is operationally UP, confirm the operationally UP status.

5. If the Intercloud Fabric link is UP, confirm that the issue is not due to intermittent connectivity issues by attempting to ping the Routing Service instance management IP address.

6. If there are no intermittent Network connectivity issues, determine whether or not the Routing Service has any internal errors when attempting to process the log in request over RESTful API.

7. After fixing the root cause, disable the Routing Service on the Intercloud Fabric cloud.

8. Once successfully disabled, reenable the Routing Service on the Intercloud Fabric cloud.

Issues Disconnecting Intercloud Fabric Routing Service from Networks

Symptom: Interface deletion errors appear when disconnecting the Routing Service from networks.

Possible Cause:

- An IP address release fails.

- Deleting an ICFC network interface fails.

- ICFC times out.

- The Routing Service is not in the system or is in a failed state.

- The Intercloud Fabric Switch or the integrated gateway cannot be reached.

Verification and Solution:

1. Review the service request logs for warnings.

2. Confirm that the Routing Service interface is properly configured and that the ICFC IP address can be reached.

3. Confirm that the Routing Service and the Intercloud Fabric Switch can be reached.

4. Confirm that the Intercloud Fabric Switch is registered with the cVSM.

Symptom: Routing Service interfaces cannot be deleted.

Possible Cause:

- All network interfaces are Layer 3 interfaces.

- The Routing Service is not in the system or is in a failed state.

Verification and Solution:

1. Review the service request logs for warnings.

2. Confirm that the data Layer 3 network (other than management and transport) is associated with the VDC.

Symptom: The Intercloud Fabric link operational status goes DOWN while the Routing Service delete request is in progress.

Failed to create the Routing Service instance <Routing Service IP address> for ICF cloud. This is primarily due to network connectivity issue between ICFC and ICS.

Possible Cause: The operational status of the Intercloud Fabric link for the Intercloud Fabric cloud on which the Routing Service is to be deleted is not UP when the Routing Service disable request is in progress. Once the Routing Service disable request starts, Intercloud Fabric waits a maximum of 6 minutes for the Intercloud Fabric link operational status to go back UP.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Review the Intercloud Fabric workflow debug logs.

3. Review the ICFC logs.

4. Fix the Intercloud Fabric link operational DOWN status.

5. After the Intercloud Fabric link is UP, retry disabling the Routing Service on the Intercloud Fabric cloud.

Symptom: The Routing Service delete request from ICFC, to the associated ICS instance, fails.

Possible Cause: The Intercloud Fabric link operation is UP but the Routing Service instance deletion request from ICFC, to the associated ICS instance, fails due to one of the following:

- Intermittent Network connectivity issue between ICFC and that ICS instance.

- Internal error within the ICS instance due to it failing to successfully process the Routing Service instance deletion request from ICFC.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Review the Intercloud Fabric workflow debug logs.

3. Review the ICFC logs.

4. If the Intercloud Fabric link is operationally UP, confirm the operationally UP status.

5. If the Intercloud Fabric link is UP, confirm that the issue is not due to intermittent connectivity issues by attempting to ping the ICS provider public IP address.

6. If there are no intermittent Network connectivity issues, determine whether or not ICS has any internal errors when attempting to process the Routing Service delete request.

7. After fixing the root cause, disable the Routing Service on the Intercloud Fabric cloud.

Issues with Intercloud Fabric Routing Policy

Symptom: Configuring a routing policy fails when the Routing Service is being created.

Possible Cause: The Routing Service login fails.

Verification and Solution: Review the service request logs for warnings.

Symptom: Updating the routing policy is successful but the workflow fails.

Routing Policy Not Configured. <error string>.

Possible Cause:

- The workflow did not properly start.

- ICFC times out or returns an error.

Verification and Solution:

1. From the Manage Service Request screen, verify that the service request named icfRoutingPolicyConfiguration started.

2. Review the service request logs for warnings:

Symptom: Intercloud Fabric and ICFC suffer a communication error while sending a policy create request to ICFC.

Routing Policy Not Configured. <error-details>.

Possible Cause: Intercloud Fabric fails to send the policy create request to ICFC. This can be due to one of the following reasons:

- Authentication failure.

- ICFC service is down.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Fix the issue due to Intercloud Fabric failing to communicate with ICFC.

3. Once the root cause is fixed, redeploy the infrastructure.

Symptom: Intercloud Fabric and ICFC suffer a communication error while sending an update request to ICFC.

Routing Policy Update Failed.

Possible Cause: Intercloud Fabric fails to send the routing policy update request to ICFC. This can be due to one of the following reasons:

- Authentication failure.

- ICFC service is down.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Fix the issue due to Intercloud Fabric failing to communicate with ICFC.

3. Once the root cause is fixed, perform another routing policy update.

Symptom: Intercloud Fabric and ICFC suffer a communication error while sending an update request to ICFC.

Routing Policy Update Failed.

Possible Cause: Intercloud Fabric fails to send the routing policy update request to ICFC. This can be due to one of the following reasons:

- Authentication failure.

- ICFC service is down.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Fix the issue due to Intercloud Fabric failing to communicate with ICFC.

3. Once the root cause is fixed, perform another routing policy update.

Symptom:ICFC fails to send the routing policy configuration to Routing Service instance(s).

Possible Cause: The configuration attempt fails due to one of the following reasons: Intercloud Fabric fails to send the routing policy update request to ICFC. This can be due to one of the following reasons:

1. Intermittent network connectivity issue between ICFC and that Routing Service instance.

2. An internal error within Routing Service instance occurs due to it failing to successfully process the request from ICFC.

3. The Intercloud Fabric link operational status is DOWN.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Verify if the Intercloud Fabric link is operationally UP.

3. If the Intercloud Fabric link is UP, verify that the issue is not due to intermittent connectivity issues between ICFC and the Routing Service.

4. If the Intercloud Fabric link is operationally UP and there are no intermittent network connectivity issues, check if the Routing Service has any internal errors when attempting to process the request.

5. Once the root cause is fixed, ICFC automatically pushes the latest routing policy configuration to the Routing Service instance(s).

6. Intercloud Fabric detects the successful configuration of the routing policy by verifying it with ICFC. It then updates the routing policy configuration status for the Routing Service instance to be configured.

Symptom: Intercloud Fabric and ICFC suffer a communication error while polling to verify the results of policy configuration on the Routing Service instance(s).

Routing Service Update Failed [ <link name>: ErrorCode:<code> Details:<ICFC communication exception message> ].

Possible Cause: Intercloud Fabric fails to receive the Routing Service instance status to verify if the configuration attempt is successful or not. This can be due to the following reasons:

- Authentication failure.

- ICFC service is down.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Fix the issue due to Intercloud Fabric failing to communicate with ICFC.

3. Once the root cause is fixed, ICFC automatically pushes the latest routing policy configuration to the Routing Service instance(s).

4. Intercloud Fabric detects the successful configuration of the routing policy by verifying it with ICFC. It then updates the routing policy configuration status for the Routing Service instance to be configured.

Symptom: Intercloud Fabric times out waiting for ICFC to configure the policy on its Routing Service instance(s).

Update Routing Policy FailedConfiguring Routing Policy on Routing Service on ICF link <link name> timed out: <poll timeout message in ICF>.

Possible Cause: Intercloud Fabric gives up waiting for ICFC to configure the routing policy. ICFC suffers an internal error. This prevents ICFC itself from timing out the policy configuration flow.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Fix the issue due to ICFC failing to complete the operation in time by restarting ICFC services.

3. Once the root cause is fixed, ICFC automatically pushes the latest routing policy configuration to the Routing Service instance(s).

4. Intercloud Fabric detects the successful configuration of the routing policy by verifying it with ICFC. It then updates the routing policy configuration status for the Routing Service instance to be configured.

- Network delete request.

- Network update request (dummy update).

- VDC delete request.

Issues with Intercloud Fabric Virtual Data Center

Symptom: Traffic between cloud VMs fails.

Possible Cause: The Routing Service fails to connect the cloud VMs.

Verification and Solution:

1. Confirm that the Routing Service can reach both of the cloud VM NICs by pinging the cloud VM IP addresses from the Routing Service.

Perform a test ping:

<Menu Options list>

SELECT> 1

Enter IP Adress/Hostname:

After the IP address is entered, the ping command is executed for 5 packets:

Enter IP Address/Hostname : 10.36.10.198
PING 10.36.10.198 (10.36.10.198) 56(84) byts of data.
64 bytes from 10.36.10.198: icmp_seq=1 ttl=64 time=5.28 ms
64 bytes from 10.36.10.198: icmp_seq=2 ttl=64 time=5.78 ms
64 bytes from 10.36.10.198: icmp_seq=3 ttl=64 time=5.38 ms
64 bytes from 10.36.10.198: icmp_seq=4 ttl=64 time=6.66 ms
64 bytes from 10.36.10.198: icmp_seq=5 ttl=64 time=5.35 ms

--- 10.36.10.198 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 5.285/5.694/6.660/0.518 ms

Press return to continue ...

2. If the connection to the Routing Service fails, determine whether or not the site-to-site tunnel status on the Intercloud Fabric Switch is operational.

AWS-712-Link-ics-1# show intercloud tunnel brief
--------------------------------------------------------------
Tunnel   Tunnel          Tunnel   Control   Peer
ID       Type            Status   Status    Name
--------------------------------------------------------------
1        Site-to-Site    Up       Up        AWS-712-Link-ics-1

3. Confirm that the Intercloud Fabric Switch is registered with the cVSM.

vsm598859926# show mod
Mod Ports Module-Type                   Model              Status
--- ----- ------------------------      -----------        ----------
1   0     Virtual Supervisor Module     Nexus1000V         active*
3   1002  Virtual Cloud Extender Module InterCloudExtender ok
4   1022  Virtual Cloud Extender Module InterCloudSwitch   ok

Mod Sw              Hw
--- -----------     -----------------------
1   5.2(1)SK3(1.3)  0.0
3   5.2(1)SK3(1.3)  Linux 3.14.27-ics10
4   5.2(1)SK3(1.3)  Linux 3.14.27-ics10

Mod Server-IP         Server-UUID                          Server-Name
--- -------------     ------------------------             ----------------
1   10.36.5.90        NA                                   NA
3   10.36.5.91        2F50E5A0-47DF-5877-3C6D-3573F3351C94 AWS-712-Link-icx-1
4   10.36.5.92        485642DC-00AB-C149-F5CF-088930A4ADBB AWS-712-Link-icx-1

* this terminal session

4. Determine whether or not the secure tunnel to cloud VM is operational on the Intercloud Fabric Switch.

AWS-712-Link-ics-1# show intercloud tunnel brief
--------------------------------------------------------------------
Tunnel    Tunnel          Tunnel    Control    Peer
ID        Type            Status    Status     Name
--------------------------------------------------------------------
1         Site-to-Site    Up        Up         AWS-712-Link-ics-1
2         Access          Up        Up         vm-a27f5
3         Access          Up        Up         vm-6e5de
4         Access          Up        Up         vm-17029
5         Access          Up        Up         vm-24875
6         Access          Up        Up         vm-ae865
--------------------------------------------------------------------

5. In the Routing Service, verify that the interfaces are created for both networks and are in an operational state.

6. Review the display network details to confirm that the data VLAN for both VLANs is in operation and contains the IP addresses from their respective networks.

Interfaces
-----------
data
ip6gre0
ip6tn10
lo
mgmt
pCeth

Enter the interfaces name details. Press return for all interfaces, or 0 to exit.

Output for details on 1 interface:

Enter the interfaces name details. Press return for all interfaces, or 0 to exit: mgmt

Interface details
-----------------
13: mgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/either 7e:00:18:70:00:03 brd ff:ff:ff:ff:ff:ff promiscuity 0
veth
RX: bytes  packets  errors  dropped overun  mcast
2243384597 13757572 0       4870857 0       0
TX: bytes  packets  errors  dropped carrier collsns
4676361    29527    0       0       0       0

Address
-------
13: mgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> qdisc pfifo_fast state UP group default qlen 1000
link/either 7e:00:18:70:00:03 brd ff:ff:ff:ff:ff:ff
inet 10.36.70.28/16 brd 10.36.255.255 scope global mgmt
valid_lft forever preferred_lft forever
inet6 fe80::7c00::18ff:fe70:3/64 scope link
valid_lft forever preferred_lft forever

Output for all interfaces:

data    Link encap:Ethernet  HWaddr 7e:00:18:70:00:04
inet6 addr: fe80::7c00:18ff:fe70:4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1352 Metric:1
RX packets:13724218 errors:0 dropped:4871814 overruns:0 frame:0
TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2239552998 (2.2 GB)  TX bytes:738 (738.0 B)

ip6gre0 Link ecap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
NOARP  MTU:1448  Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 GB)  TX bytes:0 (0.0 B)

ip6tn10 Link ecap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
NOARP  MTU:1452  Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 GB)  TX bytes:0 (0.0 B)

lo      Link ecap:Local loopback
inet addr:127.0.0.1  Mask:255.0.0.0
inet6 addr: ::1/128  Scope:Host
UP LOOPBACK RUNNING  MTU:65536 Metric:1
RX packets:3473237 errors:0 dropped:0 overruns:0 frame:0
TX packets:3473237 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:181018327 (181.0 GB)  TX bytes:181018327 (181.0 B)

mgmt    Link encap:Ethernet  HWaddr 7e:00:18:70:00:03
inet addr:10:36:70:28  Bcast:10:36:255:255  Mask:255:255:0:0
inet6 addr: fe80::7c00:18ff:fe70:3/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1352 Metric:1
RX packets:13759826 errors:0 dropped:4871612 overruns:0 frame:0
collisions:0 txqueuelen:1000
RX bytes:2243746072 (2.2 GB)  TX bytes:4678855 (4.6 B)

pCeth   Link encap:Ethernet  HWaddr a2:e6:22:1a:20:f2
inet addr:172:17:10:2  Bcast:172:17:10:255  Mask:255:255:0
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 GB)  TX bytes:0 (0.0 B)

7. Log in to the Intercloud Fabric Switch and verify that the data trunk port-profile of 0-data-vem has the VLANs of both cloud VM NICs.

link1-ics-1# show intercloud services
Routing
===========================================================
Enabled: Yes
Status: Running <--- Status should be Running
Management IP:10:193:180:237

link1-ics-1# vemcmd show port
LTL  VSM Port Admin Link State PC-LTL SGID   Vem Port Type ORG svcpath Owner
50     Veth3    UP   UP   FWD      0        eth1-vem        1       0
51     Veth4    UP   UP   F/B*     0         veth1-0        0       0 <---+
52     Veth5    UP   UP   F/B*     0      0-data-vem        0       0 <---+ Should be in F/B state. Note VSM Port name for the next command
53     Veth6    UP   UP   FWD      0      0-mgmt-vem        1       0

F/B: Port is BLOCKED on some of the vlans.
One or more vlans are either not created or
not in the list of allowed vlans for this port.
Please run "vemcmd show port vlans" to see the details.
link1-ics-1# vemcmd show port vlans
Native   VLAN   Allowed
LTL  VSM Port  Mode    VLAN     State* Vlans
50     Veth3   A        280    FWD    280
51     Veth4   T          1    FWD    280,
52     Veth5   T          1    FWD    280, 1161-1163 <--- Should have the VLANs of both the CVMs.
53     Veth6   A        280    FWD    280

VLAN State: VLAN State represents the state of allowed vlans.
link1-ics-1# vemcmd show dvport
LTL  VSM Port  DVPortID      DVPortGroup  Vem Port
50     Veth3         0 e52e4c4c-38e7-4774-84b0-16dc6a5063a3_ICS_Access_36  eth1-vem
51     Veth4         0 e52e4c4c-38e7-4774-84b0-16dc6a5063a3_ICS_Trunk_Tunnel  veth1-0
52     Veth5         0 e52e4c4c-38e7-4774-84b0-16dc6a5063a3_ICS_Trunk_Tunnel  0-data-vem <-- Check this port profile in VSM has all the L3 vlan-s of CVSM.
53     Veth6         0 e52e4c4c-38e7-4774-84b0-16dc6a5063a3_ICS_Access_36  0-mgmt-vem

8. If 0-data-vem is in BLK state, determine whether or not the Intercloud Fabric Switch is registered with the cVSM.

9. If 0-data-vem does not include both VLANs, determine whether or not the VLANs have been created in the cVSM and are part of the trunk port-profile.

10. If the cloud VM network is stretched across Layer 3, verify that the ARP filter entry is created in the Intercloud Fabric Switch for the private cloud gateway with the MAC address of the Routing Service interface.

link1-ics-1#vemcmd show arp all
Flags: D-Dynamic S-Static d-Delete s-Sticky P-Proxy B-Public C-Create
X-Exlusive
VLAN/SEFID     IP Address     MAC Address        Flags     Expiry
1161           192.168.61.1   8e.00:04:00:00:04  S P       0   <--- Make sure that the flags S and P are set.

11. Log in to the cloud VM and verify that it is configured with the correct IP address for the default gateway (ip route show on Linux cloud VMs, route print on Windows cloud VMs). For Layer 3 cloud-only networks, the default gateway should be the same as the cloud gateway; for networks stretched across Layer 3, the default gateway should be the private cloud gateway.

12. Log in to the cloud VMs and verify that they are configured with the private cloud gateway IP address for the default gateway (ip route show on Linux cloud VMs, route print on Windows cloud VMs). For Layer 3 cloud-only networks, the default gateway should be the same as the cloud gateway; for networks stretched across Layer 3, the default gateway should be the private cloud gateway.

Symptom: Traffic between the cloud VM and the private cloud VM fails.

Possible Cause: The Routing Service fails to connect the cloud VM and the private cloud VM IP addresses from the Routing Service.

Verification and Solution:

1. Confirm that the Routing Service can reach both of the cloud VM NICs by pinging the cloud VM IP addresses from the Routing Service.

Perform a test ping:

<Menu Options list>

SELECT> 1

Enter IP Adress/Hostname:

After the IP address is entered, the ping command is executed for 5 packets:

Enter IP Address/Hostname : 10.36.10.198
PING 10.36.10.198 (10.36.10.198) 56(84) byts of data.
64 bytes from 10.36.10.198: icmp_seq=1 ttl=64 time=5.28 ms
64 bytes from 10.36.10.198: icmp_seq=2 ttl=64 time=5.78 ms
64 bytes from 10.36.10.198: icmp_seq=3 ttl=64 time=5.38 ms
64 bytes from 10.36.10.198: icmp_seq=4 ttl=64 time=6.66 ms
64 bytes from 10.36.10.198: icmp_seq=5 ttl=64 time=5.35 ms

--- 10.36.10.198 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 4006ms
rtt min/avg/max/mdev = 5.285/5.694/6.660/0.518 ms

Press return to continue ...

2. Check if the site-to-site tunnel status on the Intercloud Fabric Switch is operational.

AWS-712-Link-ics-1# show intercloud tunnel brief
--------------------------------------------------------------
Tunnel   Tunnel          Tunnel   Control   Peer
ID       Type            Status   Status    Name
--------------------------------------------------------------
1        Site-to-Site    Up       Up        AWS-712-Link-ics-1

3. Check if the Intercloud Fabric Switch is registered with the cVSM.

vsm598859926# show mod
Mod Ports Module-Type                   Model              Status
--- ----- ------------------------      -----------        ----------
1   0     Virtual Supervisor Module     Nexus1000V         active*
3   1002  Virtual Cloud Extender Module InterCloudExtender ok
4   1022  Virtual Cloud Extender Module InterCloudSwitch   ok

Mod Sw              Hw
--- -----------     -----------------------
1   5.2(1)SK3(1.3)  0.0
3   5.2(1)SK3(1.3)  Linux 3.14.27-ics10
4   5.2(1)SK3(1.3)  Linux 3.14.27-ics10

Mod Server-IP         Server-UUID                          Server-Name
--- -------------     ------------------------             ----------------
1   10.36.5.90        NA                                   NA
3   10.36.5.91        2F50E5A0-47DF-5877-3C6D-3573F3351C94 AWS-712-Link-icx-1
4   10.36.5.92        485642DC-00AB-C149-F5CF-088930A4ADBB AWS-712-Link-icx-1

* this terminal session

4. Confirm that the secure tunnel to the cloud VM is operational on the Intercloud Fabric Switch.

AWS-712-Link-ics-1# show intercloud tunnel brief
--------------------------------------------------------------------
Tunnel    Tunnel          Tunnel    Control    Peer
ID        Type            Status    Status     Name
--------------------------------------------------------------------
1         Site-to-Site    Up        Up         AWS-712-Link-ics-1
2         Access          Up        Up         vm-a27f5
3         Access          Up        Up         vm-6e5de
4         Access          Up        Up         vm-17029
5         Access          Up        Up         vm-24875
6         Access          Up        Up         vm-ae865
--------------------------------------------------------------------

5. If the private cloud VM is on a stretched network, verify in the Routing Service that interfaces are created for both networks and are in an operational state.

6. Review the display network details to confirm that the data VLAN for both VLANs is in operation and contains the IP addresses from their respective networks.

Interfaces
-----------
data
ip6gre0
ip6tn10
lo
mgmt
pCeth

Enter the interfaces name details. Press return for all interfaces, or 0 to exit.

Output for details on 1 interface:

Enter the interfaces name details. Press return for all interfaces, or 0 to exit: mgmt

Interface details
-----------------
13: mgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
link/either 7e:00:18:70:00:03 brd ff:ff:ff:ff:ff:ff promiscuity 0
veth
RX: bytes  packets  errors  dropped overun  mcast
2243384597 13757572 0       4870857 0       0
TX: bytes  packets  errors  dropped carrier collsns
4676361    29527    0       0       0       0

Address
-------
13: mgmt: <BROADCAST,MULTICAST,UP,LOWER_UP> qdisc pfifo_fast state UP group default qlen 1000
link/either 7e:00:18:70:00:03 brd ff:ff:ff:ff:ff:ff
inet 10.36.70.28/16 brd 10.36.255.255 scope global mgmt
valid_lft forever preferred_lft forever
inet6 fe80::7c00::18ff:fe70:3/64 scope link
valid_lft forever preferred_lft forever

Output for all interfaces:

data    Link encap:Ethernet  HWaddr 7e:00:18:70:00:04
inet6 addr: fe80::7c00:18ff:fe70:4/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1352 Metric:1
RX packets:13724218 errors:0 dropped:4871814 overruns:0 frame:0
TX packets:9 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2239552998 (2.2 GB)  TX bytes:738 (738.0 B)

ip6gre0 Link ecap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
NOARP  MTU:1448  Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 GB)  TX bytes:0 (0.0 B)

ip6tn10 Link ecap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
NOARP  MTU:1452  Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 GB)  TX bytes:0 (0.0 B)

lo      Link ecap:Local loopback
inet addr:127.0.0.1  Mask:255.0.0.0
inet6 addr: ::1/128  Scope:Host
UP LOOPBACK RUNNING  MTU:65536 Metric:1
RX packets:3473237 errors:0 dropped:0 overruns:0 frame:0
TX packets:3473237 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:181018327 (181.0 GB)  TX bytes:181018327 (181.0 B)

mgmt    Link encap:Ethernet  HWaddr 7e:00:18:70:00:03
inet addr:10:36:70:28  Bcast:10:36:255:255  Mask:255:255:0:0
inet6 addr: fe80::7c00:18ff:fe70:3/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1352 Metric:1
RX packets:13759826 errors:0 dropped:4871612 overruns:0 frame:0
collisions:0 txqueuelen:1000
RX bytes:2243746072 (2.2 GB)  TX bytes:4678855 (4.6 B)

pCeth   Link encap:Ethernet  HWaddr a2:e6:22:1a:20:f2
inet addr:172:17:10:2  Bcast:172:17:10:255  Mask:255:255:0
BROADCAST MULTICAST MTU:1500 Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:0 (0.0 GB)  TX bytes:0 (0.0 B)

7. If the private cloud VM is on a private cloud-only network, confirm that the transport network interface is created on the Routing Service and is in an operational state.

8. Review the display routing table details to confirm that a route exists to the private cloud VM network by way of the transport network gateway.

9. Log in to the Intercloud Fabric Switch and verify that the data trunk port-profile of 0-data-vem includes the VLAN of the cloud VM.

10. If the private cloud VM is on a stretched network, verify that the data trunk port-profile that contains 0-data-vem and veth 1-0 includes the VLAN of the private cloud VM.

11. If the private cloud VM is on a private cloud-only network, verify that 0-data-vem and veth 1-0 include the VLAN of the transport network.

link1-ics-1# show intercloud services
Routing
===========================================================
Enabled: Yes
Status: Running <--- Status should be Running
Management IP:10:193:180:237

link1-ics-1# vemcmd show port
LTL  VSM Port Admin Link State PC-LTL SGID   Vem Port Type ORG svcpath Owner
50     Veth3    UP   UP   FWD      0        eth1-vem        1       0
51     Veth4    UP   UP   F/B*     0         veth1-0        0       0 <---+
52     Veth5    UP   UP   F/B*     0      0-data-vem        0       0 <---+ Should be in F/B state. Note VSM Port name for the next command
53     Veth6    UP   UP   FWD      0      0-mgmt-vem        1       0

F/B: Port is BLOCKED on some of the vlans.
One or more vlans are either not created or
not in the list of allowed vlans for this port.
Please run "vemcmd show port vlans" to see the details.
link1-ics-1# vemcmd show port vlans
Native   VLAN   Allowed
LTL  VSM Port  Mode    VLAN     State* Vlans
50     Veth3   A        280    FWD    280
51     Veth4   T          1    FWD    280, 1162 <--- Should have the VLAN EVM/transport network
52     Veth5   T          1    FWD    280, 1161-1163 <--- Should have the VLANs of both the CVM and EVM/transport network.
53     Veth6   A        280    FWD    280

VLAN State: VLAN State represents the state of allowed vlans.
link1-ics-1# vemcmd show dvport
LTL  VSM Port  DVPortID      DVPortGroup  Vem Port
50     Veth3         0 e52e4c4c-38e7-4774-84b0-16dc6a5063a3_ICS_Access_36  eth1-vem
51     Veth4         0 e52e4c4c-38e7-4774-84b0-16dc6a5063a3_ICS_Trunk_Tunnel  veth1-0
52     Veth5         0 e52e4c4c-38e7-4774-84b0-16dc6a5063a3_ICS_Trunk_Tunnel  0-data-vem <-- Check this port profile in VSM has all the L3 vlan-s of both the CVM and EVM/transport network.
53     Veth6         0 e52e4c4c-38e7-4774-84b0-16dc6a5063a3_ICS_Access_36  0-mgmt-vem

12. If 0-data-vem is in BLK state, confirm that the Intercloud Fabric Switch is registered with cVSM.

13. If 0-data-vem does not have both VLANs, determine whether or not the VLANs have been created in cVSM and are part of the trunk port-profile.

14. If the network is stretched across Layer 3, verify that the ARP filter entry is created in the Intercloud Fabric Switch for the private cloud gateway with the MAC address of the Routing Service interface. Make sure flags S and P are set.

link1-ics-1# vemcmd show arp all
Flags: D-Dynamic S-Static d-Delete s-Sticky P-Proxy B-Public C-Create
X-Exlusive
VLAN/SEFID     IP Address     MAC Address        Flags     Expiry
1161           192.168.61.1   8e.00:04:00:00:04  S P       0

15. Log in to ICX and verify that both trunk ports carry the private cloud VM VLAN.

link-icx-1# vemcmd show port vlans
NATIVE    VLAN     Allowed
LTL   VSM Port   Mode   VLAN      State*   Vlans
50     Veth1    T      1         FWD      280, 1162
51     Veth2    T      1         FWD      280, 1162

16. Log in to the cloud VM and verify that it is configured with the correct IP address for the default gateway (ip route show on Linux cloud VMs, route print on Windows cloud VMs). For Layer 3 cloud-only networks, the default gateway should be the same as the cloud gateway; for networks stretched across Layer 3, the default gateway should be the private cloud gateway.

17. Log in to the cloud VMs and verify that they are configured with the private cloud gateway IP address for the default gateway (ip route show on Linux cloud VMs, route print on Windows cloud VMs). If the private cloud VM is on private cloud-only network, verify that the transport network gateway can be reached from the private cloud VM.

Symptom: Connectivity between a private VM and a cloud VM in different VLANs fails intermittently.

Possible Causes:

- A Cisco Nexus 1000V is used in the private data center with Unknown-Unicast-Flooding-Block enabled.

- The private VM and cloud VM are in different VLANs, thereby causing inter-VLAN routing to fail.

Verification and Solution:

1. Confirm that packets are being dropped by entering the following command on the VEM:

vemcmd show port-drops

2. Determine whether Unknown-Unicast-Flooding-Block is enabled or disabled by entering the following command on the Cisco Nexus 1000V:

privateVSM# show run | include uufb
uufb enable

3. If Unknown-Unicast-Flooding-Block is enabled, disable it by entering the following command in config mode:

no uufb enable

If Unknown-Unicast-Flooding-Block is already disabled, see other topics in this section for more information about troubleshooting inter-VLAN routing issues.

Symptom: For cloud VMs whose default gateway points to the Routing Service that is in the cloud, inter-VLAN routing does not work, but communication within the same VLAN does work.

Possible Cause:

- The Routing Service subinterface IP addresses are incorrect.

- The default gateway for the cloud VM does not point to the Routing Service subinterface IP address.

- The Routing Service does not have the appropriate routes.

- The Intercloud Fabric Switch is routing a VLAN that is not in the list of allowed VLANs in the Cisco Nexus 1000V default trunk.

- The cloud VM cannot reach the default gateway.

Verification and Solution:

1. In ICFC, ensure that the Routing Service subinterfaces are configured with the correct IP address.

2. Log in to the cloud VM and verify that its default gateway points to the Routing Service subinterface IP address (from Step 1).

3. Log in to the Routing Service and verify that the appropriate routes are present.

4. Log in to the Intercloud Fabric Switch and verify that the VLAN being routed is in the list of allowed VLANs in the Cisco Nexus 1000V default trunk:

# vemcmd show port
# vemcmd show port vlans

5. Verify that the cloud VM can reach the default gateway, which is the Routing Service subinterface.

Symptom: For cloud VMs whose default gateway points to the Routing Service that is in the private data center, inter-VLAN routing does not work, but communication within the same VLAN does work.

Possible Cause:

- The Routing Service subinterface IP addresses are incorrect.

- The Intercloud Fabric Switch is routing a VLAN that is not in the list of allowed VLANs in the Cisco Nexus 1000V default trunk.

- The Routing Service does not have the appropriate routes.

- The default gateway for the cloud VM is a Routing Service that is not in the cloud.

- The cloud VM cannot reach the default gateway.

- Inter-VLAN routing is not working.

Verification and Solution:

1. In ICFC, confirm that the Extend Default Gateway option is checked for the Routing Service subinterface and that the IP addresses are correct.

2. Log in to the Intercloud Fabric Switch and verify that the VLAN being routed is in the list of allowed VLANs in the Cisco Nexus 1000V default trunk:

# vemcmd show port
# vemcmd show port vlans

3. From the Routing Service, confirm that the IP address from Step 1 is displayed in the output of the following command:

# vemcmd show arp all

4. Log in to the Routing Service and verify that the appropriate routes are present.

5. In ICFC, uncheck the Extend Default Gateway box.

6. Log in to the cloud VM and change the default gateway to the Routing Service that is in the cloud.

7. Verify that the cloud VM can reach the default gateway, which is the Routing Service subinterface.

8. Verify that inter-VLAN routing works.

Issues with Intercloud Fabric Routing Service Internal Errors

Symptom: Intercloud Fabric fails to communicate with ICFC.

Routing Service creation request to ICFC failed. <ICFC exception message>.

Possible Cause: Intercloud Fabric fails to successfully communicate with ICFC.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Review the Intercloud Fabric workflow debug logs.

3. Fix the root cause failure and disable the Routing Service on the Intercloud Fabric cloud.

4. Once successfully disabled, reenable the Routing Service on that Intercloud Fabric cloud.

Symptom: Intercloud Fabric times out during the Routing Service create instance.

Routing Service creation has timed out. ICFC internal error.

Possible Cause: Intercloud Fabric times out waiting for ICFC to create the Routing Service instance due to an ICFC internal error that prevents ICFC to itself time out the Routing Service create instance.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Fix the root cause failure by restarting the ICFC services.

3. Disable the Routing Service on the Intercloud Fabric cloud.

4. Once successfully disabled, reenable the Routing Service on that Intercloud Fabric cloud.

Symptom: ICFC rejects the Routing Service create request.

Possible Cause: Intercloud Fabric and ICFC are not synchronized after a Routing Service is deleted without being successfully cleaned up from ICFC and a new Routing Service is created with the same name (same as the Intercloud Fabric link name).

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Do one of the following:

- Create the Intercloud Fabric cloud and Intercloud Fabric link with different names.

- Contact Cisco Customer Support to assist in getting the ICFC database to be synchronized with Intercloud Fabric.

Symptom: Intercloud Fabric times out during the Routing Service delete instance.

Possible Cause: Intercloud Fabric times out waiting for ICFC to delete the Routing Service instance due to an ICFC internal error that prevents ICFC to itself time out the Routing Service delete instance.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Contact Cisco Customer Support to debug and fix the ICFC internal error problem.

3. After fixing the root cause, perform one of the following:

- If the Intercloud Fabric link deletion request failed, delete the Intercloud Fabric link again.

- If the Routing Service disable request failed, disable it again.

Issues with the Intercloud Fabric Routing Service Interface

This section includes symptoms, possible causes, and solutions for issues associated with the Routing Service interface.

- Issues Enabling and Configuring Intercloud Fabric Routing Service Interface

- Issues Disabling Intercloud Fabric Routing Service Interface

- Issues Connecting Intercloud Fabric Routing Service Interface to Networks

- Issues Disconnecting Intercloud Fabric Routing Service Interface from Networks

- Issues with Intercloud Fabric Routing Service Interface Internal Creation Errors

- Issues with Intercloud Fabric Routing Service Interface Internal Deletion Errors

Issues Enabling and Configuring Intercloud Fabric Routing Service Interface

Symptom: The Layer 3 data network, which is a stretched network, does not have a free IP address.

Invalid Interface IP for Network <network name>.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Add at least one more IP pool to the network.

3. Perform a dummy update on the network without changing any of its properties.

Symptom: The Layer 3 data network, which is a cloud-only network, does not have a reserved cloud gateway IP address.

Invalid Interface IP for Network <network name>.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Delete and recreate the cloud-only network.

Issues Disabling Intercloud Fabric Routing Service Interface

Symptom: The Routing Service interface IP address release fails.

Failed to release Mgmt IP <ips> for IG Intance <Routing Service name> (error message).

Possible Cause: The allocation failure can occur on one or more of the following interface types:

- Management network interface.

- Transport network interface.

- Data-only network interface.

Verification and Solution: Review the Intercloud Fabric service request logs.

Issues Connecting Intercloud Fabric Routing Service Interface to Networks

Symptom: ICFC fails to send the interface create request to the Routing Service instance(s).

Failed to configure Routing Service interface for Network (<network name>) on ICFCloud (<cloud name>). Failed to configure Routing Service interface.

Possible Cause: The interface create request, from Intercloud Fabric to ICFC, fails even though the Intercloud Fabric link is operationally UP. The creation fails for one of the following reasons:

- Intermittent network connectivity between ICFC and that Routing Service instance.

- An internal error within the Routing Service instance fails to successfully process the request from ICFC.

- Prior to the Intercloud Fabric link being sent to the Routing Service instance(s), its operational status goes DOWN after the interface create request is sent to ICFC.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Verify that the Intercloud Fabric link is operationally UP.

3. If the Intercloud Fabric link is UP, make sure that the issue is not due to intermittent connectivity issues between ICFC and the Routing Service.

4. If the Intercloud Fabric link is operationally UP and there are no intermittent network connectivity issues, determine if the Routing Service has any internal errors when attempting to process the request.

5. Once the root cause is fixed, perform the appropriate operation corresponding to the previous operation that initially triggered the interface creation. The following are the possible operations:

- Network update request (dummy update): network create/update as the initial request.

- VDC delete followed by VDC create: VDC create as the initial request.

Symptom: ICFC fails to successfully send the ARP filter create request to the Intercloud Fabric Switch instance(s).

Failed to configure Routing Service interface for Network (<network name>) on ICFCloud (<cloud name>). Enabling proxy ARP for Gateway (<enterprise gateway IP>) in ICS failed.

Possible Cause: The ARP filter create request, from Intercloud Fabric to ICFC, fails even though the Intercloud Fabric link is operationally UP. the creation fails for one of the following reasons:

- Intermittent network connectivity between ICFC and that Intercloud Fabric Switch instance.

- An internal error within the Intercloud Fabric Switch instance fails to successfully process the request from ICFC.

- Prior to the Intercloud Fabric link being sent to the Intercloud Fabric Switch instance(s), its operational status goes DOWN after the interface create request is sent to ICFC.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Verify that the Intercloud Fabric link is operationally UP.

3. If the Intercloud Fabric link is UP, make sure that the issue is not due to intermittent connectivity issues between ICFC and the Intercloud Fabric Switch.

4. If the Intercloud Fabric link is operationally UP and there are no intermittent network connectivity issues, determine if Intercloud Fabric Switch has any internal errors when attempting to process the request.

5. Once the root cause is fixed, perform the appropriate operation corresponding to the previous operation that initially triggered the interface creation. The following are the possible operations:

- Network update request (dummy update): network create/update as the initial request.

- VDC delete followed by VDC create: VDC create as the initial request.

Issues Disconnecting Intercloud Fabric Routing Service Interface from Networks

Symptom: ICFC fails to successfully send the ARP filter delete request to the Intercloud Fabric Switch instance(s).

Disabling proxy ARP in ICS for Gateway (<enterprise gateway IP>) failed.

Possible Cause: The ARP filter delete request, from Intercloud Fabric to ICFC, fails even though the Intercloud Fabric link is operationally UP. The deletion fails for one of the following reasons:

- Intermittent network connectivity between ICFC and that Intercloud Fabric Switch instance.

- An internal error within the Intercloud Fabric Switch instance fails to successfully process the request from ICFC.

- Prior to the Intercloud Fabric link being sent to the Intercloud Fabric Switch instance(s), its operational status goes DOWN after the ARP filter delete request is sent to ICFC.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Verify that the Intercloud Fabric link is operationally UP.

3. If the Intercloud Fabric link is UP, make sure that the issue is not due to intermittent connectivity issues between ICFC and the Intercloud Fabric Switch.

4. If the Intercloud Fabric link is operationally UP and there are no intermittent network connectivity issues, determine if Intercloud Fabric Switch has any internal errors when attempting to process the request.

5. Once the root cause is fixed, repeat the operation that initially triggered the interface deletion. The following are the possible operations:

- Network delete request.

- Network update request (dummy update).

- VDC delete request.

Symptom: ICFC fails to successfully send the interface delete request to the Routing Service instance(s).

Deleting Routing Service interface from Routing Service for Network (<network name>) failed. Failed to delete Routing Service interface.

Deleting Routing Service interface from Routing Service for Network (<network name>) failed. Failed to delete Routing Service interface.

Possible Cause: The interface delete request, from Intercloud Fabric to ICFC, fails even though the Intercloud Fabric link is operationally UP. The deletion fails for one of the following reasons:

- Intermittent network connectivity between ICFC and that Routing Service instance.

- An internal error within the Routing Service instance fails to successfully process the request from ICFC.

- Prior to the Intercloud Fabric link being sent to the Routing Service instance(s), its operational status goes DOWN after the interface create request is sent to ICFC.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Verify that the Intercloud Fabric link is operationally UP.

3. If the Intercloud Fabric link is UP, make sure that the issue is not due to intermittent connectivity issues between ICFC and the Routing Service.

4. If the Intercloud Fabric link is operationally UP and there are no intermittent network connectivity issues, determine if the Routing Service has any internal errors when attempting to process the request.

5. Once the root cause is fixed, repeat the operation that initially triggered the interface deletion. The following are the possible operations:

- Network delete request.

- Network update request (dummy update).

- VDC delete request.

Issues with Intercloud Fabric Routing Service Interface Internal Creation Errors

Symptom: Intercloud Fabric and ICFC suffer a communication error while sending an interface create request to ICFC.

Failed to create Routing Service interface for Network (<network name) on ICFCloud(<cloud name>). Creating Routing Service Interface for Network (network name>) failed. Error: <message from ICFC>.

Possible Cause: Intercloud Fabric fails to send the interface create request to ICFC. This can be due to one of the following reasons:

- Authentication failure.

- ICFC service is down.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Fix the issue due to Intercloud Fabric failing to communicate with ICFC.

3. Once the root cause is fixed, perform the appropriate operation corresponding to the previous operation that initially triggered the interface creation. The following are the possible operations:

- Network update request (dummy update): network create/update as initial request.

- VDC delete followed by VDC create: VDC create an initial request.

Symptom: Intercloud Fabric and ICFC suffer a communication error when polling to review the outcome of the interface create operation in ICFC.

Failed to create Routing Service interface for Network (<network name>) on ICFCloude(<cloud name>). Creating Routing Service Interface for Network (<network name>) failed. Error: <message from ICFC>.

Possible Cause: Intercloud Fabric fails to receive the status of the Routing Service instance to verify if the create operation is successful or not. This is due to the following reasons:

- Authentication failure.

- ICFC service is down.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Fix the issue due to Intercloud Fabric failing to communicate with ICFC.

3. Once the root cause is fixed, perform the appropriate operation corresponding to the previous operation that initially triggered the interface creation. The following are the possible operations:

- Network update request (dummy update): network create/update as initial request.

- VDC delete followed by VDC create: VDC create an initial request.

Symptom: Intercloud Fabric times out waiting for ICFC to create the interface.

Failed to create Routing Service interface for Network (<network name>) on ICFCloude(<cloud name>). Creating Routing Service Interface for Network (<network name>) failed. Error: <message from ICFC>.

Possible Cause: Intercloud Fabric gives up waiting for ICFC to create the interface. This only occurs when ICFC suffers an internal error. This internal error prevents ICFC itself from timing out the interface deletion flow.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Fix the issue due to ICFC failing to complete the operation in time by restarting ICFC services.

3. Once the root cause is fixed, perform the appropriate operation corresponding to the previous operation that initially triggered the interface creation. The following are the possible operations:

- Network update request (dummy update): network create/update as initial request.

- VDC delete followed by VDC create: VDC create an initial request.

Symptom: Intercloud Fabric and ICFC suffer a communication error while sending an ARP filter create request to ICFC.

Failed to create Routing Service interface for Network (<network name>) on ICFCloude(<cloud name>). Creating Routing Service Interface for Network (<network name>) failed. Error: <message from ICFC>.

Possible Cause: Intercloud Fabric fails to send the ARP filter create request to ICFC. This is due to one of the following reasons:

- Authentication failure.

- ICFC service is down.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Fix the issue due to Intercloud Fabric failing to communicate with ICFC.

3. Once the root cause is fixed, perform the appropriate operation corresponding to the previous operation that initially triggered the interface creation. The following are the possible operations:

- Network update request (dummy update): network create/update as initial request.

- VDC delete followed by VDC create: VDC create an initial request.

Symptom: Intercloud Fabric and ICFC suffer a communication error while polling to review the outcome of an ARP filter create operation in ICFC.

Failed to create Routing Service interface for Network (<network name>) on ICFCloude(<cloud name>). Creating Routing Service Interface for Network (<network name>) failed. Error: <message from ICFC>.

Possible Cause: Intercloud Fabric fails to receive the status of the Intercloud Fabric Switch instance to verify if the create operation is successful or not. This is due to the following reasons:

- Authentication failure.

- ICFC service is down.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Fix the issue due to Intercloud Fabric failing to communicate with ICFC.

3. Once the root cause is fixed, perform the appropriate operation corresponding to the previous operation that initially triggered the interface creation. The following are the possible operations:

- Network update request (dummy update): network create/update as initial request.

- VDC delete followed by VDC create: VDC create an initial request.

Symptom: Intercloud Fabric times out waiting for ICFC to create the ARP filter.

Failed to create Routing Service interface for Network (<network name>) on ICFCloude(<cloud name>). Creating Routing Service Interface for Network (<network name>) failed. Error: <message from ICFC>.

Possible Cause: Intercloud Fabric gives up waiting for ICFC to create the ARP filter. This only occurs when ICFC suffers an internal error. This internal error prevents ICFC itself from timing out the ARP filter deletion flow.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Fix the issue due to ICFC failing to complete the operation in time by restarting ICFC services.

3. Once the root cause is fixed, perform the appropriate operation corresponding to the previous operation that initially triggered the interface creation. The following are the possible operations:

- Network update request (dummy update): network create/update as initial request.

- VDC delete followed by VDC create: VDC create an initial request.

Issues with Intercloud Fabric Routing Service Interface Internal Deletion Errors

Symptom: Intercloud Fabric and ICFC suffer a communication error while sending an interface delete request to ICFC.

Deleting Routing Service interface from Routing Service for Network (<network name) failed. Error: <message indicating type of ICFC communication exception>.

Possible Cause: Intercloud Fabric fails to send the interface delete request to ICFC. This can be due to one of the following reasons:

- Authentication failure.

- ICFC service is down.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Fix the issue due to Intercloud Fabric failing to communicate with ICFC.

3. Once the root cause is fixed, repeat the operation that initially triggered the interface deletion. The following are the possible operations:

- Network delete request.

- Network update request (dummy update).

- VDC delete request.

Symptom: Intercloud Fabric and ICFC suffer a communication error while polling to review the outcome of the interface delete operation in ICFC.

Deleting Routing Service interface from Routing Service for Network (<network name) failed. Error: <message indicating type of ICFC communication exception>.

Possible Cause: Intercloud Fabric fails to receive the status of the Routing Service instance to verify if the delete operation is successful or not. This is due to the following reasons:

- Authentication failure.

- ICFC service is down.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Fix the issue due to Intercloud Fabric failing to communicate with ICFC.

3. Once the root cause is fixed, repeat the operation that initially triggered the interface deletion. The following are the possible operations:

- Network delete request.

- Network update request (dummy update).

- VDC delete request.

Symptom: Intercloud Fabric times out waiting for ICFC to delete the interface.

Deleting Routing Service interface from Routing Service for Network (<network name) failed. Error: <timeout message generated within Intercloud Fabric>.

Possible Cause: Intercloud Fabricgives up waiting for ICFC to delete the interface. ICFC has suffered an internal error. This internal error prevents ICFC itself from timing out the interface deletion flow.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Fix the issue due to ICFC failing to complete the operation in time by restarting ICFC services.

3. Once the root cause is fixed, repeat the operation that initially triggered the interface deletion. The following are the possible operations:

- Network delete request.

- Network update request (dummy update).

- VDC delete request.

Symptom: Intercloud Fabric and ICFC suffer a communication error while sending an ARP filter delete request to ICFC.

Disabling proxy ARP for Gateway (<enterprise gateway IP>) in ICS failed. Error: <message indicating type of ICFC communication exception>.

Possible Cause: Intercloud Fabric fails to send the ARP filter delete request to ICFC. This is due to one of the following reasons:

- Authentication failure.

- ICFC service is down.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Fix the issue due to Intercloud Fabric failing to communicate with ICFC.

3. Once the root cause is fixed, repeat the operation that initially triggered the interface deletion. The following are the possible operations:

- Network delete request.

- Network update request (dummy update).

- VDC delete request.

Symptom: Intercloud Fabric and ICFC suffer a communication error while polling to review the outcome of an ARP filter delete operation in ICFC.

Disabling proxy ARP for Gateway (<enterprise gateway IP>) in ICS failed. Error: <message indicating type of ICFC communication exception>.

Possible Cause: Intercloud Fabric fails to receive the status of the Intercloud Fabric Switch instance to verify if the delete operation is successful or not. This is due to the following reasons:

- Authentication failure.

- ICFC service is down.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Fix the issue due to Intercloud Fabric failing to communicate with ICFC.

3. Once the root cause is fixed, repeat the operation that initially triggered the interface deletion. The following are the possible operations:

- Network delete request.

- Network update request (dummy update).

- VDC delete request.

Symptom: Intercloud Fabric times out waiting for ICFC to delete the ARP filter.

Disabling proxy ARP for in ICS Gateway (<enterprise gateway IP>) in ICS failed. Error: <timeout message generated within Intercloud Fabric>.

Possible Cause: Intercloud Fabric gives up waiting for ICFC to create the ARP filter. ICFC suffers an internal error. This internal error prevents ICFC itself from timing out the ARP filter deletion flow.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Fix the issue due to ICFC failing to complete the operation in time by restarting ICFC services.

3. Once the root cause is fixed, repeat the operation that initially triggered the interface deletion. The following are the possible operations:

- Network delete request.

- Network update request (dummy update).

- VDC delete request.

Issues with the Intercloud Fabric Advanced Routing Service

This section includes symptoms, possible causes, and solutions for issues associated with the Advanced Routing Service.

- Issues Enabling and Configuring Intercloud Fabric Advanced Routing Service

- Issues Disabling Intercloud Fabric Advanced Routing Service

- Issues Connecting Intercloud Fabric Advanced Routing Service to Networks

- Issues Configuring Intercloud Fabric Advanced Routing Service Static Route

Issues Enabling and Configuring Intercloud Fabric Advanced Routing Service

Symptom: The management network does not have two (both management and service) unallocated IP addresses.

Advanced Routing Service management and service IP address allocation from network <network name> failed.

Possible Cause: Advanced Routing Service failed due to a lack of IP addresses in management network IP pool.

Verification and Solution:

1. Allocate at least two more IP addresses to the management network.

2. Delete the failed Intercloud Fabric link.

3. Create a new Intercloud Fabric link.

Symptom: The AWS marketplace End User License Agreement (EULA) not accepted for Advanced Routing Service images.

Possible Cause: The Advanced Routing Service instance creation fails due to EULA not being accepted for Advanced Routing Service images in AWS marketplace.

Verification and Solution:

1. Accept AWS EULA for Advanced Routing Service images.

2. Delete the failed Intercloud Fabric link.

3. Create a new Intercloud Fabric link.

Symptom: Intercloud Fabric and ICFC communication error.

Advanced Routing Service creation request to ICFC failed. <ICFC exception message>.

Possible Cause: The Advanced Routing Service creation fails due to an internal error within Intercloud Fabric. This can be due to one of the following reasons:

- Authentication failure.

- ICFC service is down.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Fix the issue due to Intercloud Fabric failing to communicate with ICFC.

3. Once the root cause is fixed, delete the Intercloud Fabric link.

4. Once successfully deleted, recreate the Intercloud Fabric link for the Advanced Routing Service.

Symptom: Intercloud Fabric times out during Advanced Routing Service creation.

Advanced Routing Service creation has timed out. ICFC internal error.

Possible Cause: Intercloud Fabric times out waiting for ICFC to create the Advanced Routing Service instance. This only occurs when ICFC suffers an internal error. This internal error prevents ICFC to timeout the Advanced Routing Service instance create flow.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Contact Cisco Customer Support to debug and fix the ICFC internal error.

3. Once successfully deleted, recreate the Intercloud Fabric link for the Advanced Routing Service.

Issues Disabling Intercloud Fabric Advanced Routing Service

Symptom: Intercloud Fabric and ICFC communication error.

Possible Cause: The Advanced Routing Service deletion fails due to an internal error within Intercloud Fabric. This can be due to one of the following reasons:

- Authentication failure.

- ICFC service is down.

Verification and Solution:

1. Review the Intercloud Fabric service request logs.

2. Fix the issue due to Intercloud Fabric failing to communicate with ICFC.

3. Once the root cause is fixed, if the Intercloud Fabric link deletion request fails, delete the Intercloud Fabric link again.

Symptom: Intercloud Fabric times out during the Advanced Routing Service delete attempt.

Undeploy Advanced Routing Service from Icflink <iclink name> has timed out: <timeout message in ICFD>.

Possible Cause:

- Intercloud Fabric times out waiting for ICFC to delete the Advanced Routing Service instance.

- ICFC suffers an internal error.

Verification and Solution:

1. Contact Cisco Customer Support to debug and resolve the ICFC internal error.

2. Once the root cause is resolved, if the Intercloud Fabric link deletion request fails, delete the Intercloud Fabric link again.

Symptom: The Advanced Routing Service interface IP address release fails.

Failed to release IP <ip address> assigned to Advanced Routing Service instance <name> nic with role <management/service>.

Failed to release ips (<ip address(s)>) associated with Advanced Routing Service interface <name>.

Possible Cause: Even if the deallocation of the resources within Intercloud Fabric fails, the Advanced Routing Service is successfully deleted. The allocation failure can occur for one or more of the following interface types:

- Data only Network interface

- Transport Network interface

- Management Network interface

Verification and Solution: Contact Cisco Customer Support to get the IP addresses released.

Issues Connecting Intercloud Fabric Advanced Routing Service to Networks

Symptom: The Intercloud Fabric link operational status is DOWN at the same time as the Advanced Routing Service create request is processing.

Advanced Routing Service <name> failed to create since link <name> is not-operational.

Possible Cause: Advanced Routing Service creation fails due to a network connection problem preventing Intercloud Fabric from reaching the Advanced Routing Service instance in the cloud for initial configuration:

- The operational status of the Intercloud Fabric link for the Intercloud Fabric cloud on which Advanced Routing Service is to be created must be UP before an attempt to create the Advanced Routing Service can be made.

- Once an Intercloud Fabric link is successfully deployed, Intercloud Fabric waits for a maximum of 15 minutes for the Intercloud Fabric link operational status to be UP.

Verification and Solution:

1. Resolve the Intercloud Fabric link operational DOWN status.

2. Delete the failed Intercloud Fabric link.

3. Create a new Intercloud Fabric link.

Note

For more information on troubleshooting the Intercloud Fabric link, see Issues with the Intercloud Fabric Link.

Issues Configuring Intercloud Fabric Advanced Routing Service Static Route

Symptom: Failed to complete prerequisites for configuring Advanced Routing Service static route.

Possible Cause:

- If the Routing Service interface creation fails and is still pending, static route will not be added on Advanced Routing Service.

- Intercloud Fabric link status must be operational.

- Routing Service must be present.

- Advanced Routing Service must be successfully created.

- If static route configuration fails, static route status will be marked as FAILED_TO_CONFIGURE and corresponding service requests will go to a failed state.

Verification and Solution:

1. Ensure the Intercloud Fabric link status is operational.

2. Resubmit a network update on the corresponding network to retrigger the static route creation.

Symptom: Advanced Routing Service rejects the REST request to create, update, delete, and save the static routes.

Possible Cause: Review the SR logs to view the HTTP error from Advanced Routing Service.

Verification and Solution: Review the Advanced Routing Service troubleshooting steps for creation failure and then attempt to reconfigure the static route.

Chapter 7. Obtaining Help from Cisco

- Cisco Intercloud Fabric Support Community

- Contacting Cisco Customer Support

- Obtaining Logs for Cisco Customer Support

Cisco Intercloud Fabric Support Community

For more information about troubleshooting issues with Cisco Intercloud Fabric, visit the Cisco Intercloud Fabric support community:

Click Here

Contacting Cisco Customer Support

If you cannot solve a problem after using the troubleshooting suggestions in this guide, contact a customer service representative for assistance and further instructions. Before contacting Cisco, collect the following information so that Cisco Customer Support can assist you as quickly as possible:

- Version of Cisco Intercloud Fabric software that you are running

- Contact phone number

- Brief description of the problem

- Brief explanation of the steps you have already taken to isolate and resolve the problem

After collecting this information, see for information on how to submit a service request.

Obtaining Logs for Cisco Customer Support

If you contact Cisco Customer Support for assistance, you might need to provide logs for them to assist in determining the cause of the problem. This procedure describes how to obtain logs for Cisco Intercloud Fabric.

Procedure

To obtain Intercloud Fabric logs:

1. In Intercloud Fabric, choose Administration > Support Information.

2. From the Support Information drop-down list, choose Show Log.

3. From the Show Log drop-down list, choose Infra Manager and click Show Logs.

4. Save the information in a file that you can send to Cisco Customer Support.

Chapter 8. Additional Information

- Related Documentation

- Documentation Feedback

- Obtaining Documentation and Submitting a Service Request

Related Documentation

Cisco Intercloud Fabric for Business

The Cisco Intercloud Fabric for Business documentation is available at the following URL:

Click Here

Cisco Intercloud Fabric for Provider

The Cisco Intercloud Fabric for Provider documentation is available at the following URL:

Click Here

Documentation Feedback

To provide technical feedback on this document, or to report an error or omission, please send your comments to: intercloud-fabric-doc-feedback@cisco.com.

We appreciate your feedback.

Obtaining Documentation and Submitting a Service Request

For information on obtaining documentation, using the Cisco Bug Search Tool (BST), submitting a service request, and gathering additional information, see What's New in Cisco Product Documentation.

To receive new and revised Cisco technical content directly to your desktop, you can subscribe to the What's New in Cisco Product Documentation RSS feed. RSS feeds are a free service.

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB's public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS" WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: Click Here . Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R)