This section includes symptoms, possible causes, and solutions for issues encountered while setting up an Intercloud Fabric cloud.

Symptom: Creating an Intercloud Fabric cloud failed.

Possible Cause: The network does not have enough available IP addresses.

Verification and Solution: Modify the IP address pools that are associated with the specific Intercloud Fabric cloud and recreate the Intercloud Fabric cloud.

Symptom: Resources required for this REST API operation are not available.

Possible Cause: Locks are not available for the required associated objects.

Verification and Solution:

1. From the REST API/UI, review the service request API page for log information and each lock acquired and released. Use the log details to identify the ongoing operation or any previous service requests that did not release the locks.

2. Review the MongoDB locks collection for details on unreleased locks.

3. If the operation, after waiting for some time, does not complete or the locks are not released, restart services from the administrator console. This will result in a release of the locks and move the stalled operation in a system aborted state.

Symptom: The REST API operation cannot be performed because the referenced object with OID {0} cannot be found.

Possible Cause: The referenced object may have been deleted by a concurrent operation running at approximately the same time and is no longer present.

Verification and Solution:

1. From the Rest API/UI, review the associated objects to verify if they are available.

2. Query MongoDB for the object and identify, from the input object or ‘additionalLockableEntities’, the object being referred.

3. To resolve, refer to an alternate object that is present or create the associated object.

Symptom: Credentials validation failed.

Possible Cause: The cloud provider credentials were entered incorrectly.

Verification and Solution: Reenter the cloud provider credentials.

Symptom: From the VMware vCenter client, you cannot determine which VSM virtual machine (VM) configuration is active (primary) and which configuration is standby (secondary).

Possible Cause: While deploying the Intercloud Fabric Switch in high availability mode, you created the VMs with names similar to the Intercloud Fabric VSM high availability VM names.

Verification and Solution: As a best practice, prefix active VM names with "pri" (primary) and standby VM names with "sec" (secondary).

Symptom: Intercloud Fabric cloud creation fails with a message stating that SCP is not present on the VM.

Error Message:

Jan 23 2015 20:32:15 GMT Error occurred while Template Creation: ICFC Task State: Failed. Task Message: OS not compatible:scp not present on VM
Jan 23, 2015 20:33:49 GMT Handler failed with error - ICFC Task State: failed. Task Message: OS not compatible:scp not present on VM, selectedContext=<None>
Jan 23, 2015 20:33:49 GMT Task #1 (InterCloud Create Template (Create Template in Cloud)) failed after 456 seconds

Possible Causes:

• SCP is not installed or is installed in the wrong location.

• SSH is not fully installed.

Verification and Solution:

1. Confirm that SCP is installed and in the correct location by entering the command whereis -b scp.

2. Confirm that SSH is fully installed. If it is not, use the yum command to update the SSH package by entering yum update package-name.

Symptom: Creating an Intercloud Fabric cloud fails with the error, "Failed to create instance".

Error Message:

Jan 23, 2015 07:28:08 GMT [Deploy IcSwitchInst : icldCiscoITDD3-iclink-ics-1] [Step 1 of 4 : Instantiate IcSwitch][Create instance failed for instance: Failed to create instance: 00050a12-fab0-43d6-005-a12fab043d6Error not populated][Retry Count:5]
Jan 23, 2015 07:28:08 GMT (Deploy IcSwitchInst : icldCiscoITDD3-iclink-ics-1 failed: Create instance failed for instance: Failed to create instance: 00050a12-fab0-43d6-0005-0a12fab043d6Error not populated)
Jan 23, 2015 07:28:08 GMT 'icldCiscoITDD3' is the last IcfCloud for Provider Account 'CiscoITDDwedd'.

Possible Cause: Multiple situations can cause this problem.

Verification: The most likely cause is that the networking infrastructure on the VMware vCenter client that controls how the VMs communicate prevents the Intercloud Fabric VSM from establishing a connection with the Intercloud Fabric Switch instance that is being brought up on the cloud provider.

Symptom: After rebooting the host on which Intercloud Fabric, ICFC, Intercloud Fabric Extender, and Intercloud Fabric VSM are deployed and powering on all VMs, there is no communication between private VMs and cloud VMs.

Possible Cause: The Intercloud Fabric Extender host was rebooted out-of-band, resulting in a loss of communication and modules being unable to register with the Intercloud Fabric VSM.

Verification and Solution:

1. Check module status by logging into the Intercloud Fabric VSM console and entering the show module command. The results should indicate that modules are not registered and that communication has been lost.

2. In the Intercloud Fabric UI, reboot the Intercloud Fabric Extender and the Intercloud Fabric Switch. Because this reboot occurs in-band, the Intercloud Fabric VSM can register the modules and communication is restored.

Symptom: The Intercloud Fabric cloud is deployed but the tunnel is not operational.

Possible Cause: The control channel port or data tunnel port is not open in the firewall.

Verification and Solution:

1. Confirm that the required protocol or port is configured for the Intercloud tunnel by entering the following command:

   icx-1# show intercloud tunnel config
   ICS-Profile:
   uuid                     2B59D348-9E48-F950-92B2-5258A9E2999F
   public-ip                54.151.47.163
   ICE-uuid                 855FD5BF-2BD0-7864-57DB-954DA517D887
   Site-to-Site-tunnel-profile:
         Heartbeat-interval       1
         Heartbeat-retries        120
         Tunnel-Protocol          UDP
         Tunnel-Port              6644                    
   Site-to-site-key-profile [Configured]
   Encryption Algorithm     AES-128-CBC
   Hash Algorithm           SHA1
   Key-Profile-Checksum     B722147B-276704CF-FBA8ACB6-13371C99-ECE116BF
   ReKey-State              Not-In-Progress
   Epoch                    0

2. Do one of the following:

   • If the desired protocol or port is not shown in the preceding command, change the protocol for the site-to-site tunnel in tunnel-profile in Intercloud Fabric:

     • If the default tunnel-profile is used in Intercloud Fabric cloud, it cannot be changed. In that case, create or clone a new Intercloud Fabric cloud with a new tunnel profile with the desired protocol.

     • If the nondefault tunnel profile is used in the Intercloud Fabric cloud, edit the tunnel profile with the correct protocol in the site-to-site tunnel section. To enforce the modified tunnel profile, you must reboot or switchover the Intercloud Fabric cloud. If the Intercloud Fabric cloud is in high availability mode, you must reboot or switch it over twice.

   • If the correct protocol or port is shown in the preceding command, use the following command to verify that ports are open in the firewall and establish a site-to-site tunnel between ICX and ICS:

     icx-1# intercloud test ics-reachability
     PORT       STATE                     SERVICE         REASON
     6644/udp   open                      data-tunnel     success
     6646/tcp   closed                    data-tunnel     Connection refused
     22/tcp     open                      ssh             success
     443/tcp    open                      https           success
     icx-1#

     • For a TCP HTTPS-based tunnel, make sure that 22/tcp, and 443/tcp are open.

     • For a TCP-based tunnel, make sure that 22/tcp, 443/tcp, and 6646/tcp are open.

     • For a UDP-based tunnel, make sure that 22/tcp, 443/tcp, and 6644/udp are open.

     If the data tunnel in HTTPS mode (6646/tcp) is unused, you can ignore its status.

Symptom: Creating an Intercloud Fabric cloud in AWS fails.

Error Message:

 Unable to create tempVM

Possible Cause: No subnet or default subnet is attached on the VPC.

Verification and Solution: On the AWS console, verify that the VPC has a subnet attached and whether or not the attached subnet ID is a default subnet. If a default subnet does not exist, contact AWS support to recreate it. If a default subnet exists, associate it with the VPC.

Symptom: Intercloud Fabric link creation fails during deployment.

Possible Cause: There are several causes for Intercloud Fabric link creation failure. Use the following steps to identify where the failure occurs.

Verification and Solution:

1. Check the Service Request (SR) event log in Intercloud Fabric for error messages detailing at which step the Intercloud Fabric link failed and why.

2. Check this guide for Intercloud Fabric log error messages.

If the issue does not match any of the documented log file error messages, use ICFC for further debugging:

1. Log in to Intercloud Fabric.

2. Choose InterCloud Management > Infrastructure > VPCs and select the Intercloud Fabric link from the VPC list you are debugging.

3. Click the Recent Jobs tab and look for any failure associated with ICX or ICS deployment.

4. To verify configuration information, go to (ICFC) Visore by choosing hcloudEp > hcloud > tab.

For additional debugging, do the following:

1. SSH to ICFC.

2. If the failure occurs at the private cloud data center, review the ICFC logs by entering the following information:

   /var/log/resource-mgr

   For example:

   /var/log/resource-mgr/svc_res_dme.log
   
   /var/log/resource-mgr/svc_res_enterpriseAG.log

3. If the failure occurs at the cloud provider, review the ICFC logs by entering the following information:

   /var/log/cloud/cloudprovider-mgr

   For example:

   /var/log/cloud/cloudprovider-mgr/cpm.log
   
   /var/log/resource-mgr/svc_res_cloudprovider.AG.log

Symptom: Unable to allocate IP addresses.

Possible Cause: Intercloud Fabric cannot allocate the correct number of IP addresses for the ICX/ICS interfaces.

Verification and Solution: Verify that there are enough available IP addresses. Additional IP addresses can be copied from the error message shown in the Service Request (SR) logs.

Symptom: Port profile creation fails on cloud VSM (cVSM).

Possible Cause: There are varying causes for port profile creation failure.

Verification and Solution: For each possible cause, first check the cVSM Fault tab to identify the possible root cause, such as:

• NTP configuration: ICFC cannot communicate with cVSM if the ICFC clock is set ahead of the cVSM clock.

• Limit on the number of port profiles: cVSM cannot create more than 128 port profiles using the same system vLAN. Determine whether or not this limit has been reached.

Symptom: Intercloud Fabric flow times out during the port profile operation.

Possible Cause: Port profile operations can time out for multiple reasons:

• NTP configuration: ICFC cannot communicate with cVSM if the ICFC clock is set ahead of the cVSM clock.

• REST connectivity to cVSM is not working.

• cVSM service is down.

• Port profile configuration errors.

Verification and Solution:

1. Compare the time set for ICFC and cVSM. If the cVSM time is ahead of the ICFC time, the certificate that ICFC uses to authenticate itself to cVSM is not yet valid. Configure NTP on cVSM and retry the failed configuration. Also ensure that ESXi hosts are configured for NTP .

2. From the ICFC debug shell, enter the following curl command to check connectivity to cVSM REST API:

   $ curl --cert /opt/cisco/cert/CACertificate.pem --key 
   /opt/cisco/cert/privKey.pem -k https://cVSM_IP/nsc-api

3. Check the service status on ICFC to confirm that vsm-service is running. Vsm-service is the application that talks to the cVSM REST API:

   • CLI command:

     # service status

   • Shell command:

     $ /opt/cisco/bin/state.sh

   • Send ICFC logs to Cisco.

4. Check for errors that occurred during port profile configuration. Choose Intercloud Management > cVSM > Edit > Faults and look for any active errors. From ICFC Visore, search for "vaswVsm", navigate to its children, and look for "faultInst" objects and any active errors.

Symptom: ICFC rejects ICF port profile requests.

Possible Cause: There are varying causes for rejected ICF port profile requests.

Verification and Solution:

1. Confirm that you did not create a duplicate port profile.

2. Check if one of the ICFC validations on port profile was violated. If so, the error message is visible in the SR or ICFC logs.

3. Send log information to Cisco for further debugging.

Symptom: An ICX VM creation on vCenter failure occurs during ICX deployment.

Possible Cause: ICFC cannot connect with vCenter, or the host or data store where ICFC is deploying the ICX VM is unreachable.

Verification and Solution:

1. Check the ICF Service Request (SR) logs.

2. Review the ICFC logs (such as /var/log/resource-mgr/svc_res_enterpriseAG.log) for ICX VM instantiation failures.

3. Debug connectivity between ICFC and vCenter.

4. Restore connectivity to the host or data store or use an alternate host or data store.

Symptom: The Intercloud Fabric Switch (ICS) VM was created in ICFC, but ICX cannot register with ICFC.

Possible Cause: Communication issue between ICFC and ICX.

Verification and Solution: Using the management IP address, confirm that the ICX and ICFC can communicate with each other. If they are both reachable, review the ICX management interface port group configuration and confirm that the port group that is configured is the correct one. In addition, confirm that the ICX VM settings in the vCenter client are correct.

Symptom: The ICS upload or register image fails.

Possible Cause: Image creation fails while uploading ICFC or registering the ICS image on the public cloud.

Verification and Solution: Review the Service Request (SR) event log in ICF for error message details. If there is not sufficient information, check the ICFC CPM logs (/var/log/cloudprovider-mgr/cpm.log).

Symptom: ICFC fails to generate SSH certificates and pushes them to ICS using ICS's public IP address.

Possible Cause: Either the ICS public IP address is not reachable or port 22 is not open on the cloud.

Verification and Solution: Check the gateway IP address configured in ICFC and confirm that the gateway is reachable by doing the following:

1. Log in to the public cloud portal and check the security group and ACL configuration.

2. Review the public and private provider IP address configurations.

3. If you use an AWS provider VPC, confirm that the internet gateway is configured and associated with a VPC so that it can add a route to the routing table. For more information on creating a VPC, see http:/​/​docs.aws.amazon.com/​AmazonVPC/​latest/​GettingStartedGuide/​getting-started-create-vpc.html.

Symptom: ICS and ICX are registered with ICFC, but the site-to-site tunnel is not operational.

Possible Cause: ICFC fails to push tunnel and device profiles to ICS to form a site-to-site tunnel.

Verification and Solution: Do the following:

1. Review the ICFC and ICX tabs for any configuration errors.

2. SSH to the ICX console through its management IP address for further debugging. For example:

   ssh admin@icx_mgmt_ip

3. Examine the ICX log files for further analysis of the Intercloud Fabric link failure.

Symptom: Intercloud Fabric link redundancy (high availability) fails.

Possible Cause: Intercloud Fabric link redundancy is not available if one of the two pairs belonging to ICX or ICS, named primary pair (IcLink-1) and secondary pair (IcLink-2), is not established and in operation.

Verification and Solution: Review the ICX log files and the internal system event-log redundancy-mgr for further analysis of the Intercloud Fabric link failure.

Symptom: Both primary and secondary ICF link come up in Active state instead of one in Active and the other in Standby.

Possible Cause: A Network connectivity problem between primary and secondary ICXs in ICF link can cause this issue.

Verification and Solution: Reboot secondary in ICF link to restore ICF link to normal Active/Standby state.

Symptom: ICF link becomes partial_operational or not_operational after out-of-band (from provider portal) reboot operation, or a stop operation followed by a start operation, of ICS instance(s).

Possible Cause: An out-of-band reboot operation, or a stop operation followed by a start operation, can cause this issue.

Verification and Solution: Using the Intercloud Fabric UI, reboot the ICF link pair for the tunnel or module whose status is not showing up or online.

This section includes symptoms, possible causes, and solutions for issues encountered while setting up an Intercloud Fabric link.