Cisco Security Cloud Control: Network Devices with Generic SSH Access Management

PDF

Cisco Security Cloud Control: Network Devices with Generic SSH Access Management

Troubleshoot a Secure Device Connector

Want to summarize with AI?

Log in

Learn how to troubleshoot Secure Device Connector issues, including unreachable SDCs, inactive status, IP address changes, device connectivity problems, system time errors, version issues, and AWS certificate or connection errors.


Use these topics to troubleshoot an on-premises Secure Device Connector (SDC).

If none of these scenarios match yours, open a case with Cisco Technical Assistance Center.


SDC is Unreachable

An SDC is in the state "Unreachable" if it has failed to respond to two heartbeat requests from Security Cloud Control in a row. If your SDC is unreachable, your tenant will not be able to communicate with any of the devices you have onboarded.

Security Cloud Control indicates that an SDC is unreachable in these ways:

  • You see the message, “Some Secure Device Connectors (SDC) are unreachable. You will not be able to communicate with devices associated with these SDCs.” on the Security Cloud Control home page.

  • The SDC's status in the Services page is "Unreachable."

Frist, attempt to reconnect the SDC to your tenant to resolve this issue:

  1. Check that the SDC virtual machine is running and can reach a Security Cloud Control IP address in your region.

    See Allow inbound access for direct cloud connectivity.

  2. Attempt to reconnect Security Cloud Control and the SDC by requesting a heartbeat manually. If the SDC responds to a heartbeat request, it will return to "Active" status. To request a heartbeat manually:

    1. In the left pane, choose Administration > Integrations > Secure Connectors.

    2. Click the SDC that is unreachable.

    3. In the Actions pane, click Request Heartbeat.

    4. Click Reconnect.

  3. If the SDC does not return to the Active status after manually attempting to reconnect it to your tenant, follow the instructions in SDC Status not Active on Security Cloud Control After Deployment.

    .

SDC Status not Active on Security Cloud Control After Deployment

If Security Cloud Control does not indicate that your SDC is active in about 10 minutes after deployment, connect to the SDC VM using SSH using the Security Cloud Control user and password you created when you deployed the SDC.

Procedure

1.

In Security Cloud Control, click on the SDC that isn't showing as active, and click Request Heartbeat.

2.

Ensure that the container is running using the command:

docker ps

3.

View the SDC logs and check for errors using the command:

sdc show logs

4.

Restart the container using the command:

sdc restart


Changed IP Address of the SDC is not Reflected in Security Cloud Control

To view the updated IP address of an SDC, request a heartbeat using the steps provided earlier, and then refresh your browser.

Request a Heartbeat

In Security Cloud Control, click on the SDC that isn't showing as active, and, in the sidebar, click Request Heartbeat.

  1. Make sure the container is running using the command: docker ps

  2. View the SDC logs and check for errors using the command sdc show logs

  3. Restart the container using the command sdc restart


Troubleshoot Device Connectivity with the SDC

Use this tool to test connectivity from Security Cloud Control, through the Secure Device Connector (SDC) to your device. You may want to test this connectivity if your device fails to onboard or if you want to determine, before on-boarding, if Security Cloud Control can reach your device.

Procedure

1.

In the left pane, click Administration > Integrations > Firewall Management Center, and click the Secure Connectors tab.

2.

Select the SDC.

3.

In the Troubleshooting pane on the right, click Device Connectivity.

4.

Enter a valid IP address or FQDN and port number of the device you are attempting to troubleshoot, or attempting to connect to, and click Go. Security Cloud Control performs the following verifications:

  1. DNS Resolution - If you provide a FQDN instead of an IP address, this verifies the SDC can resolve the domain name and acquires the IP address.

  2. Connection Test - Verifies the device is reachable.

  3. TLS Support - Detects the TLS versions and ciphers that both the device and the SDC support.

    • Unsupported Cipher - If there are no TLS version that are supported by both the device and the SDC, Security Cloud Control also tests for TLS versions and ciphers that are supported by the device, but not the SDC.

  4. SSL Certificate - The troubleshoot provides certificate information.

5.

If you continue to have issues onboarding or connecting to the device, contact Security Cloud Control support.


Intermittent or No Connectivity with SDC

The Secure Device Connector (SDC) has been installed on CentOS 7 virtual machines up to now. However, as CentOS has reached its end-of-life and is no longer supported by Firewall in Security Cloud Control, we recommend migrating all SDCs from CentOS 7 to an Ubuntu virtual machine.

For more information, see Migrate an On-Premises Secure Device Connector and Secure Event Connector from a CentOS 7 Virtual Machine to an Ubuntu Virtual Machine.

This procedure applies only to an on-premise SDC and if you are still using CentOS.

Symptom: Intermittent or no connectivity with SDC.

Diagnosis: This problem may occur if the disk space is almost full (above 80%).

Perform the following steps to check the disk space usage.

  1. Open the console for your Secure Device Connector (SDC) VM.

  2. Log in with the username cdo.

  3. Enter the password created during the initial login.

  4. First, check the amount of free disk space by typing df -h to confirm that there is no free disk space available.

    You can confirm that the disk space was consumed by the Docker. The normal disk usage is expected to be under 2 Gigabytes.

  5. To see the disk usage of the Docker folder,

    execute sudo du -h /var/lib/docker | sort -h.

    You can see the disk space usage of the Docker folder.

Procedure

If the disk space usage of the Docker folder is almost full, define the following in the docker config file:

  • Max-size: To force a log rotation once the current file reaches the maximum size.

  • Max-file: To delete excess rotated log files when the maximum limit it reached.

Perform the following:

  1. Execute sudo vi /etc/docker/daemon.json.

  2. Insert the following lines to the file.

    {

    "log-driver": "json-file",

    "log-opts": {"max-size": "100m", "max-file": "5" }

    }

  3. Press ESC and then type :wq! to write the changes and close the file.

    Note

    You can execute sudo cat /etc/docker/daemon.json to verify the changes made to the file.

  4. Execute sudo systemctl restart docker to restart the docker file.

    It will take a few minutes for the changes to take effect. You can execute sudo du -h /var/lib/docker | sort -h to see the updated disk usage of the docker folder.

  5. Execute df -h to verify that the free disk size has increased.

  6. Before your SDC status can change from Unreachable to Active, you must go to the Secure Connectors tab which you can navigate to from Administration > Integrations > Firewall Management Center and click Request Reconnect from the Actions menu.


Container Privilege Escalation Vulnerability Affecting Secure Device Connector: cisco-sa-20190215-runc

The Cisco Product Security Incident Response Team (PSIRT) published the security advisory cisco-sa-20190215-runc which describes a high-severity vulnerability in Docker. Read the entire PSIRT team advisory for a full explanation of the vulnerability.

This vulnerability impacts all Security Cloud Control customers:

  • Customers using Security Cloud Control's cloud-deployed Secure Device Connector (SDC) do not need to do anything as the remediation steps have already been performed by the Security Cloud Control Operations Team.

  • Customers using an SDC deployed on-premise need to upgrade their SDC host to use the latest Docker version. They can do so by using the following instructions:


Updating a Security Cloud Control-Standard SDC Host

Procedure

1.

Connect to your SDC host using SSH or the hypervisor console.

2.

Check the version of your Docker service by running this command:

docker version
3.

If you are running one of the latest virtual machines (VMs) you should see output like this:

 > docker version
Client:
     Version: 18.06.1-ce
     API version: 1.38
     Go version: go1.10.3
     Git commit: e68fc7a
     Built: Tue Aug 21 17:23:03 2018
     OS/Arch: linux/amd64
     Experimental: false
It's possible you may see an older version here.
4.

Run the following commands to update Docker and restart the service:

> sudo yum update docker-ce
> sudo service docker restart
Note

There will be a brief connectivity outage between Security Cloud Control and your devices while the docker service restarts.

5.

Run the docker version command again. You should see this output:

> docker version
Client:
    Version: 18.09.2
    API version: 1.39
    Go version: go1.10.6
    Git commit: 6247962
    Built: Sun Feb XX 04:13:27 2019
    OS/Arch: linux/amd64
    Experimental: false
6.

You are done. You have now upgraded to the latest, and patched, version of Docker.


Updating a Custom SDC Host

The Secure Device Connector (SDC) has been installed on CentOS 7 virtual machines up to now. However, as CentOS has reached its end-of-life and is no longer supported by Firewall in Security Cloud Control, we recommend migrating all SDCs from CentOS 7 to an Ubuntu virtual machine.

For more information, see Migrate an On-Premises Secure Device Connector and Secure Event Connector from a CentOS 7 Virtual Machine to an Ubuntu Virtual Machine.

This procedure applies if you are still using CentOS.

  • If you have created your own SDC host you will need to follow the instructions to update based on how you installed Docker. If you used CentOS, yum and Docker-ce (the community edition) the preceding procedure will work.

  • If you have installed Docker-ee (the enterprise edtion) or used an alternate method to install Docker, the fixed versions of Docker may be different. You can check the Docker page to determine the correct versions to install: Docker Security Update and Container Security Best Practices.


Bug Tracking

Cisco is continuing to evaluate this vulnerability and will update the advisory as additional information becomes available. After the advisory is marked Final, you can refer to the associated Cisco bug for further details:

CSCvo33929-CVE-2019-5736: runc container breakout


Invalid System Time

The Secure Device Connector (SDC) has been installed on CentOS 7 virtual machines up to now. However, as CentOS has reached its end-of-life and is no longer supported by Firewall in Security Cloud Control, we recommend migrating all SDCs from CentOS 7 to an Ubuntu virtual machine.

For more information, see Migrate an On-Premises Secure Device Connector and Secure Event Connector from a CentOS 7 Virtual Machine to an Ubuntu Virtual Machine.

This procedure applies if you are still using CentOS.

  • Security Cloud Control is adapting a new way of communicating with the Secure Device Connector (SDC). To facilitate this, Security Cloud Control must migrate your existing SDC to the new communication method by February 1, 2024.

    Note

    If your SDC is not migrated by February 1, 2024, Security Cloud Control will no longer be able to communicate with your devices through the SDC.

  • Security Cloud Control's operations team attempted to migrate your SDC but was unsuccessful because your SDC system time was 15 minutes ahead or behind the AWS system time.

Follow the steps below to correct the system time issue. Once this problem is resolved, you can proceed with the migration.

Procedure

1.

Login to your SDC VM throught the VM terminal or by making an SSH connection.

2.

At the prompt, enter sudo sdc-onboard setup and authenticate.

3.

You are now going to respond to the SDC setup questions as if you are were setting up the SDC for the first time. Re-enter all the same passwords and network information as you had before, except take special note of the NTP server address:

  1. Reset the root and Security Cloud Control user passwords with the same passowrds you used to setup the SDC.

  2. When prompted, enter y to re-configure the network.

  3. Enter the value for IP address/CIDR as you had before.

  4. Enter the value for the network gateway as you had before.

  5. Enter the value for the DNS Server as you had before.

  6. When prompted for the NTP server, be sure to provide a valid NTP server address, such as time.aws.com.

  7. Review the values you provided and enter y if they are correct.

4.

Validate that your time server is reachable and synchronized with your SDC by entering date at the prompt. The UTC date and time are displayed and you can compare it to your SDC time.

What to do next

Contact the Cisco Technical Assistance Center (TAC) once you have completed these steps, or in case you encounter any errors. Once you have successfully completed these steps, the Security Cloud Control operations team can complete your SDC migration to the new communication method.


SDC version is lower than 202311****

The Secure Device Connector (SDC) has been installed on CentOS 7 virtual machines up to now. However, as CentOS has reached its end-of-life and is no longer supported by Firewall in Security Cloud Control, we recommend migrating all SDCs from CentOS 7 to an Ubuntu virtual machine.

For more information, see Migrate an On-Premises Secure Device Connector and Secure Event Connector from a CentOS 7 Virtual Machine to an Ubuntu Virtual Machine.

This procedure applies if you are still using CentOS.

  • Security Cloud Control is adapting a new way of communicating with the Secure Device Connector (SDC). To facilitate this, Security Cloud Control must migrate your existing SDC to the new communication method by February 1, 2024.

    Note

    If your SDC is not migrated by February 1, 2024, Security Cloud Control will no longer be able to communicate with your devices through the SDC.

  • Security Cloud Control's operations team attempted to migrate your SDC but was unsuccessful because your tenant is running a version lower than 202311****.

  • The current version of your SDC is listed on the Secure Connectors page by navigating from the Security Cloud Control navigation menu, Administration > Integrations > Secure Connectors. After selecting your SDC, its version number is found in the Details pane on the right of the screen.

Please follow the steps below to upgrade the SDC version. Once this problem is resolved, Security Cloud Control operations will run the migration process again.

Procedure

1.

Log in to the SDC VM and authenticate.

2.

At the prompt, enter sudo su - sdc and authenticate.

3.

At the prompt, enter crontab -r.

If you receive the message no crontab for sdc you can ignore it and move to the next step.

4.

At the prompt, enter ./toolkit/toolkit.sh upgrade. Security Cloud Control will determine if you need an upgrade and upgrade the toolkit. Ensure that no errors were reported in the console.

5.

Verify the new version of the SDC:

  1. Log in to Security Cloud Control.

  2. Navigate to the Secure Connectors page by navigating from the Security Cloud Control menu bar, Tools & Services > Secure Connectors.

  3. Select your SDC and click Request Heartbeat in the Actions pane.

  4. Validate that the SDC version is 202311**** or later.

What to do next

Contact the Cisco Technical Assistance Center (TAC) once you have completed these steps, or in case you encounter any errors. Once you have successfully completed these steps, the Security Cloud Control operations team can run the migration process again.


Certificate or Connection errors with AWS servers

Security Cloud Control is adapting a new way of communicating with the Secure Device Connector (SDC). To facilitate this, Security Cloud Control must migrate your existing SDC to the new communication method by February 1, 2024.

Note

If your SDC is not migrated by February 1, 2024, Security Cloud Control will no longer be able to communicate with your devices through the SDC.

Security Cloud Control's operations team attempted to migrate your SDC but was unsuccessful because they experienced a connection issue.

Please follow the steps below to correct the connection issue. Once this problem is resolved, we will be able to proceed with the migration.

Procedure

1.

Create firewall rules that allow outbound proxy connections, on port 443, to the domains in your region:

  • Production tenants in the Australia region:

    • cognito-identity.ap-southeast-2.amazonaws.com

    • cognito-idp.ap-southeast-2.amazonaws.com

    • sns.ap-southeast-2.amazonaws.com

    • sqs.ap-southeast-2.amazonaws.com

  • Production tenants in the India region:

    • cognito-identity.ap-south-1.amazonaws.com

    • cognito-idp.ap-south-1.amazonaws.com

    • sns.ap-south-1.amazonaws.com

    • sqs.ap-south-1.amazonaws.com

  • Production tenants in the US region:

    • cognito-identity.us-west-2.amazonaws.com

    • cognito-idp.us-west-2.amazonaws.com

    • sns.us-west-2.amazonaws.com

    • sqs.us-west-2.amazonaws.com

  • Production tenants in the EU region:

    • cognito-identity.eu-central-1.amazonaws.com

    • cognito-idp.eu-central-1.amazonaws.com

    • sns.eu-central-1.amazonaws.com

    • sqs.eu-central-1.amazonaws.com

  • Production tenants in the APJ region:

    • cognito-identity.ap-northeast-1.amazonaws.com

    • cognito-idp.ap-northeast-1.amazonaws.com

    • sqs.ap-northeast-1.amazonaws.com

    • sns.ap-northeast-1.amazonaws.com

2.

You can determine the full list of IP addresses you need to add to your firewall's "allow list" by using one of the commands below.

Note

The commands below are for users that have jq installed. The IP addresses will be displayed in a single list.

  • Production tenants in the US region:
    curl -s https://ip-ranges.amazonaws.com/ip-ranges.json | jq -r '.prefixes[] | select( (.service == "AMAZON" ) and .region == "us-west-2") | .ip_prefix'
  • Production tenants in the EU region:
    curl -s https://ip-ranges.amazonaws.com/ip-ranges.json | jq -r '.prefixes[] | select( (.service == "AMAZON" ) and .region == "eu-central-1") | .ip_prefix'
  • Production tenants in the APJ region:
    curl -s https://ip-ranges.amazonaws.com/ip-ranges.json | jq -r '.prefixes[] | select( (.service == "AMAZON" ) and .region == "ap-northeast-1") | .ip_prefix'
Note
If you don't have jq installed, you can use this shortened version of the command:
curl -s https://ip-ranges.amazonaws.com/ip-ranges.json

What to do next

Contact the Cisco Technical Assistance Center (TAC) once you have completed these steps, or in case you encounter any errors. Once you have successfully completed these steps, the Security Cloud Control operations team can complete your SDC migration to the new communication method.