Cisco Security Cloud Control: Secure Firewall Management

PDF

Cisco Security Cloud Control: Secure Firewall Management

Troubleshoot a Secure Firewall ASA device

Want to summarize with AI?

Log in

Learn how to troubleshoot Secure Firewall ASA device issues including connectivity, configuration, synchronization, and deployment problems.



ASA Fails to Reconnect to Security Cloud Control After Reboot

If Security Cloud Control and your ASA do not connect after an ASA reboot, it may be because the ASA has fallen back to using an OpenSSL cipher suite that is not supported by Security Cloud Control's Secure Device Connector (SDC). This troubleshooting topic tests for that case and provides remediation steps.

Symptoms

  • ASA reboots and Security Cloud Control and the ASA fail to reconnect. Security Cloud Control displays the message, "Failed to reconnect."

  • When attempting to onboard an ASA, Security Cloud Control displays the message: Certificate could not be retrieved for <ASA_IP_Address>.


Cannot onboard ASA due to certificate error

Environment: ASA is configured with client-side certificate authentication.

Solution: Disable client-side certificate authentication.

Details: ASAs support credential-based authentication as well as client-side certificate authentication. Security Cloud Control cannot connect to ASAs that use client-side certificate authentication. Before onboarding your ASA to Security Cloud Control, make sure it does not have client-certificate authentication enabled by using this procedure:

Procedure

1.

Open a terminal window and connect to the ASA using SSH.

2.

Enter global configuration mode.

3.

At the hostname (config)# prompt, enter this command:

no ssl certificate-authentication interfaceinterface-nameport 443

The interface name is the name of the interface Security Cloud Control connects to.


Determine the OpenSSL Cipher Suite Used by your ASA

Use this procedure to identify the OpenSSL cipher suite being used by your ASA. If the cipher suite named in the command output is not in the list of supported cipher suites, the SDC doesn't support that cipher suite and you will need to update the cipher suites on your ASA.

Procedure

1.

Open a console window on a computer that can reach the SDC.

2.

Connect to your SDC using SSH. You can log in as a regular user such as Security Cloud Control or SDC or some other user you created. You don't need to be logged in as root.

Tip

To find your SDC IP address:

  1. Open Security Cloud Control.

  2. From the user menu, select Secure Device Connectors.

  3. Click the SDC displayed in the table. The IP address of the SDC is displayed in the details pane for the device.

3.

At the command prompt enter: openssl s_client -showcerts -connect ASA_IP_Address:443

4.

Look for these lines in the command output.

New, TLSV1/SSLv3, Cipher is DES-CB3-SHA 
or 
SSL-Session:
            Protocol: TLSv1.2 
            Cipher: DES-CB3-SHA

In this example, the cipher suite being used by the ASA is DES-CB3-SHA.


Cipher Suites Supported by Security Cloud Control's Secure Device Connector

Security Cloud Control's Secure Device Connector uses node.js which only accepts the latest and most secure ciphers. As a result, Security Cloud Control's SDC only supports this list of ciphers:

  • ECDHE-RSA-AES128-GCM-SHA256

  • ECDHE-ECDSA-AES128-GCM-SHA256

  • ECDHE-RSA-AES256-GCM-SHA384

  • ECDHE-ECDSA-AES256-GCM-SHA384

  • DHE-RSA-AES128-GCM-SHA256

  • ECDHE-RSA-AES128-SHA256

  • DHE-RSA-AES128-SHA256

  • ECDHE-RSA-AES256-SHA384

  • DHE-RSA-AES256-SHA384

  • ECDHE-RSA-AES256-SHA256

  • DHE-RSA-AES256-SHA256

If the cipher suite you use on your ASA is not in this list, SDC does not support it and you will need to update the cipher suite on your ASA.


Updating your ASA's Cipher Suite

To update the TLS cipher suites on an ASA:

Procedure

1.

Connect to the ASA using SSH.

2.

Once connected to the ASA, elevate your privileges to global configuration mode. Your prompt should look like this: asaname(config)#

3.

At the prompt, enter a command similar to this:

ssl cipher tlsv1.2 custom "ECDHE-RSA-AES128-GCM-SHA256 ECDHE-ECDSA-AES128-GCM-SHA256 ECDHE-RSA-AES256-GCM-SHA384 ECDHE-ECDSA-AES256-GCM-SHA384 DHE-RSA-AES128-GCM-SHA256 ECDHE-RSA-AES128-SHA256 DHE-RSA-AES128-SHA256 ECDHE-RSA-AES256-SHA384 DHE-RSA-AES256-SHA384 ECDHE-RSA-AES256-SHA256 DHE-RSA-AES256-SHA256"
Note

The cipher suites this command configures your ASA to support are contained between quotes and after the word custom. In this command, the cipher suites specified begin with ECDHE-RSA-AES128-GCM-SHA256 and end with DHE-RSA-AES256-SHA256. When you enter the command on your ASA, remove any cipher suites you know your ASA will not support.

4.

After you submit the command, enter write memory at the prompt to save the local configuration. For example: asaname(config)#write memory


Troubleshoot ASA using CLI commands

This section discusses some of the important commands you may want to use to troubleshoot the ASA and test basic connectivity. See CLI Book 1: Cisco ASA Series General Operations CLI Configuration Guide to learn about other troubleshooting scenarios and CLI commands. In the 'System Administration' section, navigate to the 'Testing and Troubleshooting' chapter.

You can use the Security Cloud Control CLI interface available for each ASA device to execute these commands.

See ASA command line interface to learn about how to use the CLI interface in Security Cloud Control.

NAT Policy Settings

Some of the important commands to determine the NAT settings are as follows:

  • To determine NAT policy statistics, use show nat.

  • To determine the NAT pools, including the addresses and ports allocated, and how many times they were allocated, use show nat pool.

For more commands related to NAT, see CLI Book 2: Cisco ASA Series Firewall CLI Configuration Guide, and navigate to the 'Network Address Translation (NAT)' chapter.

Test Basic Connectivity: Pinging Addresses

You can ping the ASA device using the ping <IP address> command using the ASA CLI interface. To learn about

Display the Routing Table

Use the show route command to view the entries in the routing table.

ciscoasa# show route

Example output for a routing table of an ASA:

Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP

D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area

N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2

E1 - OSPF external type 1, E2 - OSPF external type 2, V - VPN

i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2

ia - IS-IS inter area, * - candidate default, U - per-user static route

o - ODR, P - periodic downloaded static route, + - replicated route

SI - Static InterVRF

Gateway of last resort is 192.168.0.254 to network 0.0.0.0

S* 0.0.0.0 0.0.0.0 [1/0] via 192.168.0.254, management

C 10.0.0.0 255.0.0.0 is directly connected, Outside

L 10.10.10.1 255.255.255.255 is directly connected, Outside

C 192.168.0.0 255.255.255.0 is directly connected, management

L 192.168.0.118 255.255.255.255 is directly connected, management

Monitor Switch Ports

  • show interface

    Displays interface statistics.

  • show interface ip brief

    Displays interface IP addresses and status.

  • show arp

Shows dynamic, static, and proxy ARP entries. Dynamic ARP entries include the age of the ARP entry in seconds.

Example output of ARP entries:

management 10.10.32.129 0050.568a.977b 0

management 10.10.32.136 0050.568a.5387 21

LANFAIL 20.20.21.1 0050.568a.4d70 96

outsi 10.10.16.6 0050.568a.e6d3 3881

outsi 10.10.16.1 0050.568a.977b 5551


Troubleshoot ASA Remote Access VPN

This section discusses some of the troubleshooting issues that may occur when configuring remote access VPN on an ASA device.

Missing Information on the RA VPN Monitoring Page

This issue may occur if the outside interface is not enabled for Webvpn.

Resolution:

  1. In the left pane, click Security Devices and then click the ASA tab.

  2. Select the RA VPN headend ASA device that is having issues.

  3. In the Management pane on the right, click Configuration.

  4. Click Edit and search for ‘webvpn’.

  5. Press Enter and add enable interface_name. Here, the interface_name is the name of the outside interface to which users connect when making the remote access VPN connection. Although this is normally the outside (internet-facing) interface, choose whichever interface is between the device and the end-users you are supporting with this connection profile.

    For example:

    webvpn

    enable outside

  6. Click Save.

  7. Preview and deploy the configuration to the device.


ASA Real-time Logging

Use real-time logging to display the last 20 seconds of logged data or the last 10 KB of logged data, whichever limit is reached first. When Security Cloud Control retrieves the real-time data, it reviews the existing logging configuration on your ASDM, changes it to request debugging-level data, and then returns the logging configuration to your configuration. The logging Security Cloud Control displays reflects any logging filters you may have set in ASDM.

You can see the commands that Security Cloud Control sends to perform logging by reviewing the change log. Below is an example of a change log entry. The first entry (on the bottom) indicates that Security Cloud Control "turned on" logging with the logging enable command and changed the ASDM logging level to debugging. The second entry (on the top) shows that the logging configuration was returned to its previous state. Logging was "turned off" with the no logging enable command and the ASDM logging level was returned to informational.


View ASA Real-time Logs

Procedure

1.

In the left pane, click Security Devices.

2.

Click the appropriate device type tab and select the device for which you want to view real-time data.

3.

Click Troubleshoot .

4.

(Optional) Before clicking View Real-time Log, you can define a filter in the left pane to refine the results of your logging search.

5.

Click View Real-time Log. Security Cloud Control retrieves the real-time logging data based on your filterin criteria and displays it.

6.

To see an additional 20 seconds of logged data or the last 10 KB of logged data. click View Real-Time Log again.


ASA Packet Tracer

Packet tracer allows you to send a synthetic packet into the network and evaluate how the existing routing configuration, NAT rules, and policy configurations, affect that packet. Use this tool to troubleshoot these kinds of issues:

  • Users report that they cannot reach resources that they should be able to.

  • Users report that they can reach resources they should not be able to.

  • Test a policy to determine if it works as you expect.

Packet tracer can be used on a live, online, ASA device either physical or virtual. Packet Tracer does not work on ASA model devices. Packet tracer evaluates packets based on the saved configuration on the ASA. Staged changes on Security Cloud Control are not evaluated by packet tracer.

We consider it a best-practice to run packet tracer on an ASA that is in the synced state. Though packet tracer will run if the device is not synced, you could encounter some unexpected results. For example, if you deleted a rule in the staged configuration on Security Cloud Control, and this same rule was triggered on the ASA during packet tracing, Security Cloud Control won't be able to show you the result of the packet's interaction with that rule.

Troubleshooting with ASA Packet Tracer

As packet tracer sends the packet through the routing configuration, NAT rules, and security policies of your ASA, it displays the packet's status at each step. If the packet is allowed by the policy it receives a green checkmark . If a packet is denied and dropped, Security Cloud Control displays a red X .

Packet tracer also displays a real time log of the result of the packet trace. In the example below, you can see where a rule denied a tcp packet.


Troubleshoot an ASA Device Security Policy

Procedure

1.

In the left pane, click Security Devices.

2.

Select your ASA and view troubleshooting details in the Troubleshooting pane.

3.

In the pane, select the interface and packet type you want to send virtually through your ASA.

4.

(Optional) If you want to trace a packet where the security group tag value is embedded in the Layer 2 CMD header (Trustsec), check SGT number and enter the security group tag number, 0-65535.

5.

Specify the source and destination. You can specify IPv4 or IPv6 addresses, fully-qualified domain names (FQDN), or security group names or tags if you use Cisco Trustsec. For the source address, you can also specify a username in the format Domain\username.

6.

Specify other protocol characteristics:

  • ICMP-Enter the ICMP type, ICMP code (0-255), and optionally, the ICMP identifier.

  • TCP/UDP/SCTP-Enter the source and destination ports by selecting them from the list or entering a value in the port combo box.

  • IP-Enter the protocol number, 0-255.

7.

Click Run Packet Tracer.

8.

Continue with Analyze Packet Tracer Results.


Troubleshoot an Access Rule

Procedure

1.

From the Security Cloud Control Home page, click Firewall.

2.

In the left pane, click Policies > ASA > Access Policies.

3.

Select a policy that is associated with your ASA.

4.

Select a rule in the network policy to troubleshoot and click Troubleshoot in the details pane. Notice that in the values panel of the troubleshoot page, many of the fields are pre-populated with the attributes of the rule you chose.

5.

Enter information in the remaining required fields. Once you have completed all the required fields the Run Packet Tracer button becomes active.

6.

Click Run Packet Tracer.

7.

Continue with Analyze Packet Tracer Results.


Troubleshoot a NAT Rule

Procedure

1.

In the left pane, click Security Devices and then click the ASA tab.

2.

Select your ASA, and click View NAT Rules in the Action pane.

3.

Select the rule from the NAT Rules table that you want to troubleshoot and click Troubleshoot in the details pane. Notice that in the values panel of the Troubleshoot page, many of the fields are pre-populated with the attributes of the rule you chose.

4.

Enter information in the remaining required fields. Once you have completed all the required fields the Run Packet Tracer becomes active.

5.

Click Run Packet Tracer.

6.

Continue with Analyze Packet Tracer Results.


Troubleshoot a Twice NAT Rule

Procedure

1.

In left pane, click Security Devices.

2.

Click the ASA tab, select your ASA, and click View NAT Rules in the Action pane.

3.

Select the rule from the NAT Rules table that you want to troubleshoot and click Troubleshoot in the details pane. For a bi-directional Twice NAT rule, this opens a dropdown where you choose to troubleshoot the source packet translation or the destination packet translation.

4.

Enter information in the remaining required fields. Once you have completed all the required fields the Run Packet Tracer becomes active.

5.

Click Run Packet Tracer.


Analyze Packet Tracer Results

Whether the packet is dropped or allowed, you can learn why by expanding a row in the packet trace table and reading the rule or logging information related to that action. In the example below, packet tracer identified an access list policy that included a rule to deny an IP packet coming from any source and going to any destination. If this is not the action you want, you can click the View in Network Policies link and edit that rule immediately. After you edit the rule, be sure to deploy that configuration change to the ASA and then re-run packet tracer to ensure that you get the access results you expect.

Along with the packet tracer results, Security Cloud Control displays the real-time logs from the ASA.


Cisco ASA Advisory cisco-sa-20180129-asa1

The Cisco Product Security Incident Response Team (PSIRT) published the security advisory cisco-sa-20180129-asa1 which describes a critical-severity ASA and Firepower security vulnerability. Read the entire PSIRT team advisory for a full explanation of what ASA and Firepower hardware, software, and configurations are affected.

If you determine that your ASAs are impacted by the advisory, you can upgrade your ASAs to the patched version using Security Cloud Control. Use this process:

Procedure

1.

Configure a DNS server on each ASA that is affected.

2.

Return to the advisory to determine which software patch you need.

3.

See Upgrade ASA and ASDM images on a single ASA for topics that describe how to use Security Cloud Control to upgrade your ASAs to the fixed releases listed in the ASA advisory. Start with the upgrade prerequisites and then read about upgrading individual ASAs, upgrading ASAs in an active-standby configuration, or upgrading ASAs in bulk.

For your convenience, here is the summary of the security advisory that Cisco reported:

UPDATED 2/5/2018: After further investigation, Cisco has identified additional attack vectors and features that are affected by this vulnerability. In addition, it was also found that the original fix was incomplete so new fixed code versions are now available. Please see the Fixed Software section for more information. A vulnerability in the XML parser of Cisco Adaptive Security Appliance (ASA) Software could allow an unauthenticated, remote attacker to cause a reload of the affected system or to remotely execute code. It was also possible that the ASA could stop processing incoming Virtual Private Network (VPN) authentication requests due to a low memory condition. The vulnerability is due to an issue with allocating and freeing memory when processing a malicious XML payload. An attacker could exploit this vulnerability by sending a crafted XML packet to a vulnerable interface on an affected system. An exploit could allow the attacker to execute arbitrary code and obtain full control of the system, cause a reload of the affected device or stop processing of incoming VPN authentication requests. To be vulnerable the ASA must have Secure Socket Layer (SSL) services or IKEv2 Remote Access VPN services enabled on an interface. The risk of the vulnerability being exploited also depends on the accessibility of the interface to the attacker. For a comprehensive list of vulnerable ASA features please refer to the table in the Vulnerable Products section. Cisco has released software updates that address this vulnerability. There are no workarounds that address all the features that are affected by this vulnerability. This advisory is available at the following link: https://sec.cloudapps.cisco.com/security/center/content/CiscoSecurityAdvisory/cisco-sa-20180129-asa1

Confirming ASA Running Configuration Size

To confirm the size of your running configuration file, follow this procedure:

Procedure

1.

Access the ASA's command line interface in one of these ways:

  • Open a terminal window and log into your ASA using SSH. Elevate your privilege to "privileged EXEC" mode so you see the prompt with hostname# .

  • If you managed to get your ASA onboarded, open the Security Devices page, select the device you want to connect to, and click >_ Command Line Interface button in the Device Actions pane.

2.

At the prompt type copy running-config flash

3.

When prompted for the Source filename, don't type anything and press <Enter>

4.

When prompted for the destination filename, enter a name for the output file. After ASA copies the running configuration the file you specified, it returns you to the privileged EXEC prompt.

5.

At the prompt, type show flash

6.

Look in the length column. If your file is over 4718592 bytes, it is larger than 4.5 MB.

Here is a sample set of commands and output:

asa1# copy running-config flash
Source filename [running-config]? 
Destination filename [running-config]? running-config-output
Cryptochecksum: 725f4c1c 4adfb8a9 8b3e7a6d 49e3420d 
23648 bytes copied in 1.380 secs (23648 bytes/sec)
asa1# show flash
--#-- --length-- -----date/time------ path
 107 110325428 Feb 28 2019 15:41:42 asdm-8826067.bin
 122 5018592 Apr 30 2019 21:00:59 running-config-output
 111 102647808 Mar 12 2019 14:26:10 asa9-12-1-smp-k8.bin

Container Privilege Escalation Vulnerability Affecting Secure Device Connector: cisco-sa-20190215-runc

The Cisco Product Security Incident Response Team (PSIRT) published the security advisory cisco-sa-20190215-runc which describes a high-severity vulnerability in Docker. Read the entire PSIRT team advisory for a full explanation of the vulnerability.

This vulnerability impacts all Security Cloud Control customers:

  • Customers using Security Cloud Control's cloud-deployed Secure Device Connector (SDC) do not need to do anything as the remediation steps have already been performed by the Security Cloud Control Operations Team.

  • Customers using an SDC deployed on-premise need to upgrade their SDC host to use the latest Docker version. They can do so by using the following instructions:


Updating a Security Cloud Control-Standard SDC Host

Procedure

1.

Connect to your SDC host using SSH or the hypervisor console.

2.

Check the version of your Docker service by running this command:

docker version
3.

If you are running one of the latest virtual machines (VMs) you should see output like this:

 > docker version
Client:
     Version: 18.06.1-ce
     API version: 1.38
     Go version: go1.10.3
     Git commit: e68fc7a
     Built: Tue Aug 21 17:23:03 2018
     OS/Arch: linux/amd64
     Experimental: false
It's possible you may see an older version here.
4.

Run the following commands to update Docker and restart the service:

> sudo yum update docker-ce
> sudo service docker restart
Note

There will be a brief connectivity outage between Security Cloud Control and your devices while the docker service restarts.

5.

Run the docker version command again. You should see this output:

> docker version
Client:
    Version: 18.09.2
    API version: 1.39
    Go version: go1.10.6
    Git commit: 6247962
    Built: Sun Feb XX 04:13:27 2019
    OS/Arch: linux/amd64
    Experimental: false
6.

You are done. You have now upgraded to the latest, and patched, version of Docker.


Updating a Custom SDC Host

The Secure Device Connector (SDC) has been installed on CentOS 7 virtual machines up to now. However, as CentOS has reached its end-of-life and is no longer supported by Firewall in Security Cloud Control, we recommend migrating all SDCs from CentOS 7 to an Ubuntu virtual machine.

For more information, see Migrate an On-Premises Secure Device Connector and Secure Event Connector from a CentOS 7 Virtual Machine to an Ubuntu Virtual Machine.

This procedure applies if you are still using CentOS.

  • If you have created your own SDC host you will need to follow the instructions to update based on how you installed Docker. If you used CentOS, yum and Docker-ce (the community edition) the preceding procedure will work.

  • If you have installed Docker-ee (the enterprise edtion) or used an alternate method to install Docker, the fixed versions of Docker may be different. You can check the Docker page to determine the correct versions to install: Docker Security Update and Container Security Best Practices.


Bug Tracking

Cisco is continuing to evaluate this vulnerability and will update the advisory as additional information becomes available. After the advisory is marked Final, you can refer to the associated Cisco bug for further details:

CSCvo33929-CVE-2019-5736: runc container breakout


Large ASA Running Configuration Files

Behavior in Security Cloud Control

You may see behavior such as the ASA failing to onboard, Security Cloud Control not displaying all of the configuration defined in the ASA's running configuration file, or Security Cloud Control failing to write to the change log.

Possible Cause

The running configuration file of your ASA may be "too large" for Security Cloud Control.

When you an onboard an ASA to Security Cloud Control, Security Cloud Control stores a copy of the ASA's running configuration file in its database. Generally, if that running configuration file is too large (4.5 MB or larger), or it contains too many lines (approximately 22,000 lines), or there are too many access-list entries for a single access group, Security Cloud Control will not be able to predictably manage that device.

To confirm the size of your running configuration file, see Confirming ASA Running Configuration Size.

Workaround or Solution

Contact your Cisco account team for help safely reducing the size of your configuration file without disrupting your security policies.