This document describes how to troubleshoot Cisco Secure Access Control Server (ACS) and resolve error messages.
For information on how to troubleshoot Cisco Secure Access Control System (ACS 5.x and later), refer to Secure Access Control System (ACS 5.x and later) Troubleshooting.
There are no specific requirements for this document.
The information in this document is based on the Cisco Secure Access Control Server (ACS) version 3.3 and 4.x.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Refer to Cisco Technical Tips Conventions for more information on document conventions.
You can experience this problem when you upgrade your ACS server.
If you have too many old log files, you need to clear the "Local Logging Configuration" logs.
Modify the logging of ACS to keep the last three files.
On the ACS GUI, choose System Configuration > Service Control. Check the Manage Directory box and select to keep only the last three files. Then restart ACS and test the upgrade.
If option #1 does not work, you can try to manually remove some log files.
You must always copy the files to a dedicated folder before you delete them.
On the local drive of the Windows server, where ACS for Windows is installed, choose Program Files > Cisco Secure ACS folder.
Delete all the logs under each of these folders:
Restart the PC and retest the upgrade.
The Cannot Delete AAA Server, AAA Server is a Synchronization Partner error message can appear when you delete the entry under Network Configuration.
Complete these steps in order resolve this issue:
Choose Interface Configuration, and check the RDBMS Synchronization check box
Choose System Configuration > RDBMS Synchronization and remove the AAA server that cannot be deleted from the AAA group that is on the Synchronization Partner
You can now delete the AAA server group.
You have two units of ACS SE 1113 and want to replicate the internal database from primary to secondary, but you notice this error message in the secondary unit:
Inbound database replication from ACS <secondary ACS unit name> denied - shared
When you try to modify the key of AAA Server Self under Network Configuration the error message is returned.
In order to resolve the 127.0.0.1 self problem, you can backup and restore the .DMP files on a fresh installation of ACS for Windows 4.2 and modify the 127.0.0.1 entry with the desired IP address.
Note: Cisco bug ID CSCso36620 (registered customers only) states that the toggle nic command changes the AAA server IP address to 127.0.0.1 in the GUI. In order to restore the original IP address on the appliance, issue the set ip command.
Nexus 5010 authentication does not work with TACACS+. This error message can also appear:
Message-Type : Authen failed
Authen-Failure-Code : Key Mismatch
The shared secret defined under the NDG takes precedence over the individually configured device. Look at the shared secret configured under the NDG Century PROD FSW, and make sure it matches with the one configured on Nexus switch.
This issue occurs when you are unable to configure the static IPaddress on ACS 1113 SE.
In order to resolve this issue, install the applACS-4.1-set-ip-CSCsm73656-Patch.zip patch, which is available from Cisco Downloads (registered customers only) . The patch suits all ACS SE 4.1 versions.
When the primary ACS servers goes down, you authenticate users with the secondary server. When the primary is up again, your users are still authenticated against the secondary, even though the primary is running again.
By default, the ASA works in depletion mode. Change it to timed mode so that when the primary ACS server becomes active you can return the authentication to the primary.
You can use:
host(config)# aaa-server <tag> protocol radius
host(config)# reactivation mode timed
host(config)# aaa-server acsgroup deadtime 0
Optional: Specify the amount of time in minutes with the deadtime, between zero and 1440, that elapses between when the last server in the group is disabled and when all the servers are re-enabled. The default is ten minutes.
This issue occurs when you configure the static IP address on ACS 1113 SE.
In order to resolve this issue, try to reimage the software.
The ACS Folder is Locked by Another Application error message appears during an ACS software upgrade, such as the upgrade from version 3.3 to 4.0
Use these solutions in order to solve the problem.
Complete these steps:
In the ACS Window, check the System Configuration > Service Control > Check the Manage Directory check box.
Enter a value, such as 3, in the Keep only the last __ files box.
Restart. The upgrade is likely to work.
If Solution 1 does not resolve the issue, complete these steps:
Backup the current ACS database.
Refer to the Cisco Secure ACS Backup section of User Guide for Cisco Secure ACS for Windows Server for more information on how to perform the backup of the ACS database.
Run the clean.exe file in order to uninstall ACS 3.3 (or your existing version). This file is located on the CD under ACS Utilities/support/clean.
Reinstall ACS 3.3 from the CD.
Restore your database from the file that you saved in Step 1.
Refer to the Cisco Secure ACS System Restore section of User Guide for Cisco Secure ACS for Windows Server for more information on how to restore the ACS database.
Upgrade the ACS to version 4.0.
Refer to Installation Guide for Cisco Secure ACS for Windows Server Version 4.0 for more information on upgrade procedures.
During startup, the ACS SE receives the At least one service or driver failed during startup. use event viewer to examine the event log for details error message.
This error on the ACS SE does not affect any of the ACS functionalities. It is a Microsoft Windows error . This error appears because the monitor, mouse and keyboard cannot be used on the appliance and are disabled by default.
The ACS appliance is a hardened, locked-down system and is designed with security in mind. The appliance uses windows strengthen image, which has all redundant services and connections stopped. It is made to keep all viruses, worms, and DDOS attackers out. Hence there is no VNC, DOS prompt, or any other way to reach the windows configuration. Services like the mouse, keyboard and monitor are closed.
On rare occasions, it indicates that something is corrupted on the appliance image. If you re-image the appliance, it fixes the issue in the majority of instances. You can try to re-image the ACS as well.
This error message appears:
Bad request from NAS
Authen-Failure-Code=Invalid message authenticator in EAP request
This error message usually appears because of a mismatch in the shared secret key or like in this case NDG defined with a key overriding the AAA client key.
Unable to install images earlier than version 4.0 on ACS SE 1113.
Only ACS 4.0 and later can run on ACS SE 1113. Refer to Upgrading and Migrating to Cisco Secure ACS Solution Engine for more information on how to upgrade ACS SE.
When you open the ACS page, you can receive this error: Reason: is currently being edited elsewhere..
Restart the ACS services in order to resolve this issue.
The user is not able to run the remote agent service.
The user must be a local admin user for the service to start.
The Auth type not supported by External DB error appears during user authentication.
This error appears because the CHAP Authentication protocol is not supported on the Microsoft Windows database Active Directory (AD) when you use ACS version 3.3. In order to resolve this issue, use PAP instead of CHAP. Refer to Authentication Protocol-Database Compatibility for more information on Protocol-Database Compatibility for ACS version 3.3.
Unable to ping ACS SE.
Turn off the CSA Agent in System Configuration --> Appliance Configuration in order to enable ping response on ACS SE versions earlier to 4.2. For ACS versions 4.2 and later download and install the patch available from Cisco.com. Refer to Turning Ping On and Off for more information.
The Appliance upgrade in progress message appears, even after the ACS upgrade is complete.
ACS is struck after upgrade and cannot start or stop any services.
In order to resolve this issue, complete these steps:
Log into the ACS Appliance with a different Admin account.
On the Appliance Upgrade present under the System Configuration tab, press the Refresh or the Download button.
Refer to Cisco bug ID CSCsg89042 (registered customers only) for more information.
If you are unable to use the GUI, try to reboot the ACS appliance in order to resolve the issue.
After the replication, the new password gets reset to the old password.
This issue occurs because users do not authenticate to the primary ACS. Once the replication occurs, the primary pushes its policies to the secondary ACS because the replication is not bidirectional. This causes the password to be reset to the old password.
In order to resolve this issue, authenticate the user to the primary ACS, if possible.
DST issues are seen on ACS.
In order to resolve the Daylight Saving Time (DST) issue with ACS, download and install these patches:
Note: Apply the csupdate patch first. Then install the cumulative patch.
The Error: Failed to get NIC configuration: (null) (FFFFFFFF) error appears on the ACS appliance.
This error usually appears if the right version of the ACS image is not used on ACS appliance. It is more of a compatibility issue. Re-image the ACS appliance in order to resolve this issue.
Refer to Re-Imaging the Appliance Hard Drive for more information on how to re-image the ACS appliance.
Unable to disable SSHv1 and leave only the SSHv2 enabled on the ACS appliance.
Right now it is not possible to disable SSHv1 and leave only SSHv2 enabled. Both SSHv1 and SSHv2 are enabled together and cannot be disabled individually.
This section details what to do if you are unable to reset the ACS appliance to factory default settings.
The acs reset-config command includes an option to reset the configuration that, when issued, resets all ACS configuration information, but retains the appliance settings such as network configuration. If you want it to look exactly like the factory default, you need to re-image the appliance.
This section explains why authentication fails with TACACS+ when a Network Device Group (NDG) is configured..
The same AAA client is mapped to two different NDGs, one as a RADIUS client and the other as a TACACS client, and NDG level external database authentication is enabled for the NDG with the RADIUS client.
TACACS+ users are configured in the ACS internal database. When the TACACS+ authentication request comes, ACS looks in the NDG, where the same client is configured as RADIUS.
In order to avoid this problem, remove the external database authentication check box from the RADIUS NDG.
This section explains why some user authentication fails with external a database not operational error.
Here is a list of possible causes and their solutions:
The Remote Agent (RA) version dies not match the ACS version. Install the correct version of RA.
The Remote Agent services are stopped. Restart the RA services.
Upgrade the ACS to the latest available version.
This section explains why you receive the External DB user invalid or bad password error for authentication on ACS.
Review these troubleshooting tips in order to resolve this issue:
If any changes related to the AD membership or the system name are made on the ACS server, make sure to reboot it for changes to take effect.
Check the connectivity between the ACS and the Domain Server.
Security policies on the Domain Server must allow the ACS to Query Username on the Active Directory.
Make sure that there is a two-way trust that exists between the ACS and the Domain Server.
Make sure that the ACS is installed on a server that has Local and DomainAdmin Privileges.
Make sure the username and password is correct.
The faultCode:Server.Error.Request faultString:'HTTP request error' faultDetail:'Error: [IOErrorEvent type="ioError" bubbles=false cancelable=false eventPhase=2 text="Error #2032"]. URL: /acsview/LoadAuthenticationTrendsPortlet.do' error occurs on the ACS when the ACS is accessed using Internet Explorer 8 (IE8).
This error occurs because IE8 is not supported by ACS. Use another browser in order to resolve this issue.
The eap_peap type not configured error occurs on the ACS when you attempt to perform a wireless authentication.
This error occurs on the ACS due to one of these reasons:
The supplicant requesting for EAP-PEAP authentication is not configured on the ACS. Enable EAP-MSCHAPv2 and EAP-GTC from the Global Authentication page, and disable NAP on the primary server in order to resolve the issue.
When a wireless user tries to authenticate through the ACS server, the login fails and the error message is EAP_PEAP Type not configured. This occurs when authenticating with a user configured in the Microsoft Windows AD database, as well as when authenticating with a user in the local ACS database.
When the WLC uses key-wrap for FIPS, but the ACS has not been configured for the same. Configure the same on the ACS in order to resolve the issue.
The issue is the inability to perform local logging on the Cisco Secure ACS Solution Engine instead of using the remote logging capability of the Cisco Secure ACS remote agent.
It is possible to perform local logging on the Cisco Secure ACS Solution Engine instead of using the remote logging capability of the Cisco Secure ACS remote agent. However, local logging on the Cisco Secure ACS Solution Engine is constrained in size. This forces log files to be recycled after seven days. The Cisco Secure ACS remote agent provides full, unconstrained logging capability to a remote server.
With ACS 4.2, the users are authenticated by different methods such as Windows/LDAP/OTP. Is there a way to prepare a complete list of the users with their password authentication methods?
This is time-consuming if performed manually. There is a way to perform this automatically with ACS release 22.214.171.124.
Complete these steps:
Take the backup of the ACS internal database.
Run the CSUtil.exe -dumpUSERS command.
This generates a text file "userauditinfo.txt" that contains the password authentication method used for all the available users.
The ACS is unable to control the delimiter of the mac-address. The delimiter cannot be changed or added.
The ACS is not designed to control the delimeter of the mac-address and it cannot change or add delimeter. The client or the WLC controls the delimiter.
The problem is the backup database cannot be restored when upgrading the ACS for Windows. An insufficient disk space error message is received.
Complete this workaround:
Collect a backup from your database.
Uninstall the ACS software by using the clean utility which is available on the FULL packages of the installation files of the ACS version.
Reinstall the software with the same version.
Perform a restore of the database.
Upgrade the ACS version again.
The ACS cannot join the Active Directory domain and the user cannot authenticate. A clock skew error is received.
This issue can be resolved by changing the time-zone and time on the ACS to match the time-zone and time on the Active Directory.
The Could not generate valid password to perform the Auth test error message appears on the ACS.
In order to resolve this issue, go to System Configuration and click Local Password Management. Make sure the password length is not more than 9 characters. If it is then make sure to change the length to between 4 and 8 characters.
When authenticating as an administrator, a successful message is received. Then, you are quickly forwarded to a page that shows Cannot login to CiscoSecure ACS, all Administration ports are currently in use. Contact the System Administrator for more details. This occurs in ACS 4.X.
This error message indicates that the range of ports allocated for GUI auto redirect are totally reserved and being used by others. In order to resolve this, complete this procedure:
Stop the csadmin service and then try to login.
Verify the HTTP port allocation policy for the Administrator. The complete path is shown here:
Administration Control > Access Policy > HTTP port Allocation > Restrict Administration Sessions to the following port range From Port n to Port n
Increase the range of the ports as per the requirement. For more information, refer to HTTP Configuration.
Specify a lesser Session idle-time-out in the Session Policy. The complete path is shown here:
Administration Control > Session Policy > Session idle timeout
For more information, refer to Session Policy.
Sometimes, reloading the ACS can also help to resolve this issue.
This error is received on ACS version 4.X: ODBC operation failed with following information: message=[Sybase][ODBC Driver][Adaptive Server Anywhere]......
ACS version 4.0 does not install properly if the Sybase server is installed on the same machine. In certain cases, when CiscoWorks and the ACS are used on the same machine, this error message appears and ACS installation problems arise. This occurs because CiscoWorks uses the Sybase for a database. In order to avoid this error, you need to ensure there is no other application that uses SQL Anywhere on that PC in order to successfully install the software. Refer to the Notes section in Preparation for Install or Upgrade ACS for more information.
Unable to integrate ACS with Active Directory, and the Samba Port Status Error error message is received.
In order to resolve this problem, make sure these ports are open to support Active Directory functionality:
ACS needs to be able to reach all the DCs in the domain in order for the ACS-AD integration to be complete. Even if one of the DCs is not reachable from the ACS, the integration would not happen. Refer to Cisco bug ID CSCte92062 (registered customers only) for more information.
Why do I receive the CSCOacs_Internal_Operations_Diagnostics ERROR Could not start message bus error message on ACS?
This is a cosmetic error and it is not a serious problem as long as none of the authentication/authorizations/ACS performance is affected and it only indicates that the internal message bus connection is being re-established.
Why do I receive the 13017 Received TACACS+ packet from unknown Network Device or AAA Client error message on ACS?
This error usually comes up when either the right interface is not configured as the AAA client on ACS, or when the IP address configured on ACS is getting natted. In other words, the right IP address is not contacting ACS which is causing this error. This can also come up if the ip tacacs source-interface <interface-name/id> command is issues on the router, but some other IP address is used on ACS as the AAA client address. Also, disabling single-connect on IOS might help resolve this problem.
Unable to delete Authentication History (RADIUS Successes or Failures) and the syslogs from the ACS.
It is not possible to delete the Authentication History from the ACS. Also, the logs that are sent as syslogs to the ACS itself cannot be deleted.
The management process is not running and the management process shows running (HTTP is nonresponsive).
This issue can be resolved by restoring an older backup of the configuration followed by reimaging and reloading the ACS.
No, this is not possible. SFTP needs a static user name/password. When using a secure ID, it cannot provide a static user name/password.
When trying to filter ACS reports using the Interactive Viewer; all the buttons are greyed out and the right-click menu options are not populated properly. Internet Explorer 8 is the browser used.
This could be a browser related problem. Try other browsers like Firefox in order to get this to work. You could also try to enable the "Compatitibility View" on IE8 to make things appear properly.
When a Windows XP host sends across an 802.1x requests to the ACS via a 3750G switch, there is an authentication prompt only the first time the device attempts to connect to the switch. All subsequent connections are made without an authentication. Why does this happen and how can the authentication prompt be made to appear each time a connection is made?
In order to resolve the issue, go to Network Connections > Local Area Connection > Properties > Authentication, and make sure the Cache user information for subsequent connections to this network option is unchecked.
Why does an Authorization prompt to validate the certificate appear while using Apple devices with ACS? Can I stop this authorization prompt from appearing?
The Authorization prompt is generated by Apple iDevices and not the ACS. There is no way to configure the ACS in such a way that the Apple device will stop showing the Authorization prompt.
Why is the Not all user Active Directory groups are retrieved successfully. One or more of the group's canonical name was not retrieved error message seen on ACS?
This issue occurs because unicode characters are used in the group name on AD. Since ACS sees AD groups as ASCII text, the unicode characters are not translated correctly. As a result, the group membership is not retrieved. Remove the unicode character from the AD configuration in order to resolve this issue.
ACS does not log proxy authentication requests even though Radius proxying has been enabled.
ACS does not log proxy authentication requests. ACS only takes the request and forwards it to the proxy server. The logs will only be visible on the proxy radius server. ACS does not contribute anything to the processing of authentication/accounting of the packet. As a result, no messages are logged on ACS for proxied packets.
ACS loses the configuration when repository is created from the GUI after modifications are done on the CLI.
If you create the repository from the GUI, after modifications are done using CLI, ACS loses the configuration and this is the expected behavior. When you stop and start ACS, the repository will be recreated based on the configuration stored by the GUI. The modifications made on the CLI to a repository created by the GUI will not be transported to the ACS application configuration.
Unable to use an SSH session for the RADIUS IETF attribute "Login-Service".
It is not possible to use an SSH session for the RADIUS IETF attribute "Login-Service" as ACS IETF attributes are a per-RFC standard and there is no way any changes can be made in it.
The value too long (<ACS Server Name>,TacacsAuthentication), Alarm details is "Please see the collector log for details" error message is received.
Check each of these items in order to resolve this problem:
Verify that the console port of the ACS has a cable connected to it.
Remove any unnecessary cables.
Reseat the cable if it is connected to a terminal server.
When a user connects with SSH to the system and uses an expired TACACS password, they are prompted to change their password. However, this password change is not working correctly.
In order to fix this issue, you need to have SSH v2 with "Keyboard interactive" authentication for the SSH v2 set. Cisco bug ID CSCin91851 (registered customers only) discusses this behavior.
ACS Remote Agent is unable to log messages from the ACS Solution Engine, and this error message is received:
CSLogAgent - Can't get max number of connections maxNumberOfConnections using default 32
Try to un-install the Remote Agent from the member server, and re-install it with the domain user account.
In what scenario would an IOS® device using TACACS+ in order to authenticate the users from ACS fallback to its local database?
An IOS device using TACACS+ in order to authenticate the users from ACS will fallback to its local database in these two scenarios:
When the ACS (TACACS+) server is not responding. In the IOS debug messages, you will see "timeout" in contacting the TACACS+ server. In this case, if a fallback is configured to a local database, the IOS will fallback to the local database users.
When the TACACS+ server sends an ERROR message in response to an authentication request.
This error message is seen while adding a new UDV and their VSAs on ACS 4.1 using CSUTIL or RDBMS sync:
Config Error: Illegal
enumeration value 'Name' in key CiscoACS\Dictionaries\005\002\Enumerations -
wrong type, must be int
This issue occurs when the VSAs are added in ACS 126.96.36.199. Refer to Cisco bug ID CSCsq36428 (registered customers only) for more information.
ACS Solution Engine 188.8.131.52 has been scanned and found to be vulnerable to the vulnerability MS08-67, described as:Microsoft Windows Server Service Could Allow Remote Code Execution .
Apply the patch filename appl_w2K3_hotfix_kb958644.zip (registered customers only) on the ACS Solution Engine. Refer to Cisco bug ID CSCsy71711 (registered customers only) for more information.
The Can not initialize SchemeLayer error message is received when executing CSUtil.exe -u.
Complete these steps:
Rename the folder under the admin account that installed the application present in the location documents and settings\administrator\applicationdata\Microsoft\Crypto\RSA\S-1-5xxxxxxxxxx.
Restart the ACS services.
This creates another folder and fixes the broken crypto api. Refer to Cisco bug ID CSCse90116 (registered customers only) for more information.