-
Using Management Center for VPN Routers 1.3
-
Preface
-
Introduction
-
Getting Started with Router MC
-
Router MC Scenarios
-
Working with Activities
-
Managing Devices
-
Configuring VPN Settings
-
Configuring an IOS Firewall on your VPN Router
-
Defining IKE Policies
-
Defining VPN Tunnel Policies
-
Configuring Translation Rules
-
Working with Building Blocks
-
Uploading Device Configurations
-
Deploying Configurations
-
Viewing Reports
-
Router MC Administration
-
Troubleshooting
-
Router MC Operating Prerequisites
-
Router MC User Permissions
-
Table Of Contents
Changing to Debug Mode for Troubleshooting
Collecting Troubleshooting Information (Diagnostics)
Troubleshooting Login Problems
Troubleshooting Device Access Problems
Troubleshooting Device Management Problems
Troubleshooting Activity Problems
Troubleshooting Import Problems
Troubleshooting Problems with Dynamic IP Devices
Troubleshooting Policy Configuration Problems
Troubleshooting Preshared Key or CA Policy Problems
Troubleshooting Deployment Problems
Troubleshooting Problems with Generated CLI Commands
Troubleshooting VPN Tunnel Connectivity Problems
Troubleshooting Performance Problems
Troubleshooting User Interface Anomalies
Troubleshooting Runtime Errors in the Browser
Troubleshooting Backup/Restore Problems
Troubleshooting Dial Backup Problems
Troubleshooting
This section provides explanations and solutions for problems you might encounter when using Router MC. It also provides information about working with log files to locate the source of problems.
Problems are divided per type, as follows:
•
Changing to Debug Mode for Troubleshooting
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Note
Changing to Debug Mode for Troubleshooting
If you encounter problems while working with Router MC, you can enter debug mode to try and track the problems. You can view the debug log messages in the Common Services user interface, by selecting VPN Security Management Solution > Admin > Logging > Operation. You can also send the generated log files to TAC, if necessary, using the command-line utility. See Collecting Troubleshooting Information (Diagnostics) for more information.
Note
To enter debug mode on a Windows station:
1.
2.
3.
To enter debug mode on a Solaris station:
Run the setlogger utility, as follows:
1.
cd /opt/CSCOpx/MDC/iosmdc/bin2.
perl setloggermode -debug3.
4.
Collecting Troubleshooting Information (Diagnostics)
If you encounter a problem that you cannot solve yourself and you request help from Cisco technical support (TAC), support personnel might ask you to submit system configuration information to assist in diagnosing the problem.
CiscoWorks Common Services includes a command-line utility, MDCSupport.exe for Windows and mdcsupport for Solaris. The utility collects configuration and system information in a .zip file called MDCSupportInformation.zip (Windows), or a .tar file called MDCSupportInformation.tar.Z (Solaris). You can send these files to the Cisco Technical Assistance Center (TAC) support staff.
If necessary, you can also send your Router MC database and your device configuration files to the TAC support staff, since these are not included in the MDCSupportInformation files.
To collect troubleshooting information on a Windows system:
1.
If you want to save the file to a different location, enter MDCSupport drive:\path at the DOS prompt.2.
3.
To collect troubleshooting information on a Solaris system:
1.
2.
If you want to save the file to a different location, enter ./mdcsupport at the prompt or ./mdcsupport path.3.
4.
5.
To send your Router MC database to TAC:
1.
2.
iosmdcDB.db
iosmdcDB.log
3.
4.
To send your device configuration files to TAC:
1.
2.
Troubleshooting Login Problems
Explanations and recommended actions are provided for the following potential problems:
•
I cannot log into Router MC.
Explanation—The required services might not be running.
Recommended Action—Wait a few minutes for the processes to start, then try to log in again. If you still cannot log in, check that the required services are running, as follows:
1.
2.
–
–
–
–
–
3.
Note
I encounter a certification problem when trying to log in using the Netscape browser
If you encounter a certification problem that prevents you from invoking Router MC when using Netscape, try the following:
•
•
Troubleshooting Device Access Problems
I cannot access device by Telnet or any other means after deploying access rules to the device.
Explanation—If an ACL is attached to the device's identifying interface, all IP traffic on that interface will be blocked by default.
Recommended Action—From the device console, remove the ACL attachment from the interface. Alternatively, you can add entries to the ACL to allow the relevant routing publication through this interface and/or to enable the SSH session initiated by Router MC.
After you have solved the problem manually on the device, you must modify the access rule policy in Router MC, otherwise the next time you deploy to the device, the same problem will occur. Add ACEs to permit the relevant routing protocol. See Editing an Access Rule, page 1-50 for more information.
Note
Troubleshooting Device Management Problems
Explanations and recommended actions are provided for the following potential problems:
•
•
I cannot create a new device group.
Explanation—A device group of the same name might already exist in the system.
Recommended Action—Make sure you give the new device group a unique name.
Some imported devices are not showing up in the device hierarchy.
Explanation—In rare cases, devices you imported into Router MC are no longer visible in the user interface.
Recommended Action—Log out of Router MC and CiscoWorks. On the server running Router MC, restart the CW2000 Daemon Manager from the Windows Control Panel > Administrative Tools > Services. After a few minutes, log into CiscoWorks and Router MC again.
Troubleshooting Activity Problems
Explanations and recommended actions are provided for the following potential problems:
•
•
•
•
I cannot access my activity after closing and re-opening Router MC.
Explanation—If you close Router MC using the browser's close button and you do not log out of Router MC, your activity remains open. When you log into Router MC again, you will be unable to access it.
Recommended Action—Wait until the activity times out or ask your administrator to close the activity for you. In general, always close your open activity before closing Router MC.
I cannot create a new activity.
Explanation—An activity of the same name might already exist in the system.
Recommended Action—Make sure you give the new activity a unique name.
Some activities are not showing up in the list of activities.
Explanation—In rare cases, previously existing activities are no longer visible in the user interface.
Recommended Action—Log out of Router MC and CiscoWorks. On the server running Router MC, restart the CW2000 Daemon Manager from the Windows Control Panel > Administrative Tools > Services. After a few minutes, log into CiscoWorks and Router MC again.
I deleted an activity but some of the changes made within the activity were retained.
Explanation—When you delete an activity, all the configurations that were made within the activity should be deleted. However, changes that are made to a device group name, device model, or device IOS version are not activity sensitive. This means that if the activity under which the changes were made is deleted, the changes will still be committed to Router MC's database.
Recommended Action—Create a new activity and reverse the changes.
Troubleshooting Import Problems
Explanations and recommended actions are provided for the following potential problems:
•
Import of configuration file fails.
Explanation—The maximum configuration file size might have been exceeded.
Recommended Action—Make sure that the configuration file does not exceed 583 kilobytes.
Import of live device(s) fails and an error message indicates that Router MC could not log into the device.
There are several possible explanations for this problem:
•
Recommended Action—Ping the device. If this is successful, try to establish an SSH connection with the device using a third party SSH client. If this is not successful, verify that you have enabled SSH on the device. See Enabling SSH for Live Device Deployment, page A-1 for more information.
•
Recommended Action—Check that the usernames and passwords specified for device login is correct.
•
Recommended Action—Check which SSH version is enabled on the device by typing sh ip ssh at the command line. If version 2 is enabled, change to version 1 using the following command:
ip ssh version 1
Troubleshooting Problems with Dynamic IP Devices
Import of dynamic IP device fails.
There are several possible explanations for this problem:
•
Recommended Action—Make sure that the AUS settings under the Admin tab are defined correctly. Then, try importing the device again. See Defining Configuration Support Settings, page 1-3 for more information.
•
Recommended Action—Configure the device with the necessary commands to enable it to communicate with AUS. See Prerequisites for Working With Dynamically Addressed Devices, page A-10 for more information.
•
Recommended Action—Add an ACE to the ACL permitting CNS server traffic on port 2976 under TCP.
•
Recommended Action—Wait a few minutes, then try importing the device again.
Troubleshooting Policy Configuration Problems
Explanations and recommended actions are provided for the following potential problems:
After deleting an activity in which devices were moved/deleted, no interfaces are available for selection for inside interface definition, hub assignment, or other tasks requiring interface selection.
Explanation—This might occur if you perform multiple move/delete actions to a device within an activity, and then you delete the activity. The device appears in the Object Selector but its interfaces are no longer available.
Recommended Action—Reimport the device.
When I click apply after making a configuration change, I get a message that the object is locked by another activity.
Explanation—Another user might be configuring that device or device group in a different activity, in which case it will be locked to other users. A red lock icon next to the selected object indicates that the object is locked by another activity.
Recommended Action—Point your mouse at the red lock icon next to the selected object. A tooltip will indicate which activity is locking the object.
Troubleshooting Preshared Key or CA Policy Problems
Explanations and recommended actions are provided for the following potential problems:
•
CA enrollment fails. The device receives the CA's certificates but not its own certificate.
Explanation—The clock settings on the CA machine and the device might not be synchronized and they might differ by more than 24 hours.
Recommended Action—Check the clock settings on the device and change them if necessary. Make sure that the clock settings on the device and on the CA machine do not differ by more than 24 hours. Redeploy the CA configurations after you have made the change.
Aggressive mode commands are not deployed to the device, although they appear in the View Configs page.
Explanation—If there is an existing preshared key defined on the device, the aggressive mode commands might not be written to the device. Router MC generates the commands and deploys them, but they are not reflected in the device's show-run.
Recommended Action—Reboot the device. Then, reimport the device into Router MC, and deploy to the device.
Troubleshooting Deployment Problems
Explanations and recommended actions are provided for the following potential problems:
•
•
•
•
•
•
•
I changed a device's role but the new configs were not deployed to the device.
Explanation—In general, if policy changes are made on a device, the device is automatically selected for inclusion in the next job. However, the device is not selected if the only change was to the device's role, e.g., from spoke to spoke/firewall. The additional required commands for this action will not be generated and deployed to the device unless it is included in the job.
Recommended Action—Manually select the device to be included in the job.
Settings defined on Catalyst with VPN Services Module were not deployed to the device.
Explanation—Router MC only generates commands for the VPN Services Module device when at least one of its assigned spokes is included in the deployment.
Recommended Action—Select the VPN Services Module device and at least one of its assigned spokes in the next deployment.
The VPN policy changes I made were not deployed to the device.
Explanation—Router MC might be set up to manage preshared keys only. In this case, only preshared key configurations are generated and deployed even though other VPN policy configuration changes have been made.
Recommended Action—Go to Configuration > IKE > Preshared Key. A note at the bottom of the Preshared Key page indicates whether the application is set up to managed preshared keys only. You can change this setting from here, if necessary.
When creating a job, the required device does not appear in the device tree so it cannot be selected.
There are two possible explanations for this problem:
•
Recommended Action—Cancel the job creation, approve the activity, and then create the job again. The device should now be available for selection.
•
Recommended Action—Cancel the job creation, deploy or reject the other job to free its devices, and then create the job again. The device should now be available for selection.
Deployment to a device fails because Router MC could not log into the device.
The explanations and recommended actions for this problem are the same as the ones provided for Import of live device(s) fails and an error message indicates that Router MC could not log into the device..
Deployment to a dynamic IP device fails.
There are several possible explanations for this problem:
•
Recommended Action—Make sure that the AUS settings under the Admin tab are defined correctly. Then, try importing the device again. See Defining Configuration Support Settings, page 1-3 for more information.
•
Recommended Action—Configure the device with the necessary commands to enable it to communicate with AUS. See Prerequisites for Working With Dynamically Addressed Devices, page A-10 for more information.
•
Recommended Action—Add an ACE to the ACL permitting CNS server traffic on port 2976 under TCP.
•
Recommended Action—Wait a few minutes, then redeploy.
Deployment Hangs
When deploying IPSec (without GRE), the job deployment remains in progress for a long time (approximately 10 minutes), after which the deployment to the device fails. The job status then also changes to Failed.
There are several possible explanations for this problem:
•
Recommended Action—Redefine either the VPN interface or the inside interface on the device. Select Configuration > Settings to access the options for defining interfaces on hubs and spokes.
•
Recommended Action—Delete the device from the inventory and then import it again using the IP address of its external interface as the device name. Select the Devices tab to access the options for device management and import.
Job action does not end and the job remains in transient status (e.g., generating, deploying, and so forth).
A job action that does not end prevents the job from being deployed. The job remains active and the devices in the job cannot be included in other jobs.
Explanation—This might be due to the failure of one of the services associated with Router MC.
Recommended Action—On the server running Router MC, restart the CW2000 Daemon Manager from the Windows Control Panel > Administrative Tools > Services. This causes the job action to fail, thereby freeing its devices to be included in other jobs. After restarting the Daemon Manager, log into CiscoWorks and Router MC again.
Generation of configurations for a hub fails.
When deploying a large number of devices to a file, the generation of configurations for a hub fails. The proposed configuration in the View Configs page contains an empty string.
Explanation—The maximum configuration file size might have been exceeded.
Recommended Action—Make sure that the configuration file does not exceed 583 kilobytes.
After rolling back a job, no current configurations are displayed for the device under View Configs.
Explanation—If a device has been included in multiple jobs, its previous configuration is the configuration prior to the deployment of the last job in which the device was included. When you perform rollback on a job, the devices' previous configuration becomes the current configuration, and the devices no longer have a previous configuration. Therefore, you can only perform successful rollback for a device once. If you roll back another job in which the same device was included, no current configurations will be available for that device when you select View Configs.
Recommended Action—Before rolling back a job, verify that the devices in the job have previous configurations to which to revert. Do this by opening the job, selecting a device, and then selecting Deployment > View Configs > Previous.
When deploying to a large number of devices using Router MC (Solaris), deployment to several devices fails for no apparent reason.
Explanation—Router MC does not support the deployment of more than 100 devices at one time, on a Solaris station.
Recommended Action—Either redeploy the devices that failed, or try the following workaround:
1.
/etc/init.d/dmgtd stop
2.
In the /etc/rc.config.d/CiscoRMCtrl file, change the value of PX_OPENFILES from 256 to 1024.
3.
/etc/init.d/dmgtd start
Log into CiscoWorks and Router MC again.
Troubleshooting Problems with Generated CLI Commands
Explanations and recommended actions are provided for the following potential problems:
•
The proposed device configurations shown in the Configuration tab are different from those shown in the Workflow tab.
Explanation—The View Configs page in the Configuration tab includes configurations for both committed policies and uncommitted policies (meaning policies defined in the current activity that has not yet been approved).
The View Configs page in the Workflow tab shows only the configurations generated for committed policies (meaning policies from approved activities).
Some of the hub configurations are not shown in the View Configs page in the Configuration tab.
Explanation—Peer-oriented policies (such as preshared key and tunnel policies) are defined on spokes. Router MC mirrors these policies on the hub and generates the required configurations for the relevant hubs. This is done during job creation, not during policy definition. Therefore, the proposed configurations for the hub cannot be seen in the View Configs page in the Configuration tab.
Recommended Action—Define the required policies, approve the activity, and then create a job that includes the spokes associated with the hub for which you want to see configurations. The View Configs page in the Deployment tab will show the configurations that Router MC has generated for the hub based on the policies defined on its assigned spokes.
Troubleshooting GRE Problems
Explanations and recommended actions are provided for the following potential problems:
•
•
•
•
•
Deployment of GRE configurations to devices fails
When deploying GRE configurations to devices, the job fails. In the deployment report, the last command deployed to the device is no router EIGRP #
Explanation— This might occur if the device is already configured to use an IGP routing process that is within the routing process range specified in the Admin tab but is different from the routing process number specified in the GRE settings. In this case, Router MC removes the existing (unsecured) IGP routing process. As soon as this occurs, Router MC loses connectivity with the device and the job fails.
Recommended Action—Choose one of the following recommended actions:
•
•
After switching from regular IPSec to GRE, some flows are transmitted unsecured
If you switch to GRE after previously having configured regular IPSec on your devices, the deployment succeeds but some of the flows are not transmitted in the VPN tunnel.
Explanation—Router MC does not manage the networks beyond the peers for GRE. With GRE, any custom ACE included in the tunnel policy's filter is discarded and only traffic between the peers' attached networks is secured.
Recommended Action—Do the following:
•
•
After deployment of GRE configurations, flows are transmitted unsecured.
Explanation— The inside interfaces on your devices might be configured to use an IGP routing process other than the IGP routing process specified in your GRE settings (meaning that the interfaces belong to an unsecured IGP).
Recommended Action
•
•
Otherwise, you can try to ensure that the spoke peers use only the secured IGP for transmission of flows by manually adding the auto-summary command for the unsecured IGP, or by creating higher cost on the unsecured path so that the secured path will be chosen. Note that these actions should only be performed as a last resort.
After deploying GRE over Frame Relay, the VPN tunnel is disconnected.
Explanation—The configuration on your devices might not meet the prerequisites for configuring GRE over Frame Relay with Router MC. See Prerequisites for Configuring a VPN Over Frame Relay, page A-7 for a list of the prerequisites for Frame Relay.
Recommended Action—Manually change the configuration on the devices to meet the requirements for configuring a VPN over Frame Relay.
After deploying GRE Dynamic IP, VPN connectivity is lost.
Following are some possible explanations for this problem:
•
Recommended Action—Define a group preshared key on the hub. See Creating Group Preshared Keys, page 1-23.
•
Recommended Action—Make sure that the preshared key on the spokes matches the key on the peering hub.
•
Recommended Action—Change the specified subnet in the group preshared key settings. See Creating Group Preshared Keys, page 1-23 for more information.
After removing the device inside interface from the unsecured IGP, I couldn't ping the device and the routing table was not updated.
Explanation—A requirement for migrating to GRE is to manually remove the inside interface on the device from the unsecured IGP (because Router MC will add it to the secured IGP). Following this change, the routing table must be updated with the new path to the inside interface.
Recommended Action—Run the command clear ip route to update the routing table.
If the router is connected to a fast port switch that is configured to avoid spanning tree, the routing table update might take some time. In such a case, a "show route" might indicate that the inside interface is still published by the unsecured IGP. In this case, temporarily remove the spanning tree configuration from the fast port switch.
GRE tunnel is not stable
Check that the ip route commands on the device are logical
Troubleshooting VPN Tunnel Connectivity Problems
Explanations and recommended actions are provided for the following potential problems:
•
•
Secured flow from hub to spoke does not reach the spoke
In a specific secure hub-spoke connection, traffic from the hub to the spoke does not reach the spoke, while traffic from the spoke to the hub reaches its destination successfully.
Explanation—The tunnel policy for this spoke might be defined on the device group level and its filter might include a custom ACE with an IP address specified as the source on the spoke side. In this case, when the filter is mirrored on the hub, all the VPN tunnels created for this group would have the specified IP address as the destination. Therefore, a flow that matches the filter definition might go to the wrong spoke.
Recommended Action—There are two possible recommended actions:
•
•
Note
After deployment, VPN connections that previously existed between the device and other devices not managed by Router MC, are disconnected.
Explanation—Router MC might be set up to remove CLI commands that were not configured using Router MC. In this case, if you have VPN tunnels from the device to other devices not managed by Router MC, they will be disconnected because the manually defined crypto map commands will be removed.
Recommended Action—Go to Admin > Configuration Support Settings. Make sure that the Maintain non-RMC defined CLI check box is selected.
VPN is configured successfully but the flow is either blocked or is transmitted unencrypted.
Explanation—This might occur in an environment in which a firewall is configured, and its ACL does not permit ISAKMP, ESP, and GRE.
Recommended Action—Make sure that the device's role is defined as Hub/Firewall or Spoke/Firewall so that Router MC can generate the ACEs required to allow these flows. Alternatively, manually add ACEs to the ACL on the external interface on the router or firewall to permit ISAKMP and ESP (to enable SA negotiation and encrypted flow), and GRE (if you are using GRE), for the incoming flow.
Note
Troubleshooting Performance Problems
Disturbances in hub-spoke communication, manifesting in high jitter, delay, or packet loss.
Explanation—This might be due to a fragmentation problem. By default, Router MC uses end-to-end MTU discovery using ICMP messages. If ICMP is blocked, MTU discovery will fail and packets will either be lost (if the DF bit is set) or will only be fragmented after encryption (if the DF bit is not set).
Recommended Action—In your fragmentation settings (under Configuration > Settings > General), select Local MTU Handling.
Troubleshooting User Interface Anomalies
Many buttons in the UI are grayed out.
Explanation— You might not have the correct user privileges to perform the tasks associated with the grayed out buttons, or for the group or device currently selected in the object selector. For example, assume you have privileges to manage devices but some functions in the Devices tab are disabled. This might be because a group or device for which you have restricted privileges is selected in the object selector.
Recommended Action—Verify your user permissions in the CiscoWorks desktop or in ACS (depending on what method is being used for user authentication). See the CiscoWorks or ACS documentation for information. Alternatively, consult your system administrator for information about your user permissions.
For more information regarding user permissions in Router MC, see Appendix A, "Router MC User Permissions."
Troubleshooting Runtime Errors in the Browser
Runtime error encountered while working with Router MC.
Explanation—The Java plug-in installed on the server running Router MC must be version 1.3.1. You might have a different version installed. For example, version 1.3.1_02.
Recommended Action—Verify the Java plug-in version, as follows:
1.
2.
3.
4.
If selecting the correct Java Plug-in version does not help, try uninstalling and reinstalling the Java Plug-in, and then restart the server.
Troubleshooting Backup/Restore Problems
Explanations and recommended actions are provided for the following potential problems:
•
•
•
Router MC is not included in the restore operation.
Explanation—This occurs if you do a second restore and you have not rebooted the server after the first restore.
Recommended Action—After restoring data, reboot the Router MC server.
The restore operation failed but all my recent changes were lost.
Explanation—When you start a restore, the current database is cleared. Therefore, even if the restore fails, your most recent changes within the Router MC GUI will not be retained.
Recommended Action
1.
2.
After doing a backup and restore operation, I get an internal error when launching Router MC.
Explanation—The .
Recommended Action—After restoring data, reboot the Router MC server.
Router MC Stops Working
Explanations and recommended actions are provided for the following potential problems:
•
•
Router MC crashes after a database password change.
Explanation—This occurs if you change the database password using CiscoWorks Common Services and you do not restart the server running Router MC.
Recommended Action—After making a database password change, reboot the Router MC server.
While working with Router MC, the user interface suddenly disappears and I get a message that I can no longer access Router MC.
Explanation—When working in Workflow Disabled mode, only one Router MC session per user can be open on the same server at any given time. If you are working with Router MC and another user with the same username and password logs in, your access to Router MC is blocked.
Recommended Action—Close your browser and log into Router MC again.
•
•
When working in Workflow Disabled mode, it is recommended to have only one user per username and password. If necessary, create new users from the CiscoWorks desktop. Go to Server Configuration > Setup > Security > Add Users and add a user with the appropriate permissions. Click the Help button in the Adding Users page for more information.
A security warning popup starts loading, then the Router MC browser freezes suddenly.
Explanation—When using Java plug-in 1.4.x, a security warning popup appears when you first navigate to a page that uses the Object Selector. If you try to navigate to another page or perform an action while the security warning popup is loading, the browser might freeze.
Recommended Action—Close the browser and log out of CiscoWorks. Then, log into CiscoWorks and Router MC again. The next time you get the security warning popup, wait a few seconds for it to load, then click Yes or Always to accept the certificate. See Managing Java Plug-In Security Warnings, page 1-34 for more information.
Troubleshooting Dial Backup Problems
Explanations and recommended actions are provided for the following potential problems:
•
•
Dial backup is not working although the deployment of dial backup configurations succeeded.
Explanation—The Service Assurance Agent (SAA), used to detect loss of network performance on the primary route, might be down.
Recommended Action—Check that the SAA is up and running. If not, try rebooting the device running the SAA.
Deployment of dial backup configurations fails and connectivity to the device(s) is lost.
Explanation—The primary route might have been down when you deployed the dial backup configurations.
Recommended Action—Access the device's CLI manually and get the primary route up, if it was down. Then, reimport the device and deploy again.
