Alert and Error Messages
This appendix identifies all alert and error messages that Broadband Access Center (BAC) generates, specifically:
•Syslog Alert Messages
–RDU Alerts
–Solaris DPE Alerts
–Watchdog Alerts
–Network Registrar Extension Point Alerts
•RDU Error Messages with CCM
Syslog Alert Messages
BAC generates alerts through the Syslog service. Syslog is a client-server protocol that manages the logging of information on Solaris. BAC syslog alerts are not a logging service; they provide a notification that a problem exists, but do not necessarily define the specific cause of the problem. you might find this information in the appropriate BAC log files.
Message Format
When BAC generates an alert message, the format is:
XXX-#-####: Message
•XXX—Identifies the facility code, which can include:
–RDU (Regional Distribution Unit)
–DPE (Device Provisioning Engine)
–AGENT (rduSnmpAgent or dpeSnmpAgent)
–NR_EP (Network Registrar extension points)
–KDC
•#—Identifies the severity level in use. Table A-1 describes the different levels.
Table A-1 Severity Levels for Alert Messages
|
|
1 |
Identifies an alert |
2 |
Identifies a critical alert |
3 |
Identifies an error |
6 |
Identifies an informational message |
•###—Identifies the numeric error code.
•Message—Provides the alert text or message.
RDU Alerts
Table A-2 identifies the RDU alerts.
Table A-2 RDU Alerts
|
|
RDU-1-101: RDU ran out of disk space |
Indicates that the storage partition of the RDU server ran out of space. After encountering this error, the RDU attempts to restart automatically, but will typically encounter the same error again until more storage space is available. You can remove or compress some of the log files. See Support Tools and Advanced Concepts, page 13-1, for additional information on upgrading the disk. |
RDU-1-103: RDU ran out of memory |
Indicates that the RDU ran out of memory. After encountering this error, the RDU server restarts automatically. |
RDU-1-111: Evaluation key for technology [technology_name] expired |
Indicates that an evaluation key for the technology specified expired. You must contact Cisco sales or TAC for a new license key. |
RDU-1-115: You have used [] percent of available [technology_name] licenses. |
Identifies, in percentage, the quantity of licenses used out of the total number of allowable licenses. Appears when you reach 80 percent of the license capacity. |
RDU-1-122: DNS took [] seconds for lookup of address [ip/hostname]. Check DNS configuration and health of servers |
Indicates that BAC performance may be slow due to delayed response from the DNS. The alert is generated whenever IP address lookup takes more than 60 seconds. |
RDU-2-119: Directory [] that contains the RDU database has a filesystem block size of [] bytes that does not match the required size of [] bytes. Corruption may occur. |
Indicates that the BAC database may not be reliable because the file system that contains the database files is not configured to support an 8-KB or greater block size. For details on configuring the file system block size, refer to the Installation and Setup Guide for the Cisco Broadband Access Center, 2.7.1. |
RDU-2-200: Directory [] that contains the RDU database transaction logs has a filesystem block size of [] bytes that does not match the required size of [] bytes. Corruption may occur. |
Indicates that the BAC database may not be reliable because the file system that contains the database log files is not configured to support an 8-KB or greater block size. For details on configuring the file system block size, refer to the Installation and Setup Guide for the Cisco Broadband Access Center, 2.7.1. |
Note Whenever an RDU syslog alert is sent, additional details (if any) can be found in the log file, BAC_data/rdu/logs/rdu.log. |
Solaris DPE Alerts
Whenever a DPE syslog alert is sent, you can find additional details in the DPE logs.
You can use the show log command to access the DPE logs. For additional information, refer to the Cisco Broadband Access Center DPE CLI Reference, 2.7.1.
Some DPE errors are also propagated to the RDU server log files. You can find these in the BAC_data/rdu/logs/rdu.log file.
Table A-3 identifies the Solaris DPE alerts.
Table A-3 DPE Alerts
|
|
DPE-1-102: DPE ran out of disk space |
The storage partition that the DPE server uses ran out of space. You have three options: a. Clear out any excess support bundles that may reside on the disk. You can do this by moving those support bundles to another machine and then running the clear bundles command from the DPE CLI. b. Run the clear logs command from the DPE CLI to clear more disk space. c. As a last resort, run the clear cache command from the DPE CLI to remove any cache files and force the DPE to resynchronize with the RDU server. |
DPE-1-104: DPE ran out of memory |
The DPE process ran out of memory. After encountering this error condition, the DPE restarts automatically. Determine how many device configurations are on the DPE; the larger the number of device configurations, the more memory is used. To reduce the number of device configurations, limit the number of devices in the provisioning groups, either primary or secondary, that the DPE serves. |
DPE-1-109: Failed to connect to RDU |
The RDU cannot be contacted. You must: a. Verify that the DPE network is configured and connected correctly. b. Check that the DPE is configured to connect to the proper RDU, and that the connecting port is configured properly by using the dpe rdu-server command. c. Check that the RDU process is running on the correct server and listening on the correct port. The DPE attempts to reconnect to the RDU process every few seconds until a connection is established. |
DPE-1-117: DPE license nodes have been exceeded or there is no valid DPE license |
Indicates that the BAC watchdog agent, which starts the DPE, did not detect a license for the DPE. Enter the license key for the DPE using the administrator user interface. If you do not have a license, contact your Cisco representative. |
DPE-1-116: DPE evaluation license has expired. Dropping DPE connections and deleting DPEs from database |
Indicates that an evaluation license key for the DPE expired. You must contact Cisco sales or TAC for a new license key. |
DPE-2-118: Directory [] that contains the DPE's cache has a filesystem block size of [] bytes that does not match the required size of [] bytes. Corruption may occur. |
Indicates that the DPE cache may not be reliable because the file system is not configured to support an 8-KB or greater block size. For details on configuring the file system block size, refer to the Installation and Setup Guide for the Cisco Broadband Access Center, 2.7.1. |
DPE-1-121: Cannot start the server due to an invalid encryption key. |
Indicates that the DPE could not be started because of an invalid encryption key. |
Watchdog Alerts
Whenever the watchdog agent process sends a syslog alert, you can find error details (if any) in the BPR_DATA/agent/logs/agent_console.log file and the log files corresponding to the specific component mentioned in the alert (if any). For example, if you receive an alert similar to The rdu unexpectedly terminated, you would check the RDU server log file (BPR_DATA/rdu/logs/rdu.log) for additional information. Table A-4 identifies the watchdog agent alerts.
Table A-4 Watchdog Agent Alerts
|
|
AGENT-3-9001: Failed to start the [component] |
Indicates that the watchdog has failed to start the specified component. |
AGENT-3-9002: The [component] unexpectedly terminated |
Indicates that the specified component, monitored by the agent process, has unexpectedly failed. |
AGENT-6-9004: The [component] has started |
Generated any time a component is successfully started by the watchdog agent. This message is for informational purposes only. |
AGENT-6-9005: The [component] has stopped |
Generated any time a component is successfully stopped through the watchdog agent. This message is for informational purposes only. |
AGENT-3-9003: Failed to stop the [component] |
Indicates that a component did not stop when the watchdog agent attempted to stop it. |
AGENT-3-9003: Failed to create listener thread; [error no]Failed to close listen socket; [error no] Failed to cancel listen thread, and so on |
Indicates errors that are not defined in other alert messages. |
The [component] variable presented in the watchdog agent alerts list shown in Table A-4 represents any of these component values:
•rdu
•dpe
•tomcat
•cli
•snmpAgent
•kdc
Network Registrar Extension Point Alerts
Whenever a BAC Network Registrar extension point syslog alert is sent, you can find additional details in the Network Registrar log file. Table A-5 identifies the watchdog agent alerts.
Table A-5 Network Registrar Extension Alerts
|
|
NR_EP-1-106: Failed to connect to RDU |
The Network Registrar server cannot connect to the RDU. You should verify that the RDU process is running and, if it is not already running, start the RDU. If the RDU is running, use the Network Registrar computer to ping the RDU. If you are unable to ping the RDU, fix the routing tables or other communication parameters, between the two devices. If this alert is frequently repeated, you may have an unstable connection between the two hosts. Use generally accepted network troubleshooting techniques to improve the connectivity between the two hosts. |
NR_EP-1-107: Failed to connect to any DPEs |
The Network Registrar extension cannot connect to the DPEs. Check that there are DPEs in the provisioning group for each Network Registrar extension. If not, change the Network Registrar provisioning group to one that has DPEs available. If DPEs are in the provisioning group, ensure that the Network Registrar extension has registered with the RDU, if it has not, it will not recognize any of the DPEs. If, after completing the check, the alert continues, check that there is network connectivity between the Network Registrar extension and the DPEs in the provisioning group. If this alert is frequently repeated, you may have an unstable connection between the two hosts. Use generally accepted network troubleshooting techniques to improve the connectivity between the two hosts. |
NR_EP-6-108: The BAC NR extensions have started |
The Network Registrar extensions have been started. |
NR_EP-6-109: The BAC NR extensions have stopped |
The Network Registrar extensions have been stopped. |
NR_EP-6-110: Registered with RDU [address and port] |
The Network Registrar extensions have been registered with the RDU. The address and port identifies the address of the RDU that has registered the Network Registrar extensions. |
NR_EP-1-111: Failed to find usable (best) DPEs |
The Network Registrar extensions are unable to find a usable DPE. |
RDU Error Messages with CCM
The RDU error messages described in this section are accompanied by corrective actions that you perform to remedy specified problems. These error messages are recorded in the rdu.log file, and are a direct result of the Lease Reservation feature. Consequently, if you are not using the Network Registrar Regional CCM feature, these error messages should not appear.
BAC generates three types of lease reservation-related error messages, including:
•Failure to add a reservation request failure. These messages are preceded by the following information, which identifies several aspects of the error:
rdu.cisco.com: 2005 02 09 13:25:11 EST: %BPR-RDU-3-1146: PACE-13: Failed to add
reservation [10.10.10.1] for [1,6,03:03:14:00:00:21] using client-class
[unprovisioned-docsis] and selection-tag [NULL]; [AddReservation: FORWARD_FAILED]
•Failure to remove a reservation request failure. These messages are preceded by the following information, which identifies several aspects of the error:
rdu.cisco.com: 2005 02 09 13:25:11 EST: %BPR-RDU-3-1147: Failed to remove reservation
[10.10.10.1] for [1,6,03:03:14:00:00:21];
•Selection-criteria exclusion tag errors. These messages are preceded by the following information, which identifies several aspects of the error:
rdu.cisco.com: 2005 02 09 13:25:11 EST: %BPR-RDU-4-1145: Use of selection-criteria
exclusion tags [black] is not allowed when adding a lease reservation. They will be
ignored.
Where, from the preceding examples:
•rdu.cisco.com:—Identifies the RDU FQDN
•2005 02 09 13:25:11 EST:—Identifies the date and time that the error took place.
•%BPR-RDU-#-####:—Identifies the RDU error number (#) and error message ID (####).
•[10.10.10.1]—Identifies the IP address being reserved.
•1,6,03:03:14:00:00:21]—Identifies the MAC address that is attempting to reserve an IP address. Note that the second example illustrates BAC support for variable length MAC addresses.
Each of these examples contain a generalized description of the error as well as the error message that is returned from Network Registrar.
The following sections describe error messages that BAC generates.
[OBJECT_EXISTS]
A user is trying to make reservation [11.100.14.21] for MAC [1,6,03:03:14:00:00:21], but the reservation request failed due to duplicate IP used (IP is already being reserved by another device in the database).
Corrective Action:
Use a different IP address or free up the desired IP address.
[RemoveReservation: NOT_FOUND]
A user is trying to remove a reservation [11.100.0.103] for MAC device [1,6,01:02:03:04:05:06]; the request failed due to nonexistence of reservation in CCM database. In other words, CCM is unable to find the reservation in the database.
Corrective Action:
Verify if the reservation exists in the CCM database.
[AddReservation: INVALID_PARENT]
This means that CCM did not find any scope for the IP address of the reservation. CCM local has to find a scope that could contain the reservation. CCM displays this error message when reserved IP is out of scope. You get this error message if you try to add a reservation to a nonexisting subnet or scope.
Corrective Action:
Recheck your DHCP server configuration (scope, subnet, and so on).
[AddReservation: INVALID_SECOND_PARENT]
This should be returned only by a local CCM. It means that no scope existing in the configuration could contain the reservation. Recheck your DHCP server configuration (scope, subnet, and so on).
You get this error if you try to add a reservation to a subnet that exists at the Regional Cluster but no such scope/subnet exists at the local cluster(s):
Or CCM cannot find a scope that contains both the reserved IP address and matching clientclass/selection-tags criteria.
Corrective Action:
Recheck your DHCP server configuration (scope, subnet, clientclass, selection-tags, and so on).
[AddReservation: FORWARD_FAILED]
CCM regional was not able to forward the reservation request to any local CCM. This happens when the CCM cannot find a scope that contains both the reserved IP address and matching clientclass/selection-tags criteria.
The following two use cases illustrate solutions to potential causes of this error:
Example Case #1
Assume that there are three scopes with these attributes/tags; scope A is red and gray, scope B is blue and red, and scope C is blue and green. If the AddReservation API call specifies the inclusion of red and gray tags, but the scope that contains the IP address requested is scope B.
The outcome of this situation is that the reservation request fails and this error is logged in the rdu.log file.
Corrective Action
Use the same setup or configuration, but modify the AddReservation API call to specify the inclusion only red tags, rather than red and gray as described above. This should be successful and any scope that contains the IP address, and matches all of the selection tags, is used.
Note In this example, it is expected that the device gets the reservation from scope B.
Example Case #2
Assume that there are three scopes with these attributes/tags; scope A is red, scope B is blue, and scope C is green. Also assume that the AddReservation API call specifies a client class of test1, Network Registrar defines test1 with the green selection criteria, and the IP being requested is contained in the scope B.
The outcome of this situation is that the reservation request fails and this error is logged in the rdu.log file.
Corrective Action
Use the same setup or configuration, but reconfigure the test1 clientclass in Network Registrar with selection criteria blue. This should successfully correct the problem so that any scope containing IP address, and matching all of the selection tags, is used.
[AddReservation: AX_ETIME]
This indicates that a reservation request timeout has occurred. This could be due to the CCM being overloaded, or busy with tasks and requests, and was not able to process the reservation request.
Corrective Action
Use the administrator user interface to configure a longer timeout value. See RDU Defaults, page 11-19, for information on submitting reservation requests.
[AddReservation: INVALID_OBJECT]
The reservation itself is invalid: the MAC address is invalid, the IP address is missing, or it supplied an invalid client-class name or selection tags.
Corrective Action:
Check the reservation requests: the MAC address, IP address, client class, selection tags, and so on.
Selection-criteria exclusion tags will be ignored
The use of selection-criteria exclusion tags [black] is not allowed when adding a lease reservation. They will be ignored.
Corrective Action:
You should not configure selection-criteria exclusion tags in the DHCP criteria since the use of selection-criteria exclusion tags is not allowed.
[AX_EIO]
This indicates that the connection, or session, between the RDU and the CCM is broken.
Corrective Action:
There is no user interaction required when this error occurs. The RDU automatically establishes another connection to CCM.
[AX_EPIPE]
This indicates that the connection, or session, between the RDU and the CCM is broken.
Corrective Action:
There is no user interaction required when this error occurs. The RDU automatically establishes another connection to CCM.