-
Troubleshooting Guide
-
Preface
-
Chapter 1 - Troubleshooting Overview
-
Chapter 2 - Audit Troubleshooting
-
Chapter 3 - Billing Troubleshooting
-
Chapter 4 - Call Processing Troubleshooting
-
Chapter 5 - Configuration Troubleshooting
-
Chapter 6 - Database Troubleshooting
-
Chapter 7 - Maintenance Troubleshooting
-
Chapter 8 - Operations Support System Troubleshooting
-
Chapter 9 - Security Troubleshooting
-
Chapter 10 - Signaling Troubleshooting
-
Chapter 11 - Statistics Troubleshooting
-
Chapter 12 - System Troubleshooting
-
Chapter 13 - Network Troubleshooting
-
Chapter 14 - General Troubleshooting
-
Chapter 15 - Diagnostic Tests
-
Chapter 16 - Disaster Recovery Procedures
-
Chapter 17 - Disk Replacement
-
Appendix A - Recoverable and Nonrecoverable Error Codes
-
Appendix B - System Usage of MGW Keepalive Parameters
-
Appendix C - Overload Control
-
Glossary
-
Feedback
Table Of Contents
Database Management Update Failure: Master/Slave Database Out of Sync—Database (2)
Element Management System Database HeartBeat: Replication PUSH Job Broken—Database (4)
Element Management System Database HeartBeat Process Died—Database (5)
Element Management System Database Replication DefTranDest Queue Overloaded—Database (6)
Element Management System Database DefTran Queue is Overloaded—Database (7)
Element Management System Database Tablespace is Out of Free Space—Database (8)
Urgent: Element Management System Database Archive Log Directory is Getting Full—Database (9)
Element Management System Database: Backup Fails—Database (10)
Element Management System Database Alert.log Alerts—Database (11)
Element Management System Database Process Died—Database (12)
Element Management System Database Performance Alert—Database (13)
Table Size Exceeds Minor Threshold Limit—Database (14)
Table Size Exceeds Major Threshold Limit—Database (15)
Table Size Exceeds Critical Threshold Limit—Database (16)
Data Replication Failed—Database (17)
Unexpected Runtime Data Interaction—Database (18)
Daily Database Backup Completed Successfully—Database (19)
Replication Data Flush Timeout During Switchover—Database (20)
Database Statistics Collection Exception—Database (21)
Unprovisioned Language—Database (22)
Element Management System Oracle Database-Minor Error—Database (23)
Element Management System Oracle Database-Major Error—Database (24)
Secure File Transfer Protocol Transfer Failed—Database (25)
File Write Error—Database (26)
Failure Setting the Index Table Soft Limit—Database (27)
Troubleshooting Database Alarms
Element Management System Database HeartBeat: Replication PUSH Job Broken—Database (4)
Element Management System Database HeartBeat Process Died—Database (5)
Element Management System Database Replication DefTranDest Queue Overloaded—Database (6)
Element Management System Database DefTran Queue is Overloaded—Database (7)
Element Management System Database Tablespace is Out of Free Space—Database (8)
Urgent: Element Management System Database Archive Log Directory is Getting Full—Database (9)
Element Management System Database: Backup Fails—Database (10)
Element Management System Database Alert.log Alerts—Database (11)
Element Management System Database Process Died—Database (12)
Element Management System Database Performance Alert—Database (13)
Table Size Exceeds Minor Threshold Limit—Database (14)
Table Size Exceeds Major Threshold Limit—Database (15)
Table Size Exceeds Critical Threshold Limit—Database (16)
Data Replication Failed—Database (17)
Replication Data Flush Timeout During Switchover—Database (20)
Database Statistics Collection Exception—Database (21)
Unprovisioned Language—Database (22)
Element Management System Oracle Database-Minor Error—Database (23)
Element Management System Oracle Database-Major Error—Database (24)
Secure File Transfer Protocol Transfer Failed—Database (25)
File Write Error—Database (26)
Failure Setting the Index Table Soft Limit—Database (27)
Database Troubleshooting
Revised: July 22, 2009, OL-8723-19Introduction
This chapter provides the information needed to monitor and troubleshoot database events and alarms. This chapter is divided into the following sections:
•
Database Events and Alarms—Provides a brief overview of each database event and alarm.
•
•
Database Events and Alarms
This section provides a brief overview of the database events and alarms for the Cisco BTS 10200 Softswitch in numerical order. Table 6-1 lists all of the database events and alarms by severity.
Note
DATABASE (1)
For additional information, refer to the "Test Report—Database (1)" section.
DATABASE (2)
To monitor and correct the cause of the event, refer to the "Database Management Update Failure: Master/Slave Database Out of Sync—Database (2)" section.
DATABASE (3)
To troubleshoot and correct the cause of the alarm, refer to the "There Are Errors in Element Management System Database DefError Queue; Contact Database Administrator—Database (3)" section.
Note
Prior to contacting Cisco TAC, collect the following information:
On one EMS server:
su - oracledbadm -C repOn both EMS servers:
nodestatdbadm -r get_deferrordbadm -r get_deferrdbadm -r get_deftrandestdbadm -r get_defcall_order
Note
DATABASE (4)
To troubleshoot and correct the cause of the alarm, refer to the "Element Management System Database HeartBeat: Replication PUSH Job Broken—Database (4)" section.
DATABASE (5)
To troubleshoot and correct the cause of the alarm, refer to the "Element Management System Database HeartBeat Process Died—Database (5)" section.
DATABASE (6)
To troubleshoot and correct the cause of the alarm, refer to the "Element Management System Database Replication DefTranDest Queue Overloaded—Database (6)" section.
DATABASE (7)
To troubleshoot and correct the cause of the alarm, refer to the "Element Management System Database DefTran Queue is Overloaded—Database (7)" section.
DATABASE (8)
To troubleshoot and correct the cause of the alarm, refer to the "Element Management System Database Tablespace is Out of Free Space—Database (8)" section.
DATABASE (9)
To troubleshoot and correct the cause of the alarm, refer to the "Urgent: Element Management System Database Archive Log Directory is Getting Full—Database (9)" section.
DATABASE (10)
To troubleshoot and correct the cause of the alarm, refer to the "Element Management System Database: Backup Fails—Database (10)" section.
DATABASE (11)
To troubleshoot and correct the cause of the alarm, refer to the "Element Management System Database Alert.log Alerts—Database (11)" section.
Note
DATABASE (12)
To troubleshoot and correct the cause of the alarm, refer to the "Element Management System Database Process Died—Database (12)" section.
Note
DATABASE (13)
To troubleshoot and correct the cause of the alarm, refer to the "Element Management System Database Performance Alert—Database (13)" section.
Note
DATABASE (14)
To troubleshoot and correct the cause of the alarm, refer to the "Table Size Exceeds Minor Threshold Limit—Database (14)" section.
Note
Prior to contacting Cisco TAC, collect the following information:
show db-usage table_name=<string>DATABASE (15)
To troubleshoot and correct the cause of the alarm, refer to the "Table Size Exceeds Major Threshold Limit—Database (15)" section.
DATABASE (16)
To troubleshoot and correct the cause of the alarm, refer to the "Table Size Exceeds Critical Threshold Limit—Database (16)" section.
DATABASE (17)
To troubleshoot and correct the cause of the alarm, refer to the "Data Replication Failed—Database (17)" section.
DATABASE (18)
To monitor and correct the cause of the event, refer to the "Unexpected Runtime Data Interaction—Database (18)" section.
Note
DATABASE (19)
For additional information, refer to the "Daily Database Backup Completed Successfully—Database (19)" section.
DATABASE (20)
To troubleshoot and correct the cause of the alarm, refer to the "Replication Data Flush Timeout During Switchover—Database (20)" section.
DATABASE (21)
To troubleshoot and correct the cause of the alarm, refer to the "Database Statistics Collection Exception—Database (21)" section.
DATABASE (22)
To troubleshoot and correct the cause of the alarm, refer to the "Unprovisioned Language—Database (22)" section.
DATABASE (23)
To troubleshoot and correct the cause of the alarm, refer to the "Element Management System Oracle Database-Minor Error—Database (23)" section.
Note
DATABASE (24)
To troubleshoot and correct the cause of the alarm, refer to the "Element Management System Oracle Database-Major Error—Database (24)" section.
Note
DATABASE (25)
To troubleshoot and correct the cause of the alarm, refer to the "Secure File Transfer Protocol Transfer Failed—Database (25)" section.
DATABASE (26)
To troubleshoot and correct the cause of the alarm, refer to the "File Write Error—Database (26)" section.
Note
DATABASE (27)
To troubleshoot and correct the cause of the alarm, refer to the "Failure Setting the Index Table Soft Limit—Database (27)" section.
Monitoring Database Events
This section provides the information needed to monitor and correct database events. Table 6-2 lists all of the database events in numerical order and provides cross reference to each subsection in this section.
Test Report—Database (1)
The Test Report is for testing the database event category. The event is informational and no further action is required.
Database Management Update Failure: Master/Slave Database Out of Sync—Database (2)
The Database Management Update Failure: Master/Slave Database Out of Sync event functions as a warning that master and slave databases are out of sync. The primary cause of the event is that the master database under Oracle control in the EMS was successfully updated, but the subsequent update of the shared memory tables in the Call Agent (CA) servers and/or Feature Servers (FS) were not properly updated. To correct the primary cause of the event, perform an audit of the database in question to correct the data stored in shared memory. Additionally, use the CLI to show and delete the transaction queue, and to audit and manage the queue.
There are Errors in Element Management System Database DefError Queue; Contact Database Administrator—Database (3)
The There are Errors in Element Management System Database DefError Queue; Contact Database Administrator alarm (critical) indicates that there are errors in the EMS database DefError queue. To troubleshoot and correct the cause of the There are Errors in Element Management System Database DefError Queue; Contact Database Administrator alarm, refer to the "There Are Errors in Element Management System Database DefError Queue; Contact Database Administrator—Database (3)" section.
Element Management System Database HeartBeat: Replication PUSH Job Broken—Database (4)
The Element Management System Database HeartBeat: Replication PUSH Job Broken alarm (critical) indicates that the replication PUSH job is broken. To troubleshoot and correct the cause of the of the Element Management System Database HeartBeat: Replication PUSH Job Broken alarm, refer to the "Element Management System Database HeartBeat: Replication PUSH Job Broken—Database (4)" section.
Element Management System Database HeartBeat Process Died—Database (5)
The Element Management System Database HeartBeat Process Died alarm (critical) indicates that the EMS database heartbeat process has died. To troubleshoot and correct the cause of the Element Management System Database HeartBeat Process Died alarm, refer to the "Element Management System Database HeartBeat Process Died—Database (5)" section.
Element Management System Database Replication DefTranDest Queue Overloaded—Database (6)
The Element Management System Database Replication DefTranDest Queue Overloaded alarm (major) indicates that the EMS database replication DefTranDest queue is overloaded. To troubleshoot and correct the cause of the Element Management System Database Replication DefTranDest Queue Overloaded alarm, refer to the "Element Management System Database Replication DefTranDest Queue Overloaded—Database (6)" section.
Element Management System Database DefTran Queue is Overloaded—Database (7)
The Element Management System Database DefTran Queue is Overloaded alarm (minor) indicates that the EMS database DefTran queue is overloaded. To troubleshoot and correct the cause of the Element Management System Database DefTran Queue is Overloaded alarm, refer to the "Element Management System Database DefTran Queue is Overloaded—Database (7)" section.
Element Management System Database Tablespace is Out of Free Space—Database (8)
The Element Management System Database Tablespace is Out of Free Space alarm (major) indicates that the EMS database table space is out of free space. To troubleshoot and correct the cause of the Element Management System Database Tablespace is Out of Free Space alarm, refer to the "Element Management System Database Tablespace is Out of Free Space—Database (8)" section.
Urgent: Element Management System Database Archive Log Directory is Getting Full—Database (9)
The Urgent: Element Management System Database Archive Log Directory is Getting Full alarm (critical) indicates that the EMS database archive log directory is getting full. To troubleshoot and correct the cause of the Urgent: Element Management System Database Archive Log Directory is Getting Full alarm, refer to the "Urgent: Element Management System Database Archive Log Directory is Getting Full—Database (9)" section.
Element Management System Database: Backup Fails—Database (10)
The Element Management System Database: Backup Fails alarm (major) indicates that the EMS database backup has failed. To troubleshoot and correct the cause of the Element Management System Database: Backup Fails alarm, refer to the "Element Management System Database: Backup Fails—Database (10)" section.
Element Management System Database Alert.log Alerts—Database (11)
The Element Management System Database Alert.log Alerts alarm (major) indicates that the EMS database alerts are being received and logged into the alert log. To troubleshoot and correct the cause of the Element Management System Database Alert.log Alerts alarm, refer to the "Element Management System Database Alert.log Alerts—Database (11)" section.
Element Management System Database Process Died—Database (12)
The Element Management System Database Process Died alarm (critical) indicates that the EMS database process has died. To troubleshoot and correct the cause of the Element Management System Database Process Died alarm, refer to the "Element Management System Database Process Died—Database (12)" section.
Element Management System Database Performance Alert—Database (13)
The Element Management System Database Performance Alert alarm (major) indicates that the EMS database performance has degraded. To troubleshoot and correct the cause of the Element Management System Database Performance Alert alarm, refer to the "Element Management System Database Performance Alert—Database (13)" section.
Table Size Exceeds Minor Threshold Limit—Database (14)
The Table Size Exceeds Minor Threshold Limit alarm (minor) indicates that the table size has exceeded the minor threshold crossing limit. To troubleshoot and correct the cause of the Table Size Exceeds Minor Threshold Limit alarm, refer to the "Table Size Exceeds Minor Threshold Limit—Database (14)" section.
Table Size Exceeds Major Threshold Limit—Database (15)
The Table Size Exceeds Major Threshold Limit alarm (major) indicates that the table size has exceeded the major threshold crossing limit. To troubleshoot and correct the cause of the Table Size Exceeds Major Threshold Limit alarm, refer to the "Table Size Exceeds Major Threshold Limit—Database (15)" section.
Table Size Exceeds Critical Threshold Limit—Database (16)
The Table Size Exceeds Critical Threshold Limit alarm (critical) indicates that the table size has exceeded the critical threshold crossing limit. To troubleshoot and correct the cause of the Table Size Exceeds Critical Threshold Limit alarm, refer to the "Table Size Exceeds Critical Threshold Limit—Database (16)" section.
Data Replication Failed—Database (17)
The Data Replication Failed alarm (major) indicates that the data replication failed. To troubleshoot and correct the cause of the Data Replication Failed alarm, refer to the "Data Replication Failed—Database (17)" section.
Unexpected Runtime Data Interaction—Database (18)
The Unexpected Runtime Data Interaction event functions as a warning that an unexpected runtime data interaction has occurred. The primary cause of the event is that an unexpected data interaction has been detected at runtime in the call agent or feature server. To correct the primary cause of the event, collect the logs and contact Cisco TAC.
Note
Daily Database Backup Completed Successfully—Database (19)
The Daily Database Backup Completed Successfully event functions as an informational alert that the daily database backup has completed successfully. The event is informational and no further action is required.
Replication Data Flush Timeout During Switchover—Database (20)
The Replication Data Flush Timeout During Switchover alarm (major) indicates that the replication data flush timed out during a switchover. To troubleshoot and correct the cause of the Replication Data Flush Timeout During Switchover alarm, refer to the "Replication Data Flush Timeout During Switchover—Database (20)" section.
Database Statistics Collection Exception—Database (21)
The Database Statistics Collection Exception alarm (minor) indicates that the database statistics collection process had an exception. To troubleshoot and correct the cause of the Database Statistics Collection Exception alarm, refer to the "Database Statistics Collection Exception—Database (21)" section.
Unprovisioned Language—Database (22)
The Unprovisioned Language alarm (major) indicates that the operator has missed provisioning the language in this alarm in the language table on the EMS. To troubleshoot and correct the cause of the Unprovisioned Language alarm, refer to the "Unprovisioned Language—Database (22)" section.
Element Management System Oracle Database-Minor Error—Database (23)
The Element Management System Oracle Database-Minor Error alarm (minor) indicates that a minor error has occurred in an Oracle background process. To troubleshoot and correct the cause of the Element Management System Oracle Database-Minor Error alarm, refer to the "Element Management System Oracle Database-Minor Error—Database (23)" section.
Element Management System Oracle Database-Major Error—Database (24)
The Element Management System Oracle Database-Major Error alarm (major) indicates that a major error has occurred in an Oracle background process. To troubleshoot and correct the cause of the Element Management System Oracle Database-Major Error alarm, refer to the "Element Management System Oracle Database-Major Error—Database (24)" section.
Secure File Transfer Protocol Transfer Failed—Database (25)
The Secure File Transfer Protocol Transfer Failed alarm (alarm) indicates that a SFTP file transfer had failed. To troubleshoot and correct the cause of the Secure File Transfer Protocol Transfer Failed alarm, refer to the "Secure File Transfer Protocol Transfer Failed—Database (25)" section.
File Write Error—Database (26)
The File Write Error alarm (major) indicates that a file write error has occurred. To troubleshoot and correct the cause of the File Write Error alarm, refer to the "File Write Error—Database (26)" section.
Failure Setting the Index Table Soft Limit—Database (27)
The Failure Setting the Index Table Soft Limit alarm (major) indicates that a corruption of the IDX framework for the table has occurred. To troubleshoot and correct the cause of the Failure Setting the Index Table Soft Limit alarm, refer to the "Failure Setting the Index Table Soft Limit—Database (27)" section.
Troubleshooting Database Alarms
This section provides the information needed to monitor and correct database alarms. Table 6-3 lists all of the database alarms in numerical order and provides cross reference to each subsection in this section.
There Are Errors in Element Management System Database DefError Queue; Contact Database Administrator—Database (3)
The There Are Errors in Element Management System Database DefError Queue; Contact Database Administrator alarm (critical) indicates that there are errors in the EMS database DefError queue. The primary cause of the alarm is that replication data conflicts have occurred. The additional causes of the alarm are that a request for update or delete on non-existing data occurred, or an unique constraint (primary key) was violated. Correcting the cause of the alarm may require a manual update on database tables. Contact Cisco TAC for assistance.
Note
Prior to contacting Cisco TAC, collect the following information:
On one EMS server:
su - oracledbadm -C repOn both EMS servers:
nodestatdbadm -r get_deferrordbadm -r get_deferrdbadm -r get_deftrandestdbadm -r get_defcall_order
Note
Element Management System Database HeartBeat: Replication PUSH Job Broken—Database (4)
The Element Management System Database HeartBeat: Replication PUSH Job Broken alarm (critical) indicates that the replication PUSH job is broken. The primary cause of the alarm is that the remote database is down or the remote database is not accessible. To correct the primary cause of the alarm, re-start or restore remote database. The secondary cause of the alarm is that network connections in broken. To correct the secondary cause of the alarm, correct the network connection problem. The ternary cause of the alarm is that the remote Oracle Listener process died. To correct the ternary cause of the alarm, restart the remote Listener process.
For additional troubleshooting information, execute the following:
On one EMS server:
su - oracledbadm -C repOn both EMS servers:
nodestatdbadm -r get_deferrordbadm -r get_deferrdbadm -r get_deftrandestdbadm -r get_defcall_order
Note
Element Management System Database HeartBeat Process Died—Database (5)
The Element Management System Database HeartBeat Process Died alarm (critical) indicates that the EMS database heartbeat process has died. The primary cause of the alarm is that the EMS DBHeartBeat Process was terminated by SMG, or stopped by the platform. To correct the primary cause of the alarm, restart the DBHeartBeat by executing the dbinit -H -i start command as an Oracle user, or by executing the platform start command as a root user.
For additional troubleshooting information, execute the following:
On one EMS server:
su - oracledbadm -C repOn both EMS servers:
nodestatdbadm -r get_deferrordbadm -r get_deferrdbadm -r get_deftrandestdbadm -r get_defcall_order
Note
Element Management System Database Replication DefTranDest Queue Overloaded—Database (6)
The Element Management System Database Replication DefTranDest Queue Overloaded alarm (major) indicates that the EMS database replication DefTranDest queue is overloaded. The primary cause of the alarm is that the replication PUSH job is broken. To correct the primary cause of the alarm, correct the problems on remote database. The secondary cause of the alarm is that the remote database is not accessible. To correct the secondary cause of the alarm, verify that the db_heart_beat process is up. The ternary cause of the alarm is that the database is overloaded. To correct the ternary cause of the alarm, troubleshoot database performance.
For additional troubleshooting information, execute the following:
On one EMS server:
su - oracledbadm -C repOn both EMS servers:
nodestatdbadm -r get_deferrordbadm -r get_deferrdbadm -r get_deftrandestdbadm -r get_defcall_order
Note
Element Management System Database DefTran Queue is Overloaded—Database (7)
The Element Management System Database DefTran Queue is Overloaded alarm (minor) indicates that the EMS database DefTran queue is overloaded. The primary cause of the alarm is that the replication DefTranDest queue is overloaded. To correct the primary cause of the alarm, resume replication activities. The secondary cause of the alarm is that there are too many errors in DefError queue. To correct the secondary cause of the alarm, correct the replication errors. The ternary cause of the alarm is the replication PURGE job is broken or overloaded. To correct the ternary cause of the alarm, enable the replication PURGE job.
For additional troubleshooting information, execute the following:
On one EMS server:
su - oracledbadm -C repOn both EMS servers:
nodestatdbadm -r get_deferrordbadm -r get_deferrdbadm -r get_deftrandestdbadm -r get_defcall_order
Note
Element Management System Database Tablespace is Out of Free Space—Database (8)
The Element Management System Database Tablespace is Out of Free Space alarm (major) indicates that the EMS database table space is out of free space. The primary cause of the alarm is that there has been an increase in data volume or transactions. To correct the primary cause of the alarm, add more space to the tablespace.
For additional troubleshooting information, execute the following:
On one EMS server:
su - oracledbadm -C repOn both EMS servers:
nodestatdbadm -r get_deferrordbadm -r get_deferrdbadm -r get_deftrandestdbadm -r get_defcall_order
Note
Urgent: Element Management System Database Archive Log Directory is Getting Full—Database (9)
The Urgent: Element Management System Database Archive Log Directory is Getting Full alarm (critical) indicates that the EMS database archive log directory is getting full. The primary cause of the alarm is that transaction volume has increased. To correct the primary cause of the alarm, backup and cleanup the archive log files. Additionally, add more space to archive log directory.
Element Management System Database: Backup Fails—Database (10)
The Element Management System Database: Backup Fails alarm (major) indicates that the EMS database backup has failed. The primary cause of the alarm is that the system or hardware is unstable. To correct the primary cause of the alarm, re-start the backup process.
Element Management System Database Alert.log Alerts—Database (11)
The Element Management System Database Alert.log Alerts alarm (major) indicates that the EMS database alerts are being received and logged into the alert log. The probable cause of the ORA- errors report in alert.log file is documented in the $ORACLE_HOME/rdbms/mesg/oraus.msg file. Login to EMS system as oracle user (or su - oracle) to view this file. If more information is needed, contact Cisco TAC for database support, or query Oracle metalink library at http://metalink.oracle.com. The corrective action is documented in the $ORACLE_HOME/rdbms/mesg/oraus.msg file. Login to EMS system as oracle user (or su - oracle) to view this file. If more information is needed, contact Cisco TAC for database support, or query the Oracle metalink library at http://metalink.oracle.com. The alert.log file is the global message file for errors issued by all Oracle background processes. The majority of error conditions may require administrator's investigation and manual correction. Thus the administrator should manually clear this alarm.
Note
Element Management System Database Process Died—Database (12)
The Element Management System Database Process Died alarm (critical) indicates that the EMS database process has died. The primary possible causes of the alarm are:
•
•
•
To correct the possible causes of the alarm, re-start process and contact Cisco TAC.
Note
Element Management System Database Performance Alert—Database (13)
The Element Management System Database Performance Alert alarm (major) indicates that the EMS database performance has degraded. To identify the primary cause of the alarm, check the "StatEventName" dataword information. To correct the primary cause of the alarm, perform database performance tuning and contact Cisco TAC.
Note
Table Size Exceeds Minor Threshold Limit—Database (14)
The Table Size Exceeds Minor Threshold Limit alarm (minor) indicates that the table size has exceeded the minor threshold crossing limit. The primary cause of the alarm is that the pre-provisioned size for the stated table is nearing the licensed limit on the number of entries it can hold. To correct the primary cause of the alarm, contact Cisco TAC to purchase additional entry space for this particular table.
Note
Prior to contacting Cisco TAC, collect the following information:
show db-usage table_name=<string>Table Size Exceeds Major Threshold Limit—Database (15)
The Table Size Exceeds Major Threshold Limit alarm (major) indicates that the table size has exceeded the major threshold crossing limit. The primary cause of the Table Size Exceeds Major Threshold Limit alarm is the major threshold crossing limit has been exceeded. No further action is required.
Table Size Exceeds Critical Threshold Limit—Database (16)
The Table Size Exceeds Critical Threshold Limit alarm (critical) indicates that the table size has exceeded the critical threshold crossing limit. The primary cause of the alarm is that the critical threshold limit was exceeded. No corrective action is required.
Data Replication Failed—Database (17)
The Data Replication Failed alarm (major) indicates that the data replication failed. The primary cause of the alarm is that an index is out of range. The secondary cause of the alarm is that a record size mismatch occurred. No corrective action is required.
Replication Data Flush Timeout During Switchover—Database (20)
The Replication Data Flush Timeout During Switchover alarm (major) indicates that the replication data flush timed out during a switchover. The primary cause of the alarm is that a Replication Module software problem has occurred. To correct the primary cause of the alarm, a database restore procedure needs to be executed on the side which goes active after a switchover. The alarm should be cleared manually after recovery action is taken.
Database Statistics Collection Exception—Database (21)
The Database Statistics Collection Exception alarm (minor) indicates that the database statistics collection process had an exception. To identify the primary cause of the alarm, check the information listed in the "Exception" dataword field. The correction action varies and is determined by the type of exception. For more information about the ORA-xxxxx errors, execute an oerr ora xxxxx command as an Oracle user.
Unprovisioned Language—Database (22)
The Unprovisioned Language alarm (major) indicates that the operator has missed provisioning the language in this alarm in the language table on the EMS. To correct the cause of the alarm, the operator has to provision the missing language and update the LANGUAGE table.
Use the following procedures to the correct the cause of the Unprovisioned Language alarm:
This section describes general troubleshooting procedures.
Ensure the subscriber has MLS using report billing_record:
DIALEDDIGITS=*56CALLTERMINATIONCAUSE=NORMAL_CALL_CLEARING*56 is the VSC entered by the subscriber to start MLS. NORMAL_CALL_CLEARING shows the IVR successfully completed its service.
If NORMAL_CALL_CLEARING does not return, check both the service and subscriber_service_profile tables:
btsadmin>show service id=mlstestID=mlstestFNAME1=MLSbtsadmin>show subscriber_service_profile sub-id=2212437211SUB_ID=2212437211SERVICE_ID=mlstestIf you hear a reorder-tone from a SIP phone ensure the status of the BTS announcement table is all of the following:
btsadmin>status tt tgn-id=889;cic=all889 1 ADMIN_INS TERM_ACTIVE_IDLE ACTV IDLE NON_FAULTYIf you hear a click from an MGCP phone ensure the status of the BTS announcement table is all of the following:
btsadmin>status tt tgn-id=889;cic=all889 1 ADMIN_INS TERM_ACTIVE_IDLE ACTV IDLE NON_FAULTYIf you hear a reorder tone instead of audio ensure the release_cause table routes to correct MS:
btsadmin>show release_causeID=1ANNC_ID=18btsadmin>show announcement...ANNOUNCEMENT_FILE=ann_id_18.auROUTE_GUIDE_ID=10013Ensure the IVR script points to the correct MS and that the MLS has an FNAME:
btsadmin>show ivr_script_profileFNAME=MLSIVR_ACCESS_MODE=IVRIVR_ROUTE_GUIDE_ID=10013IVR_SCRIPT_PKG_TYPE=BAUEnsure the annc-tg-profile table is correct:
ANNC_LANG_FORMAT_SUPPORTED=N for IPUnityANNC_LANG_FORMAT_SUPPORTED=Y for CognitronicsTurn on trace in the BTS Call Agent (CA) for MLS, set MGCP on the CA to info5 level and examine the BAU code from the MS:
TC_11.3.1_CA.log:. . MGA 00-00. |<<<< RECV FROM: 10.1.31.2FROM-PORT=2427 TO-PORT=2727 <<<<|TC_11.3.1_CA.log-.. MGA 00-00. |notify 717annc/1@sj-ms1-s4.sjc-devtest.com MGCP 1.0 NCS 1.0^M|TC_11.3.1_CA.log-.. MGA 00-00. |X: 2B00000007^M|TC_11.3.1_CA.log-.. MGA 00-00. |O: A/of(rc=601)^M|TC_11.3.1_CA.log-.. MGA 00-00. ||snd_rcv.c:260Error: NEEDExplanation:
The mls-annc-mult-factor token value is lower than the number of announcements existing on the MS.
Recommended Action:
Provision the mls-annc-mult-factor token value greater than the number of announcements on the MS.
Error: Return Code 601: File not foundExplanation:
MSs are limited to 40 character filenames. These 40 characters include the extension (typically a.wav) and the announcement-file-prefix: for example fra_, eng_ and spa_.
Recommended Action:
Change the filename length to less than 40 characters.
Element Management System Oracle Database-Minor Error—Database (23)
The Element Management System Oracle Database-Minor Error alarm (minor) indicates that a minor error has occurred in an Oracle background process. The probable cause of the ORA- errors report in alert.log file is documented in the $ORACLE_HOME/rdbms/mesg/oraus.msg file. Login to EMS system as oracle user (or su - oracle) to view this file. If more information is needed, contact Cisco TAC for database support, or query Oracle metalink library at http://metalink.oracle.com. The corrective action is documented in the $ORACLE_HOME/rdbms/mesg/oraus.msg file. Login to EMS system as oracle user (or su - oracle) to view this file. If more information is needed, contact Cisco TAC for database support, or query the Oracle metalink library at http://metalink.oracle.com. The alert.log file is the global message file for errors issued by all Oracle background processes. The majority of error conditions may require administrator's investigation and manual correction. Thus the administrator should manually clear this alarm.
Note
Element Management System Oracle Database-Major Error—Database (24)
The Element Management System Oracle Database-Major Error alarm (major) indicates that a major error has occurred in an Oracle background process. The probable cause of the ORA- errors report in alert.log file is documented in the $ORACLE_HOME/rdbms/mesg/oraus.msg file. Login to EMS system as oracle user (or su - oracle) to view this file. If more information is needed, contact Cisco TAC for database support, or query Oracle metalink library at http://metalink.oracle.com. The corrective action is documented in the $ORACLE_HOME/rdbms/mesg/oraus.msg file. Login to EMS system as oracle user (or su - oracle) to view this file. If more information is needed, contact Cisco TAC for database support, or query the Oracle metalink library at http://metalink.oracle.com. The alert.log file is the global message file for errors issued by all Oracle background processes. The majority of error conditions may require administrator's investigation and manual correction. Thus the administrator should manually clear this alarm.
Note
Secure File Transfer Protocol Transfer Failed—Database (25)
The Secure File Transfer Protocol Transfer Failed alarm (major) indicates that a secure file transfer has failed. The primary cause of the alarm is that the SFTP was unable to connect between active and standby call agents. To troubleshoot and correct the primary cause of the alarm, verify communication between primary and secondary call agent (CA). On each CA, ping the other node. The secondary cause of the alarm is that the system was unable to login to the remote host. To troubleshoot and correct the secondary cause of the alarm, verify that the SSH keys have been pre-configured for user root on both active and standby call agents. The ternary cause of the alarm is that a file transfer error has occurred. To troubleshoot and correct the ternary cause of the alarm, check the Error dataword to see if it gives an indication of the kind of error that occurred. It could be a file-system error on the remote host, or a communication failure between the active and standby call agents.
File Write Error—Database (26)
The File Write Error alarm (major) indicates that a file write error has occurred. The primary cause of the alarm is that a system error has occurred and that the system may be out of file descriptors. To troubleshoot and correct the primary cause of the alarm, call Cisco TAC technical support.
Note
Failure Setting the Index Table Soft Limit—Database (27)
The Failure Setting the Index Table Soft Limit alarm (major) indicates that a corruption of the IDX framework for the table has occurred. To troubleshoot and correct the primary cause of the alarm, run the tiat command to verify that a corruption has occurred. Once the corruption is verified, fix the corruption.
