Table Of Contents

Database Troubleshooting

Introduction

Database Events and Alarms

DATABASE (1)

DATABASE (2)

DATABASE (3)

DATABASE (4)

DATABASE (5)

DATABASE (6)

DATABASE (7)

DATABASE (8)

DATABASE (9)

DATABASE (10)

DATABASE (11)

DATABASE (12)

DATABASE (13)

DATABASE (14)

DATABASE (15)

DATABASE (16)

DATABASE (17)

DATABASE (18)

DATABASE (19)

DATABASE (20)

DATABASE (21)

DATABASE (22)

DATABASE (23)

DATABASE (24)

DATABASE (25)

DATABASE (26)

DATABASE (27)

Monitoring Database Events

Test Report—Database (1)

Database Management Update Failure: Master/Slave Database Out of Sync—Database (2)

There are Errors in Element Management System Database DefError Queue; Contact Database Administrator—Database (3)

Element Management System Database HeartBeat: Replication PUSH Job Broken—Database (4)

Element Management System Database HeartBeat Process Died—Database (5)

Element Management System Database Replication DefTranDest Queue Overloaded—Database (6)

Element Management System Database DefTran Queue is Overloaded—Database (7)

Element Management System Database Tablespace is Out of Free Space—Database (8)

Urgent: Element Management System Database Archive Log Directory is Getting Full—Database (9)

Element Management System Database: Backup Fails—Database (10)

Element Management System Database Alert.log Alerts—Database (11)

Element Management System Database Process Died—Database (12)

Element Management System Database Performance Alert—Database (13)

Table Size Exceeds Minor Threshold Limit—Database (14)

Table Size Exceeds Major Threshold Limit—Database (15)

Table Size Exceeds Critical Threshold Limit—Database (16)

Data Replication Failed—Database (17)

Unexpected Runtime Data Interaction—Database (18)

Daily Database Backup Completed Successfully—Database (19)

Replication Data Flush Timeout During Switchover—Database (20)

Database Statistics Collection Exception—Database (21)

Unprovisioned Language—Database (22)

Element Management System Oracle Database-Minor Error—Database (23)

Element Management System Oracle Database-Major Error—Database (24)

Secure File Transfer Protocol Transfer Failed—Database (25)

File Write Error—Database (26)

Failure Setting the Index Table Soft Limit—Database (27)

Troubleshooting Database Alarms

There Are Errors in Element Management System Database DefError Queue; Contact Database Administrator—Database (3)

Element Management System Database HeartBeat: Replication PUSH Job Broken—Database (4)

Element Management System Database HeartBeat Process Died—Database (5)

Element Management System Database Replication DefTranDest Queue Overloaded—Database (6)

Element Management System Database DefTran Queue is Overloaded—Database (7)

Element Management System Database Tablespace is Out of Free Space—Database (8)

Urgent: Element Management System Database Archive Log Directory is Getting Full—Database (9)

Element Management System Database: Backup Fails—Database (10)

Element Management System Database Alert.log Alerts—Database (11)

Element Management System Database Process Died—Database (12)

Element Management System Database Performance Alert—Database (13)

Table Size Exceeds Minor Threshold Limit—Database (14)

Table Size Exceeds Major Threshold Limit—Database (15)

Table Size Exceeds Critical Threshold Limit—Database (16)

Data Replication Failed—Database (17)

Replication Data Flush Timeout During Switchover—Database (20)

Database Statistics Collection Exception—Database (21)

Unprovisioned Language—Database (22)

Element Management System Oracle Database-Minor Error—Database (23)

Element Management System Oracle Database-Major Error—Database (24)

Secure File Transfer Protocol Transfer Failed—Database (25)

File Write Error—Database (26)

Failure Setting the Index Table Soft Limit—Database (27)

Database Troubleshooting

---

Revised: December 2, 2008, OL-8723-17

Introduction

This chapter provides the information needed to monitor and troubleshoot database events and alarms. This chapter is divided into the following sections:

•Database Events and Alarms—Provides a brief overview of each database event and alarm.

•Monitoring Database Events—Provides the information needed to monitor and correct the database events.

•Troubleshooting Database Alarms—Provides the information needed to troubleshoot and correct the database alarms.

Database Events and Alarms

This section provides a brief overview of the database events and alarms for the Cisco BTS 10200 Softswitch in numerical order. Table 6-1 lists all of the database events and alarms by severity.

---

Note Click the database message number in Table 6-1 to display information about the event or alarm.

---

Table 6-1 Database Events and Alarms by Severity 

Critical	Major	Minor	Warning	Info	Not Used
DATABASE (3)	DATABASE (6)	DATABASE (7)	DATABASE (2)	DATABASE (1)
DATABASE (4)	DATABASE (8)	DATABASE (14)	DATABASE (11)	DATABASE (19)
DATABASE (5)	DATABASE (10)	DATABASE (21)	DATABASE (18)
DATABASE (9)	DATABASE (13)	DATABASE (23)
DATABASE (12)	DATABASE (15)
DATABASE (16)	DATABASE (17)
	DATABASE (20)
	DATABASE (22)
	DATABASE (24)
	DATABASE (25)
	DATABASE (26)
	DATABASE (27)

DATABASE (1)

For additional information, refer to the "Test Report—Database (1)" section.

DESCRIPTION	Test Report
SEVERITY	Information (INFO)
THRESHOLD	10000
THROTTLE	0

DATABASE (2)

To monitor and correct the cause of the event, refer to the "Database Management Update Failure: Master/Slave Database Out of Sync—Database (2)" section.

DESCRIPTION	Database Management Update Failure: Master/Slave Database Out of Sync (DBM Update Failure: Master/Slave Database Out of Sync)
SEVERITY	WARNING
THRESHOLD	100
THROTTLE	0
DATAWORDS	Error Code-TWO_BYTES Error String-STRING [20] Provisioning String-STRING [80]
PRIMARY CAUSE	The master database under Oracle control in the Element Management System (EMS) was successfully updated, but the subsequent update of the shared memory tables in the Call Agents (CAs) and/or Feature Servers (FSs) were not properly updated.
PRIMARY ACTION	Perform an audit of the database in question to correct the data stored in shared memory.
SECONDARY ACTION	Use command line interface (CLI) to show and delete the transaction queue, and to audit and manage the queue.

DATABASE (3)

To troubleshoot and correct the cause of the alarm, refer to the "There Are Errors in Element Management System Database DefError Queue; Contact Database Administrator—Database (3)" section.

DESCRIPTION	There Are Errors in Element Management System Database DefError Queue; Contact Database Administrator (There are Errors in EMS Database DefError Queue; Contact DBA)
SEVERITY	CRITICAL
THRESHOLD	100
THROTTLE	0
DATAWORDS	Host Name-STRING [30] Database Name-STRING [10] Error Count-ONE_BYTE Time Stamp-STRING [20]
PRIMARY CAUSE	Replication data conflicts have occurred.
PRIMARY ACTION	The replication data conflicts may require a manual update on the database tables. Call Cisco Support. (Contact Cisco Technical Assistance Center (TAC).)
SECONDARY CAUSE	An update or delete attempt on non-existing data has occurred.
TERNARY CAUSE	The unique constraint (primary key) has been violated.

---

Note Refer to the "Obtaining Documentation and Submitting a Service Request" section on page lvi for detailed instructions on contacting Cisco TAC and opening a service request.

---

Prior to contacting Cisco TAC, collect the following information:

On one EMS server:

su - oracle 

dbadm -C rep 

On both EMS servers:

nodestat

dbadm -r get_deferror

dbadm -r get_deferr

dbadm -r get_deftrandest

dbadm -r get_defcall_order

---

Note Do not perform an EM01 switchover until the deferrors are removed.

---

DATABASE (4)

To troubleshoot and correct the cause of the alarm, refer to the "Element Management System Database HeartBeat: Replication PUSH Job Broken—Database (4)" section.

DESCRIPTION	Element Management System Database HeartBeat: Replication PUSH Job Broken (EMS DB_Heart_Beat: Replication PUSH Job Broken)
SEVERITY	CRITICAL
THRESHOLD	100
THROTTLE	0
DATAWORDS	Host Name-STRING [30] Local Database-STRING [10] Remote Database-STRING [10] Job-STRING [5] Time Stamp-STRING [20]
PRIMARY CAUSE	The remote database is not accessible.
PRIMARY ACTION	Re-start up or restore the remote database.
SECONDARY CAUSE	The remote database is down.
SECONDARY ACTION	Restart the remote listener process.
TERNARY CAUSE	The remote Oracle listener process has died.
TERNARY ACTION	Correct the network connection problem.
SUBSEQUENT CAUSE	The network connection is broken.

DATABASE (5)

To troubleshoot and correct the cause of the alarm, refer to the "Element Management System Database HeartBeat Process Died—Database (5)" section.

DESCRIPTION	Element Management System Database HeartBeat Process Died (EMS DBHeartBeat Process Died)
SEVERITY	CRITICAL
THRESHOLD	100
THROTTLE	0
DATAWORDS	Host Name-STRING [30] Database Name-STRING [10] Time Stamp-STRING [20]
PRIMARY CAUSE	The DBHeartBeat process has been terminated by the system manager program (SMG) or stopped by the platform.
PRIMARY ACTION	Restart the DBHeartBeat process by entering the dbinit -H -i start command as an Oracle user, or by entering the platform start command as a root user.

DATABASE (6)

To troubleshoot and correct the cause of the alarm, refer to the "Element Management System Database Replication DefTranDest Queue Overloaded—Database (6)" section.

DESCRIPTION	Element Management System Database Replication DefTranDest Queue Overloaded (EMS Database Replication DefTranDest Queue Overloaded)
SEVERITY	MAJOR
THRESHOLD	100
THROTTLE	0
DATAWORDS	Host Name-STRING [30] Database Name-STRING [10] Threshold-FOUR_BYTES Time Stamp-STRING [20]
PRIMARY CAUSE	The replication PUSH job is broken.
PRIMARY ACTION	Correct problems on the remote database.
SECONDARY CAUSE	The remote database is not accessible.
SECONDARY ACTION	Make sure that the DBHeartBeat process is up.
TERNARY CAUSE	The database is overloaded.
TERNARY ACTION	Troubleshoot the database performance.

DATABASE (7)

To troubleshoot and correct the cause of the alarm, refer to the "Element Management System Database DefTran Queue is Overloaded—Database (7)" section.

DESCRIPTION	Element Management System Database DefTran Queue is Overloaded (EMS Database DefTran Queue is Overloaded)
SEVERITY	MINOR
THRESHOLD	100
THROTTLE	0
DATAWORDS	Host Name-STRING [30] Database Name-STRING [10] Threshold-TWO_BYTES Time Stamp-STRING [20]
PRIMARY CAUSE	The replication DefTranDest queue is overloaded.
PRIMARY ACTION	Resume the replication activities.
SECONDARY CAUSE	There are too many errors in the DefError queue.
SECONDARY ACTION	Correct the replication errors.
TERNARY CAUSE	The replication PURGE job is broken or overloaded.
TERNARY ACTION	Enable the replication PURGE job.

DATABASE (8)

To troubleshoot and correct the cause of the alarm, refer to the "Element Management System Database Tablespace is Out of Free Space—Database (8)" section.

DESCRIPTION	Element Management System Database Tablespace is Out of Free Space (EMS Database Tablespace is Out of Free Space)
SEVERITY	MAJOR
THRESHOLD	100
THROTTLE	0
DATAWORDS	Host Name-STRING [30] Database Name-STRING [10] Tablespace Name-STRING [30] Total Free Space-TWO_BYTES Time Stamp-STRING [20]
PRIMARY CAUSE	An increase data volume or transactions has occurred.
PRIMARY ACTION	Add more space to the tablespace.

DATABASE (9)

To troubleshoot and correct the cause of the alarm, refer to the "Urgent: Element Management System Database Archive Log Directory is Getting Full—Database (9)" section.

DESCRIPTION	Urgent: Element Management System Database Archive Log Directory is Getting Full (URGENT: EMS Database Archive Log Directory is Getting Full)
SEVERITY	CRITICAL
THRESHOLD	100
THROTTLE	0
DATAWORDS	Host Name-STRING [30] Database Name-STRING [10] Directory Name-STRING [100] Free Space-TWO_BYTES Time Stamp-STRING [20]
PRIMARY CAUSE	The transaction volume has increased.
PRIMARY ACTION	Backup and cleanup the archive log files.
SECONDARY ACTION	Add more space to the archive log directory.

DATABASE (10)

To troubleshoot and correct the cause of the alarm, refer to the "Element Management System Database: Backup Fails—Database (10)" section.

DESCRIPTION	Element Management System Database: Backup Fails (EMS Database: Backup Fails)
SEVERITY	MAJOR
THRESHOLD	100
THROTTLE	0
DATAWORDS	Host Name-STRING [30] Database Name-STRING [10] Message 1-STRING [200] Message 2-STRING [200] Time Stamp-STRING [20]
PRIMARY CAUSE	The system or hardware are unstable.
PRIMARY ACTION	Re-start backup process.

DATABASE (11)

To troubleshoot and correct the cause of the alarm, refer to the "Element Management System Database Alert.log Alerts—Database (11)" section.

DESCRIPTION	Element Management System Database Alert.log Alerts (EMS Database Alert.log Alerts)
SEVERITY	MAJOR
THRESHOLD	100
THROTTLE	0
DATAWORDS	Host Name-STRING [30] Database Name-STRING [10] Message 1-STRING [200] Message 2-STRING [200] Time Stamp-STRING [20]
PRIMARY CAUSE	The probable cause of the ORA- errors report in alert.log file is documented in the $ORACLE_HOME/rdbms/mesg/oraus.msg file. Login to EMS system as oracle user (or su-oracle) to view this file. If more information is needed, contact Cisco TAC for database support, or query Oracle metalink library at http://metalink.oracle.com.
PRIMARY ACTION	The corrective action is documented in the $ORACLE_HOME/rdbms/mesg/oraus.msg file. Login to EMS system as oracle user (or su-oracle) to view this file. If more information is needed, contact Cisco TAC for database support, or query Oracle Metalink library at http://metalink.oracle.com. The alert.log file is the global message file for errors issued by all Oracle background processes. The majority of error conditions may require administrator's investigation and manual correction.

---

Note Refer to the "Obtaining Documentation and Submitting a Service Request" section on page lvi for detailed instructions on contacting Cisco TAC and opening a service request.

---

DATABASE (12)

To troubleshoot and correct the cause of the alarm, refer to the "Element Management System Database Process Died—Database (12)" section.

DESCRIPTION	Element Management System Database Process Died (EMS Database Process Died)
SEVERITY	CRITICAL
THRESHOLD	100
THROTTLE	0
DATAWORDS	Host Name-STRING [30] Database Name-STRING [10] Error Source-STRING [40] Message-STRING [200] Time Stamp-STRING [20]
PRIMARY CAUSE	Possible Error Source: 1. Process Name, if the local process is not running. 2. Cannot_connect_database if the local database (DB) is unreachable. 3. Cannot_connect if the remote DB is unreachable.
PRIMARY ACTION	Re-start the process.
SECONDARY ACTION	Contact Cisco Support. (Contact Cisco TAC.)

---

Note Refer to the "Obtaining Documentation and Submitting a Service Request" section on page lvi for detailed instructions on contacting Cisco TAC and opening a service request.

---

DATABASE (13)

To troubleshoot and correct the cause of the alarm, refer to the "Element Management System Database Performance Alert—Database (13)" section.

DESCRIPTION	Element Management System Database Performance Alert (EMS Database Performance Alert)
SEVERITY	MAJOR
THRESHOLD	100
THROTTLE	0
DATAWORDS	Host Name-STRING [30] Database Name-STRING [10] Stat Event Name-STRING [80] Value 1-STRING [50] Value 2-FOUR_BYTES Message-STRING [200] Time Stamp-STRING [20]
PRIMARY CAUSE	See the Stat Event Name dataword.
PRIMARY ACTION	Contact Cisco Support. (Contact Cisco TAC.)
SECONDARY ACTION	Perform a database performance tuning.

---

Note Refer to the "Obtaining Documentation and Submitting a Service Request" section on page lvi for detailed instructions on contacting Cisco TAC and opening a service request.

---

DATABASE (14)

To troubleshoot and correct the cause of the alarm, refer to the "Table Size Exceeds Minor Threshold Limit—Database (14)" section.

DESCRIPTION	Table Size Exceeds Minor Threshold Limit
SEVERITY	MINOR
THRESHOLD	100
THROTTLE	0
DATAWORDS	Table Name-STRING [32]
PRIMARY CAUSE	The pre-provisioned size for the stated table is nearing the licensed limit on the number of entries it can hold.
PRIMARY ACTION	Contact Cisco to purchase additional entry space for this particular table. (Contact Cisco TAC.)

---

Note Refer to the "Obtaining Documentation and Submitting a Service Request" section on page lvi for detailed instructions on contacting Cisco TAC and opening a service request.

---

Prior to contacting Cisco TAC, collect the following information:

show db-usage table_name=<string>

DATABASE (15)

To troubleshoot and correct the cause of the alarm, refer to the "Table Size Exceeds Major Threshold Limit—Database (15)" section.

DESCRIPTION	Table Size Exceeds Major Threshold Limit
SEVERITY	MAJOR
THRESHOLD	100
THROTTLE	0
DATAWORDS	Table Name-STRING [32]
PRIMARY CAUSE	The major threshold limit has been exceeded.
PRIMARY ACTION	N/A

DATABASE (16)

To troubleshoot and correct the cause of the alarm, refer to the "Table Size Exceeds Critical Threshold Limit—Database (16)" section.

DESCRIPTION	Table Size Exceeds Critical Threshold Limit
SEVERITY	CRITICAL
THRESHOLD	100
THROTTLE	0
DATAWORDS	Table Name-STRING [32]
PRIMARY CAUSE	The critical threshold limit has been exceeded.
PRIMARY ACTION	N/A

DATABASE (17)

To troubleshoot and correct the cause of the alarm, refer to the "Data Replication Failed—Database (17)" section.

DESCRIPTION	Data Replication Failed
SEVERITY	MAJOR
THRESHOLD	100
THROTTLE	0
DATAWORDS	Replication-Stage-STRING [40] Table Name-STRING [40] Index-FOUR_BYTES Table ID-TWO_BYTES
PRIMARY CAUSE	The index size is out of range.
SECONDARY CAUSE	Record the index size mismatch.

DATABASE (18)

To monitor and correct the cause of the event, refer to the "Unexpected Runtime Data Interaction—Database (18)" section.

DESCRIPTION	Unexpected Runtime Data Interaction
SEVERITY	WARNING
THRESHOLD	100
THROTTLE	0
DATAWORDS	Internal/External In-STRING [10] Table Name-STRING [32] Table Entry-STRING [10] Table Field Name-STRING [32] Descriptive Data 1-STRING [64] Descriptive Data 2-STRING [64] Descriptive Data 3-STRING [64] Descriptive Data 4-STRING [64]
PRIMARY CAUSE	An unexpected data interaction has been detected at the runtime in the call agent or the feature server.
PRIMARY ACTION	Collect the logs and contact Cisco Support. (Contact Cisco TAC.)

---

Note Refer to the "Obtaining Documentation and Submitting a Service Request" section on page lvi for detailed instructions on contacting Cisco TAC and opening a service request.

---

DATABASE (19)

For additional information, refer to the "Daily Database Backup Completed Successfully—Database (19)" section.

DESCRIPTION	Daily Database Backup Completed Successfully
SEVERITY	INFO
THRESHOLD	0
THROTTLE	0
DATAWORDS	Host Name-STRING [60] ORACLE_SID-STRING [30] Process-STRING [60] Message 1-STRING [100] Message 2-STRING [100] Message 3-STRING [100]
PRIMARY CAUSE	Normal operation.
PRIMARY ACTION	N/A

DATABASE (20)

To troubleshoot and correct the cause of the alarm, refer to the "Replication Data Flush Timeout During Switchover—Database (20)" section.

DESCRIPTION	Replication Data Flush Timeout During Switchover
SEVERITY	MAJOR
THRESHOLD	100
THROTTLE	0
DATAWORDS	Tables Failed-STRING [20]
PRIMARY CAUSE	An replication module software problem has occurred.
PRIMARY ACTION	The database restore procedure needs to be executed on the side of the system which goes active after the switchover. Any alarms should be cleared manually after the recovery action is taken.

DATABASE (21)

To troubleshoot and correct the cause of the alarm, refer to the "Database Statistics Collection Exception—Database (21)" section.

DESCRIPTION	Database Statistics Collection Exception (DB Statistics Collection Exception)
SEVERITY	MINOR
THRESHOLD	100
THROTTLE	0
DATAWORDS	Host Name-STRING [30] Database Name-STRING [10] Schema Name-STRING [32] Object Name-STRING [64] Task Name-STRING [64] Exception-STRING [256]
PRIMARY CAUSE	Check the messages in the Exception dataword field to identify the cause of the error.
PRIMARY ACTION	The correction action varies and is determined by the type of exception. For more information about the ORA-xxxxx errors, execute the oerr ora xxxxx command as an Oracle user.

DATABASE (22)

To troubleshoot and correct the cause of the alarm, refer to the "Unprovisioned Language—Database (22)" section.

DESCRIPTION	Unprovisioned Language
SEVERITY	MAJOR
THRESHOLD	100
THROTTLE	0
DATAWORDS	Language ID-STRING [4]
PRIMARY CAUSE	The operator has missed provisioning the language for this alarm in the language table on the EMS.
PRIMARY ACTION	The operator has to provision the missing language and update the LANGUAGE table.

DATABASE (23)

To troubleshoot and correct the cause of the alarm, refer to the "Element Management System Oracle Database-Minor Error—Database (23)" section.

DESCRIPTION	Element Management System Oracle Database-Minor Error (EMS Oracle Database (ORA)-Minor Error)
SEVERITY	MINOR
THRESHOLD	100
THROTTLE	0
DATAWORDS	HostName-STRING [30] DatabaseName-STRING [10] Message1-STRING [200] Message2-STRING [200] TimeStamp-STRING [20]
PRIMARY CAUSE	This ORA error is issued by an Oracle background process. The probable cause of the ORA- errors is documented in the $ORACLE_HOME/rdbms/mesg/oraus.msg file. To view the file, login to EMS system as root, then su-oracle. If more information is needed, contact Cisco TAC for database support, or query Oracle Metalink library at http://metalink.oracle.com.
PRIMARY ACTION	The corrective action is documented in the $ORACLE_HOME/rdbms/mesg/oraus.msg file. To view the file, login to EMS system as root, then su-oracle. If more information is needed, contact Cisco TAC for database support, or query Oracle Metalink library at http://metalink.oracle.com. Many ORA- errors may need administrator to investigate and resolve the problem. When the problem is resolved, this alarm should be manually cleared by the operator.

---

Note Refer to the "Obtaining Documentation and Submitting a Service Request" section on page lvi for detailed instructions on contacting Cisco TAC and opening a service request.

---

DATABASE (24)

To troubleshoot and correct the cause of the alarm, refer to the "Element Management System Oracle Database-Major Error—Database (24)" section.

DESCRIPTION	Element Management System Oracle Database-Minor Error (EMS Oracle Database (ORA)-Major Error)
SEVERITY	MAJOR
THRESHOLD	100
THROTTLE	0
DATAWORDS	HostName-STRING [30] DatabaseName-STRING [10] Message1-STRING [200] Message2-STRING [200] TimeStamp-STRING [20]
PRIMARY CAUSE	This ORA error is issued by an Oracle background process. The probable cause of the ORA- errors is documented in the $ORACLE_HOME/rdbms/mesg/oraus.msg file. To view the file, login to EMS system as root, then su-oracle. If more information is needed, contact Cisco TAC for database support, or query Oracle Metalink library at http://metalink.oracle.com.
PRIMARY ACTION	The corrective action is documented in the $ORACLE_HOME/rdbms/mesg/oraus.msg file. To view the file, login to EMS system as root, then su-oracle. If more information is needed, contact Cisco TAC for database support, or query Oracle Metalink library at http://metalink.oracle.com. Many ORA- errors may need administrator to investigate and resolve the problem. When the problem is resolved, this alarm should be manually cleared by the operator.

---

Note Refer to the "Obtaining Documentation and Submitting a Service Request" section on page lvi for detailed instructions on contacting Cisco TAC and opening a service request.

---

DATABASE (25)

To troubleshoot and correct the cause of the alarm, refer to the "Secure File Transfer Protocol Transfer Failed—Database (25)" section.

DESCRIPTION	Secure File Transfer Protocol Transfer Failed (SFTP Transfer Failed)
SEVERITY	MAJOR
THRESHOLD	100
THROTTLE	0
DATAWORDS	FileName - STRING [128] Error - STRING [50]
PRIMARY CAUSE	Unable to connect between active and standby call agents.
PRIMARY ACTION	Verify communication between primary and CA. On each CA, ping the other node.
SECONDARY CAUSE	Unable to login to remote host.
SECONDARY ACTION	Verify that secure shell (SSH) keys have been pre-configured for user root on both active and standby call agents.
TERNARY CAUSE	File transfer error.
TERNARY ACTION	Check the Error dataword to see if it gives an indication of the kind of error that occurred. It could be a file-system error on the remote host, or a communication failure between the active and standby call agents.

DATABASE (26)

To troubleshoot and correct the cause of the alarm, refer to the "File Write Error—Database (26)" section.

DESCRIPTION	File Write Error
SEVERITY	MAJOR
THRESHOLD	100
THROTTLE	0
DATAWORDS	Path Name - STRING [128]
PRIMARY CAUSE	System error, may be out of file descriptors.
PRIMARY ACTION	Call Cisco TAC technical support. (Contact Cisco TAC.)

---

Note Refer to the "Obtaining Documentation and Submitting a Service Request" section on page lvi for detailed instructions on contacting Cisco TAC and opening a service request.

---

DATABASE (27)

To troubleshoot and correct the cause of the alarm, refer to the "Failure Setting the Index Table Soft Limit—Database (27)" section.

DESCRIPTION	Failure Setting the Index Table Soft Limit (Failure Setting the IDX Table Soft Limit)
SEVERITY	MAJOR
THRESHOLD	100
THROTTLE	0
DATAWORDS	Reason - STRING [200]
PRIMARY CAUSE	A corruption of the IDX framework for the table has occurred.
PRIMARY ACTION	Running the tiat command will verify if there is corruption. Fix the corruption

Monitoring Database Events

This section provides the information needed to monitor and correct database events. Table 6-2 lists all of the database events in numerical order and provides cross reference to each subsection in this section.

Table 6-2 BTS 10200 Database Events 

Event Type	Event Name	Event Severity
DATABASE(1)	Test Report—Database (1)	INFO
DATABASE(2)	Database Management Update Failure: Master/Slave Database Out of Sync—Database (2)	WARNING
DATABASE(3)	There are Errors in Element Management System Database DefError Queue; Contact Database Administrator—Database (3)	CRITICAL
DATABASE(4)	Element Management System Database HeartBeat: Replication PUSH Job Broken—Database (4)	CRITICAL
DATABASE(5)	Element Management System Database HeartBeat Process Died—Database (5)	CRITICAL
DATABASE(6)	Element Management System Database Replication DefTranDest Queue Overloaded—Database (6)	MAJOR
DATABASE(7)	Element Management System Database DefTran Queue is Overloaded—Database (7)	MINOR
DATABASE(8)	Element Management System Database Tablespace is Out of Free Space—Database (8)	MAJOR
DATABASE(9)	Urgent: Element Management System Database Archive Log Directory is Getting Full—Database (9)	CRITICAL
DATABASE(10)	Element Management System Database: Backup Fails—Database (10)	MAJOR
DATABASE(11)	Element Management System Database Alert.log Alerts—Database (11)	MAJOR
DATABASE(12)	Element Management System Database Process Died—Database (12)	CRITICAL
DATABASE(13)	Element Management System Database Performance Alert—Database (13)	MAJOR
DATABASE(14)	Table Size Exceeds Minor Threshold Limit—Database (14)	MINOR
DATABASE(15)	Table Size Exceeds Major Threshold Limit—Database (15)	MAJOR
DATABASE(16)	Table Size Exceeds Critical Threshold Limit—Database (16)	CRITICAL
DATABASE(17)	Data Replication Failed—Database (17)	MAJOR
DATABASE(18)	Unexpected Runtime Data Interaction—Database (18)	WARNING
DATABASE(19)	Daily Database Backup Completed Successfully—Database (19)	INFO
DATABASE(20)	Replication Data Flush Timeout During Switchover—Database (20)	MAJOR
DATABASE(21)	Database Statistics Collection Exception—Database (21)	MINOR
DATABASE(22)	Unprovisioned Language—Database (22)	MAJOR
DATABASE(23)	Element Management System Oracle Database-Minor Error—Database (23)	MINOR
DATABASE(24)	Element Management System Oracle Database-Major Error—Database (24)	MAJOR
DATABASE(25)	Secure File Transfer Protocol Transfer Failed—Database (25)	MAJOR
DATABASE(26)	File Write Error—Database (26)	MAJOR
DATABASE(27)	Failure Setting the Index Table Soft Limit—Database (27)	MAJOR

Test Report—Database (1)

The Test Report is for testing the database event category. The event is informational and no further action is required.

Database Management Update Failure: Master/Slave Database Out of Sync—Database (2)

The Database Management Update Failure: Master/Slave Database Out of Sync event functions as a warning that master and slave databases are out of sync. The primary cause of the event is that the master database under Oracle control in the EMS was successfully updated, but the subsequent update of the shared memory tables in the Call Agent (CA) servers and/or Feature Servers (FS) were not properly updated. To correct the primary cause of the event, perform an audit of the database in question to correct the data stored in shared memory. Additionally, use the CLI to show and delete the transaction queue, and to audit and manage the queue.

There are Errors in Element Management System Database DefError Queue; Contact Database Administrator—Database (3)

The There are Errors in Element Management System Database DefError Queue; Contact Database Administrator alarm (critical) indicates that there are errors in the EMS database DefError queue. To troubleshoot and correct the cause of the There are Errors in Element Management System Database DefError Queue; Contact Database Administrator alarm, refer to the "There Are Errors in Element Management System Database DefError Queue; Contact Database Administrator—Database (3)" section.

Element Management System Database HeartBeat: Replication PUSH Job Broken—Database (4)

The Element Management System Database HeartBeat: Replication PUSH Job Broken alarm (critical) indicates that the replication PUSH job is broken. To troubleshoot and correct the cause of the of the Element Management System Database HeartBeat: Replication PUSH Job Broken alarm, refer to the "Element Management System Database HeartBeat: Replication PUSH Job Broken—Database (4)" section.

Element Management System Database HeartBeat Process Died—Database (5)

The Element Management System Database HeartBeat Process Died alarm (critical) indicates that the EMS database heartbeat process has died. To troubleshoot and correct the cause of the Element Management System Database HeartBeat Process Died alarm, refer to the "Element Management System Database HeartBeat Process Died—Database (5)" section.

Element Management System Database Replication DefTranDest Queue Overloaded—Database (6)

The Element Management System Database Replication DefTranDest Queue Overloaded alarm (major) indicates that the EMS database replication DefTranDest queue is overloaded. To troubleshoot and correct the cause of the Element Management System Database Replication DefTranDest Queue Overloaded alarm, refer to the "Element Management System Database Replication DefTranDest Queue Overloaded—Database (6)" section.

Element Management System Database DefTran Queue is Overloaded—Database (7)

The Element Management System Database DefTran Queue is Overloaded alarm (minor) indicates that the EMS database DefTran queue is overloaded. To troubleshoot and correct the cause of the Element Management System Database DefTran Queue is Overloaded alarm, refer to the "Element Management System Database DefTran Queue is Overloaded—Database (7)" section.

Element Management System Database Tablespace is Out of Free Space—Database (8)

The Element Management System Database Tablespace is Out of Free Space alarm (major) indicates that the EMS database table space is out of free space. To troubleshoot and correct the cause of the Element Management System Database Tablespace is Out of Free Space alarm, refer to the "Element Management System Database Tablespace is Out of Free Space—Database (8)" section.

Urgent: Element Management System Database Archive Log Directory is Getting Full—Database (9)

The Urgent: Element Management System Database Archive Log Directory is Getting Full alarm (critical) indicates that the EMS database archive log directory is getting full. To troubleshoot and correct the cause of the Urgent: Element Management System Database Archive Log Directory is Getting Full alarm, refer to the "Urgent: Element Management System Database Archive Log Directory is Getting Full—Database (9)" section.

Element Management System Database: Backup Fails—Database (10)

The Element Management System Database: Backup Fails alarm (major) indicates that the EMS database backup has failed. To troubleshoot and correct the cause of the Element Management System Database: Backup Fails alarm, refer to the "Element Management System Database: Backup Fails—Database (10)" section.

Element Management System Database Alert.log Alerts—Database (11)

The Element Management System Database Alert.log Alerts alarm (major) indicates that the EMS database alerts are being received and logged into the alert log. To troubleshoot and correct the cause of the Element Management System Database Alert.log Alerts alarm, refer to the "Element Management System Database Alert.log Alerts—Database (11)" section.

Element Management System Database Process Died—Database (12)

The Element Management System Database Process Died alarm (critical) indicates that the EMS database process has died. To troubleshoot and correct the cause of the Element Management System Database Process Died alarm, refer to the "Element Management System Database Process Died—Database (12)" section.

Element Management System Database Performance Alert—Database (13)

The Element Management System Database Performance Alert alarm (major) indicates that the EMS database performance has degraded. To troubleshoot and correct the cause of the Element Management System Database Performance Alert alarm, refer to the "Element Management System Database Performance Alert—Database (13)" section.

Table Size Exceeds Minor Threshold Limit—Database (14)

The Table Size Exceeds Minor Threshold Limit alarm (minor) indicates that the table size has exceeded the minor threshold crossing limit. To troubleshoot and correct the cause of the Table Size Exceeds Minor Threshold Limit alarm, refer to the "Table Size Exceeds Minor Threshold Limit—Database (14)" section.

Table Size Exceeds Major Threshold Limit—Database (15)

The Table Size Exceeds Major Threshold Limit alarm (major) indicates that the table size has exceeded the major threshold crossing limit. To troubleshoot and correct the cause of the Table Size Exceeds Major Threshold Limit alarm, refer to the "Table Size Exceeds Major Threshold Limit—Database (15)" section.

Table Size Exceeds Critical Threshold Limit—Database (16)

The Table Size Exceeds Critical Threshold Limit alarm (critical) indicates that the table size has exceeded the critical threshold crossing limit. To troubleshoot and correct the cause of the Table Size Exceeds Critical Threshold Limit alarm, refer to the "Table Size Exceeds Critical Threshold Limit—Database (16)" section.

Data Replication Failed—Database (17)

The Data Replication Failed alarm (major) indicates that the data replication failed. To troubleshoot and correct the cause of the Data Replication Failed alarm, refer to the "Data Replication Failed—Database (17)" section.

Unexpected Runtime Data Interaction—Database (18)

The Unexpected Runtime Data Interaction event functions as a warning that an unexpected runtime data interaction has occurred. The primary cause of the event is that an unexpected data interaction has been detected at runtime in the call agent or feature server. To correct the primary cause of the event, collect the logs and contact Cisco TAC.

---

Note Refer to the "Obtaining Documentation and Submitting a Service Request" section on page lvi for detailed instructions on contacting Cisco TAC and opening a service request.

---

Daily Database Backup Completed Successfully—Database (19)

The Daily Database Backup Completed Successfully event functions as an informational alert that the daily database backup has completed successfully. The event is informational and no further action is required.

Replication Data Flush Timeout During Switchover—Database (20)

The Replication Data Flush Timeout During Switchover alarm (major) indicates that the replication data flush timed out during a switchover. To troubleshoot and correct the cause of the Replication Data Flush Timeout During Switchover alarm, refer to the "Replication Data Flush Timeout During Switchover—Database (20)" section.

Database Statistics Collection Exception—Database (21)

The Database Statistics Collection Exception alarm (minor) indicates that the database statistics collection process had an exception. To troubleshoot and correct the cause of the Database Statistics Collection Exception alarm, refer to the "Database Statistics Collection Exception—Database (21)" section.

Unprovisioned Language—Database (22)

The Unprovisioned Language alarm (major) indicates that the operator has missed provisioning the language in this alarm in the language table on the EMS. To troubleshoot and correct the cause of the Unprovisioned Language alarm, refer to the "Unprovisioned Language—Database (22)" section.

Element Management System Oracle Database-Minor Error—Database (23)

The Element Management System Oracle Database-Minor Error alarm (minor) indicates that a minor error has occurred in an Oracle background process. To troubleshoot and correct the cause of the Element Management System Oracle Database-Minor Error alarm, refer to the "Element Management System Oracle Database-Minor Error—Database (23)" section.

Element Management System Oracle Database-Major Error—Database (24)

The Element Management System Oracle Database-Major Error alarm (major) indicates that a major error has occurred in an Oracle background process. To troubleshoot and correct the cause of the Element Management System Oracle Database-Major Error alarm, refer to the "Element Management System Oracle Database-Major Error—Database (24)" section.

Secure File Transfer Protocol Transfer Failed—Database (25)

The Secure File Transfer Protocol Transfer Failed alarm (alarm) indicates that a SFTP file transfer had failed. To troubleshoot and correct the cause of the Secure File Transfer Protocol Transfer Failed alarm, refer to the "Secure File Transfer Protocol Transfer Failed—Database (25)" section.

File Write Error—Database (26)

The File Write Error alarm (major) indicates that a file write error has occurred. To troubleshoot and correct the cause of the File Write Error alarm, refer to the "File Write Error—Database (26)" section.

Failure Setting the Index Table Soft Limit—Database (27)

The Failure Setting the Index Table Soft Limit alarm (major) indicates that a corruption of the IDX framework for the table has occurred.