Guest

Cisco BTS 10200 Softswitch

Field Notice: BTS 10200 Release 3.x Generating Security Deferrors


April 28, 2005


Products Affected

Product

Comments

BTS Other - 3.X

Patch targeted for R3.5.4v01P15 on April 22

Problem Description

Insert/update transaction on both primary and secondary at the same time are causing deferrors in Oracle on both active and standby Element Management Systems (EMS's).

Background

The USM process audits the users in the system once a day for activity. In the course of this audit, the process updates the database. This update happens without regard to whether the EM01 is ACTIVE or STANDBY. The system should check for the local state prior to issuing the audit, but since it currently does not, the audit updates are erroneously made on both sides.

DEFERR's are alarmed notifications from the EMS database indicating that a database transaction in the update queue could not be applied and that further investigation is warranted.

Problem Symptoms

Database 3 alarm There are errors in EMS database DefError queue; contact DBA. There are no negative affects when you see this alarm. You must execute the workaround to clear the alarms.

Workaround/Solution

In Oracle there is a tool called dbadm which will be used for this procedure. When executing the procedure, you will have output generated for specific Oracle queues as depicted below. The dbadm tool should never be used to truncate anything other than the deferror queue. The command dbadm -A truncate_def should never be used on a live system as this command truncates not only the deferror queue, but also the deftran and deftrandest queues.

On the standby EMS:

su - oracle 
dbadm -C rep 

optical2:test-ems-2:/opt/orahome$ dbadm -C rep 

OPTICAL2::Deftrandest is empty? YES 
OPTICAL2::dba_repcatlog is empty? YES 
OPTICAL2::Deferror is empty? NO 
OPTICAL2::Deftran is empty? NO 
OPTICAL2::Has no broken job? YES 
OPTICAL2::JQ Lock is empty? YES 
OPTICAL1::Deftrandest is empty? YES 
OPTICAL1::dba_repcatlog is empty? YES 
OPTICAL1::Deferror is empty? YES 
OPTICAL1::Deftran is empty? YES 
OPTICAL1::Has no broken job? YES 
OPTICAL1::JQ Lock is empty? YES 

Will define these queues as the following:

Deferror - Contains the ID of each transaction that could not be applied. You can use this ID to locate the queued calls associated with this transaction. These calls are stored in the DEFCALL view. In another words, deferror is used for storing the ID of each replication transaction that could not be applied because of data conflict. This queue should always be empty.

Deftran - Is used for recording all deferred transactions in the deferred transactions queue at the current BTS system. It is normal to see that this queue is not empty.

Deftrandest - Lists the destinations for each deferred transaction in the deferred transactions queue at the current BTS system. It is normal to see that this queue is not empty.

Dba_repcatlog - This is an internal debugging queue and should be empty. If it is not, then Cisco needs to be called.

Has no broken job - This is an internal debugging queue and should be empty. If it is not, then Cisco needs to be called.

JQ Lock - This is an internal debugging queue and should be empty. If it is not, then Cisco needs to be called.

Example

On the standby EMS:

su - oracle 
dbadm -C rep 

optical2:test-ems-2:/opt/orahome$ dbadm -C rep 

OPTICAL2::Deftrandest is empty? YES 
OPTICAL2::dba_repcatlog is empty? YES 
OPTICAL2::Deferror is empty? NO 
OPTICAL2::Deftran is empty? NO 
OPTICAL2::Has no broken job? YES 
OPTICAL2::JQ Lock is empty? YES 
OPTICAL1::Deftrandest is empty? YES 
OPTICAL1::dba_repcatlog is empty? YES 
OPTICAL1::Deferror is empty? YES 
OPTICAL1::Deftran is empty? YES 
OPTICAL1::Has no broken job? YES 
OPTICAL1::JQ Lock is empty? YES

Notice the deferrors present

Go to the side that has the errors, in this case optical2/standby ems

Do the following on the side with deferrors:

su - oracle 
optical2:test-ems-2:/opt/oracle/admin/rep$ dbadm -r get_deferror 

Connected. 

================================ 
List of replication error transactions with start_time 
================================ 

TRAN_ID FROM_DB START_TIME ERROR_MSG 

--------- -------- ---------------- ----------------------- 

6.95.2954 OPTICAL1 04Nov24 14:44:12 ORA-01403: no data found 
4.29.671 OPTICAL1 04Nov24 15:13:44 ORA-01403: no data found 
2.64.671 OPTICAL1 04Nov24 15:13:49 ORA-01403: no data found

This gives me the transaction ID.

Do the following and look for that transaction ID:

optical2:test-ems-2:/opt/oracle/admin/rep$ dbadm -r get_defcall_order 

Connected. 

=============================== 
Replication transaction operations (calls) in the order of start_time 
=============================== 

CALLNO TRAN_ID PACKAGENAME PROCNAME START_TIME 

------- ---------- ------------------------- --------------- ---------- 

0 6.95.2954 SECURITYLEVELS$RP REP_UPDATE 004Nov24 14:44:12 

0 4.29.671 EVENT_LOG$RP REP_DELETE 004Nov24 15:13:44 

0 2.64.671 EVENT_LOG$RP REP_DELETE 004Nov24 15:13:49

These transactions are for events and security table. Therefore, you can truncate deferror with the following commands. Do not truncate this queue if something other than alarms, events or security level transactions are present. If the deferror is not for a event, alarm or security level table, stop and open a case with Cisco.

Login to the side that is reporting deferrors, in this case OPTICAL2(side B)

su - oracle 
dbadm -A truncate_deferror 

optical2:secems05:/opt/orahome$ dbadm -A truncate_deferror 

*********************************** 
You are about to execute the following process: 

==> Truncate replication Deferred Error Queue tables. 
database: optical2 
hostname: secems05 
*********************************** 
Do you want to continue? [y/n] y 

INFO: Truncating replication deferred error queue tables... 

INFO: Truncating table SYSTEM.DEF$_ERROR 

INFO: Truncating table SYSTEM.DEF$_AQERROR 

INFO: Deferred error queue tables are truncated. 

Once the deferrors are truncated, issue the following commands to check the queue:

optical2:secems05:/opt/orahome$ dbadm -C rep 

OPTICAL2::Deftrandest is empty? YES 
OPTICAL2::dba_repcatlog is empty? YES 
OPTICAL2::Deferror is empty? YES 
OPTICAL2::Deftran is empty? YES 
OPTICAL2::Has no broken job? YES 
OPTICAL2::JQ Lock is empty? YES 
OPTICAL1::Deftrandest is empty? YES 
OPTICAL1::dba_repcatlog is empty? YES 
OPTICAL1::Deferror is empty? YES 
OPTICAL1::Deftran is empty? YES 
OPTICAL1::Has no broken job? YES 
OPTICAL1::JQ Lock is empty? YES

This issue is being tracked by CSCsa61405 (registered customers only) and it is being packaged into R3.5.4v01P15 with a targeted release date of April 22, 2005. The patch will be published out on CCO at Downloads - Voice Software.

DDTS

To follow the bug ID link below and see detailed bug information, you must be a registered user and you must be logged in.

DDTS

Description

CSCsa61405 (registered customers only)

SECURITYLEVEL TABLE CAUSING DEFERRORS

For More Information

If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:

Receive Email Notification For New Field Notices

Product Alert Tool - Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.