April 28, 2005
Products Affected
|
Product |
Comments |
|---|---|
|
BTS Other - 3.X |
Patch targeted for R3.5.4v01P15 on April 22 |
Problem Description
Insert/update transaction on both primary and secondary at the same time are causing deferrors in Oracle on both active and standby Element Management Systems (EMS's).
Background
The USM process audits the users in the system once a day for activity. In the course of this audit, the process updates the database. This update happens without regard to whether the EM01 is ACTIVE or STANDBY. The system should check for the local state prior to issuing the audit, but since it currently does not, the audit updates are erroneously made on both sides.
DEFERR's are alarmed notifications from the EMS database indicating that a database transaction in the update queue could not be applied and that further investigation is warranted.
Problem Symptoms
Database 3 alarm There are errors in EMS database DefError queue; contact DBA. There are no negative affects when you see this alarm. You must execute the workaround to clear the alarms.
Workaround/Solution
In Oracle there is a tool called dbadm which will be used for this procedure. When executing the procedure, you will have output generated for specific Oracle queues as depicted below. The dbadm tool should never be used to truncate anything other than the deferror queue. The command dbadm -A truncate_def should never be used on a live system as this command truncates not only the deferror queue, but also the deftran and deftrandest queues.
On the standby EMS:
su - oracle dbadm -C rep optical2:test-ems-2:/opt/orahome$ dbadm -C rep OPTICAL2::Deftrandest is empty? YES OPTICAL2::dba_repcatlog is empty? YES OPTICAL2::Deferror is empty? NO OPTICAL2::Deftran is empty? NO OPTICAL2::Has no broken job? YES OPTICAL2::JQ Lock is empty? YES OPTICAL1::Deftrandest is empty? YES OPTICAL1::dba_repcatlog is empty? YES OPTICAL1::Deferror is empty? YES OPTICAL1::Deftran is empty? YES OPTICAL1::Has no broken job? YES OPTICAL1::JQ Lock is empty? YES
Will define these queues as the following:
Deferror - Contains the ID of each transaction that could not be applied. You can use this ID to locate the queued calls associated with this transaction. These calls are stored in the DEFCALL view. In another words, deferror is used for storing the ID of each replication transaction that could not be applied because of data conflict. This queue should always be empty.
Deftran - Is used for recording all deferred transactions in the deferred transactions queue at the current BTS system. It is normal to see that this queue is not empty.
Deftrandest - Lists the destinations for each deferred transaction in the deferred transactions queue at the current BTS system. It is normal to see that this queue is not empty.
Dba_repcatlog - This is an internal debugging queue and should be empty. If it is not, then Cisco needs to be called.
Has no broken job - This is an internal debugging queue and should be empty. If it is not, then Cisco needs to be called.
JQ Lock - This is an internal debugging queue and should be empty. If it is not, then Cisco needs to be called.
Example
On the standby EMS:
su - oracle dbadm -C rep optical2:test-ems-2:/opt/orahome$ dbadm -C rep OPTICAL2::Deftrandest is empty? YES OPTICAL2::dba_repcatlog is empty? YES OPTICAL2::Deferror is empty? NO OPTICAL2::Deftran is empty? NO OPTICAL2::Has no broken job? YES OPTICAL2::JQ Lock is empty? YES OPTICAL1::Deftrandest is empty? YES OPTICAL1::dba_repcatlog is empty? YES OPTICAL1::Deferror is empty? YES OPTICAL1::Deftran is empty? YES OPTICAL1::Has no broken job? YES OPTICAL1::JQ Lock is empty? YES
Notice the deferrors present
Go to the side that has the errors, in this case optical2/standby ems
Do the following on the side with deferrors:
su - oracle optical2:test-ems-2:/opt/oracle/admin/rep$ dbadm -r get_deferror Connected. ================================ List of replication error transactions with start_time ================================ TRAN_ID FROM_DB START_TIME ERROR_MSG --------- -------- ---------------- ----------------------- 6.95.2954 OPTICAL1 04Nov24 14:44:12 ORA-01403: no data found 4.29.671 OPTICAL1 04Nov24 15:13:44 ORA-01403: no data found 2.64.671 OPTICAL1 04Nov24 15:13:49 ORA-01403: no data found
This gives me the transaction ID.
Do the following and look for that transaction ID:
optical2:test-ems-2:/opt/oracle/admin/rep$ dbadm -r get_defcall_order Connected. =============================== Replication transaction operations (calls) in the order of start_time =============================== CALLNO TRAN_ID PACKAGENAME PROCNAME START_TIME ------- ---------- ------------------------- --------------- ---------- 0 6.95.2954 SECURITYLEVELS$RP REP_UPDATE 004Nov24 14:44:12 0 4.29.671 EVENT_LOG$RP REP_DELETE 004Nov24 15:13:44 0 2.64.671 EVENT_LOG$RP REP_DELETE 004Nov24 15:13:49
These transactions are for events and security table. Therefore, you can truncate deferror with the following commands. Do not truncate this queue if something other than alarms, events or security level transactions are present. If the deferror is not for a event, alarm or security level table, stop and open a case with Cisco.
Login to the side that is reporting deferrors, in this case OPTICAL2(side B)
su - oracle dbadm -A truncate_deferror optical2:secems05:/opt/orahome$ dbadm -A truncate_deferror *********************************** You are about to execute the following process: ==> Truncate replication Deferred Error Queue tables. database: optical2 hostname: secems05 *********************************** Do you want to continue? [y/n] y INFO: Truncating replication deferred error queue tables... INFO: Truncating table SYSTEM.DEF$_ERROR INFO: Truncating table SYSTEM.DEF$_AQERROR INFO: Deferred error queue tables are truncated.
Once the deferrors are truncated, issue the following commands to check the queue:
optical2:secems05:/opt/orahome$ dbadm -C rep OPTICAL2::Deftrandest is empty? YES OPTICAL2::dba_repcatlog is empty? YES OPTICAL2::Deferror is empty? YES OPTICAL2::Deftran is empty? YES OPTICAL2::Has no broken job? YES OPTICAL2::JQ Lock is empty? YES OPTICAL1::Deftrandest is empty? YES OPTICAL1::dba_repcatlog is empty? YES OPTICAL1::Deferror is empty? YES OPTICAL1::Deftran is empty? YES OPTICAL1::Has no broken job? YES OPTICAL1::JQ Lock is empty? YES
This issue is being tracked by CSCsa61405 (registered customers only) and it is being packaged into R3.5.4v01P15 with a targeted release date of April 22, 2005. The patch will be published out on CCO at Downloads - Voice Software.
DDTS
To follow the bug ID link below and see detailed bug information, you must be a registered user and you must be logged in.
|
DDTS |
Description |
|---|---|
|
CSCsa61405 (registered customers only) |
SECURITYLEVEL TABLE CAUSING DEFERRORS |
For More Information
If you require further assistance, or if you have any further questions regarding this field notice, please contact the Cisco Systems Technical Assistance Center (TAC) by one of the following methods:
Receive Email Notification For New Field Notices
Product Alert Tool - Set up a profile to receive email updates about reliability, safety, network security, and end-of-sale issues for the Cisco products you specify.