Troubleshooting Guide
Preface
Downloads: This chapterpdf (PDF - 485.0KB) The complete bookPDF (PDF - 9.12MB) | Feedback

Preface

Table Of Contents

Preface

Introduction

Objective

Audience

Organization

Conventions

Updates to this Document for Release 4.5.x

New Events and Alarms (Release 3.5 to Release 4.5.1)

Modified Events and Alarms (Release 3.5 to Release 4.5.1)

Deprecated Events and Alarms (Release 4.5 to Release 4.5.1)

New Events and Alarms (Release 4.5.13 to Release 4.5.1)

Modified Events and Alarms (Release 4.5.13 to Release 4.5.1)

Deprecated Events and Alarms (Release 4.5.13 to Release 4.5.1)

Obtaining Documentation and Submitting a Service Request


Preface


Revised: July 22, 2009, OL-8000-32

Introduction

This document provides detailed troubleshooting procedures for the Cisco BTS 10200 Softswitch. It provides the procedures for troubleshooting network, subscriber, billing, operations and maintenance, system administrative problems. It also includes details for aiding troubleshooting by utilizing diagnostic and trace procedures.

Objective

The maintenance personnel/troubleshooter of a Cisco BTS 10200 Softswitch can use this document to better understand how to troubleshoot the Cisco BTS 10200 Softswitch and quickly clear network problems.

Audience

This document is designed for engineers, technicians, and system administrators who troubleshoot and repair the Cisco BTS 10200 Softswitch.

Organization

This Troubleshooting Guide contains the following chapters:

Chapter 1, "Troubleshooting Overview" - Provides an overview of troubleshooting the Cisco BTS 10200 Softswitch.

Chapter 2, "Audit Troubleshooting" - Provides the information needed to monitor and troubleshoot Audit events and alarms.

Chapter 3, "Billing Troubleshooting" - Provides the information needed to monitor and troubleshoot Billing events and alarms.

Chapter 4, "Call Processing Troubleshooting" - Provides the information needed to monitor and troubleshoot Call Processing events and alarms.

Chapter 5, "Configuration Troubleshooting" - Provides the information needed to monitor and troubleshoot Configuration events and alarms.

Chapter 6, "Database Troubleshooting" - Provides the information needed to monitor and troubleshoot Database events and alarms.

Chapter 7, "Maintenance Troubleshooting" - Provides the information needed to monitor and troubleshoot Maintenance events and alarms.

Chapter 8, "Operations Support System Troubleshooting" - Provides the information needed to monitor and troubleshoot Operations Support System events and alarms.

Chapter 9, "Security Troubleshooting" - Provides the information needed to monitor and troubleshoot Security events and alarms.

Chapter 10, "Signaling Troubleshooting" - Provides the information needed to monitor and troubleshoot Signaling events and alarms.

Chapter 11, "Statistics Troubleshooting" - Provides the information needed to monitor and troubleshoot Statistics events and alarms.

Chapter 12, "System Troubleshooting" - Provides the information needed to monitor and troubleshoot System events and alarms.

Chapter 13, "Network Troubleshooting" - Provides the information needed to conduct network troubleshooting on the Cisco BTS 10200 Softswitch.

Chapter 14, "General Troubleshooting" - Provides the general troubleshooting information needed to conduct troubleshooting on the Cisco BTS 10200 Softswitch.

Chapter 15, "Diagnostic Tests" - Describes the diagnostic tests that can be performed on media gateways, subscriber terminations, and trunk terminations.

Chapter 16, "Disaster Recovery Procedures" - Describes how to recover the database in a disaster situation, how to recover the database from another database, and how to recover data from the Call Agent shared memory.

Chapter 17, "Disk Replacement" - Describes how to manually recover a Cisco BTS 10200 Softswitch system.

Appendix A, "Recoverable and Nonrecoverable Error Codes" - Lists normal, recoverable and nonrecoverable error codes for the Cisco BTS 10200 Softswitch.

Appendix B, "System Usage of MGW Keepalive Parameters, Release 4.5.x" - Explains how the Cisco BTS 10200 Softswitch determines the connectivity status between itself and a media gateway (MGW)

Conventions

This document uses the following conventions:


Note Means reader take note. Notes contain helpful suggestions or references to material not covered in the manual.



Caution Means reader be careful. In this situation, you might do something that could result in equipment damage or loss of data.

Updates to this Document for Release 4.5.x

This document includes all of the information that was contained in the previous issue (the Release 4.5 Cisco BTS 10200 Softswitch Troubleshooting Guide, OL-8000-08), and has been updated for the Release 4.5 Cisco BTS 10200 Softswitch Troubleshooting Guide, OL-8000-09 as described in this Preface.

Updates are contained in the following chapters and appendixes of this document:

Updated Chapter 16, "Disaster Recovery Procedures"

Additionally, this document has been updated to OL-8000-10. Updates are contained in the following chapters and sections:

Updated Chapter 6, "Database Troubleshooting"

Updated Chapter 6, "DATABASE (11)"

Updated Chapter 6, "Element Management System Database Alert.log Alerts - Database (11)"

The updates to this document to OL-8000-11 include:

Removed AUDIT (9) event information

Added BILLING (60), MAINTENANCE (119), MAINTENANCE (120), SIGNALING (166), SIGNALING (167), and SYSTEM (15) event information.

The updates to this document for OL-8000-12 include:

Adding Appendix B, "System Usage of MGW Keepalive Parameters, Release 4.5.x" to the document.

The updates to this document for OL-8000-13 include:

Updated Chapter 16, "Disaster Recovery Procedures" to include the procedure for restoration of the Automatic Shared Memory Backup (ASMB)

The updates to this document for OL-8000-14 include:

Updated Chapter 16, "Disaster Recovery Procedures"

The updates to this document for OL-8000-15 included:

Updated DATABASE (11) primary cause and primary action

The updates to this document for OL-8000-16 included:

Updated Chapters 14, 15, and 16 to include troubleshooting information for Release 4.5.1.

The updates to this document for OL-8000-17 included:

Updated Chapter 7 to include the Maintenance 122 and Maintenance 123 alarms and events.

The update to this document for OL-8000-18 included:

Removal of the Release Cause Code appendix, update of Chapter 14 and Chapter 15, and update of the System Usage of MGW Keepalive Parameters appendix.

The update of this document for OL-8000-19 included:

Update of the threshold and throttle information in Chapter 1

The updates of this document for OL-8000-20 and OL-8000-21 included:

Update of the keepalive information in Appendix B.

The update of this document from OL-8000-21 to OL-8000-22 include incorporating the delta alarm and event information from Release 3.5 to Release 4.5.x.

The update of this document from OL-8000-22 to OL-8000-23 included:

Update of the keepalive information in Appendix B.

The update of this document from OL-8000-23 to OL-8000-24 included:

Adding the event and alarm differences between release 4.5.13 and 4.5.1.

The update of this document from OL-8000-24 to OL-8000-25 included:

Removing the Throttle/Threshold erroneous note

The update of this document from OL-8000-25 to OL-8000-26 included updating the disk replacement procedures in Chapter 17.

The update of this document from OL-8000-26 to OL-8000-27 included updating the recovering shared memory procedure and call agent database download and restore.

The update to OL-8000-28 included updating the information for the Audit 4 alarm.

The update to OL-8000-29 included adding the "PSTN Trunk Testing" section on page 15-80 to Chapter 15.

The update to OL-8000-30 included making changes to the "Automatic Shared Memory Backup Restore" section on page 16-30.

The update to OL-8000-31 included adding the "Oracle Database Tool Restart" section on page 13-8.

The update to OL-8000-32 included updating the "Recovering Shared Memory" section on page 16-26.

New Events and Alarms (Release 3.5 to Release 4.5.1)

lists the new events and alarms that were incorporated into the BTS 10200 system as part of the update from Release 3.5 to Release 4.5.1.

Table 1 New Events and Alarms (Release 3.5 to Release 4.5.1) 

Type and Number
Description

AUDIT (2)

Start or Stop of Signaling System 7-Circuit Identification Code Audit

AUDIT (3)

Signaling System 7 Circuit Identification Code Audit Terminated Before Successful Completion

AUDIT (4)

Call Exceeds a Long-Duration Threshold

AUDIT (6)

Major Internal Audit Failure

AUDIT (7)

Minor Internal Audit Failure

AUDIT (8)

Warning From Internal Audit

AUDIT (10)

Call Data Audit Complete

AUDIT (11)

Critical Network Time Protocol Service Failure

AUDIT (12)

Major Network Time Protocol Service Failure

AUDIT (13)

Minor Network Time Protocol Service Failure

AUDIT (14)

Network Time Protocol Service Warning

AUDIT (15)

Critical Index Shared Memory Error

AUDIT (16)

Process Heap Memory Usage Exceeds Minor Threshold Level

AUDIT (17)

Process Heap Memory Usage Exceeds Major Threshold Level

AUDIT (18)

Process Heap Memory Usage Exceeds Critical Threshold Level

AUDIT (19)

Recovered Memory of Stale Call

AUDIT (20)

Audit Found Lost Call Data Record

BILLING (52)

Bulk Data Management System Stopped Generating New Billing File

BILLING (57)

Prepaid Subscriber Call Attempt Failed Because of Balance

BILLING (58)

Signaling Prepaid Server Inaccessible

BILLING (60)

Bad File Detected During Startup

CALLP (38)

Release Cause 25 Exchange Routing Error Received

CALLP (39)

Test Call Blocked Due to Congestion or Isolation

CALLP (40)

Interactive Voice Response Real Time Transport Protocol Session Fail

CALLP (41)

INVITE Message From Unauthorized Call Agent

CONFIG (5)

Feature-Server Database and Command Line Host Mismatch

DATABASE (19)

Daily Database Backup Completed Successfully

DATABASE (20)

Replication Data Flush Timeout During Switchover

DATABASE (21)

Database Statistics Collection Exception

DATABASE (25)

Secure File Transfer Protocol Transfer Failed

DATABASE (26)

File Write Error

MAINTENANCE (86)

System Health Report Collection Error

MAINTENANCE (87)

Status Update Process Request Failed

MAINTENANCE (88)

Status Update Process Database List Retrieval Error

MAINTENANCE (89)

Status Update Process Database Update Error

MAINTENANCE (90)

Disk Partition Moderately Consumed

MAINTENANCE (91)

Internet Protocol Manager Configuration File Error

MAINTENANCE (92)

Internet Protocol Manager Initialization Error

MAINTENANCE (93)

Internet Protocol Manager Interface Failure

MAINTENANCE (94)

Internet Protocol Manager Interface State Change

MAINTENANCE (95)

Internet Protocol Manager Interface Created

MAINTENANCE (96)

Internet Protocol Manager Interface Removed

MAINTENANCE (102)

Thread Watchdog Counter Close to Expiry for a Thread

MAINTENANCE (103)

Central Processing Unit is Offline

MAINTENANCE (104)

Aggregration Device Address Successfully Resolved

MAINTENANCE (105)

Unprovisioned Aggregration Device Detected

MAINTENANCE (106)

Aggregration Device Address Resolution Failure

MAINTENANCE (107)

No Heartbeat Messages Received Through Interface From Router

MAINTENANCE (108)

A Log File Cannot be Transferred

MAINTENANCE (109)

Five Successive Log Files Cannot be Transferred

MAINTENANCE (110)

Access to Log Archive Facility Configuration File Failed or File Corrupted

MAINTENANCE (111)

Cannot Login to External Archive Server

MAINTENANCE (119)

Periodic Shared Memory Database Backup Failure

MAINTENANCE (120)

Periodic Shared Memory Database Backup Success

MAINTENANCE (122)

Northbound Provisioning Message is Retransmitted

MAINTENANCE (123)

Northbound Provisioning Message Dropped Due To Full Index Table

SECURITY (2)

Invalid Credentials Presented by a Session Initiation Protocol Phone

SECURITY (3)

Internet Protocol Security Connection Down

SECURITY (4)

Internet Protocol Security Media Terminal Adapter Key Establish Error

SECURITY (5)

Internet Protocol Security Outgoing Security Association Not Found

SECURITY (6)

Secure Session Initiation Protocol Endpoint Validation Failure

SIGNALING (109)

Stream Control Transmission Protocol Association Failure

SIGNALING (110)

Signaling Gateway Group is Out-of-Service

SIGNALING (111)

Stream Control Transmission Protocol Association Degraded (One of Two Internet Protocol Connections Down)

SIGNALING (112)

Stream Control Transmission Protocol Association Configuration Error

SIGNALING (113)

Signaling Gateway Failure

SIGNALING (114)

Signaling Gateway Process is Out-of-Service

SIGNALING (115)

Invalid Routing Context Received

SIGNALING (116)

Destination Point Code User Part Unavailable

SIGNALING (121)

Message Transfer Part 3 User Adapter Cannot Go Standby

SIGNALING (122)

Message Transfer Part 3 User Adapter Cannot Go Active

SIGNALING (124)

Remote Subsystem is Out Of Service

SIGNALING (125)

Signaling Connection Control Part Routing Error

SIGNALING (126)

Signaling Connection Control Part Binding Failure

SIGNALING (127)

Transaction Capabilities Application Part Binding Failure

SIGNALING (132)

Transaction Capabilities Application Part Reaches the Provisioned Resource Limit

SIGNALING (133)

Unable to Decode Generic Transport Descriptor Message

SIGNALING (134)

Signaling System 7 Message Encoding Failure

SIGNALING (135)

Signaling System 7 Message Decoding Failure

SIGNALING (136)

Signaling System 7 Message Invalid Received

SIGNALING (137)

Signaling System 7 Confusion Message Received

SIGNALING (138)

Number of Open Session Initiation Protocol Connections is Reaching Engineered Limit

SIGNALING (139)

Signaling System 7 Trunk was Found to be in Erroneous State

SIGNALING (140)

Unanswered Information Message

SIGNALING (141)

Address not Resolved by Domain Name System Server

SIGNALING (142)

Session Initiation Protocol Trunk Operationally Out-of-Service

SIGNALING (143)

Internet Protocol Interface Link to the Signaling System 7 Signaling Gateway is Down

SIGNALING (144)

All Internet Protocol Interface Links to Signaling System 7 Signaling Gateway are Down

SIGNALING (145)

One Internet Protocol Interface to Signaling System 7 Signaling Gateway is Down

SIGNALING (146)

All Retransmission Attempts of Session Initiation Protocol Request or Response Failed

SIGNALING (147)

Domain Name System Service Addresses Exhausted

SIGNALING (150)

Stream Control Transmission Protocol Association Congested

SIGNALING (151)

Termination Permanent Error Code Received

SIGNALING (152)

Termination Transient Error Received

SIGNALING (166)

No Routing Keys are Active

SIGNALING (167)

No Signaling Gateways are Active

STATISTICS (12)

Measurement Handshake Error - Schema Inconsistency

STATISTICS (13)

Traffic and Measurements Module Application Programming Interface Failure

SYSTEM (12)

Software Error

SYSTEM (13)

Multiple Readers and Multiple Writers Maximum Queue Depth Reached

SYSTEM (14)

Multiple Readers and Multiple Writers Queue Reached Low Queue Depth

SYSTEM (15)

Multiple Readers and Multiple Writers Throttle Queue Depth Reached


Modified Events and Alarms (Release 3.5 to Release 4.5.1)

lists the BTS 10200 modified events and alarms that were modified as part of the upgrade from Release 3.5 to Release 4.5.1.

Table 2 Modified Events and Alarms (Release 3.5 to Release 4.5.1) 

Type and Number
Description

BILLING (6)

File Transfer Protocol/Secure File Transfer Protocol Transfer Failed

BILLING (13)

Database Connection Error

BILLING (14)

File Open Error

BILLING (15)

File Write Error

BILLING (33)

File Transfer Protocol/Secure File Transfer Protocol Parameters Invalid

BILLING (41)

Message Content Error

BILLING (44)

Record Keeping System Switch Occurred

BILLING (45)

Event Message Log File Opened

BILLING (46)

Event Message Log File Closed

BILLING (53)

Event Message Disk Space 50 Percent Full

BILLING (56)

Billing Data Corruption Detected

CALLP (9)

No Route Available for Carrier Dialed

DATABASE (5)

Element Management System Database HeartBeat Process Died

DATABASE (17)

Data Replication Failed

MAINTENANCE (41)

Administrative State Change Successful with Warning

MAINTENANCE (47)

Thread Watchdog Counter Expired for a Thread

MAINTENANCE (57)

Memory and Swap are Consumed at Critical Levels

MAINTENANCE (58)

Memory and Swap are Consumed at Abnormal Levels

MAINTENANCE (65)

Disk Partition Critically Consumed

MAINTENANCE (66)

Disk Partition Significantly Consumed

MAINTENANCE (71)

Local Domain Name System Server Response Too Slow

MAINTENANCE (72)

External Domain Name System Server Response Too Slow

MAINTENANCE (73)

External Domain Name System Server not Responsive

MAINTENANCE (82)

Average Outgoing Heartbeat Period Exceeds Critical Limit

OSS (6)

Structured Query Language Connection Error

SIGNALING (23)

Destination Point Code Unavailable

SIGNALING (25)

Unanswered Blocking

SIGNALING (26)

Unanswered Unblocking Message

SIGNALING (27)

Unanswered Circuit Group Blocking

SIGNALING (28)

Unanswered Circuit Group Unblocking

SIGNALING (29)

Unanswered Circuit Query Message

SIGNALING (30)

Unanswered Circuit Validation Test

SIGNALING (31)

Unanswered Reset Circuit

SIGNALING (32)

Unanswered Group Reset

SIGNALING (33)

Unanswered Release

SIGNALING (34)

Unanswered Continuity Check Request

SIGNALING (42)

Continuity Testing Message Received on the Specified Circuit Identification Code

SIGNALING (43)

Release Complete Received in Response to Reset Circuit Message on the Specified Circuit Identification Code

SIGNALING (44)

Continuity Recheck is Performed on Specified Circuit Identification Code

SIGNALING (45)

Circuit is UNEQUIPPED on Remote Side

SIGNALING (46)

Specified Circuit Identification Code is Invalid for the Operation

SIGNALING (49)

A General Processing Error Encountered

SIGNALING (50)

Unexpected Message for the Call State is Received: Clear Call

SIGNALING (51)

Set Trunk State as Remotely Unequipped

SIGNALING (52)

Set Trunk State as NOT Remotely Blocked

SIGNALING (53)

Set Trunk State as Remotely Blocked

SIGNALING (54)

Circuit Validation Test Aborted

SIGNALING (55)

Circuit Validation Successful

SIGNALING (57)

Continuity Recheck Failed

SIGNALING (58)

Continuity Recheck Successful

SIGNALING (68)

Media Gateway Endpoints are out of Service at Gateway

SIGNALING (69)

Call Agent and Feature Server Communication Message Timeout

SIGNALING (89)

Gatekeeper not Available/Reachable

SIGNALING (90)

Alternate Gatekeeper is not Responding

SIGNALING (107)

Logical Internet Protocol Addresses not Mapped Correctly

SIGNALING (108)

Simplex Only Operational Mode

SIGNALING (118)

Circuit Verification Response Received with Failed Indication


Deprecated Events and Alarms (Release 4.5 to Release 4.5.1)

lists the BTS 10200 events and alarms that were deprecated as part of the upgrade from Release 3.5 to Release 4.5.1.

Table 3 Deprecated Events and Alarms (Release 3.5 to Release 4.5.1) 

Type and Number
Description

BILLING(9)

Duplicate Event Message Received

BILLING (10)

Database Lock Error

BILLING (11)

Database Unlock Error

BILLING (12)

Table Pointer Read Error

BILLING(16)

Time Point Calculation Error

BILLING(17)

Shared Memory Add Failure

BILLING(18)

Error Calling MKTIME Function

BILLING(19)

Call Detail Block Database Write Error

BILLING(20)

Call Detail Block Database Read Error

BILLING(21)

Unknown Billing Event Message Received

BILLING(22)

Error Releasing Staging Area in Shared Memory

BILLING(23)

Error Releasing Busy List Entry in Shared Memory

BILLING(24)

Update Table Pointer Error

BILLING(26)

Shared Memory Update Failed

BILLING(27)

Shared Memory Entry Remove Failed

BILLING(28)

Wrong Port Number Specified

BILLING(34)

All Billing Links at CA Down

BILLING(39)

RADIUS Accounting Receive Failure

BILLING(43)

Invalid Billing Configuration—Program Exiting

MAINTENANCE(59)

Disk Reports Greater Than 70% Utilization

MAINTENANCE(60)

Disk Utilization has Returned to Normal Levels

SIGNALING(2)

Message Encoding Error

SIGNALING(3)

Message Decoding Error

SIGNALING(47)

Indicated Circuit is Out of Order

SIGNALING(48)

Indicated Circuit is Back in Service

SIGNALING(56)

Circuit Validation Failure

SIGNALING(67)

Gateway/Termination Returning Transient Error for the MGCP

SIGNALING(123)

Destination Point Code is Unavailable

SIGNALING(128)

INAP Binding Failure

SIGNALING(131)

INAP Reaches the Provisioned Resource Limit


New Events and Alarms (Release 4.5.13 to Release 4.5.1)

lists the new events and alarms that were incorporated into the BTS 10200 system as part of the update from Release 4.5.13 to Release 4.5.1.

Table 4 New Events and Alarms (Release 4.5.13 to Release 4.5.1) 

Type and Number
Description

BILLING (60)

Bad File Detected During Startup

DATABASE (25)

Secure File Transfer Protocol Transfer Failed

DATABASE (26)

File Write Error

MAINTENANCE (118)

Domain Name Server Zone Database does not Match Between the Primary Domain Name Server and the Internal Secondary Authoritative Domain Name Server (DNS Zone Database does not Match Between the Primary DNS and the ISADS)

MAINTENANCE (119)

Periodic Shared Memory Database Backup Failure

MAINTENANCE (120)

Periodic Shared Memory Database Backup Success

SYSTEM (9)

Timer Start Failure


Modified Events and Alarms (Release 4.5.13 to Release 4.5.1)

lists the BTS 10200 modified events and alarms that were modified as part of the upgrade from Release 3.5 to Release 4.5.1.

Table 5 Modified Events and Alarms (Release 4.5.13 to Release 4.5.1) 

Type and Number
Description

BILLING (6)

File Transfer Protocol/Secure File Transfer Protocol Transfer Failed

BILLING (14)

File Open Error

BILLING (15)

File Write Error

BILLING (56)

Billing Data Corruption Detected

SECURITY (6)

Secure Session Initiation Protocol Endpoint Validation Failure

SIGNALING (69)

Call Agent and Feature Server Communication Message Timeout

SIGNALING (148)

Softswitch Audit Released Stale Memory

SYSTEM (7)

Index List Remove Error


Deprecated Events and Alarms (Release 4.5.13 to Release 4.5.1)

There were no deprecated events or alarms between Release 4.5.13 and Release 4.5.1.

Obtaining Documentation and Submitting a Service Request

For information on obtaining documentation, submitting a service request, and gathering additional information, see the monthly What's New in Cisco Product Documentation, which also lists all new and revised Cisco technical documentation, at:

http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html

Subscribe to the What's New in Cisco Product Documentation as a Really Simple Syndication (RSS) feed and set content to be delivered directly to your desktop using a reader application. The RSS feeds are a free service and Cisco currently supports RSS Version 2.0.