Troubleshooting Guide
Chapter 3 - Billing Troubleshooting
Downloads: This chapterpdf (PDF - 674.0KB) The complete bookPDF (PDF - 9.12MB) | Feedback

Billing Troubleshooting

Table Of Contents

Billing Troubleshooting

Introduction

Billing Events and Alarms

BILLING (1)

BILLING (2)

BILLING (3)

BILLING (4)

BILLING (5)

BILLING (6)

BILLING (7)

BILLING (8)

BILLING (13)

BILLING (14)

BILLING (15)

BILLING (29)

BILLING (30)

BILLING (31)

BILLING (32)

BILLING (33)

BILLING (35)

BILLING (36)

BILLING (37)

BILLING (38)

BILLING (40)

BILLING (41)

BILLING (42)

BILLING (44)

BILLING (45)

BILLING (46)

BILLING (47)

BILLING (48)

BILLING (49)

BILLING (52)

BILLING (53)

BILLING (54)

BILLING (55)

BILLING (56)

BILLING (57)

BILLING (58)

BILLING (60)

Monitoring Billing Events

Test Report - Billing (1)

Billing Partition Disk Usage Minor Threshold Exceeded - Billing (2)

Billing Partition Disk Usage Major Threshold Exceeded - Billing (3)

Billing Partition Disk Usage Critical Threshold Exceeded - Billing (4)

Billing Partition Disk Usage Within Normal Range - Billing (5)

File Transfer Protocol/Secure File Transfer Protocol Transfer Failed - Billing (6)

Transmission Control Protocol Connection Error - Billing (7)

Transmission Control Protocol Packet Receive Failure - Billing (8)

Database Connection Error - Billing (13)

File Open Error - Billing (14)

File Write Error - Billing (15)

Call Data Block Send Failed - Billing (29)

Domain Name Mapping Failed - Billing (30)

Port not Specified - Billing (31)

Element Management System Address not Specified - Billing (32)

File Transfer Protocol/Secure File Transfer Protocol Parameters Invalid - Billing (33)

All Billing Links at Billing Server Down - Billing (35)

Billing Link Restored - Billing (36)

Billing Link Failure - Billing (37)

Event Message Log File Access Error - Billing (38)

Event Message Encode Failure - Billing (40)

Message Content Error - Billing (41)

Error Reading Provisioned Data-Using Default - Billing (42)

Record Keeping System Switch Occurred - Billing (44)

Event Message Log File Opened - Billing (45)

Event Message Log File Closed - Billing (46)

Record Keeping System Unreachable for One Hour - Billing (47)

Record Keeping System Unreachable for Three Hours - Billing (48)

Record Keeping System Unreachable for Five Hours - Billing (49)

Bulk Data Management System Stopped Generating New Billing File - Billing (52)

Event Message Disk Space 50 Percent Full - Billing (53)

Event Message Disk Space 70 Percent Full - Billing (54)

Event Message Disk Space 100 Percent Full - Billing (55)

Billing Data Corruption Detected - Billing (56)

Prepaid Subscriber Call Attempt Failed Because of Balance - Billing (57)

Signaling Prepaid Server Inaccessible - Billing (58)

Bad File Detected During Startup - Billing (60)

Troubleshooting Billing Alarms

Billing Partition Disk Usage Minor Threshold Exceeded - Billing (2)

Billing Partition Disk Usage Major Threshold Exceeded - Billing (3)

Billing Partition Disk Usage Critical Threshold Exceeded - Billing (4)

File Transfer Protocol/Secure File Transfer Protocol Transfer Failed - Billing (6)

Transmission Control Protocol Connection Error - Billing (7)

Transmission Control Protocol Packet Receive Failure - Billing (8)

Database Connection Error - Billing (13)

File Write Error - Billing (15)

Call Data Block Send Failed - Billing (29)

Domain Name Mapping Failed - Billing (30)

Port not Specified - Billing (31)

Element Management System Address not Specified - Billing (32)

File Transfer Protocol/Secure File Transfer Protocol Parameters Invalid - Billing (33)

All Billing Links at Billing Server Down - Billing (35)

Billing Link Failure - Billing (37)

Event Message Log File Access Error - Billing (38)

Event Message Encode Failure - Billing (40)

Message Content Error - Billing (41)

Record Keeping System Switch Occurred - Billing (44)

Event Message Log File Opened - Billing (45)

Event Message Log File Closed - Billing (46)

Record Keeping System Unreachable for One Hour - Billing (47)

Record Keeping System Unreachable for Three Hours - Billing (48)

Record Keeping System Unreachable for 5 Hours - Billing (49)

Bulk Data Management System Stopped Generating New Billing File - Billing (52)

Event Message Disk Space 50 Percent Full - Billing (53)

Event Message Disk Space 70 Percent Full - Billing (54)

Event Message Disk Space 100 Percent Full - Billing (55)

Billing Data Corruption Detected - Billing (56)

Signaling Prepaid Server Inaccessible - Billing (58)

Bad File Detected During Startup - Billing (60)


Billing Troubleshooting


Revised: July 22, 2009, OL-8000-32

Introduction

This chapter provides the information needed to monitor and troubleshoot Billing events and alarms. This chapter is divided into the following sections:

Billing Events and Alarms - Provides a brief overview of each Billing event and alarm.

Monitoring Billing Events - Provides the information needed to monitor and correct Billing events.

Troubleshooting Billing Alarms - Provides the information needed to troubleshoot and correct Billing alarms.


Note The following billing records are created when a call is rejected due to overload conditions:
· SS7 termination cause code 42
· Cable signaling stop event cause code "resource unavailable"
Calls rejected by the signaling adapter will not generate a billing record.


Billing Events and Alarms

This section provides a brief overview of the Billing events and alarms for the Cisco BTS 10200 Softswitch in numerical order. Table 3-1 lists all of the billing events and alarms by severity.


Note Click the Billing message number in Table 3-1 to display information about the event or alarm.



BILLING (1)

For additional information, refer to the "Test Report - Billing (1)" section.

DESCRIPTION

Test Report

SEVERITY

Information (INFO)

THRESHOLD

10000

THROTTLE

0


BILLING (2)

To troubleshoot and correct the cause of the alarm, refer to the "Billing Partition Disk Usage Minor Threshold Exceeded - Billing (2)" section.

DESCRIPTION

Billing Partition Disk Usage Minor Threshold Exceeded

SEVERITY

MINOR

THRESHOLD

100

THROTTLE

0

DATAWORDS

Disk Usage Percentage - TWO_BYTES

PRIMARY
CAUSE

Call detail records are accumulating on the disk associated with the billing database in the Element Management System (EMS). This is because data is being written into the database faster than it is being read out of the database. The minor threshold (default value = 70%) has been exceeded.

PRIMARY
ACTION

Monitor this alarm. The read should catch up to write within a few minutes, and the alarm should not remain active.

SECONDARY
CAUSE

Some fluctuation in disk usage is to be expected as call volume rises and falls during the day. Threshold crossings might step upward (from MINOR to MAJOR to CRITICAL) when there is a rapid increase in call volume, and then step downward (CRITICAL to MAJOR to MINOR) when call volume slows.

SECONDARY
ACTION

To monitor the alarm, use the subscribe alarm-report command. To obtain a summary, use the report alarm-summary command. Verify that type = billing is entered in these commands.


BILLING (3)

To troubleshoot and correct the cause of the alarm, refer to the "Billing Partition Disk Usage Major Threshold Exceeded - Billing (3)" section.

DESCRIPTION

Billing Partition Disk Usage Major Threshold Exceeded

SEVERITY

MAJOR

THRESHOLD

100

THROTTLE

0

DATAWORDS

Disk Usage Percentage - TWO_BYTES

PRIMARY
CAUSE

Call detail records are accumulating on the disk associated with the billing database in the EMS. This is because data is being written into the database faster than it is being read out of the database. The major threshold (default value = 80%) has been exceeded.

PRIMARY
ACTION

Monitor this alarm. The read should catch up to write within a few minutes, and the alarm should not remain active.

SECONDARY
CAUSE

Some fluctuation in disk usage is to be expected as call volume rises and falls during the day. Threshold crossings might step upward (from MINOR to MAJOR to CRITICAL) when there is a rapid increase in call volume, and then step downward (CRITICAL to MAJOR to MINOR) when call volume slows.

SECONDARY
ACTION

To monitor the alarm, use the subscribe alarm-report command. To obtain a summary, use the report alarm-summary command. Verify that type = billing is entered in these commands.

TERNARY
ACTION

If the alarm does not clear (or step down to a reduced level) in a few minutes, contact Cisco Technical Assistance Center (TAC) for assistance.


Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

BILLING (4)

To troubleshoot and correct the cause of the alarm, refer to the "Billing Partition Disk Usage Critical Threshold Exceeded - Billing (4)" section.

DESCRIPTION

Billing Partition Disk Usage Critical Threshold Exceeded

SEVERITY

CRITICAL

THRESHOLD

100

THROTTLE

0

DATAWORDS

Disk Usage Percentage - TWO_BYTES

PRIMARY
CAUSE

Call detail records are accumulating on the disk associated with the billing database in the EMS. This is because data is being written into the database faster than it is being read out of the database. The major threshold (default value = 90%) has been exceeded.

PRIMARY
ACTION

Monitor this alarm. The read should catch up to write within a few minutes, and the alarm should not remain active.

SECONDARY
CAUSE

Some fluctuation in disk usage is to be expected as call volume rises and falls during the day. Threshold crossings might step upward (from MINOR to MAJOR to CRITICAL) when there is a rapid increase in call volume, and then step downward (CRITICAL to MAJOR to MINOR) when call volume slows.

SECONDARY
ACTION

To monitor the alarm, use the subscribe alarm-report command. To obtain a summary, use the report alarm-summary command. Verify that type = billing is entered in these commands.

TERNARY
ACTION

If the alarm does not clear (or step down to a reduced level) in a few minutes, contact Cisco TAC for assistance.


Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

BILLING (5)

For additional information, refer to the "Billing Partition Disk Usage Within Normal Range - Billing (5)" section.

DESCRIPTION

Billing Partition Disk Usage Within Normal Range

SEVERITY

INFO

THRESHOLD

100

THROTTLE

0

DATAWORDS

Disk Usage Percentage - TWO_BYTES


BILLING (6)

To troubleshoot and correct the cause of the alarm, refer to the "File Transfer Protocol/Secure File Transfer Protocol Transfer Failed - Billing (6)" section.


Note OpenSSH version 3.9p1 contains a bug that may cause billing file transfers over SFTP to fail.


DESCRIPTION

File Transfer Protocol/Secure File Transfer Protocol Transfer Failed

SEVERITY

MAJOR

THRESHOLD

100

THROTTLE

0

DATAWORDS

TransferType - STRING [5]
FileName - STRING [40]
RemoteAddress - STRING [40]
Error - STRING [50]

PRIMARY
CAUSE

Unable to connect to remote host.

PRIMARY
ACTION

Verify the remote host is reachable. Run "show billing-acct-addr" and verify billing-server-addr is correct. Change billing-server-addr, if necessary, via "change billing-acct-addr".

SECONDARY
CAUSE

Unable to login to remote host.

SECONDARY
ACTION

Use "show billing-acct-addr" to verify user-name is a valid user for the host specified in billing-server-addr. If user-name is correct and the "TransferType" dataword shows "File Transfer Protocol (FTP)", re-enter the password via "change billing-acct-addr". If user-name is correct and the "TransferType" dataword shows "Secure File Transfer Protocol (SFTP)", verify that secure shell (SSH) keys have been pre-configured for user-name on both the Cisco BTS 10200 Softswitch and the remote host.

TERNARY
CAUSE

File transfer error.

TERNARY
ACTION

Check the "Error" dataword to see if it gives an indication of the kind of error that occurred. It could be a file-system error on the remote host, or a communication failure between the Cisco BTS 10200 Softswitch and the remote host.

SUBSEQUENT
CAUSE

The CDB_BILLING_SUPP flag is not set to "Y" in the call-agent-profile table.

SUBSEQUENT
ACTION

Check and verify that the CDB_BILLING_SUPP flag is set to "Y" in the call-agent-profile table.


BILLING (7)

To troubleshoot and correct the cause of the alarm, refer to the "Transmission Control Protocol Connection Error - Billing (7)" section.

DESCRIPTION

Transmission Control Protocol Connection Error

SEVERITY

CRITICAL

THRESHOLD

100

THROTTLE

0

DATAWORDS

Hostname - STRING [100]

PRIMARY
CAUSE

System call error.

PRIMARY
ACTION

Check address in platform configuration.

SECONDARY
CAUSE

Not connected to the right address.

SECONDARY
ACTION

Call Cisco TAC technical support. (Contact Cisco TAC.)


Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

BILLING (8)

To troubleshoot and correct the cause of the alarm, refer to the "Transmission Control Protocol Packet Receive Failure - Billing (8)" section.

DESCRIPTION

Transmission Control Protocol Packet Receive Failure

SEVERITY

MAJOR

THRESHOLD

100

THROTTLE

0

DATAWORDS

Host Name - STRING [25]

PRIMARY
CAUSE

Peer went down; socket closed.

PRIMARY
ACTION

Check the peer's status and bring it up.



Note BILLING (9) through BILLING (12) are not used.


BILLING (13)

To troubleshoot and correct the cause of the alarm, refer to the "Database Connection Error - Billing (13)" section.

DESCRIPTION

Database Connection Error

SEVERITY

CRITICAL

THRESHOLD

100

THROTTLE

0

PRIMARY
CAUSE

Structured Query Language (SQL) server is down.

PRIMARY
ACTION

Restart server; if this does not correct the problem, call Cisco TAC technical support. (Contact Cisco TAC.)


Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

BILLING (14)

To monitor and correct the cause of the event, refer to the "File Open Error - Billing (14)" section.

DESCRIPTION

File Open Error

SEVERITY

WARNING

THRESHOLD

100

THROTTLE

0

DATAWORDS

Path Name - STRING [100]

PRIMARY
CAUSE

System error, may be out of file descriptors.

PRIMARY
ACTION

Call Cisco TAC technical support. (Contact Cisco TAC.)


Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

BILLING (15)

To troubleshoot and correct the cause of the alarm, refer to the "File Write Error - Billing (15)" section.

DESCRIPTION

File Write Error

SEVERITY

MAJOR

THRESHOLD

100

THROTTLE

0

DATAWORDS

Path Name - STRING [100]

PRIMARY
CAUSE

System error, may be out of file descriptors.

PRIMARY
ACTION

Call Cisco TAC technical support. (Contact Cisco TAC.)


Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.


Note BILLING (16) through BILLING (28) are not used.


BILLING (29)

To troubleshoot and correct the cause of the alarm, refer to the "Call Data Block Send Failed - Billing (29)" section.

DESCRIPTION

Call Data Block Send Failed

SEVERITY

MAJOR

THRESHOLD

100

THROTTLE

0

PRIMARY
CAUSE

Transmission Control Protocol (TCP) send call failure.

PRIMARY
ACTION

Check the port number and address of blg and bmg processes in platform.cfg file.

SECONDARY
CAUSE

Both the EMS servers are down.

SECONDARY
ACTION

Check if both EMS servers are down. If so, bring at least one up.


BILLING (30)

To troubleshoot and correct the cause of the alarm, refer to the "Domain Name Mapping Failed - Billing (30)" section.

DESCRIPTION

Domain Name Mapping Failed

SEVERITY

MAJOR

THRESHOLD

100

THROTTLE

0

DATAWORDS

Address - STRING [50]

PRIMARY
CAUSE

Wrong domain name system (DNS) name mapping specified in the configuration files.

PRIMARY
ACTION

Check opticall.cfg and platform.cfg for the right mapping.


BILLING (31)

To troubleshoot and correct the cause of the alarm, refer to the "Port not Specified - Billing (31)" section.

DESCRIPTION

Port not Specified

SEVERITY

MAJOR

THRESHOLD

100

THROTTLE

0

PRIMARY
CAUSE

Port not specified in the platform.cfg file.

PRIMARY
ACTION

Check platform.cfg file and add the argument to blg -port 15260.


BILLING (32)

To troubleshoot and correct the cause of the alarm, refer to the "Element Management System Address not Specified - Billing (32)" section.

DESCRIPTION

Element Management System Address not Specified

SEVERITY

MAJOR

THRESHOLD

100

THROTTLE

0

PRIMARY
CAUSE

Either the primary or secondary EMS address has not been specified in the platform.cfg file.

PRIMARY
ACTION

Check the platform.cfg for the process billing (BLG) and add the missing addresses to the file.


BILLING (33)

To troubleshoot and correct the cause of the alarm, refer to the "File Transfer Protocol/Secure File Transfer Protocol Parameters Invalid - Billing (33)" section.

DESCRIPTION

File Transfer Protocol/Secure File Transfer Protocol Parameters Invalid

SEVERITY

MAJOR

THRESHOLD

100

THROTTLE

0

DATAWORDS

TransferType - STRING [5]
BillingServerDir - STRING [100]
BillingServerAddr - STRING [100]
User Name - STRING [100]

PRIMARY
CAUSE

The billing-acct-addr table is not fully provisioned with the information needed to perform file transfers.

PRIMARY
ACTION

Check billing-acct-addr fields via "show billing-acct-addr". For FTP file transfer, ensure billing-server-addr, billing-server-directory, user-name, and password (not displayed) are provisioned. Also ensure sftp-supp is set to "N". For SFTP file transfer, ensure billing-server-addr, billing-server-directory, and user-name are provisioned. Also ensure sftp-supp is set to "Y".



Note BILLING (34) is not used.


BILLING (35)

To troubleshoot and correct the cause of the alarm, refer to the "All Billing Links at Billing Server Down - Billing (35)" section.

DESCRIPTION

All Billing Links at Billing Server Down

SEVERITY

CRITICAL

THRESHOLD

100

THROTTLE

0

PRIMARY
CAUSE

Cable connection might have been pulled out.

PRIMARY
ACTION

Restore the cable connection.

SECONDARY
CAUSE

An "ifconfig down" might have been done on the interfaces.

SECONDARY
ACTION

Do a "ifconfig up".


BILLING (36)

For additional information, refer to the "Billing Link Restored - Billing (36)" section.

DESCRIPTION

Billing Link Restored

SEVERITY

INFO

THRESHOLD

100

THROTTLE

0

DATAWORDS

Interface Name - STRING [50]

PRIMARY
CAUSE

Cable has been restored.

PRIMARY
ACTION

None


BILLING (37)

To troubleshoot and correct the cause of the alarm, refer to the "Billing Link Failure - Billing (37)" section.

DESCRIPTION

Billing Link Failure

SEVERITY

MAJOR

THRESHOLD

100

THROTTLE

0

DATAWORDS

Interface Name - STRING [5]

PRIMARY
CAUSE

Cable may have been pulled.

PRIMARY
ACTION

Restore cable connection.

SECONDARY
CAUSE

An "ifconfig down" may have been performed on the interface.

SECONDARY
ACTION

Perform "ifconfig up" on the interface.


BILLING (38)

To troubleshoot and correct the cause of the alarm, refer to the "Event Message Log File Access Error - Billing (38)" section.

DESCRIPTION

Event Message Log File Access Error

SEVERITY

MAJOR

THRESHOLD

50

THROTTLE

0

DATAWORDS

Type of Access Error - STRING [25]
Reason for Error - STRING [40]
Sequence Number of F - FOUR_BYTES
Index of Event Messa - FOUR_BYTES
Location Tag - STRING [30]

PRIMARY
CAUSE

System error--may be out of file descriptors.

PRIMARY
ACTION

Call Cisco TAC. (Contact Cisco TAC.)

SECONDARY
CAUSE

Disk may be faulty.

SECONDARY
ACTION

Cause the Bulk Data Management System (BDMS) to switch over to its mate node.


Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.


Note BILLING (39) is not used.


BILLING (40)

To troubleshoot and correct the cause of the alarm, refer to the "Event Message Encode Failure - Billing (40)" section.

DESCRIPTION

Event Message Encode Failure

SEVERITY

MINOR

THRESHOLD

100

THROTTLE

0

DATAWORDS

Location Tag - STRING [30]

PRIMARY
CAUSE

There is a problem with the format of the data to be sent to the record keeping system (RKS).

PRIMARY
ACTION

If problem persists, call Cisco TAC. (Contact Cisco TAC.)


Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

BILLING (41)

To troubleshoot and correct the cause of the alarm, refer to the "Message Content Error - Billing (41)" section.

DESCRIPTION

Message Content Error

SEVERITY

MINOR

THRESHOLD

100

THROTTLE

0

DATAWORDS

Message Type - FOUR_BYTES
Field Name - STRING [20]
Field Value (Text) - STRING [20]
Field Value (Numeric) - FOUR_BYTES
Location Tag - STRING [30]

PRIMARY
CAUSE

There is a mismatch between what the sender populated in the message and what the receiver expects.

PRIMARY
ACTION

Call Cisco TAC. (Contact Cisco TAC.)


Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

BILLING (42)

To monitor and correct the cause of the event, refer to the "Error Reading Provisioned Data-Using Default - Billing (42)" section.

DESCRIPTION

Error Reading Provisioned Data-Using Default

SEVERITY

WARNING

THRESHOLD

100

THROTTLE

0

DATAWORDS

Error Code - FOUR_BYTES
Table Name - STRING [20]
Field Name - STRING [20]
Default Value (Decim - FOUR_BYTES
Default Value (Text) - STRING [20]
Location Tag - STRING [30]

PRIMARY
CAUSE

Application was unable to read provisioned data, and had to resort to using default values.

PRIMARY
ACTION

Check to ensure a complete load has been installed on the system. If load is complete and problem persists, call Cisco TAC. (Contact Cisco TAC.)


Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.


Note BILLING (43) is not used.


BILLING (44)

To troubleshoot and correct the cause of the alarm, refer to the "Record Keeping System Switch Occurred - Billing (44)" section.

DESCRIPTION

Record Keeping System Switch Occurred

SEVERITY

MAJOR

THRESHOLD

100

THROTTLE

0

DATAWORDS

Type of RKS Switch - STRING [20]
Location Tag - STRING [30]

PRIMARY
CAUSE

Billing changed the destination RKS for transmitting Event Messages. The change could have been triggered by a communication problem with an RKS, or by an attempt to reestablish RKS communication.

PRIMARY
ACTION

No action is necessary.


BILLING (45)

To troubleshoot and correct the cause of the alarm, refer to the "Event Message Log File Opened - Billing (45)" section.

DESCRIPTION

Event Message Log File Opened

SEVERITY

MINOR

THRESHOLD

50

THROTTLE

0

DATAWORDS

Element Type - STRING [5]
File Name - STRING [60]
Location Tag - STRING [30]

PRIMARY
CAUSE

A log file has been created for the storage of Event Messages that cannot be transmitted to an RKS.

PRIMARY
ACTION

No action is necessary.


BILLING (46)

To troubleshoot and correct the cause of the alarm, refer to the "Event Message Log File Closed - Billing (46)" section.

DESCRIPTION

Event Message Log File Closed

SEVERITY

MINOR

THRESHOLD

50

THROTTLE

0

DATAWORDS

Element Type - STRING [5]
File Name - STRING [60]
Location Tag - STRING [30]

PRIMARY
CAUSE

An open Event Message log file has been closed.

PRIMARY
ACTION

No action is necessary.


BILLING (47)

To troubleshoot and correct the cause of the alarm, refer to the "Record Keeping System Unreachable for One Hour - Billing (47)" section.

DESCRIPTION

Record Keeping System Unreachable for One Hour

SEVERITY

MINOR

THRESHOLD

25

THROTTLE

0

DATAWORDS

Location Tag - STRING [30]

PRIMARY
CAUSE

Billing has not been able to communicate with any RKS for the past one hour.

PRIMARY
ACTION

Check status of the primary and secondary RKS servers; attempt to bring them into service. - Verify the radius-profile table and call-agent-profile table are provisioned such that communication with the RKS servers is possible.


BILLING (48)

To troubleshoot and correct the cause of the alarm, refer to the "Record Keeping System Unreachable for Three Hours - Billing (48)" section.

DESCRIPTION

Record Keeping System Unreachable for Three Hours

SEVERITY

MAJOR

THRESHOLD

25

THROTTLE

0

DATAWORDS

Location Tag - STRING [30]

PRIMARY
CAUSE

Billing has not been able to communicate with any RKS for the past three hours.

PRIMARY
ACTION

Check status of the primary and secondary RKS servers: attempt to bring them into service. - Verify the radius-profile table and call-agent-profile table are provisioned such that communication with the RKS severs is possible.


BILLING (49)

To troubleshoot and correct the cause of the alarm, refer to the "Record Keeping System Unreachable for 5 Hours - Billing (49)" section.

DESCRIPTION

Record Keeping System Unreachable for 5 Hours

SEVERITY

CRITICAL

THRESHOLD

25

THROTTLE

0

DATAWORDS

Location Tag - STRING [30]

PRIMARY
CAUSE

Billing has not been able to communicate with any RKS for the past five hours.

PRIMARY
ACTION

Check status of the primary and secondary RKS servers; attempt to bring them into service. - Verify the radius-profile table and call-agent-profile table are provisioned such that communication with the RKS servers is possible.



Note BILLING (50) and BILLING (51) are not used.


BILLING (52)

To troubleshoot and correct the cause of the alarm, refer to the "Bulk Data Management System Stopped Generating New Billing File - Billing (52)" section.

DESCRIPTION

Bulk Data Management System Stopped Generating New Billing File

SEVERITY

CRITICAL

THRESHOLD

100

THROTTLE

0

PRIMARY
CAUSE

Call detail records are accumulating on the disk associated with the billing files in the EMS. This is because data is being written into the billing files faster than it is being forwarded to the Billing Mediation Server. FTP to the Billing Mediation Server may not be working. The maximum disk partition for billing has been exceeded or the maximum number of files has been exceeded.

PRIMARY
ACTION

Check Billing Mediation Server node name, user name and password specified in BILLING_ACCT_ADDR table and log files. Correct any error to let FTP start again. If billing_server_directory = "/dev/null" as in the lab, primary files under billing_directory will not be forwarded or deleted automatically. In this case, files have to be manually deleted or moved out, and BDMS needs to be restarted before it start generating new billing files.


BILLING (53)

To troubleshoot and correct the cause of the alarm, refer to the "Event Message Disk Space 50 Percent Full - Billing (53)" section.

DESCRIPTION

Event Message Disk Space 50 Percent Full

SEVERITY

MINOR

THRESHOLD

100

THROTTLE

0

DATAWORDS

Number of Megabytes Used for Eve - FOUR_BYTES
Directory Containing Event Messa - STRING [30]
Location Tag - STRING [30]

PRIMARY
CAUSE

Event Message storage has reached 50% of the maximum storage allowed.

PRIMARY
ACTION

Move Event Message files out of the specified directory. Store them in another location, or discard them.


BILLING (54)

To troubleshoot and correct the cause of the alarm, refer to the "Event Message Disk Space 70 Percent Full - Billing (54)" section.

DESCRIPTION

Event Message Disk Space 70 Percent Full

SEVERITY

MAJOR

THRESHOLD

100

THROTTLE

0

DATAWORDS

Number of Megabytes Used for Eve - FOUR_BYTES
Directory Containing Event Messa - STRING [30]
Location Tag - STRING [30]

PRIMARY
CAUSE

Event Message storage has reached 70% of the maximum storage allowed.

PRIMARY
ACTION

Move Event Message files out of the specified directory. Store them in another location, or discard them.


BILLING (55)

To troubleshoot and correct the cause of the alarm, refer to the "Event Message Disk Space 100 Percent Full - Billing (55)" section.

DESCRIPTION

Event Message Disk Space 100 Percent Full

SEVERITY

CRITICAL

THRESHOLD

100

THROTTLE

0

DATAWORDS

Number of Megabytes Used for Eve - FOUR_BYTES
Directory Containing Event Messa - STRING [30]
Location Tag - STRING [30]

PRIMARY
CAUSE

Event Message storage has been completely filled. No additional Event Messages will be written to disk until more space is made available.

PRIMARY
ACTION

Move Event Message files out of the specified directory. Store them in another location, or discard them.


BILLING (56)

To troubleshoot and correct the cause of the alarm, refer to the "Billing Data Corruption Detected - Billing (56)" section.

DESCRIPTION

Billing Data Corruption Detected

SEVERITY

CRITICAL

THRESHOLD

100

THROTTLE

0

DATAWORDS

File/Table That May be Corrupt - STRING [32]
Low End of the Range of Records - FOUR_BYTES
High End of the Range of Records - FOUR_BYTES
Error Code - FOUR_BYTES
Location Tag - STRING [32]

PRIMARY
CAUSE

Billing data stored on disk may have become corrupted due to a power outage, ungraceful shutdown, or disk failure.

PRIMARY
ACTION

The BDMS that detected the problem should have gone out of service; leave it in the out-of-service state and contact Cisco TAC for assistance. (Contact Cisco TAC.)


Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

BILLING (57)

For additional information, refer to the "Prepaid Subscriber Call Attempt Failed Because of Balance - Billing (57)" section.

DESCRIPTION

Prepaid Subscriber Call Attempt Failed Because of Balance

SEVERITY

INFO

THRESHOLD

100

THROTTLE

0

DATAWORDS

Instance Name - STRING [65]
Calling Party - STRING [32]
Called Party - STRING [32]
Pop ID- STRING [32]

PRIMARY
CAUSE

Subscriber has run out of the balance.

PRIMARY
ACTION

Ask the subscriber to deposit money in the account.

SECONDARY
CAUSE

There may be a problem in billing info at prepaid server.

SECONDARY
ACTION

Verify billing info at the prepaid server.


BILLING (58)

To troubleshoot and correct the cause of the alarm, refer to the "Signaling Prepaid Server Inaccessible - Billing (58)" section.

DESCRIPTION

Signaling Prepaid Server Inaccessible

SEVERITY

MAJOR

THRESHOLD

100

THROTTLE

0

DATAWORDS

Instance Name - STRING [65]
Pop ID - STRING [32]
Radius Profile ID - STRING [32]

PRIMARY
CAUSE

All the prepaid servers are down.

PRIMARY
ACTION

Check the operating status of prepaid servers.

SECONDARY
CAUSE

Internet Protocol (IP) network between Cisco BTS 10200 Softswitch plain old telephone service (POTS) Feature Server (FS) and the prepaid servers is down.

SECONDARY
ACTION

Check and correct any problems in IP network.



Note BILLING (59) is not used.


BILLING (60)

To troubleshoot and correct the cause of the alarm, refer to the "Bad File Detected During Startup - Billing (60)" section.

DESCRIPTION

Bad File Detected During Startup

SEVERITY

MAJOR

THRESHOLD

100

THROTTLE

0

DATAWORDS

File-name of the Bad File - STRING [128]
Diagnosis - STRING [64]
Changed to File-name - STRING [128]

PRIMARY
CAUSE

A bad billing file was generated due to a CPU fatal, power outage, ungraceful shutdown, or disk failure.

PRIMARY
ACTION

The Billing subsysterm will isolate the bad file, rename the bad billing file, and continue to complete the initialization. The bad file is now out of the control by Billing subsystem and it will not FTP to the BMS. The bad file should either be deleted or the content of the file should be corrected and then the file should be uploaded to the BMS.


Monitoring Billing Events

This section provides the information needed to monitor and correct Billing events. Table 3-2 lists all Billing events in numerical order and provides cross reference to each subsection in this section.

Table 3-2 Cisco BTS 10200 Softswitch Billing Events 

Event Type
Event Name
Event Severity

BILLING(1)

Test Report - Billing (1)

INFO

BILLING(2)

Billing Partition Disk Usage Minor Threshold Exceeded - Billing (2)

MINOR

BILLING(3)

Billing Partition Disk Usage Major Threshold Exceeded - Billing (3)

MAJOR

BILLING(4)

Billing Partition Disk Usage Critical Threshold Exceeded - Billing (4)

CRITICAL

BILLING(5)

Billing Partition Disk Usage Within Normal Range - Billing (5)

INFO

BILLING(6)

File Transfer Protocol/Secure File Transfer Protocol Transfer Failed - Billing (6)

MAJOR

BILLING(7)

Transmission Control Protocol Connection Error - Billing (7)

CRITICAL

BILLING(8)

Transmission Control Protocol Packet Receive Failure - Billing (8)

MAJOR

BILLING(13)

Database Connection Error - Billing (13)

CRITICAL

BILLING(14)

File Open Error - Billing (14)

WARNING

BILLING(15)

File Write Error - Billing (15)

MAJOR

BILLING(29)

Call Data Block Send Failed - Billing (29)

MAJOR

BILLING(30)

Domain Name Mapping Failed - Billing (30)

MAJOR

BILLING(31)

Port not Specified - Billing (31)

MAJOR

BILLING(32)

Element Management System Address not Specified - Billing (32)

MAJOR

BILLING(33)

File Transfer Protocol/Secure File Transfer Protocol Parameters Invalid - Billing (33)

MAJOR

BILLING(35)

All Billing Links at Billing Server Down - Billing (35)

CRITICAL

BILLING(36)

Billing Link Restored - Billing (36)

INFO

BILLING(37)

Billing Link Failure - Billing (37)

MAJOR

BILLING(38)

Event Message Log File Access Error - Billing (38)

MAJOR

BILLING(40)

Event Message Encode Failure - Billing (40)

MINOR

BILLING(41)

Message Content Error - Billing (41)

MINOR

BILLING(42)

Error Reading Provisioned Data-Using Default - Billing (42)

WARNING

BILLING(44)

Record Keeping System Switch Occurred - Billing (44)

MAJOR

BILLING(45)

Event Message Log File Opened - Billing (45)

MINOR

BILLING(46)

Event Message Log File Closed - Billing (46)

MINOR

BILLING(47)

Record Keeping System Unreachable for One Hour - Billing (47)

MINOR

BILLING(48)

Record Keeping System Unreachable for Three Hours - Billing (48)

MAJOR

BILLING(49)

Record Keeping System Unreachable for Five Hours - Billing (49)

CRITICAL

BILLING(52)

Bulk Data Management System Stopped Generating New Billing File - Billing (52)

CRITICAL

BILLING(53)

Event Message Disk Space 50 Percent Full - Billing (53)

MINOR

BILLING(54)

Event Message Disk Space 70 Percent Full - Billing (54)

MAJOR

BILLING(55)

Event Message Disk Space 100 Percent Full - Billing (55)

CRITICAL

BILLING(56)

Billing Data Corruption Detected - Billing (56)

CRITICAL

BILLING(57)

Prepaid Subscriber Call Attempt Failed Because of Balance - Billing (57)

INFO

BILLING(58)

Signaling Prepaid Server Inaccessible - Billing (58)

MAJOR

BILLING(60)

Bad File Detected During Startup - Billing (60)

MAJOR


Test Report - Billing (1)

The Test Report event is used for testing the billing event category. The event is informational and no further action is required.

Billing Partition Disk Usage Minor Threshold Exceeded - Billing (2)

The Billing Partition Disk Usage Minor Threshold Exceeded alarm (minor) indicates that a billing partition disk usage minor threshold crossing has occurred. To troubleshoot and correct the cause of the Billing Partition Disk Usage Minor Threshold Exceeded alarm, refer to the "Billing Partition Disk Usage Minor Threshold Exceeded - Billing (2)" section.

Billing Partition Disk Usage Major Threshold Exceeded - Billing (3)

The Billing Partition Disk Usage Major Threshold Exceeded alarm (major) indicates that a billing partition disk usage major threshold crossing has occurred. To troubleshoot and correct the cause of the Billing Partition Disk Usage Major Threshold Exceeded alarm, refer to the "Billing Partition Disk Usage Major Threshold Exceeded - Billing (3)" section.

Billing Partition Disk Usage Critical Threshold Exceeded - Billing (4)

The Billing Partition Disk Usage Critical Threshold Exceeded alarm (critical) indicates that a billing partition disk usage critical threshold crossing has occurred. To troubleshoot and correct the cause of the Billing Partition Disk Usage Critical Threshold Exceeded alarm, refer to the "Billing Partition Disk Usage Critical Threshold Exceeded - Billing (4)" section.

Billing Partition Disk Usage Within Normal Range - Billing (5)

The Billing Partition Disk Usage Within Normal Range event is informational and no further action is required.

File Transfer Protocol/Secure File Transfer Protocol Transfer Failed - Billing (6)

The File Transfer Protocol/Secure File Transfer Protocol Transfer Failed alarm (major) indicates that the billing information FTP/SFTP transfer has failed. To troubleshoot and correct the cause of File Transfer Protocol/Secure File Transfer Protocol Transfer Failed alarm, refer to the "File Transfer Protocol/Secure File Transfer Protocol Transfer Failed - Billing (6)" section.


Note OpenSSH version 3.9p1 contains a bug that may cause billing file transfers over SFTP to fail.


Transmission Control Protocol Connection Error - Billing (7)

The Transmission Control Protocol Connection Error alarm (critical) indicates that an error has occurred on the TCP connection. To troubleshoot and correct the cause of the Transmission Control Protocol Connection Error alarm, refer to the "Transmission Control Protocol Connection Error - Billing (7)" section.

Transmission Control Protocol Packet Receive Failure - Billing (8)

The Transmission Control Protocol Packet Receive Failure alarm (major) indicates that a TCP packet receive failure has occurred. To troubleshoot and correct the cause of the Transmission Control Protocol Packet Receive Failure alarm, refer to the "Transmission Control Protocol Packet Receive Failure - Billing (8)" section.

Database Connection Error - Billing (13)

The Database Connection Error alarm (critical) indicates that a database connection error has occurred. To troubleshoot and correct the cause of the Database Connection Error alarm, refer to the "Database Connection Error - Billing (13)" section.

File Open Error - Billing (14)

The File Open Error event serves as a warning that a file open error has occurred. The primary cause of a file open error is a system malfunction. The system may be out of file descriptors. If a file open error has occurred, contact Cisco TAC to obtain technical assistance. Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

File Write Error - Billing (15)

The File Write Error alarm (major) indicates that a file write error has occurred. To troubleshoot and correct the cause of the File Write Error alarm, refer to the "File Write Error - Billing (15)" section.

Call Data Block Send Failed - Billing (29)

The Call Data Block Send Failed alarm (major) indicates that a call data block (CDB) send has failed. To troubleshoot and correct the cause of the Call Data Block Send Failed alarm, refer to the "Call Data Block Send Failed - Billing (29)" section.

Domain Name Mapping Failed - Billing (30)

The Domain Name Mapping Failed alarm (major) indicates that a domain name mapping has failed. To troubleshoot and correct the cause of the Domain Name Mapping Failed alarm, refer to the "Domain Name Mapping Failed - Billing (30)" section.

Port not Specified - Billing (31)

The Port not Specified alarm (major) indicates that a port has not been specified or configured. To troubleshoot and correct the cause of the Port not Specified alarm, refer to the "Port not Specified - Billing (31)" section.

Element Management System Address not Specified - Billing (32)

The Element Management System Address not Specified alarm (major) indicates that an EMS address has not been specified or configured. To troubleshoot and correct the cause of the Element Management System Address not Specified alarm, refer to the "Element Management System Address not Specified - Billing (32)" section.

File Transfer Protocol/Secure File Transfer Protocol Parameters Invalid - Billing (33)

The File Transfer Protocol/Secure File Transfer Protocol Parameters Invalid alarm (major) indicates that the FTP/SFTP parameters configuration is not valid or has not been fully configured. To troubleshoot and correct the cause of the File Transfer Protocol/Secure File Transfer Protocol Parameters Invalid alarm, refer to the "File Transfer Protocol/Secure File Transfer Protocol Parameters Invalid - Billing (33)" section.

All Billing Links at Billing Server Down - Billing (35)

The All Billing Links at Billing Server Down alarm (critical) indicates that all of the billing links to the billing server are down. To troubleshoot and correct the cause of the All Billing Links at Billing Server Down alarm, refer to the "All Billing Links at Billing Server Down - Billing (35)" section.

Billing Link Restored - Billing (36)

The Billing Link Restored event is informational and no further action is required. The primary cause of the Billing Link Restored event is that the cable to the billing server or the link to the billing server has been restored.

Billing Link Failure - Billing (37)

The Billing Link Failure alarm (major) indicates that a link to the billing server has failed. To troubleshoot and correct the cause of the Billing Link Failure alarm, refer to the "Billing Link Failure - Billing (37)" section.

Event Message Log File Access Error - Billing (38)

The Event Message Log File Access Error alarm (major) indicates that an event message (EM) log file access error has occurred. To troubleshoot and correct the cause of the Event Message Log File Access Error alarm, refer to the "Event Message Log File Access Error - Billing (38)" section.

Event Message Encode Failure - Billing (40)

The Event Message Encode Failure alarm (minor) indicates that an EM encode failure has occurred. To troubleshoot and correct the cause of the Event Message Encode Failure alarm, refer to the "Event Message Encode Failure - Billing (40)" section.

Message Content Error - Billing (41)

The Message Content Error alarm (minor) indicates that a message content error has occurred. To troubleshoot and correct the cause of the Message Content Error alarm, refer to the "Message Content Error - Billing (41)" section.

Error Reading Provisioned Data-Using Default - Billing (42)

The Error Reading Provisioned Data-Using Default event functions as a warning that an error occurred while reading provisioning data and that the default provisioning data and default values will be used. The primary cause of the error is that the application was unable to read provisioned data, and had to resort to using default values. Check to ensure a complete load has been installed on the Cisco BTS 10200 Softswitch system. If the load is complete and problem persists, contact Cisco TAC. Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

Record Keeping System Switch Occurred - Billing (44)

The Record Keeping System Switch Occurred alarm (major) indicates that a RKS switch has occurred. To troubleshoot and correct the cause of the Record Keeping System Switch Occurred alarm, refer to the "Record Keeping System Switch Occurred - Billing (44)" section.

Event Message Log File Opened - Billing (45)

The Event Message Log File Opened alarm (minor) indicates that a event message log file has been opened. To troubleshoot and correct the cause of the Event Message Log File Opened alarm, refer to the "Event Message Log File Opened - Billing (45)" section.

Event Message Log File Closed - Billing (46)

The Event Message Log File Closed alarm (minor) indicates that a event message log file has been closed. To troubleshoot and correct the cause of the Event Message Log File Closed alarm, refer to the "Event Message Log File Closed - Billing (46)" section.

Record Keeping System Unreachable for One Hour - Billing (47)

The Record Keeping System Unreachable for One Hour alarm (minor) indicates that the RKS servers have been unreachable for one hour. To troubleshoot and correct the cause of the Record Keeping System Unreachable for One Hour alarm, refer to the "Record Keeping System Unreachable for One Hour - Billing (47)" section.

Record Keeping System Unreachable for Three Hours - Billing (48)

The Record Keeping System Unreachable for Three Hours alarm (major) indicates that the RKS servers have been unreachable for three hours. To troubleshoot and correct the cause of the Record Keeping System Unreachable for Three Hours alarm, refer to the "Record Keeping System Unreachable for Three Hours - Billing (48)" section.

Record Keeping System Unreachable for Five Hours - Billing (49)

The Record Keeping System Unreachable for Five Hours alarm (critical) indicates that the RKS servers have been unreachable for five hours. To troubleshoot and correct the cause of the Record Keeping System Unreachable for Five Hours alarm, refer to the "Record Keeping System Unreachable for 5 Hours - Billing (49)" section.

Bulk Data Management System Stopped Generating New Billing File - Billing (52)

The Bulk Data Management System Stopped Generating New Billing File alarm (critical) indicates that the BDMS has stopped generating new billing files. To troubleshoot and correct the cause of the Bulk Data Management System Stopped Generating New Billing File alarm, refer to the "Bulk Data Management System Stopped Generating New Billing File - Billing (52)" section.

Event Message Disk Space 50 Percent Full - Billing (53)

The Event Message Disk Space 50 Percent Full alarm (minor) indicates that the event message disk space is 50 percent full. To troubleshoot and correct the cause of the Event Message Disk Space 50 Percent Full alarm, refer to the "Event Message Disk Space 50 Percent Full - Billing (53)" section.

Event Message Disk Space 70 Percent Full - Billing (54)

The Event Message Disk Space 70 Percent Full alarm (major) indicates that the event message disk space is 70 percent full. To troubleshoot and correct the cause of the Event Message Disk Space 70 Percent Full alarm, refer to the "Event Message Disk Space 70 Percent Full - Billing (54)" section.

Event Message Disk Space 100 Percent Full - Billing (55)

The Event Message Disk Space 100 Percent Full alarm (critical) indicates that the event message disk space is 100 percent full. To troubleshoot and correct the cause of the Event Message Disk Space 100 Percent Full alarm, refer to the "Event Message Disk Space 100 Percent Full - Billing (55)" section.

Billing Data Corruption Detected - Billing (56)

The Billing Data Corruption Detected alarm (critical) indicates that a billing data corruption has been detected. To troubleshoot and correct the cause of the Billing Data Corruption Detected alarm, refer to the "Billing Data Corruption Detected - Billing (56)" section.

Prepaid Subscriber Call Attempt Failed Because of Balance - Billing (57)

The Prepaid Subscriber Call Attempt Failed Because of Balance event functions as informational alert that a prepaid subscriber call attempt has failed of the subscriber account balance. The primary cause of the event is that the subscriber has an insufficient balance to place the attempted call. To correct the primary cause of the event, ask the subscriber to deposit more money in their account. Additionally, there may be a problem with the billing information on the prepaid server. To correct the secondary cause of the event, verify the billing information on the prepaid server.

Signaling Prepaid Server Inaccessible - Billing (58)

The Signaling Prepaid Server Inaccessible alarm (major) indicates that the signaling prepaid server has become inaccessible. To troubleshoot and correct the cause of the Signaling Prepaid Server Inaccessible alarm, refer to the "Signaling Prepaid Server Inaccessible - Billing (58)" section.

Bad File Detected During Startup - Billing (60)

The Bad File Detected During Startup alarm (major) indicates that a bad file was detected during system startup. To troubleshoot and correct the cause of the Bad File Detected During Startup alarm, refer to the "Bad File Detected During Startup - Billing (60)" section.

Troubleshooting Billing Alarms

This section provides the information needed to troubleshoot and correct Billing events. Table 3-3 lists all Billing events in numerical order and provides cross reference to each subsection in this section.

Table 3-3 Cisco BTS 10200 Softswitch Billing Alarms 

Alarm Type
Alarm Name
Alarm Severity

BILLING(2)

Billing Partition Disk Usage Minor Threshold Exceeded - Billing (2)

MINOR

BILLING(3)

Billing Partition Disk Usage Major Threshold Exceeded - Billing (3)

MAJOR

BILLING(4)

Billing Partition Disk Usage Critical Threshold Exceeded - Billing (4)

CRITICAL

BILLING(6)

File Transfer Protocol/Secure File Transfer Protocol Transfer Failed - Billing (6)

MAJOR

BILLING(7)

Transmission Control Protocol Connection Error - Billing (7)

CRITICAL

BILLING(8)

Transmission Control Protocol Packet Receive Failure - Billing (8)

MAJOR

BILLING(13)

Database Connection Error - Billing (13)

CRITICAL

BILLING(15)

File Write Error - Billing (15)

MAJOR

BILLING(29)

Call Data Block Send Failed - Billing (29)

MAJOR

BILLING(30)

Domain Name Mapping Failed - Billing (30)

MAJOR

BILLING(31)

Port not Specified - Billing (31)

MAJOR

BILLING(32)

Element Management System Address not Specified - Billing (32)

MAJOR

BILLING(33)

File Transfer Protocol/Secure File Transfer Protocol Parameters Invalid - Billing (33)

MAJOR

BILLING(35)

All Billing Links at Billing Server Down - Billing (35)

CRITICAL

BILLING(37)

Billing Link Failure - Billing (37)

MAJOR

BILLING(38)

Event Message Log File Access Error - Billing (38)

MAJOR

BILLING(40)

Event Message Encode Failure - Billing (40)

MINOR

BILLING(41)

Message Content Error - Billing (41)

MINOR

BILLING(44)

Record Keeping System Switch Occurred - Billing (44)

MAJOR

BILLING(45)

Event Message Log File Opened - Billing (45)

MINOR

BILLING(46)

Event Message Log File Closed - Billing (46)

MINOR

BILLING(47)

Record Keeping System Unreachable for One Hour - Billing (47)

MINOR

BILLING(48)

Record Keeping System Unreachable for Three Hours - Billing (48)

MAJOR

BILLING(49)

Record Keeping System Unreachable for 5 Hours - Billing (49)

CRITICAL

BILLING(52)

Bulk Data Management System Stopped Generating New Billing File - Billing (52)

CRITICAL

BILLING(53)

Event Message Disk Space 50 Percent Full - Billing (53)

MINOR

BILLING(54)

Event Message Disk Space 70 Percent Full - Billing (54)

MAJOR

BILLING(55)

Event Message Disk Space 100 Percent Full - Billing (55)

CRITICAL

BILLING(56)

Billing Data Corruption Detected - Billing (56)

CRITICAL

BILLING(58)

Signaling Prepaid Server Inaccessible - Billing (58)

MAJOR

BILLING(60)

Bad File Detected During Startup - Billing (60)

MAJOR


Billing Partition Disk Usage Minor Threshold Exceeded - Billing (2)

The Billing Partition Disk Usage Minor Threshold Exceeded alarm (minor) indicates that a billing partition disk usage minor threshold crossing has occurred. The primary cause of the alarm is that call detail records are accumulating on the disk associated with the billing database in the EMS. This is because data is being written into the database faster than it is being read out of the database. The minor threshold (default value = 70%) has been exceeded. Some fluctuation in disk usage is to be expected as call volume rises and falls during the day. Threshold crossings might step upward (from MINOR to MAJOR to CRITICAL) when there is a rapid increase in call volume, and then step downward (CRITICAL to MAJOR to MINOR) when call volume slows. To correct the primary cause of the alarm, monitor the alarm. The read should catch up to write within a few minutes, and the alarm should not remain active. To monitor the alarm, use the subscribe alarm-report command. To obtain a summary, use the report alarm-summary command. Verify that type=billing is entered in these commands.

Billing Partition Disk Usage Major Threshold Exceeded - Billing (3)

The Billing Partition Disk Usage Major Threshold Exceeded alarm (major) indicates that a billing partition disk usage major threshold crossing has occurred. The primary cause of the alarm is that call detail records are accumulating on the disk associated with the billing database in the EMS. This is because data is being written into the database faster than it is being read out of the database. The major threshold (default value = 80%) has been exceeded. Some fluctuation in disk usage is to be expected as call volume rises and falls during the day. Threshold crossings might step upward (from MINOR to MAJOR to CRITICAL) when there is a rapid increase in call volume, and then step downward (CRITICAL to MAJOR to MINOR) when call volume slows. To correct the primary cause of the alarm, monitor this alarm. The read should catch up to write within a few minutes, and the alarm should not remain active. To monitor the alarm, use the subscribe alarm-report command. To obtain a summary, use the report alarm-summary command. Verify that type=billing is entered in these commands. If the alarm does not clear (or step down to a reduced level) in a few minutes, contact Cisco TAC for assistance. Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

Billing Partition Disk Usage Critical Threshold Exceeded - Billing (4)

The Billing Partition Disk Usage Critical Threshold Exceeded alarm (critical) indicates that a billing partition disk usage critical threshold crossing has occurred. The primary cause of the alarm is that call detail records are accumulating on the disk associated with the billing database in the EMS. This is because data is being written into the database faster than it is being read out of the database. The major threshold (default value = 90%) has been exceeded. Some fluctuation in disk usage is to be expected as call volume rises and falls during the day. Threshold crossings might step upward (from MINOR to MAJOR to CRITICAL) when there is a rapid increase in call volume, and then step downward (CRITICAL to MAJOR to MINOR) when call volume slows. To correct the primary cause of the alarm, Monitor this alarm. The read should catch up to write within a few minutes, and the alarm should not remain active. To monitor the alarm, use the subscribe alarm-report command. To obtain a summary, use the report alarm-summary command. Verify that type=billing is entered in these commands. If the alarm does not clear (or step down to a reduced level) in a few minutes, contact Cisco TAC for assistance. Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

File Transfer Protocol/Secure File Transfer Protocol Transfer Failed - Billing (6)

The File Transfer Protocol/Secure File Transfer Protocol Transfer Failed alarm (major) indicates that the billing information FTP/SFTP transfer has failed. The primary cause of the alarm is that the Cisco BTS 10200 Softswitch is unable to connect to remote host. To correct the primary cause of the alarm, verify the remote host is reachable. Run "show billing-acct-addr" and verify that the billing-server-addr is correct. Change the billing-server-addr, if necessary, via "change billing-acct-addr". The secondary cause of the alarm is that the Cisco BTS 10200 Softswitch is unable to login to remote host. To correct the secondary cause of the alarm, use the "show billing-acct-addr" to verify that the user-name is a valid user for the host specified in billing-server-addr. If user-name is correct and the "TransferType" dataword shows "FTP", re-enter the password via "change billing-acct-addr". If user-name is correct and the "TransferType" dataword shows "SFTP", verify that SSH keys have been pre-configured for user-name on both the Cisco BTS 10200 Softswitch and the remote host. The ternary cause of the alarm is that a file transfer error occurred. To correct the ternary cause of the alarm, check the "Error" dataword to see if it gives an indication of the kind of error that occurred. It could be a file-system error on the remote host, or a communication failure between the Cisco BTS 10200 Softswitch and the remote host. The subsequent cause of the alarm is that the CDB_BILLING_SUPP is not set to "Y" in the call-agent-profile table. To correct the subsequent cause of the alarm, check and verify that the CDB_BILLING_SUPP flag is set to "Y" in the call-agent-profile table.


Note OpenSSH version 3.9p1 contains a bug that may cause billing file transfers over SFTP to fail.


Use the following information to check the datawords:

The datawords generated by the alarm are Filename (40), FTP address (40), and error (50), where:

file name—The name of file that the Softswitch is attempting to send to the billing server.

FTP address—The IP address/domain name of the billing server that the Cisco BTS 10200 Softswitch is attempting to reach.

error—One or more of the following data words can be displayed to indicate missing or incorrect information:

Login—The username for the billing server is missing or incorrect in the database, or the username does not have the privilege level to write to the specified directory on the remote billing server.

Password—The password for the billing server is missing or incorrect in the database.

Connection—The billing-server-addr (IP address/domain name of billing server) is missing or incorrect in the database, or the connection to the billing server is unavailable.

Repository—The billing-server-directory identifier is missing or incorrect in the database, or the specified directory is not available on the billing server.

The alarm indicates a failure in making the FTP connection to the remote billing server to transfer billing information from the EMS. This can happen if:

1. The FTP information has not been initialized.

2. The information in the Softswitch database does not match the remote billing server:

a. The login username (for the billing server) is missing or incorrect in the database.

b. The username is correct but does not have the privilege level to write to the specified directory.

c. The password (for the remote billing server) is missing or incorrect in the database.

d. The billing-server-addr (IP address/domain name) is missing or incorrect in the database.

e. The billing-server-directory identifier (repository) is missing or incorrect in the database.

3. The connection to the remote billing server is unavailable.

4. The specified directory is not available on the remote billing server.

Transmission Control Protocol Connection Error - Billing (7)

The Transmission Control Protocol Connection Error alarm (critical) indicates that an error has occurred on the TCP connection. The primary cause of the alarm is a system call error. To correct the primary cause of the alarm, check the address of the Billing server in the Cisco BTS 10200 Softswitch platform configuration. The secondary cause of the alarm is the Cisco BTS 10200 Softswitch is not connected to the right address. To correct the secondary cause of the alarm, call Cisco TAC for technical support. Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

Transmission Control Protocol Packet Receive Failure - Billing (8)

The Transmission Control Protocol Packet Receive Failure alarm (major) indicates that a TCP packet receive failure has occurred. The primary cause of the alarm is that the peer went down and the socket closed. To correct the primary cause of the alarm, check the status of the peer and bring it up if it is down.

Database Connection Error - Billing (13)

The Database Connection Error alarm (critical) indicates that a database connection error has occurred. The primary cause of the alarm is that the SQL server is down. To correct the primary cause of the alarm, restart SQL server. If restarting SQL server does not correct the problem and clear the alarm, contact Cisco TAC for technical support. Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

File Write Error - Billing (15)

The File Write Error alarm (major) indicates that a file write error has occurred. The primary cause of the alarm is that a system error has occurred. The Cisco BTS 10200 Softswitch system may be out of file descriptors. To correct the primary cause of the alarm, contact Cisco TAC for technical support. Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

Call Data Block Send Failed - Billing (29)

The Call Data Block Send Failed alarm (major) indicates that a CDB send has failed. The primary cause of the alarm is that a TCP send call has failed. To correct the primary cause of the alarm, check the port number and address of blg and bmg processes in he platform.cfg file. The secondary cause of the alarm is that both the EMS servers are down. To correct the secondary cause of the alarm, check if both EMS servers are down. If both EMS servers are down, bring at least one EMS server up.

Domain Name Mapping Failed - Billing (30)

The Domain Name Mapping Failed alarm (major) indicates that a domain name mapping has failed. The primary cause of the alarm is that the wrong DNS server name mapping is specified in the Cisco BTS 10200 Softswitch configuration files. To correct the primary cause of the alarm, check the opticall.cfg and platform.cfg files for the correct mapping information.

Port not Specified - Billing (31)

The Port not Specified alarm (major) indicates that a port has not been specified or configured. The primary cause of the alarm is that the port is not specified in platform.cfg file. To correct the primary cause of the alarm, check the platform.cfg file and add the argument to blg -port 15260.

Element Management System Address not Specified - Billing (32)

The Element Management System Address not Specified alarm (major) indicates that an EMS address has not been specified or configured. The primary cause of the alarm is that either the primary or secondary EMS address has not been specified in the platform.cfg file. To correct the primary cause of the alarm, check the platform.cfg file to verify the process BLG and to add the missing addresses to the file.

File Transfer Protocol/Secure File Transfer Protocol Parameters Invalid - Billing (33)

The File Transfer Protocol/Secure File Transfer Protocol Parameters Invalid alarm (major) indicates that the FTP/SFTP parameters configuration is not valid or has not been fully configured. The primary cause of the alarm is that the billing-acct-addr table is not fully provisioned with the information needed to perform file transfers. To correct the primary cause of the alarm, check billing-acct-addr fields via "show billing-acct-addr" command. For FTP file transfer, ensure billing-server-addr, billing-server-directory, user-name, and password (not displayed) are provisioned. Also ensure sftp-supp is set to "N". For SFTP file transfer, ensure billing-server-addr, billing-server-directory, and user-name are provisioned. Also ensure sftp-supp is set to "Y".

All Billing Links at Billing Server Down - Billing (35)

The All Billing Links at Billing Server Down alarm (critical) indicates that all of the billing links to the billing server are down. The primary cause of the alarm is that the cable connection to the Billing Server may have been pulled out. To correct the primary cause of the alarm, restore cable connection to the Billing Server. The secondary cause of the alarm is that an "ifconfig down" command may have been executed on the interfaces. To correct the secondary cause of the alarm, execute an "ifconfig up" command on the interfaces.

Billing Link Failure - Billing (37)

The Billing Link Failure alarm (major) indicates that a link to the billing server has failed. The primary cause of the alarm is that an interface cable may have been pulled. To correct the primary cause of the alarm, restore cable connection. The secondary cause of the alarm is that an "ifconfig down" command may have been executed on the interface. To correct the secondary cause of the alarm, execute and "ifconfig up" command on the interface.

Event Message Log File Access Error - Billing (38)

The Event Message Log File Access Error alarm (major) indicates that an EM log file access error has occurred. The primary cause of the alarm is that a system error has occurred. The Cisco BTS 10200 Softswitch system may be out of file descriptors. To correct primary cause of the alarm, contact Cisco TAC. Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request. The secondary cause of the alarm is that a system hard drive may be faulty. To verify the secondary cause of the alarm, cause the BDMS to switch over to its mate node.

Event Message Encode Failure - Billing (40)

The Event Message Encode Failure alarm (minor) indicates that an EM encode failure has occurred. The primary cause of the alarm is that there is a problem with the format of the data being sent to the RKS. To correct the primary cause of the alarm, contact Cisco TAC. Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

Message Content Error - Billing (41)

The Message Content Error alarm (minor) indicates that a message content error has occurred. The primary cause of the alarm is that there is a mismatch between what the sender populated in the message and what the receiver expects. To correct the primary cause of the alarm, contact Cisco TAC. Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

Record Keeping System Switch Occurred - Billing (44)

The Record Keeping System Switch Occurred alarm (major) indicates that a RKS switch has occurred. The primary cause of the alarm is that billing changed the destination RKS for transmitting Event Messages. The change could have been triggered by a communication problem with an RKS, or by an attempt to reestablish RKS communication. No further action is required to correct the primary cause of the alarm.

Event Message Log File Opened - Billing (45)

The Event Message Log File Opened alarm (minor) indicates that a event message log file has been opened. The primary cause of the alarm is that a log file has been created for the storage of Event Messages that cannot be transmitted to an RKS. No further action is required to correct the primary cause of the alarm.

Event Message Log File Closed - Billing (46)

The Event Message Log File Closed alarm (minor) indicates that a event message log file has been closed. The primary cause of the alarm is that an open Event Message log file has been closed. No further action is required to correct the primary cause of the alarm.

Record Keeping System Unreachable for One Hour - Billing (47)

The Record Keeping System Unreachable for One Hour alarm (minor) indicates that the RKS servers have been unreachable for one hour. The primary cause of the alarm is that billing has not been able to communicate with any RKS for the past one hour. To correct the primary cause of the alarm, check the status of the primary and secondary RKS servers and attempt to bring them into service. Verify that the radius-profile table and call-agent-profile table are provisioned such that communication with the RKS servers is possible.

Record Keeping System Unreachable for Three Hours - Billing (48)

The Record Keeping System Unreachable for Three Hours alarm (major) indicates that the RKS servers have been unreachable for three hours. The primary cause of the alarm is that billing has not been able to communicate with any RKS for the past three hours. To correct the primary cause of the alarm, check the status of the primary and secondary RKS servers and attempt to bring them into service. Verify that the radius-profile table and call-agent-profile table are provisioned such that communication with the RKS servers is possible.

Record Keeping System Unreachable for 5 Hours - Billing (49)

The Record Keeping System Unreachable for 5 Hours alarm (critical) indicates that the RKS servers have been unreachable for five hours. The primary cause of the alarm is that billing has not been able to communicate with any RKS for the past five hours. To correct the primary cause of the alarm, check the status of the primary and secondary RKS servers and attempt to bring them into service. Verify that the radius-profile table and call-agent-profile table are provisioned such that communication with the RKS servers is possible.

Bulk Data Management System Stopped Generating New Billing File - Billing (52)

The Bulk Data Management System Stopped Generating New Billing File alarm (critical) indicates that the BDMS has stopped generating new billing files. The primary cause of the alarm is that call detail records are accumulating on the disk associated with the billing files in the EMS. This is because data is being written into the billing files faster than it is being forwarded to the Billing Mediation Server. FTP to the Billing Mediation Server may not be working. The maximum disk partition for billing has been exceeded or the maximum number of files has been exceeded. To correct the primary cause of the alarm, check the Billing Mediation Server node name, user name and password specified in BILLING_ACCT_ADDR table and log files. Correct any errors to let FTP start again. If billing_server_directory = "/dev/null", primary files under billing_directory will not be forwarded or deleted automatically. In this case, files have to be manually deleted or moved out, and BDMS needs to be restarted before it will start generating new billing files.

Event Message Disk Space 50 Percent Full - Billing (53)

The Event Message Disk Space 50 Percent Full alarm (minor) indicates that the event message disk space is 50 percent full. The primary cause of the alarm is that the Event Message storage has reached 50% of the maximum storage space allowed. To correct the primary cause of the alarm, move the Event Message files out of the specified directory. Store them in another location, or discard them.

Event Message Disk Space 70 Percent Full - Billing (54)

The Event Message Disk Space 70 Percent Full alarm (major) indicates that the event message disk space is 70 percent full. The primary cause of the alarm is that the Event Message storage has reached 70% of the maximum storage space allowed. To correct the primary cause of the alarm, move the Event Message files out of the specified directory. Store them in another location, or discard them.

Event Message Disk Space 100 Percent Full - Billing (55)

The Event Message Disk Space 100 Percent Full alarm (critical) indicates that the event message disk space is 100 percent full. The primary cause of the alarm is that the Event Message storage has been completely filled. No additional Event Messages will be written to the disk until more space is made available. To correct the primary cause of the alarm, move the Event Message files out of the specified directory. Store them in another location, or discard them.

Billing Data Corruption Detected - Billing (56)

The Billing Data Corruption Detected alarm (critical) indicates that a billing data corruption has been detected. The primary cause of the alarm is that the billing data stored on disk may have become corrupted due to a power outage, ungraceful shutdown, or hard-drive failure. To correct the primary cause of the alarm, make sure that the BDMS that detected the problem has gone to the out-of-service state. Leave the BDMS in the out-of-service state and contact Cisco TAC for assistance. Refer to the "Obtaining Documentation and Submitting a Service Request" section on page liii for detailed instructions on contacting Cisco TAC and opening a service request.

Signaling Prepaid Server Inaccessible - Billing (58)

The Signaling Prepaid Server Inaccessible alarm (major) indicates that the signaling prepaid server has become inaccessible. The primary cause of the alarm is that all the prepaid servers are down. To correct the primary cause of the alarm, check the operating status of the prepaid servers. If possible, place the prepaid servers back into service. The secondary cause of the alarm is that the IP network between the Cisco BTS 10200 Softswitch POTS FS and the prepaid servers is down. To correct the secondary cause of the alarm, check and correct any problems in IP network.

Bad File Detected During Startup - Billing (60)

The Bad File Detected During Startup alarm (major) indicates that a bad file was detected during startup. The primary cause of the alarm is that a bad billing file was generated due to a CPU fatal, power outage, ungraceful shutdown, or disk failure. To correct the primary cause of the alarm, delete the bad file or correct the content of the file and upload the corrected file to the BMS system. If the bad file is not deleted or correct, it will remain on the system and it will not be automatically FTP'd to the BMS system.