ISDN Provisioning and Troubleshooting
Troubleshooting

Table Of Contents

Troubleshooting and Maintaining the ISDN System

Introduction

How to Troubleshoot When the ISDN Trunk Group Fails to Restore

Checking the Status of a Trunk Termination

Trunk Static State Is Set to Locally Blocked

Trunk Static State Is Set to TRNS

Trunk Static State Is Set to RBLK

Termination Status Is Faulty

Cannot Make a Call

Checking MGW Provisioning

Additional CLI Verification

Maintenance of a Call Agent Connected to an ISDN Trunk Group

D-Channel Switchover


Troubleshooting and Maintaining the ISDN System


Revised: March 12, 2009, OL-3743-37

Introduction

This chapter describes ISDN troubleshooting and maintenance for the Cisco BTS 10200 Softswitch.

How to Troubleshoot When the ISDN Trunk Group Fails to Restore

This section shows how to troubleshoot if the ISDN trunk group fails to restore. Determine if the trunk group failed to restore by entering the following command:

status trunk-grp id=nnn;

If the response shows Trunk group out-of-service (TG_OOS) for the administrative state, then enter the control command.

status trunk-grp id=100;

Reply : Success: Entry 1 of 1 returned.

TGN_ID -100
RESULT -ADM configure result in success
REASON -ADM executed successful
ADMIN_STATE -ADMIN_OOS          <================
OPER_STATE -Trunk group out-of-service
PRIMARY_OPER_STATE -NOT USED
BACKUP_OPER_STATE -NOT USED

control trunk-grp id=nnn;mode=forced; target-state=ins;

If the response does not show an operational state of Trunk group in-service for the trunk group (NFAS or D-channel backup should also show ISDN_DCHAN_STATE_INS for the D-channel operational status), then:

If the response is Trunk group out-of-service, go to Step 1.

If the response is Trunk group restore session fail normal, switchover, or maint, go to Step 2.

If the response is Trunk group restore establish fail normal, switchover, or maint, go to Step 3.

If the response is ISDN_DCHAN_STATE_WAIT or ISDN_DCHAN_STATE_MB, go to Step 4.

If the response is Trunk group down session set fail hard normal, or maint, go to Step 5.

If the response is Trunk group down establish fail hard normal, or maint, go to Step 6.

If the response is Trunk group delete graceful, go to Step 7.


Step 1 Trunk group state for FAS: Trunk group out-of-service, or NFAS: NFAS DCHAN_STATE_MOOS.

This means the administrative state was set to in-service, but the command to go to operational state was ignored due to bad provisioning.

Things to Check:

Make sure that entries exist for this trunk group in the ISDN D Channel table.

Step 2 Trunk group states for FAS: Trunk group restore session fail normal, switchover, or maint; or NFAS: NFAS ISDN_DCHAN_STATE_SDN.

This means the Call Agent and the media gateway are not communicating. The backhaul session is not coming up. The problem has to do with the UDP port numbers, IP addresses or DNS names, or network connectivity. These values are all provisioned in the RUDP Backhaul Session table in the Call Agent. For Cisco IOS gateways, these values are part of the session group groupn.....

Things to Check:

a. Do the four entries in the RUDP Backhaul Session table of the Call Agent match the values in the gateway (for Cisco IOS gateways, these values are part of session group groupn...)?

b. If the call-agent-tsap-addr, or the mgw-tsap-addr are DNS names, does nslookup on both Call Agents resolve the values to one unique IP address?

c. Can the call-agent-tsap-addr in the four entries be pinged from the media gateway? Do the Call Agents actually have the IP addresses, or DNS names, specified?

d. Can the mgw-tsap-addr be pinged from both Call Agents?

e. Does the trunk-grp isdn-profile exist (show isdn-tg-profile id=%)? Keep in mind that the profile ID is case sensitive.

f. Snoop can be used to see if the gateway and Call Agent are talking in order to establish the session. Snoop can be used to eliminate whether the problem is on the Call Agent or the media gateway side.

g. The show backhaul session all command on the media gateway can be useful in debugging.

h. Look at the trace.log on the Call Agent. Search for BSM_create_ss or bsm_create_ss. This shows the exact values used by the Call Agent.


Note Backhaul deletion is triggered by all trunk groups of a backhaul-set being in the admin-oss state. Backhaul creation is triggered by controlling one of the trunk groups to INS. The trunk group list can be determined by using the show isdn-dchan set-id command.


Step 3 Trunk group states for FAS: Trunk group restore establish fail normal, switchover, or maint or NFAS: ISDN_DCHAN_STATE_OOS.

This means the specific ISDN channel could not be restored. This is likely related to ISDN layer1 or layer2 not being up, or related to DCHAN-SLOT and DCHAN-PORT in the ISDN D Channel table. The media gateway and Call Agent are communicating successfully (session came up).

Things to Check:

a. On the media gateway, enter a show isdn status command. Is Layer 1 shown as ACTIVE? If not, there may be a cable problem, or the PBX is down.

b. Does the show isdn status command return a Network side configuration? The PBX should be set up for the user side. The gateway should be setup for the network side.

c. Was the media gateway reloaded (rebooted) after setting up parameters? There can be problems when this has not been done. This is usually reflected by having two TEI values for the T1.

d. Are the DCHAN-SLOT and DCHAN-PORT provisioned correctly in the IDSN D Channel table for this trunk group?

e. Does the T1 on the media gateway show an alarm (red or yellow lights)? These are more clues that there is an ISDN layer1 or layer2 problem.

f. Is the controller T1, or interface Serialn:23, provisioned correctly on the gateway?

g. Look at the trace.log on the Call Agent. Search for BSM_dchan_tbl_insert or bsm_dchan_tbl_insert. The slot-port value is the actual value retrieved from the database. For the AS5300, this value should be 0, 1, 2, or 3. This is a decimal value. The top 16 bits are the slot and the bottom 16 bits are the port.

Step 4 Trunk group states for NFAS: ISDN_DCHAN_STATE_WAIT/ISDN_DCHAN_STATE_MB.

This means the states at the far end are sending a SERVICE_ACK to the Cisco BTS 10200 Softswitch service message for PRI.

Things to Check:

Make sure the far end is sending a SERVICE_ACK. There are some other trunk group states that happen after the ISDN backhaul is successfully established.

Step 5 Trunk group states for FAS: Trunk group down session set fail hard normal, or maint or NFAS: ISDN_DCHAN_STATE_SDN.

This is the same as Trunk group restore establish request fail, or ISDN DChannel State SDN. However, in this case, the D channel was already successfully established, and then went out. This likely means that the Call Agent and media gateway have lost IP connectivity. Or, the media gateway has lost power. This can also happen if the provisioning was changed, particularly the session settings in the media gateway.

Step 6 Trunk group states for FAS: Trunk group down establish fail hard normal, or maint or NFAS: NFAS ISDN_DCHAN_STATE_OOS.

This is the same as Trunk group restore establish request fail, or ISDN DChannel State OOS. However, in this case, the D channel was already successfully established, and then went out.

Things to Check:

a. Is, or was, the cable unplugged between the PBX and the gateway?

b. Was the provisioning changed on the gateway?

Step 7 Trunk group state for FAS: Trunk group delete graceful.

The following command indicates you are waiting for calls to finish before going to out-of-service. If the wait time is too long, you can control the trunk group with mode=forced.

control trunk-grp id=xxxx mode=graceful target-state=oos

Checking the Status of a Trunk Termination

Check the status of a trunk termination using the following command:

status trunk-termination tgn-id=<trunk group number;cic=<cic number;

Reply : Success:

RESULT -ADM configure result in success
REASON -ADM executed successful
TGN ID -2
CIC -8
TERM ADMIN STATE -ADMIN_INS
TERM OPER STATE -Termination is idle
TERM REASON -No fault reason available
TRUNK STATIC STATE -ACTV
TRUNK DYNAMIC STATE -TRNS
TRUNK REASON -NON_FAULTY

The above status shows that the trunk is active, in idle mode, and ready for call use.

If the trunk static state is set to locally blocked, go to the "Trunk Static State Is Set to Locally Blocked" section.

If the trunk static state is set to TRNS, go to the "Trunk Static State Is Set to TRNS" section.

If the trunk static state is set to remotely blocked, go to the "Trunk Static State Is Set to RBLK" section.

If the trunk static state is set to TERM_STATUS_FAULTY, go to the "Termination Status Is Faulty" section.

If the trunk static state returns Cannot Make Call, go to the "Cannot Make a Call" section.

Trunk Static State Is Set to Locally Blocked

Perform the following steps to check the TRUNK REASON field if the TRUNK STATIC STATE is set to locally blocked (LBLK) state.


Step 1 If the TRUNK REASON is set to MAINT-OOS, then a user has manually taken the trunks out of service. The user must manually put the trunks into the in-service mode.

Use the control command on the specified trunk to put the trunk back into the in-service mode.

Clear the MAINT-OOS. Trunks should be in active or idle mode and ready for use.

Step 2 If the TRUNK REASON is set to SIGNALLING-FAULT, then the D channels are not in-service. Check the trunk group ADMIN and OPER states.

If the trunk group ADMIN state is ADMIN_OOS, use the control command to put the trunk group into in-service mode.

If the trunk group ADMIN state is ADMIN _INS, verify that the trunk group OPER-STATE is IN-SERVICE (see the How to Troubleshoot When the ISDN Trunk Group Fails to Restore section in case of any issues).

Clear any SIGNALLING-FAULT states. Trunks must be in active or idle mode and ready for use.

Step 3 If TRUNK REASON is set to TERM_FAULT, then the terminations are in FAULTY states. Verify the IP connectivity from the CA to the gateway.

a. Clear any termination faults at the gateway by turning off, then turning on, MGCP (use the mgcp, no mgcp procedure on the media gateway).

b. Clear any TERM_FAULT on the Cisco BTS 10200 Softswitch. Trunks must be in active or idle mode and ready for use.


Trunk Static State Is Set to TRNS

Perform the following steps when trunk static state is set to TRNS.


Note If trunk static state is set to TRNS, then the PBX has not responded with a SERVICE_ACK or RESTART_ACK for the service or restart message set by the Call Agent.



Step 1 Verify that the initialization procedure, as depicted by the associated isdn-tg-profile, and as used by PBX, is synchronized.

Step 2 Verify that the isdn-service-supp token is set to N if PBX does not support service messages.


Trunk Static State Is Set to RBLK

Perform the following steps if trunk static state is set to RBLK.


Note If the trunk static state is set to RBLK, and isdn-tg-profile has isdn-farend-init=Y then the PBX must start the initialization procedure, or the PBX specifically sends a service message with SERVICE-OOS mode to the Cisco BTS 10200 Softswitch to put the trunks into the remote out-of-state mode.



Step 1 Verify that the initialization procedure (ISDN-FAREND-INIT token), as depicted by associated isdn-tg-profile in the Cisco BTS 10200 Softswitch, and as used by the PBX, is in sync.

Step 2 Using the CLI command "control trunk-termination" to reset the RBLK state, perform a control trunk-termination to OOS, then control trunk-termination to INS—in case there is an error on the PBX side. Also, set the isdn-service-supp token in the ISDN trunk group profile per the procedures supported by the PBX.


Termination Status Is Faulty

The TERM_STATUS_FAULTY message indicates an MGCP problem.

Things to Check:

Is the MGW profile set up for the trunk?

Is the MGW in the in-service operational state?

Perform the "no mgcp, mgcp" commands on the gateway if an MGCP problem is indicated.

Cannot Make a Call

If you have a problem making a call, perform the following Things to Check, and then go to the "Checking MGW Provisioning" section

Things to Check:

When terminations are added, they have strings like S0/DS1-2/1. The S# refers to the slot number. The DS1-# refers to the port number. The /# at the end refers to the specific B channel (1-23). The slot and port numbers must match the dchan values (dchan-slot, dchan-port) for the related trunk group ID (tgn-id), assuming that the DS0 is on a T1 that contains a D channel.

Checking MGW Provisioning

The following example shows how a Cisco IOS gateway is provisioned.


Note The typical problem areas are in bold type. Look at the problem area after the Call Agent is provisioned.


Current configuration:
!
================
version 12.2
no service pad
service timestamps debug datetime msec
service timestamps log datetime localtime
no service password-encryption
!
hostname c2421.200
!
enable password xxxxxxx
!
clock timezone CDT -6
network-clock base-rate 56k
network-clock-select 3 T1 0
network-clock-select 1 T1 1
network-clock-select 2 system (SCB)
ip subnet-zero
!
!
ip domain-name ipclab.cisco.com
ip name-server 10.89.224.1
!
ip audit notify log
ip audit po max-events 100
backhaul-session-manager 
  set isdn1 client ft      
  group group1 set isdn1   
  group group2 set isdn1   
  session group group1 10.89.225.223 9000 10.89.227.200 9000 1  
  session group group1 10.89.226.223 9001 10.89.227.200 9001 2  
  session group group2 10.89.225.224 9000 10.89.227.200 9000 1  
  session group group2 10.89.226.224 9001 10.89.227.200 9001 2  
isdn switch-type primary-ni
isdn voice-call-failure 0
!
!
!
!
!
!
!
no voice confirmation-tone
voice-card 0
!
controller T1 0
 framing esf
 linecode b8zs
 channel-group 0 timeslots 1-24 speed 64
!
controller T1 1
 framing esf
 linecode b8zs
 pri-group timeslots 1-24 service mgcp
!
!
!
!
interface Ethernet0
 ip address 10.89.227.200 255.255.255.0
 no cdp enable
!
interface Serial0
 no ip address
 shutdown
!
interface Serial0:0
 no ip address
 ip nat outside
 encapsulation ppp
 shutdown
 no cdp enable
!
interface Serial1:23
 bandwidth 64000
 no ip address
 no logging event link-status
 isdn switch-type primary-ni
 isdn protocol-emulate network    
 isdn incoming-voice voice
 isdn bind-l3 backhaul isdn1 
 no cdp enable
!
ip classless
ip route 0.0.0.0 0.0.0.0 10.89.227.254
no ip http server
ip pim bidir-enable
!
!
access-list 1 permit 10.0.24.0 0.0.0.255
call rsvp-sync
!
voice-port 1:23
!
mgcp
mgcp call-agent mgcp-SYS01CA.ipclab.cisco.com service-type mgcp version 1.0
mgcp dtmf-relay voip codec all mode nte-gw
mgcp package-capability rtp-package
mgcp default-package dt-package
no mgcp timer receive-rtcp
!
mgcp profile default
 timeout tsmax 100
 max2 retries 3
!
dial-peer cor custom
!
!
!
dial-peer voice 1 pots
 application mgcpapp
 port 1:23
!
!
line con 0
line aux 0
line 2 3
line vty 0 4
 password xxxxxxxx
 login
!
ntp server 10.89.227.254
end

Additional CLI Verification

Perform the following steps to provide additional CLI verification of a problem:


Step 1 Use the control command to put the trunk group out of service (OOS) and back into service (INS):

Place OOS:

control tgn-id=17; target-state=oos; mode=graceful;

Place INS:

control tgn-id=17; target-state=ins; mode=forced;

Step 2 If the B channel is not IDLE, use the control command to put each circuit OOS and back INS:

Place OOS:

control trunk-termination tgn-id=17; cic=1; target-state=oos; mode=forced;

Place INS:

control trunk-termination tgn-id=17; cic=1; target-state=ins; mode=forced;

Step 3 Check the status of the trunk group:

status trunk-grp id=<TG ID number; call-agent-id=<CA ID;

If the trunk group status has still not changed to ADMIN_INS—even though the commands in Steps 1 and 2 were successful—then it means that communication to the MGW has been lost. Notify your system administrator and have the problem corrected before continuing.

Step 4 Check trunk termination and trunk status. The single command shown here queries both the trunk-termination status and trunk (channel) status:

status trunk-termination tgn-id=17; cic=all; 

Reply : Request was successful.

REPLY=1status is ....
CONFIGURATION COMMAND EXECUTED TRUNK_GRP -17 -CIC -1
TERM ADMINstatus -ADMIN_INS
TERM OPERstatus -TERM_STATE_IDLE
CIC STATIC STATE -ACTV
CIC DYNAMIC STATE -IDLE

...
...

23status is ....
CONFIGURATION COMMAND EXECUTED TRUNK_GRP -17 -CIC -23
TERM ADMINstatus -ADMIN_INS
TERM OPERstatus -TERM_STATE_IDLE
CIC STATIC STATE -ACTV
CIC DYNAMIC STATE -IDLE

status trunk-termination tgn-id=17; cic=23


Reply : Request was successful.

REPLY=CONFIGURATION COMMAND EXECUTED ISDN_TRUNK_GROUP -17 -CIC -23
TERM ADMINstatus -ADMIN_INS
TERM OPERstatus -TERM_STATE_IDLE
CIC STATIC STATE -ACTV
CIC DYNAMIC STATE -IDLE

Maintenance of a Call Agent Connected to an ISDN Trunk Group

During system operations, the operator can use the CLI control command to switch the Call Agent into one of three states:

NORMAL

FORCED-ACTIVE-STANDBY

FORCED-STANDBY-ACTIVE

When the control command is entered on a Call Agent connected to an ISDN trunk group (D channel), the Call Agent switchover time is approximately 20 seconds. During all but the first 6 seconds of this switchover time, the ISDN D channel is temporarily down. However, the D channel comes back up, and the ISDN trunk group automatically returns to in-service (INS), when the switchover completes.


Note This switchover process does not have any impact on stable calls.


D-Channel Switchover

This section describes the control command for ISDN switchover. For ISDN status, use the status trunk group command described in Trunk Group Status and Control Commands, Chapter 17, in the Cisco BTS 10200 Softswitch Command Line Interface Reference Guide. This command is only applicable to NFAS. It switches the D channel that is active to standby, and the D channel that is standby to active.

Command Types

Control

Examples

control isdn-dchan tgn-id=1; 

Reply Example:

Reply : Success

Syntax Description

The following token may be used with the ISDN D-Channel switchover control command:

WAIT (Release 4.5)—Specifies a waiting period so that previously issued provisioning commands (for example, an add command) can complete before a status, control, or equip command executes. CHAR(1): Y/N (Default = N).

Y—System checks whether there are any pending provisioning requests in the transaction queue issued before the status, control or equip command. The provisioning commands are allowed to execute before the status, control, or equip command is executed.

N—System does not check whether there are any pending provisioning requests in the transaction queue issued before the status, control or equip command. All commands execute according to their order in the transaction queue.