CORBA Adapter Interface Specification Programmer Guide Release 4.5 - Troubleshooting [Cisco BTS 10200 Softswitch]

Table Of Contents

Troubleshooting

Cadexceptions

Modify the CORBA Network Configuration

CORBA Cannot Connect to the Cisco BTS 10200 Softswitch

CORBA and EPOM Troubleshooting Steps

CORBA/EPOM Special Character Troubleshooting: Subscriber Commands

Troubleshooting

This chapter describes basic cadexceptions, debug, and network configuration procedures.

Cadexceptions

The following basic cadexceptions can be returned. The numbers given in the sample code refer to the text in the explanation. The text is returned if the sample code is used.
Error Message    No Error
Explanation    This is a placeholder since zero is not an error.
Sample Code   public static final int EM_NONE=0;

Recommended Action    No action. Not an error.
Error Message    CIS Error
Explanation    This error is used for internal processing errors that can relate to ORB interaction or other runtime exceptions.
Sample Code   public static final int EM_ERROR=1;

Recommended Action    Retry the command. If the problem persists, restart the Name Service and CORBA Adapter as described in the section Modify the CORBA Network Configuration.
Error Message    CIS No Data
Explanation    There was no data to return from a show command. This may not be a "real" error; however, it is cleaner to throw an exception than a NULL object.
Sample Code   public static final int EM_NODATA=2;

Recommended Action    Verify that there is data for that particular Noun (id) by executing the same command using the CLI. Refer to the Cisco BTS 10200 Softswitch Command Line Interface Reference Guide for appropriate syntax.
Error Message    User Security Error
Explanation    A fault was found in the user security. This can result from an invalid login by a password or username. This can also result from some internal error to the security system that failed in validating the user identity.
Sample Code   public static final int EM_USERSEC=5;

Recommended Action    Verify that the username and password are correct and valid for access to the Cisco BTS 10200 Softswitch. Cross-check by logging into the CLI with the same username and password.
Error Message    Permission Error
Explanation    A command was attempted that failed the authorization tests for the command. The user does not have permission to execute this command.
Sample Code   public static final int EM_PERMISSION=6;

Recommended Action    Verify that the user has the appropriate command privilege to execute the request. Cross-check by executing the same command using the CLI as the same user. Refer to the Cisco BTS 10200 Softswitch Command Line Interface Reference Guide for command privilege information.
Error Message    Error Message: Block Error
Explanation    All provisioning on the switch is blocked because the Cisco BTS 10200 Softswitch is in a maintenance mode. The command may be perfectly well formed and the connection is still valid.
Sample Code   public static final int EM_BLOCK=7;

Recommended Action    Try again later.
Error Message    Linkage Error
Explanation    The linkage failed.
Sample Code   public static final int EM_LINKAGE=10;

Recommended Action    This is an installation error. Call Cisco TAC.
Error Message    Exception In Initializer Error
Explanation    The initialization method failed.
Sample Code   public static final int EM_INIT=11;

Recommended Action    Retry the command again. If the problem persists, restart the Name Service and CORBA Adapter as described in the section Modify the CORBA Network Configuration.
Error Message    Class Not Found Exception
Explanation    Class cannot be located.
Sample Code   public static final int EM_NOTFOUND=12;

Recommended Action    Verify that the CLASSPATH is set up and all jar files exist where specified in the CLASSPATH. The CLASSPATH is set up in /opt/BTScis/bin/cis3 and /opt/ems/bin/Runtime.sh on the EMS.
Error Message    Illegal Access Exception
Explanation Class or initializer is not accessible.
Sample Code public static final int EM_ACCESS=13;
Error Message    Instantiation Exception
Explanation    Thrown if this class represents an abstract class, an interface, an array class, a primitive type, or void; or if the instantiation fails for some other reason.
Sample Code   public static final int EM_INSTANCE=14;

Recommended Action    Verify that the CLASSPATH is set up and all jar files exist where specified in the CLASSPATH. The CLASSPATH is set up in /opt/BTScis/bin/cis3 and /opt/ems/bin/Runtime.sh on the EMS.
Error Message    Security Exception
Explanation    There is no permission to create a new instance.
Sample Code   public static final int EM_SECURITY=15;

Recommended Action    Verify that the CORBA adapter is running as root. Verify by performing the following command on the EMS: ps -ef | grep cis3 and not the user.
Error Message    Invalid Request Exception
Explanation    Request is not valid or cannot be initialized.
Sample Code   public static final int EM_REQUEST=16;

Recommended Action    Verify that the XML request is syntactically correct according to the Cisco BTS 10200 Softswitch standard.
Error Message    Invalid Noun Exception
Explanation    Command noun is not found or is invalid.
Sample Code   public static final int EM_NOUN=17;

Recommended Action    Verify that the Noun (id) in the request is valid and correct for the current software release of the Cisco BTS 10200 Softswitch. Cross-check by executing the same command using CLI on the EMS. Refer to the Cisco BTS 10200 Softswitch Command Line Interface Reference Guide for syntax information.
Error Message    Invalid Verb Exception
Explanation    Command verb is not found or is invalid.
Sample Code   public static final int EM_VERB=18;

Recommended Action    Verify that the Verb in the request is valid and correct for the current software release of the Cisco BTS 10200 Softswitch. Cross-check by executing the same command using the CLI on the EMS. Refer to the Cisco BTS 10200 Softswitch Command Line Interface Reference Guide for syntax information.
Error Message    SQL Exception
Explanation    Thrown if the database cannot be accessed, if the constraints are violated, if there is another table conflict, if there is a resource issue, or any other Oracle-related cause.
Sample Code   public static final int EM_DATABASE=19;

Recommended Action    Verify that the relational databases are currently running on the EMSs. Verify by using the following commands:

1) ps -ef | grep mysqld. Verify there's a process running.
2) as root, perform a nodestat and verify that the Oracle database is running.
If either are not running, refer to the Cisco BTS 10200 Operations and Maintenance Manual to restart these processes.
Error Message    Invalid Value Exception
Explanation    A parameter value exceeds the range or some other restrictions like text length, pick-list, and so forth.
Sample Code   public static final int EM_VALUE=20;

Recommended Action    Verify that all the values within the XML request conforms to the restrictions for those values. Cross-check by executing the same command with the same values using the CLI.
Error Message    Invalid Key Exception
Explanation    An invalid key or token was used to describe some data value.
Sample Code   public static final int EM_KEY=21;

Recommended Action    Verify that all keys and values are valid within the XML request. Cross-check by executing the same command with the same values using the CLI.
Error Message    Missing Parameter Exception
Explanation    One or more required parameters were not included in the command parameter data.
Sample Code   public static final int EM_PARAM=22;

Recommended Action    Verify that all mandatory keys and values are valid. Cross-check by executing the same command with the same values using the CLI.

Modify the CORBA Network Configuration

In previous releases, the operator was asked for either an IP address or hostname. In this 4.5.1 release, the CORBA Installation automatically selects VIP (Virtual IP) if the VIP is configured. Otherwise, the first Management IP address is selected. This requires adding the hostname on the client machine. The following procedure can also be used to modify the name usage in the Cisco BTS 10200 Softswitch. In most cases, it is better to change to an IP address to prevent name resolution issues in the client-side network.

Note The hostname/IP address is used as part of the interoperable object reference (IOR) that is sent to the client side. That is why the name/IP address must resolve and route to the client.

Determine the hostname or IP address that works best for the client side of the network and perform the following steps to modify the CORBA network configuration:

Step 1 Edit the /etc/inittab file and modify the names on the last two entries to match the desired configuration. For example:
ns:3:respawn:/sbin/ins3 <VIP or EMS Management IP or hostname> 
cs:3:respawn:/sbin/cis3 <VIP or EMS Management IP or hostname> <EMS Management IP #1>  
< EMS Management IP #2> 
Step 2 In the directory /opt/BTSoorb/config/. (a period (.) fully qualifies the domain directory), execute the setConfig.sh script. The XML configuration file is a bundled resource file in the OpenORB architecture. This command updates the JAR files with the modified OpenORB.xml file.

Step 3 Kill and restart the NameService (ins3) and CORBA Servant programs in the listed order.
pkill ins3
pkill cis3
The NameService (ins3) and CORBA Servant (cis3) programs restart automatically.

Step 4 Kill the Java children using the ptree and kill commands. Orphaned Java programs can cause unpredictable behavior.

CORBA Cannot Connect to the Cisco BTS 10200 Softswitch

If CORBA cannot connect to the Cisco BTS 10200 Softswitch, client-side hostname resolution may be required to match what was configured in the Element Management System (EMS).

Perform the following steps to troubleshoot:

Step 1 Determine the name used in the /etc/inittab file when invoking the entries:
ns:3:respawn:/sbin/ins3 <VIP or EMS Management IP or hostname> 
cs:3:respawn:/sbin/cis3 <VIP or EMS Management IP or hostname> <EMS Management IP #1>  
< EMS Management IP #2> 
Step 2 Add the following arguments to the turn on debug-trace command to get the debug information on the client-side. These are the Java arguments:
-Dopenorb.debug.trace=DEBUG -Dopenorb.debug.level=HIGH
Step 3 If the hostname of the EMS server does not resolve, then an exception is thrown. If a default name was used such as "priems01" or "priems_nms1", these names do not resolve in the client network. Uninstall and then reinstall the CORBA package to modify the EMS to accept IP addresses.

CORBA and EPOM Troubleshooting Steps

This section describes various procedures for troubleshooting CORBA with the Extensible Provisioning and Operations Manager (EPOM). This section requires a knowledge of Unix and Java commands.

Caution EPOM cannot be co-resident with the EMS.

Perform the following steps when a problem is encountered:

Step 1 Verify that the CORBA application is running.
ps -ef | grep cis 
Step 2 Check the cis.log for errors or clues.
more /opt/ems/log/CIS.log
Step 3 Check the EPOM logs for errors or clues.
more var/opt/CSCOepom/logs
If the problem is with the way the Cisco BTS 10200 Softswitch CORBA services are configured, perform the following steps:

Step 1 Ensure that the Cisco BTS 10200 Softswitch hostname is provided in the CORBA configuration.

a. If the hostname is provided, change it to the IP address of the Cisco BTS 10200 Softswitch and restart the CORBA services on the Cisco BTS 10200 Softswitch.

b. If the hostname is not provided, stop and reinstall the CORBA application before doing anything else.

Step 2 Check if ports 683 and 14001 are listening on the Cisco BTS 10200 Softswitch server by using the following commands:
netstat -a |grep 683 
netstat -a |grep 14001. 
Step 3 Do a tail -f on the CIS.log on the Cisco BTS 10200 Softswitch server.
tail -f /opt/ems/log/CIS.log
Note EPOM log files can be found at: /var/opt/CSCOepom/logs. Check for trace.log and localhost files on the EPOM server.

Step 4 After clicking on the Config button on the EPOM GUI, (you may need to click more than once):
netstat -a |grep 14001 and netstat -a |grep 683 
Step 5 Make sure that the username and password are the same as given when adding the Cisco BTS 10200 Softswitch inventory to EPOM. The username and password are optiuser/optiuser by default. Verify this by logging into the system using the CLI.

CORBA/EPOM Special Character Troubleshooting: Subscriber Commands

Table 5-1 shows the CORBA and EPOM responses to special characters in Subscriber commands.
Table 5-1 CORBA and EPOM Responses to Special Characters
Character

Usage Example

CORBA/EPOM Response
` (single quote)
btsadmin>change subscriber 
id=x1-6-00-00-ca-ac-ef-98_02;name=Joe'
Error Message    BtsException: 
IDL:oam.sswitch.com/cad/CadExceptions:1.0, 
Invalid parameter value. name=Joe'; 
contains one of the following invalid 
characters: ("')
Explanation The single quote is a reserved character for delimiting strings. It is used this way in caller-ID messages to the MTA. A single quote cannot be used in a name because it causes an error in the MTA that is parsing the caller-ID message.
" (double quote)
btsadmin>change subscriber 
id=x1-6-00-00-ca-ac-ef-98_02;name=Joe" 
Error Message    org.omg.CORBA.UNKNOWN: Server 
Exception: Unregistered vendor exception #0 
vmcid: 0x0 minor code: 0 completed
Explanation Invalid parameter value.
name=Joe"; contains one of the following invalid characters: ("')

The double quote is a reserved character for delimiting strings. It is used this way in caller-ID messages to the MTA. A single quote cannot be used in a name because it causes an error in the MTA that is parsing the caller-ID message.
; (semi-
colon)
Example 1:
btsadmin>change subscriber 
id=x1-6-00-00-ca-ac-ef-98_02;name=Joe;; 
Reply : Success: Transaction 914661109242987301 was processed.

btsadmin>show subscriber id=x1-6-00-00-ca-ac-ef-98_02

ID=x1-6-00-00-ca-ac-ef-98_02

CATEGORY=INDIVIDUAL

NAME=Joe

STATUS=ACTIVE

COUNTRY=USA

PRIVACY=NONE

RING_TYPE_DN1=1

TERM_ID=aaln/S1/1

MGW_ID=iad-11

PIC1=NONE

PIC2=NONE

PIC3=NONE

GRP=N

USAGE_SENS=Y

SUB_PROFILE_ID=subpf1

TERM_TYPE=TERM

IMMEDIATE_RELEASE=N

TERMINATING_IMMEDIATE_REL=N

SEND_BILLING_DN=N

SEND_BDN_AS_CPN=N

SEND_BDN_FOR_EMG=N

Example 2:
btsadmin>change subscriber 
id=x1-6-00-00-ca-ac-ef-98_02;name=Joe; 
Reply : Success: Transaction 914661178134430501 was processed.

btsadmin>show subscriber id=x1-6-00-00-ca-ac-ef-98_02

ID=x1-6-00-00-ca-ac-ef-98_02

CATEGORY=INDIVIDUAL

NAME=Joe

STATUS=ACTIVE

COUNTRY=USA

PRIVACY=NONE

RING_TYPE_DN1=1

TERM_ID=aaln/S1/1

MGW_ID=iad-11

PIC1=NONE

PIC2=NONE

PIC3=NONE

GRP=N

USAGE_SENS=Y

SUB_PROFILE_ID=subpf1

TERM_TYPE=TERM

IMMEDIATE_RELEASE=N

TERMINATING_IMMEDIATE_REL=N

SEND_BILLING_DN=N

SEND_BDN_AS_CPN=N

SEND_BDN_FOR_EMG=N

Reply : Success: Entry 1 of 1 returned.
Error Message    ID=x1-6-00-00-ca-ac-ef-98_02

CATEGORY=INDIVIDUAL

NAME=Joe;

STATUS=ACTIVE

COUNTRY=USA

PRIVACY=NONE

RING_TYPE_DN1=1

TERM_ID=aaln/S1/1

MGW_ID=iad-11

PIC1=NONE

PIC2=NONE

PIC3=NONE

GRP=N

USAGE_SENS=Y

SUB_PROFILE_ID=subpf1

TERM_TYPE=TERM

IMMEDIATE_RELEASE=N

TERMINATING_IMMEDIATE_REL=N

SEND_BILLING_DN=N

SEND_BDN_AS_CPN=N

SEND_BDN_FOR_EMG=N



Reply : Success: Entry 1 of 1 returned.
Explanation In the first example, the two semicolons delimit only two null parameters. In the second example, the last parameter is a null parameter terminated by the <enter> key. The <enter> key always terminates the last parameter in the CLI.

The fact that the semicolons terminate the command, without parameters between them, makes this possible. Placing the semicolons anywhere else causes the queue to hang.
% (percent sign)
btsadmin>change subscriber 
id=x1-6-00-00-ca-ac-ef-98_02;name=Joe%
Error Message    BtsException: 
IDL:oam.sswitch.com/cad/CadExceptions:1.0, 
Invalid parameter value. name=Joe%; Enter 
at least 1 character, but not more than 32 
characters.
Explanation The percent sign is a wildcard: for example, show subscriber id=x1-6-00% shows all subscribers whose id begins with x1-6-00. Thus the percent sign is a valid character for the subscriber noun, but is not valid when used with the add or change verbs.
- (hyphen)
btsadmin>change subscriber 
id=x1-6-00-00-ca-ac-ef-98_02;name=Joe-
Error Message    Success: Transaction 
914661231396356901 was processed.
Explanation This is a valid character but impacts the way the caller id is displayed. Using this character is not recommended.
_ (under-
score)
btsadmin>change subscriber 
id=x1-6-00-00-ca-ac-ef-98_02;name=Joe_
Error Message    Failure: NAME cannot contain an 
_ character
Explanation This is a valid character but impacts the way the caller id is displayed. Using this character is not recommended.
&
btsadmin>change subscriber 
id=x1-6-00-00-ca-ac-ef-98_02;name=Joe&
Reply : Success: Transaction 
914661254499435301 was processed.
Error Message    org.omg.CORBA.UNKNOWN: Server 
Exception: Unregistered vendor exception #0 
vmcid: 0x0 minor code: 0 completed
Explanation The ampersand is a reserved character in XML that causes this transaction to fail when sent using XML over a CORBA interface.
@#$^&*()}{|\/<>,.:[]~!

(With ampersand)
btsadmin>change subscriber 
id=x1-6-00-00-ca-ac-ef-98_02;name=Joe!@#$^&*()}
{|\/<>,.:[]~
Reply : Success: Transaction 
914661290390225701 was processed.
Error Message    org.omg.CORBA.UNKNOWN: Server 
Exception: Unregistered vendor exception #0 
vmcid: 0x0 minor code: 0 completed.
Explanation The ampersand is a reserved character in XML that causes this transaction to fail when sent using the XML over a CORBA interface.
@#$^*()}{|\/>,.:[]~! (No ampersand)
btsadmin>change subscriber 
id=x1-6-00-00-ca-ac-ef-98_02;name=Joe!@#$^*()}{
|\/<>,.:[]~
Error Message    Successful
Explanation These characters are valid.