Cisco BTS 10200 Softswitch CORBA Adapter Interface Specification Programmer s Guide, Release 7.0

Troubleshooting

This chapter describes basic cadexceptions, debug actions, and network configuration procedures.

Cadexceptions

The following basic cadexceptions can be returned. The numbers given in the sample code refer to the text in the explanation. The text is returned if the sample code is used.

Error Message    No Error

Explanation    This is a placeholder since 0 (zero) is not an error.

        Sample Code

public static final int EM_NONE=0;

Recommended Action    No action. Not an error.

Error Message    CIS Error

Explanation    This error is used for internal processing errors that can be related to ORB interaction or other runtime exceptions.

        Sample Code

public static final int EM_ERROR=1;

Recommended Action    Retry the command. If the problem persists, restart the Name Service and CORBA Adapter as described in the "Modify the CORBA Network Configuration" section.

Error Message    CIS No Data

Explanation    There was no data to return from a show command. This might not be a "real" error; however, it is cleaner to throw an exception than a NULL object.

         Sample Code

public static final int EM_NODATA=2;

Recommended Action    Verify that there is data for this particular Noun (id) by executing the same command using the CLI. Refer to the Cisco BTS 10200 Softswitch CLI Database for appropriate syntax.

Error Message    User Security Error

Explanation    A fault was found in the user security. This can result from an invalid login (password or username). This can also result from some internal error in the security system that caused a failure during an attempt to validate user identity.

        Sample Code

public static final int EM_USERSEC=5;

Recommended Action    Verify that the username and password are correct and valid for access to the BTS 10200. Cross-check by logging in to the CLI with the same username and password.

Error Message    Permission Error

Explanation    A command was attempted that failed the authorization tests for the command. The user does not have permission to execute this command.

        Sample Code

public static final int EM_PERMISSION=6;

Recommended Action    Verify that the user has the appropriate command privilege to execute the request. Cross-check by executing the same command using the CLI and the same user. Refer to the Cisco BTS 10200 Softswitch CLI Database for command privilege information.

Error Message    Error Message: Block Error

Explanation    All provisioning on the switch is blocked because the BTS 10200 is in a maintenance mode. The command might be perfectly well formed and the connection valid.

        Sample Code

public static final int EM_BLOCK=7;

Recommended Action    Try again later.

Error Message    Linkage Error

Explanation    The linkage failed.

         Sample Code

public static final int EM_LINKAGE=10;

Recommended Action    This is an installation error. Call Cisco TAC.

Error Message    Exception In Initializer Error

Explanation    The initialization failed.

        Sample Code

public static final int EM_INIT=11;

Recommended Action    Retry the command again. If the problem persists, restart the Name Service and CORBA Adapter as described in the "Modify the CORBA Network Configuration" section.

Error Message    Class Not Found Exception

Explanation    Class cannot be located

        Sample Code

public static final int EM_NOTFOUND=12;

Recommended Action    Verify that the CLASSPATH is set up and all jar files exist where specified in the CLASSPATH. The CLASSPATH is set up in /opt/BTScis/bin/cis3 and /opt/ems/bin/Runtime.sh on the EMS.

Error Message    Illegal Access Exception

Explanation    Class or initializer is not accessible.

        Sample Code

public static final int EM_ACCESS=13;

Error Message    Instantiation Exception

Explanation    Thrown if this class represents an abstract class, an interface, an array class, a primitive type, or void; or if the instantiation fails for some other reason.

        Sample Code

public static final int EM_INSTANCE=14;

Recommended Action    Verify that the CLASSPATH is set up and all jar files exist where specified in the CLASSPATH. The CLASSPATH is set up in /opt/BTScis/bin/cis3 and /opt/ems/bin/Runtime.sh on the EMS.

Error Message    Security Exception

Explanation    There is no permission to create a new instance.

        Sample Code

public static final int EM_SECURITY=15;

Recommended Action    Verify that the CORBA adapter is running as root. Verify by performing the following command on the EMS: ps -ef | grep cis3 and not the user.

Error Message    Invalid Request Exception

Explanation    Request is not valid or cannot be initialized.

        Sample Code

public static final int EM_REQUEST=16;

Recommended Action    Verify that the XML request is syntactically correct according to the BTS 10200 standard.

Error Message    Invalid Noun Exception

Explanation    Command noun is not found or is invalid.

        Sample Code

public static final int EM_NOUN=17;

Recommended Action    Verify that the Noun (id) in the request is valid and correct for the current software release of the BTS 10200. Cross-check by executing the same command using CLI on the EMS. Refer to the Cisco BTS 10200 Softswitch CLI Database for syntax information.

Error Message    Invalid Verb Exception

Explanation    Command verb is not found or is invalid.

        Sample Code

public static final int EM_VERB=18;

Recommended Action    Verify that the Verb in the request is valid and correct for the current software release of the BTS 10200. Cross-check by executing the same command using the CLI on the EMS. Refer to the Cisco BTS 10200 Softswitch CLI Database for syntax information.

Error Message    SQL Exception

Explanation    Thrown if the database cannot be accessed, if the constraints are violated, if there is a conflict with another table, if there is a resource issue, or any other Oracle-related cause.

        Sample Code

public static finel int EM_DATABASE=19;

Recommended Action    Verify that the relational databases are currently running on the EMSs. Verify by using the following commands:

1. ps -ef | grep mysqld. Verify there's a process running.

2. As root, perform a nodestat and verify that the Oracle database is running.

If either are not running, refer to the Cisco BTS 10200 Operations and Maintenance Manual to restart these processes.

Error Message    Invalid Value Exception

Explanation    A parameter value exceeds the range or some other restriction like text length or pick-list.

        Sample Code

public static final int EM_VALUE=20;

Recommended Action    Verify that all the values within the XML request conform to the restrictions for those values. Cross-check by executing the same command with the same values using the CLI.

Error Message    Invalid Key Exception

Explanation    An invalid key or token was used to describe some data value.

        Sample Code

public static final int EM_KEY=21;

Recommended Action    Verify that all keys and values within the XML request are valid. Cross-check by executing the same command with the same values using the CLI.

Error Message    Missing Parameter Exception

Explanation    One or more required parameters were not included in the command parameter data.

        Sample Code

public static final int EM_PARAM=22;

Recommended Action    Verify that all mandatory keys and values are valid. Cross-check by executing the same command with the same values using the CLI.

Error Message    Session in Use

Explanation    Indicates the user session is valid but busy.

        Sample Code

public static final EM_SESS_INUSE = 23;

Recommended Action    Try again later, as a command corresponding to this user session is already executing in BTS.

Error Message    Session invalid

Explanation    Indicates that the user is no longer logged into the system or the session key is not valid.

        Sample Code

public static final EM_SESS_INVALID = 24;

Recommended Action    Create a new session and then proceed with command execution using new session key.

Modify the CORBA Network Configuration

In previous releases, the operator was asked for either an IP address or hostname. In this release, the CORBA Installation automatically selects VIP (Virtual IP) if the VIP is configured. Otherwise, the first Management IP address is selected. This requires adding the hostname on the client machine. The following procedure can also be used to modify the name usage in the BTS 10200. In most cases, it is better to change to an IP address to prevent name resolution issues in the client-side network.

Note The hostname/IP address is used as part of the interoperable object reference (IOR) that is sent to the client side. That is why the name/IP address must resolve and route to the client.

Determine the hostname or IP address that works best for the client side of the network and perform the following steps to modify the CORBA network configuration:

Step 1 Edit the /opt/BTSsmg/bin/jpm file and modify the following entries to match the desired configuration. For example:

/sbin/ins3 <VIP or EMS Management IP or hostname> /sbin/ins3

/sbin/cis3_IIOP <VIP or EMS Management IP or hostname> <EMS Management IP #1>  
< EMS Management IP #2> 2>&1 &

/sbin/cis3_SSLIOP <VIP or EMS Management IP or hostname> <EMS Management IP #1>  
< EMS Management IP #2> 2>&1 &

Step 2 In the directory /opt/BTSoorb/config/. (a period (.) fully qualifies the domain directory), execute the setConfig.sh script. The XML configuration file is a bundled resource file in the OpenORB architecture. This command updates the JAR files with the modified OpenORB.xml file.

Step 3 Get the process ID and kill the NameService program using the following commands:

ps -eaf | grep D_INS

kill -KILL <pid>

The NameService (INS) and CORBA Servant (CIS_IIOP and CIS_SSLIOP) programs restart automatically.

Step 4 Terminate the Java children using the ptree and kill commands. Orphaned Java programs can cause unpredictable behavior.

CORBA Cannot Connect to the BTS 10200

If CORBA cannot connect to the BTS 10200, client-side hostname resolution might be required to match what was configured in the Element Management System (EMS).

Perform the following steps to troubleshoot:

Step 1 Verify that the IP/hostname used by the client (for CORBA connection) matches the IP/hostname in the /opt/BTSsmg/bin/jpm file:

/sbin/ins3 <VIP or EMS Management IP or hostname> /sbin/ins3

/sbin/cis3_IIOP <VIP or EMS Management IP or hostname> <EMS Management IP #1>  
< EMS Management IP #2> 2>&1 &

/sbin/cis3_SSLIOP <VIP or EMS Management IP or hostname> <EMS Management IP #1>  
< EMS Management IP #2> 2>&1 &

Step 2 Add the following arguments to turn on debug-trace command to get the debug information on the client-side. These are the Java arguments:

-Dopenorb.debug.trace=DEBUG -Dopenorb.debug.level=HIGH

Step 3 If the hostname of the EMS server does not resolve, an exception is thrown. If a default name was used such as "priems01" or "priems_nms1", these names do not resolve in the client network. Uninstall and then reinstall the CORBA package to modify the EMS to accept IP addresses.

CORBA and EPOM Troubleshooting Steps

This section describes various procedures for troubleshooting CORBA with the Extensible Provisioning and Operations Manager (EPOM). This section requires a knowledge of UNIX and Java commands.

Caution EPOM cannot be co-resident with the EMS.

Perform the following steps when a problem is encountered:

Step 1 Verify that the CORBA application is running.

•For non-secure CORBA, use the following command:

ps -eaf | grep CIS_IIOP 

•For secure CORBA, use the following command:

ps -eaf | grep CIS_SSLIOP 

Step 2 Check the logs for errors or clues.

•For non-secure CORBA, use the following command:

more /opt/ems/log/CIS.log 

•For secure CORBA, use the following command:

more /opt/ems/log/CIS-SSL.log 

Step 3 Check the EPOM logs for errors or clues.

more var/opt/CSCOepom/logs

If you determine that the problem pertains to how the BTS 10200 CORBA services are configured, perform the following steps:

Step 1 Ensure that the BTS 10200 hostname is provided in the CORBA configuration.

a. If the hostname is provided, change it to the IP address of the BTS 10200 and restart the CORBA services on the BTS 10200.

b. If the hostname is not provided, stop and reinstall the CORBA application before doing anything else.

Step 2 Check if ports 683, 684, and 14001 are listening on the BTS 10200 server by using the following commands:

netstat -a |grep 683 
netstat -a |grep 684 
netstat -a |grep 14001 

Note Port 683 is used for non-secure CORBA communication, and port 684 is used for secure CORBA communication.

Step 3 Do a tail -f on the CIS.log on the BTS 10200 server.

tail -f /opt/ems/log/CIS.log

Note EPOM log files can be found at /var/opt/CSCOepom/logs. Check for trace.log and localhost files on the EPOM server.

Step 4 After clicking on the Config button on the EPOM GUI (you might need to click more than once):

netstat -a |grep 14001  
netstat -a |grep 683  
netstat -a |grep 684 

Step 5 Ensure that the username and password are the same asthose you provided when you added the BTS 10200 inventory to EPOM. The username and password are optiuser/optiuser by default. Verify this by logging in to the system using the CLI.

CORBA/EPOM Special Character Troubleshooting: Subscriber Commands

Table 5-1 shows the CORBA and EPOM responses to special characters in Subscriber commands.

btsadmin>change subscriber 
id=x1-6-00-00-ca-ac-ef-98_02;name=Joe'

Error Message    BtsException: 
IDL:oam.sswitch.com/cad/CadExceptions:1.0, 
Invalid parameter value. name=Joe'; 
contains one of the following invalid 
characters: ("')

btsadmin>change subscriber 
id=x1-6-00-00-ca-ac-ef-98_02;name=Joe" 

Error Message    org.omg.CORBA.UNKNOWN: Server 
Exception: Unregistered vendor exception #0 
vmcid: 0x0 minor code: 0 completed

btsadmin>change subscriber 
id=x1-6-00-00-ca-ac-ef-98_02;name=Joe;; 

btsadmin>change subscriber 
id=x1-6-00-00-ca-ac-ef-98_02;name=Joe; 

Error Message    ID=x1-6-00-00-ca-ac-ef-98_02

CATEGORY=INDIVIDUAL

NAME=Joe;

STATUS=ACTIVE

COUNTRY=USA

PRIVACY=NONE

RING_TYPE_DN1=1

TERM_ID=aaln/S1/1

MGW_ID=iad-11

PIC1=NONE

PIC2=NONE

PIC3=NONE

GRP=N

USAGE_SENS=Y

SUB_PROFILE_ID=subpf1

TERM_TYPE=TERM

IMMEDIATE_RELEASE=N

TERMINATING_IMMEDIATE_REL=N

SEND_BILLING_DN=N

SEND_BDN_AS_CPN=N

SEND_BDN_FOR_EMG=N



Reply : Success: Entry 1 of 1 returned.

btsadmin>change subscriber 
id=x1-6-00-00-ca-ac-ef-98_02;name=Joe%

Error Message    BtsException: 
IDL:oam.sswitch.com/cad/CadExceptions:1.0, 
Invalid parameter value. name=Joe%; Enter 
at least 1 character, but not more than 32 
characters.

btsadmin>change subscriber 
id=x1-6-00-00-ca-ac-ef-98_02;name=Joe-

Error Message    Success: Transaction 
914661231396356901 was processed.

btsadmin>change subscriber 
id=x1-6-00-00-ca-ac-ef-98_02;name=Joe_

Error Message    Failure: NAME cannot contain an 
_ character

btsadmin>change subscriber 
id=x1-6-00-00-ca-ac-ef-98_02;name=Joe&

Reply : Success: Transaction 
914661254499435301 was processed.

Error Message    org.omg.CORBA.UNKNOWN: Server 
Exception: Unregistered vendor exception #0 
vmcid: 0x0 minor code: 0 completed

btsadmin>change subscriber 
id=x1-6-00-00-ca-ac-ef-98_02;name=Joe!@#$^&*()}
{|\/<>,.:[]~

Reply : Success: Transaction 
914661290390225701 was processed.

Error Message    org.omg.CORBA.UNKNOWN: Server 
Exception: Unregistered vendor exception #0 
vmcid: 0x0 minor code: 0 completed.

btsadmin>change subscriber 
id=x1-6-00-00-ca-ac-ef-98_02;name=Joe!@#$^*()}{
|\/<>,.:[]~

Error Message    Successful

CORBA Operations Support System Events

This section provides the information that you need to monitor and correct Operations Support System events. Table 8-2 lists all Operations Support System events in numerical order and provides a cross reference to each subsection in this section.

User Session Count Is Approaching Threshold Limit (OSS 12)

The "User Session Count is approaching Threshold Limit" warning event indicates that the number of CORBA login sessions is over 90 percent of the maximum number of logins allowed. Remove idle or long duration sessions using the stop client-session command to reduce the number of active sessions. You should also check the number of users in the /opt/BTScis/etc/bts.properties file and set it to a higher value to increase the maximum limit. (The maximum number of users has a hard limit of 50.)

User Session Count Exceeds Major Threshold Limit (OSS 14)

The "User Session Count Exceeds Major Threshold Limit" alarm indicates that the maximum number of CORBA login sessions has been reached and no new users can log in. Remove idle or long duration sessions using the stop client-session command to reduce the number of active sessions. You should also check the number of users in the /opt/BTScis/etc/bts.properties file and set it to a higher value to increase the maximum limit. (The maximum number of users has a hard limit of 50.)

Alternatively, you can use the add ems-session-control-policy command to add control policies and to remove any long running CORBA sessions. After taking these actions, try the CORBA login again.

Session Has Been Removed By Session Control Policy (OSS 16)

The "Session has been removed by Session Control Policy" information event indicates that a CORBA session was removed for violating a control policy. Inspect the event contents in the policy id. Adjust the policy using the change ems-session-control-policy command as needed.

Session Removed But No OOS (16) Returned

If a CORBA session is removed but no OSS(16) event was returned, the session may have been idle for too long. Check the CORBA log file to verify if the "Removing idle user session from the server. Session Detail:" message appeared at the time the session was removed. Adjust the idle.timeout in /opt/BTScis/etc/bts.properties file to a higher value if needed.

Session Has Been Removed (OSS 17)

The Session Has Been Removed event serves as an information alert that the session has been removed. The primary cause of the informational alert is that the session was removed because it was idle over the timeout limit.

Interface Is Active and Operational (OSS 19)

The Interface Is Active and Operational event serves as an informational alert that the application interface is active and operational. The event is informational only and no further action is required.

Interface Is Not Started or Not Operational (OSS 20)

The Interface Is Not Started or Is Not Operational alarm (minor) indicates that application interface has failed to start or is not operational. To troubleshoot and correct the cause of the alarm, restart the application interface.

Resource Reset (OSS 21)

The Resource Reset event serves as an informational alert that the resource has been reset. The event is informational only and no further action is required.

Error Messages

The following error messages can be returned. The number given in the sample code refers to the text in the explanation. The text is returned if the sample code is used.

Error Message    Password Aging Warning

Explanation    This exception indicates that the user's password will expire in a few days. Login is successful.

Sample Code   public static final int EM_PW_WARN = 3;

Recommended Action    This exception is informational. The login was successful. No action to take. If desired, change the password using the loginResetPassword() API instead of the loginWithStatus() API to log in and change the password.

Error Message    Password Expired Error

Explanation    This exception indicates that the user's password has expired. Login was rejected.

Sample Code   public static final int EM_PW_EXP = 4;

Recommended Action    Change the password using the loginResetPassword() API to log in and change the password.

Alarms

The following alarms might be returned by this feature.

OSS (12)

To monitor and correct the cause of the event, refer to the "User Session Count Is Approaching Threshold Limit (OSS 12)" section.

OSS (14)

To monitor and correct the cause of the event, refer to the "User Session Count Exceeds Major Threshold Limit (OSS 14)" section.

OSS (16)

To monitor and correct the cause of the event, refer to the "Session Has Been Removed By Session Control Policy (OSS 16)" section.

OSS (17)

To monitor and correct the cause of the event, refer to the "Session Has Been Removed (OSS 17)" section.

OSS (19)

To monitor and correct the cause of the event, refer to the "Interface Is Active and Operational (OSS 19)" section.

OSS (20)

To monitor and correct the cause of the event, refer to the "Interface Is Not Started or Not Operational (OSS 20)" section.

OSS (21)

To monitor and correct the cause of the event, refer to the "Resource Reset (OSS 21)" section.