Troubleshooting the SSC

This chapter briefly describes troubleshooting and monitoring information, known issues and their work around if available.

You may face issues while working with a Subscriber Service Controller (SSC) deployment, as well as while installing or un-installing an SSC instance. For troubleshooting such issues SSC deployment provides following categories of support data:
  • System logs that record deployment infrastructure information such as web servers and active sessions.
  • Audit or event logs that record events related to subscriber profile and service usage.
  • Scripts that can be used to generate system verification information, such as generating eth and host IPs as well as generating their IP maps.
  • Current status of processes associated with any SSC component.
  • Scripts that can be used to dump database.
  • Scripts that can be used to view grid status and re-attach a grid if it is detached.
  • Scripts to view network connectivity status.

IMPORTANT:

You can access system statistics for monitoring and troubleshooting the SSC instance, refer section Monitoring System Statistics in SSC Administration chapter as well as schema statistics appendices.

Following log files can be used for troubleshooting purpose:
  • /var/log/message: This log file records all the installation related errors and alert messages.
  • /localhome/ssc/log/sn_SSC.log: This log file records, alarms, application errors as well as database query failure issues related to the SSC instance.
  • /localhome/ssc/log/startup.log: This log file records the segmentation errors received by any SSC related process along with start-up and incremental log.
  • /localhome/ssc/log/SscHaTool_yyymmdd_hh:mm:ss.log : This log file records information related to High Availability (HA) activities. This file is created only if the HA feature is enabled for the deployment.
Following scripts can be used for troubleshooting purpose:
  • maintenacemode.sh: By default after performing the installation and upgrade procedures SSC deployment is in the maintenance mode. Cron job execution is halted when the maintenance mode is enabled. Use Maintenacemode.sh script from /localhome/ssc/tools directory to enable or disable the maintenance mode or to view its status.
  • ./hbtool: Execute this script, from /localhome/ssc directory, with SSC administrator privileges. It displays list of all active processes except profile manager process. For each process it displays PID, facility, instance, start time and total restarts. This tool also displays the restart count for the specific process that has died and restarted.
  • ./createCronjobForDumpingmonitoringInfo.sh: Execute this script, with privileges of a root user, from localhome/ssc/tools/MonitoringCron directory. This cron job is executed after every fifteen minutes. It logs parameters such as, disk, memory and CPU usage as well as outputs of sscadm status, imdb status and imdb check commands. It also logs semaphore details and ping results for host names. These details are stored in monitoring.log file.
  • ./removeCronjobForDumpingmonitoringInfo.sh: Execute this script with privileges of a root user, from localhome/ssc/tools/monitoringCron directory. This command removes the cron job created by the command ./createCronJobForDumpingMonitoringInfo.sh .
  • ./dumpMonitoringInfo.sh: Execute this script, with privileges of a root user, from localhome/ssc/tools/monitoringCron directory. This command logs the parameters described in command ./createCronjobForDumpingMonitoringInfo.sh. These parameters are logged in monitoring.log file, instead of creating a cron job.
  • ./sscdbstatus: Execute this script from /localhome/ssc/install/spr_install/tools directory with database user privileges.This script lists the status of database and IMDB application.
  • ./sscdb_shutdown: Execute this script from /localhome/ssc/install/spr_install/tools directory. This script shuts down the SSC database.
  • ./sscdb_backup: Execute this script from /localhome/ssc/install/spr_install/tools directory. This script creates a back-up of SSC database. Before executing this script, ensure that the status of SSC application, database and IMDB is not active.
  • ./sscdb_startup: Execute this script from /localhome/ssc/install/spr_install/tools directory. This script start ups down the SSC database.
Following scripts can be used for troubleshooting the IMDB or database grid issues:
  • checkGridAttach.sh: Execute this script with privileges of a root user, from llocalhome/ssc/install/spr_install/tools/GridAttachCron directory. It displays current status of the IMDB grid.
  • checkT10Health.sh: Execute this script with privileges of a root user, from localhome/ssc/install/spr_install/tools/GridAttachCron directory. It displays current status of the IMDB application.
Following scripts can be used for verifying installed SSC set-up:
  • common.sh
  • compare_eth_ip.sh
  • compare_host_ip.sh
  • generate_eth_ip_map.sh
  • generate-host_ip_map.sh
  • ping_all_hosts_from_all_machines.sh
  • ping_hostnames.sh
  • system_verification.sh

IMPORTANT:

These scripts are located in /localhome/ssc/tools/System Verification directory.

Following trouble shooting sections provide information about possible cause and work around if available for such issues. The issues are categorized as:

Issues Pertaining to SSC Installation

This section includes the issues that you may face during single host as well as cluster installation for your SSC deployment.

Problem:

RDBMS database creation fails.
Possible Cause(s):

Installer is not able to create the database schema.
Action(s):
  • Log-in as root user.
  • Access the installation log by issuing following command tailf /var/log/messages
  • The log file should contain following entries.: Result of installing post install steps for database 0 and SSC installation is done.
  • Absence of the above mentioned entries in log suggests that display issues may be interrupting execution of database utilities.
  • As a root user issue following command: su -database- c “/localhome/install/database/runInstaller -silent -noconfig -responseFile /localhome/install/database/ssc_database_sw_install.rsp”
  • This command will update log file in ../var/log/messages directory.
  • Un-install SSC on this blade.
  • Again install SSC on this blade.


Problem:

Installer hangs while installing database.

Possible Cause(s):

This may be due to some issue related to X11 forwarding.

Action(s):
  • Log-in as root user.
  • Access .. /var/log/message directory.
  • Ensure that this directory contains a message - Host created for searching xauth list is datablade2/unix:10.
  • Ensure that this directory does not contain any message stating – Searching xauth list using key.
  • Delete .Xauthority file from home/root directory.
  • Issue following command: mv /var/.com.zerog.registry.xml var/.com.zerog.registry.xml.
  • Start the installation process again.


Problem:

During installation process lost connectivity to installer.

Possible Cause(s):

You may lose connectivity to installer if you try to configure management interface using the GUI based installer.
Action(s):
  • Exit installation process.
  • Delete.. /tmp/ssc folder.
  • Re-install SSC without configuring management interface.


Issues Pertaining to SSC Startup

This section includes the issues that you may face during initiating the SSC using the administrative scripts, provided during installation.

Problem:

Neither SSC starts nor does it gives any specific error message.

Possible Cause(s):

SSC is not able to display the cause for not starting.
Action(s):
  • Log in as SSC admin user.
  • Access the logs in file logs/startup.log directory.
  • Read the messages and fix the issues mentioned in the log file.
  • Re-start SSC.


Problem:

SSC does not start while giving apache error message.

Possible Cause(s):

SSC is not able to start and displays following message Error: Syntax error on line 40 of /localhome/ssc/3rdparty/apache/conf/httpd.conf

Action(s):
  • Log in as SSC admin user.
  • Access the etc/hosts file, and get the ip address of the blade where SSC is not starting.
  • Edit the file localhome/ssc/3rdparty/apache/conf/httpd.conf and add this IP address before the port as follows: Listen 192.168.10.2:8080 where 192.168.10.2 is the host IP address.
  • Stop SSC using command sscadmin stop.
  • Start SSC using command sscadmin start.
  • Configure the profile config using system administration option of SSC Administration Console to prevent the reoccurrence of this error.


Problem:

After installation Sh controller is not coming up.

Possible Cause(s):

Sh controllers do not start until you bind these controllers using SSC administration console.
Action(s):
  • Log-in to console is an SSC administrator.
  • Use the Interface Management option to bind Sh controllers.
  • Refer to Managing Interfaces section, in SSC administration chapter of the SSC Installation & Administration guide.


Problem:

SSC does not stop properly.

Possible Cause(s):

Heart beat daemon is not able to kill all the processes related to application and database of this SSC instance.
Action(s):
  • Stop SSC instance using script ./sscadm stop.
  • List all the SSC processes that are still active by using command ps –e | grep sn_*.
  • Kill each of this SSC related process.
  • Re-start SSC using script ./sscadm start.


Problem:

File size of Start-up log file keeps on increasing.

Possible Cause(s):

The start-up log file may keep on increasing in size due to increased number of debug or error log entries.
Action(s):
Ensure that following script and system parameter are added to a cron job:
  • purgefile.shscript is available which rotates the start-up log file when the file size exceeds 1 Gb.
  • In the system.cfgfile a parameters MaxSscStartupLogFileCount is added, this parameter controls the number of start-up log files the can be present on a system.
Following are the locations of these scripts:
  • create cron job – scripts/LogFilesPurgeCron/createCronJobForPurgingLogFiles.sh
  • remove cron job - scripts/LogFilesPurgeCron/removeCronJobForPurgingLogFiles.sh


Problem:

In a two blade cluster HA deployment, after restoring back-up, system may not be able to bind the profile controller interface.
Possible Cause(s):

Some parameter settings in the cluster configuration file may cause this issue.
Action(s):
  • Login as SSC administrator.
  • Stop SSC application on both the blades using script ./sscadm stop.
  • Access /etc/cluster/ cluster.config file on both the blades.
  • Ensure that parameter <cman expected_vosts> has value 1.
  • Ensure that the parameter two_node has value 1.
  • Ensure that parameter broadcast has value yes. You may need to add this parameter.


Issues Pertaining to SSC Database

This section includes the issues that you may face when SSC is trying to access the subscriber or subscription information from database.

Problem:

Not able to start database listener.
Possible Cause(s):

database listener process is not running.

Action(s):
  • From the location $database_HOME/bin execute lsnrctl.
  • At lsncrtl prompt enter start to initiate the listnerl process.
  • Enter quit.


Problem:

12519:database-12519:TNS: no appropriate service handler found – error listed in log/sn_SSCn.log file.

Possible Cause(s):

Listener process may not have been registered for the database.
Action(s):
  • Ensure that the listener has started.
  • Login as databasesuper user, by issuing command: su –database.
  • Access the directory ../localhome/ssc/install/spr_install .
  • Check the database status by issuing following command:./sscdbstatus.sh SSC .
  • If database status shows errors, then check whether the listener process is owned by database.
  • If databaseno longer owns the listener process, and this process is owned by daemon, then kill listener process, as a root user and using kill -9 <pid> command.
  • Login as database administrator and start listener, by issuing command:lsnrctl start .
  • Now, again check the status of database, by accessing../ssc/install/spr_install directory and by issuing command: ./sscdbstatus.sh SSC .


Problem:

Oracle database creation hangs.
Possible Cause(s):

This issue may be observed for some deployments, when executing SSC installer using Xming from Windows 7.

Action(s):
  • Exit SSC installer.
  • Delete /var/.com.*xml file
  • Restart SSC installer.


Problem:

Oracle database creation fails.
Possible Cause(s):

For some deployments display issue may prevent the execution of the database utilities.
Action(s):
  • Access var/log/messages file.
  • In this file search for the statement Result of installing post installation steps for oracle is 0 If this statement is not appearing in the log then it indicates that the database creation failure is due to display issue.
  • Log in with root administrative privileges and execute following command su-l oracle -c “/localhome/install/oracle/runInstaller -responseFile /localhome/install/oracle/ssc_oracle_sw_install.rsp -nowelcome –silent .
  • Above mentioned command creates a log file that logs the failure messages along with suggested work around.
  • After fixing the errors using work around, un-install SSC.
  • Install SSC on the same blade again.


Issues Pertaining to In Memory Database (IMDB) Application

This section includes the issues that you may face when SSC is trying to access the subscriber or subscription information from database using the IMDB application.

Problem:

Grid creation fails.
Possible Cause(s):

Installer is not able to create the IMDB_App grid.
Action(s):
  • Login as a root user.
  • Access the following file on the blade where database is configured: /localhome/database/app/database/product/11.2.0/dbhome_1/network/admin/tnsnames.ora.
  • Check the SSC entry in this file that contains following fields(DESCRIPTION =, (ADDRESS_LIST = (ADDRESS= PROTOCOL = TCP) (HOST = datablade1) (PORT = 1521)) CONNECT_ DATA = (SID =SSC))).
  • Ensure that you can log-in to database as sqlplus spradm/spr_adm@SSC .