ESS Maintenance and Troubleshooting

This chapter includes the following topics:

Using the Maintenance Utility

A shell script utility called serv is included with the ESS distribution at the <less_install_dir>/ess/ directory. This serv script can be used to manage the following processes of ESS Server:

  • PS Monitor Application (PSMON)
  • ESS

This utility can report the status of the ESS process on the system or it can be used to stop an instance of ESS process.

IMPORTANT:

ESS must always be started with the serv script command.

Following are the options available with the serv script:

./serv { start | stop | forcestart | forcestop | switch | version | hoststatus | status[ <resource_name | resourcegroup_name> ] }

Keyword

Description

start

Use this command to start each ESS process and PSMON.

stop

Stops the running ESS process and PSMON.

forcestart

Use this command to restart ESS and PSMON. This command will first stop already running ESS processes and then restart each process.

forcestop

Use this command to forcibly stop ESS process and PSMON. If ESS is not stopped by serv stop command, this command will be used.

switch

Use this command to switchover the resource group.

IMPORTANT:

This option can be used only in cluster mode installation.

hoststatus

Displays the status of each source and destination host.

status

Displays the status of each of process/resource/resourcegroup.

For stand-alone mode:
  • Process monitor tool
  • Local External Storage Server

version

This command will show the version of ESS build. It will also show the revision date.



The following is a sample output of the serv status command:

PID      Process                             Status
---------------------------------------------------------
 4270   PS
Monitor Application               Running
  -     Local
External Storage Server         Not running

The following is a sample output of the serv hoststatus command:

===================================================================
Host ID        State       Status      LastListedCount  ProcessedCount  FailedCount
===================================================================
clustems2_edr_1269433964.51   
  Disabled   -          -    
             0                  0
clustems1_edr_1269434568.52   
  Enabled    Pulling    0    
             294                6
10.4.4.93_edr_1269434878.55   
  Enabled    Pulling    0  
               143            
   35


===================================================================
Host ID        State       Status      LastListedCount  ProcessedCount  FailedCountt
==================================================================
local_edr_1269433964.54   
      Enabled    Pushing   
1                  294         
      0
local_data_1269434878.62   
     Enabled    Pushing    1   
              143                0
where,
  • Host ID – source or destination name
  • State – Source enabled/disabled for pull, or destination enabled/disabled for push
  • Status – Current status of the source or destination
  • LastListedCount – The latest count of files listed by the source or destination
  • ProcessedCount – The number of files pulled/pushed successfully
  • FailedCount – The number of files failed during the push/pull process

Using ESS Logs

The PSMON process logs memory usage threshold crossing alerts and other error and warning messages in the watchdog.log file located in the <less_install_dir>/ess/log directory.

The PSMON process also sends alerts and messages to the configured e-mail address.

ESS stores all logs and other error and warning messages in a directory path configurable during installation. If this path is incorrect, then logs are stored in the <less_install_dir>/ess/log directory. See the Installing ESS Application in Stand-alone Mode section in this guide for details.

The ESS creates separate log files for each ESS process (one file/ESS instance).

In 14.0 and later releases, the log file size can be a maximum of 50 MB.

In 9.0 and earlier releases, the log file size can be a maximum of 5 MB.

Each time ESS starts, a new directory is created under the log path directory. This directory uses the following naming conventions:

SERVER_LOG<Current date>_<Current time>

Paramiko related logs are also stored at the same location with file name paramiko.log.

ESS Server Scripts

This section provides the function of the scripts available in the <less_install_dir>/ess directory, and the information on the usage of scripts. These scripts are mainly used for configuration management and maintenance purposes.

This section includes the following topics:

Using the add_project Script

To avoid the impact of other applications running simultaneously on ESS Server, ESS tasks can be separated by creating it as a Solaris project. This script is designed to add a dedicated project for ESS. The script adds project "lessPrj" with ID 1001 and for the user essadmin. Depending on the underlying platform this script enables workload division mechanism. On Netra 210 or Netra 245 server Fair Share Scheduler (FSS) mechanism is enabled and on T5220 server resource pool mechanism is enabled.

IMPORTANT:

The script must be executed with superuser login before starting the ESS Server. In the cluster mode, this script must be executed on both the nodes of the cluster.

To use the script, enter the following command from the <less_install_dir>/ess directory:

./add_project.sh

Using FSS Scheduler

The ESS project is allocated with two CPU shares to avoid starvation of ESS due to other concurrent processes that might be running on the server. These shares are considering the default configuration the system.

Avoid configuring another project on the ESS Server or if added allocate sufficient shares to ESS project. To alter the project name, project ID or user name, this script should be edited to change the required parameters.

This script also makes FSS as a default scheduler for the system and also forces all existing processes using TS scheduler to use FSS scheduler. Hence, this script should always be started if you accept FSS as a default scheduler for the system.

IMPORTANT:

In case of T5220 server, Veritas cluster configuration should not be modified to start ESS process using FSS scheduler. Hence, the following entry must be removed from the configuration file types.cf located in the /etc/VRTSvcs/conf/config directory if available: static str ScriptClass = FSS

Using Resource Pool Facility

Resource pools enable you to separate the workload so that the workload consumption of certain resources does not overlap. This resource reservation aids to achieve predictable performance on systems with mixed workloads. Resource pools provide a persistent configuration mechanism for processor set (pset) configuration. In case of a multi-processor machine few CPUs can be dedicated to ESS and rest can be left for other processes. The configuration related to ESS resource pool is available in less_pool.cfg file.

IMPORTANT:

ESS must be started using start_serv.sh script instead of serv script to get the benefits of resource pool.

Using the start_serv Script

This script is specifically designed to start ESS Server in configured projects environment. The script assumes that "lessPrj" project is configured on the system and is allocated sufficient shares. If the project entry is not added and if the user starting ESS is not privileged for configured project, then script will not start ESS.

ESS must always be started using this script to get the benefits of dedicated CPU shares. Path of this script, without any argument must be configured in VCS config file, main.cf.

To start the ESS manually enter the following command from the <less_install_dir>/ess directory:

./start_serv.sh

Configuring Veritas Cluster to Start ESS Using FSS Scheduler

In the default configuration, VCS starts application using TS scheduler. This configuration must be changed to use FSS scheduler for allocating CPU shares to ESS. For this "static str ScriptClass = FSS" variable must be added in Application module of VCS config file, types.cf.

Alternatively, this parameter can also be set using GUI client of VCS.

Using the Cleanup Script

Use the deleteLocalFiles.sh script to delete files from local paths as a cleanup process. This script is required so that older files from the local destination, such as mediation, can be removed periodically. This ensures that there is no unnecessary disk space usage.

If the local destination deletes the file after picking up, then this script may not be required.

Files from ESS specific directories are deleted as soon as a fetched file is transferred to all of the configured destinations. However, if the file is not pushed towards the destination, these skipped files keep accumulating under the destination’s local temporary directory. You can use the cleanup script to regularly remove these older files from the temporary directories.

IMPORTANT:

You should run the cleanup script from the ESS base directory.

How the Cleanup Script Works

Use this procedure to start and kill the deletelocalfiles.sh script manually.

  1. Provide the local paths from where files should be deleted periodically. These paths are taken as base paths and all of the older files below the base path are deleted at the configured time. If you want to delete files from directory /home/ess/udr and /home/ess/edr, you can provide base path as /home/ess. In other words if you provide path as /home/ess all of the older files from the directories below ess such as edr or udr will be deleted.
  2. You can provide more than one path at a time so that the script deletes the files from more than one path.
    ./deleteLocalFiles.sh /home/ess /home/mediation /home/RESS
    

    DANGER:

    Since all older files below the base directory are deleted, make sure that you are providing only required paths for the script.

    The script reads the required parameters from the ess/ess config. The parameters and definitions are listed below:
    • essdellocalrecordsexpirytime: Indicates the number of days after which the file is treated as an older file. If this parameter is set to 0, the files will not be deleted.
    • essdellocalrecordsstarttime: This value in hours indicates the starting hour when the ESS will start deleting older files.

    IMPORTANT:

    The above parameters read from config file are applied to all paths provided to the script.

  3. Start this script from /ess directory with the following command:
      ./deleteLocalFiles.sh
    path1 [path2 path3 .. pathn ]
    

    IMPORTANT:

    The logs for the cleanup script are generated in a file located at ess/log/deleteLocalFiles_%timestamp%.log.

Troubleshooting the ESS

In the event problems are experienced while using the ESS application, refer to the following table for troubleshooting information.


Table 1. Troubleshooting ESS

Problem

Troubleshooting Method

The ESS application cannot connect/login to the ASR 5x00 platform.

  • Make sure that you are supplying the correct user name, password and chassis’ host name or IP address.
  • Make sure that you created an admin or config-admin account that is enabled for SFTP in the correct context.
  • Make sure that you have created SSH keys on the chassis.
  • Make sure that you have enabled the SFTP subsystem on the chassis.
  • Make sure that you can manually create SFTP connection from ESS to chassis with same configured user name, password and host name/IP address of the chassis.
    For example:
    sftp lessadmin@qain5
    Connecting to qain5...
    lessadmin@qain5's
    password: sftp>
  • If SNMP support is configured trap notification raised due to connection failure may provide additional information on why ESS could not connect to chassis.

The ESS application cannot connect/login to the Remote destination.

  • Make sure that you are supplying the correct user name, password and remote destination host name or IP address.
  • Make sure that the supplied user is already created on remote destination.
  • Make sure that SSH daemon / SFTP Server is running on remote destination.
  • Make sure that you can manually create SFTP connection from ESS to remote destination with same configured user name, password and host name/IP address of the remote destination.
    For example:
    sftp lessadmin@qain5
    Connecting to qain5...
    lessadmin@qain5's
    password: sftp>
  • If SNMP support is configured trap notification raised due to connection failure may provide additional information on why ESS could not connect to remote destination.

ESS is not retrieving any files.

  • Make sure that you have specified correct source directory paths for chassis.
  • Make sure that you have specified the correct destination directory path for configured destinations.
  • Make sure that you have configured ESS to fetch respective files and have configured destination for respective files.
  • Make sure that you have configured ECS to generate billing files correctly. Try fetching file manually from chassis.
  • Make sure that the disk on which ESS directories reside and disk where destination directories reside have sufficient free space.
  • If SNMP feature is enabled then trap notification raised for ESS file transfer failure may provide additional information on the reasons for the failure.
  • Check if compression related parameters are in sync at ESS side and at the chassis side.

ESS is not starting

  • Make sure no ESS Server is already running on same machine.
  • Make sure that TCP port configured in ESS config file as 'ServerPort' is not blocked by any other process.
  • Make sure that disk is not full.
  • If config file is modified after installation, check if it is correctly modified.

ESS is generating alarm 'starLESSPullIntervalMissed'

  • Check if network latency between chassis and ESS is higher than usual.
  • Check if file size is larger than expected.
  • Check if file poll interval is correctly set.

Files are accumulating on chassis

  • Make sure that there is no high network latency between ESS and chassis.
  • Make sure that file generation rate is not too high on chassis.
  • Check if ESS is loosing connections repeatedly.
  • Check if CPU consumption of ESS is higher than expected.
  • Check if SSH Daemon on chassis is busier than expected.
  • Check if any other application residing on ESS is causing heavy system resource consumption.
  • Make sure that ESS processes are running in FSS scheduler if ESS is started with priority based solution. This can be tested by using ps -cafe command. For example, all below processes are running under FSS scheduler.
    #ps -cafe | grep ess
    root 15154     1  FSS
      1 20:12:44 ?        0:00 /less/ess/3rdparty/python/solaris/bin/python2.5 /less/ess/bin/lr_ess_push.py
    -i
    
    root 15166     1  FSS
      1 20:12:45 ?        0:00 /less/ess/3rdparty/python/solaris/bin/python2.5 /less/ess/bin/lr_ess_transfer.p
    
    root 15160     1  FSS
      1 20:12:44 ?        0:00 /less/ess/3rdparty/python/solaris/bin/python2.5 /less/ess/bin/lr_ess.py
    -i 1
    
    root 15147     1  FSS
     57 20:12:43 pts/1    0:00 /less/ess/3rdparty/perl/solaris/bin/perl
    -w /less/ess/template/psmon
    --daemon -
    
    root 15207 14990  FSS
     59 20:12:55 pts/1    0:00 grep ess
    

ESS is generating alarm 'starLESSThreshDiskUsage'

  • Free the local disk containing ESS directories if it is over utilized.
  • Check if disk threshold is properly configured.
  • Check if cleanup script is running and is periodically removing the files from intended paths.

ESS is generating alarm 'connectionfail'

  • Check if the IP/host name address is properly configured.
  • Check if the user name is properly configured.
  • Check if the password is properly configured.


In case of any failure, check the ESS logs for additional information. If the problem still persists, contact your system administrator.

Capturing Server Logs Using Script

In the event additional troubleshooting assistance is required, debugging information can be collected using a script called getSupportDetails.sh. This script collects different log files and captures the output of certain system commands that aid in troubleshooting issues. This script is packaged with the server in the <less_install_dir>/ess/tools/supportdetails/ directory.

This script refers to an XML file to get the list of logs. This XML file resides in the same directory as the script. Once executed, the script retrieves the contents of logs, files, folders, and output of certain commands and prepares a zipped file (lesssupportDetails.tar.gz), by default it is placed in /tmp/log directory.

Requirements

Perl 5.8.5 and above is required for running the script.

Apart from standard Perl modules (which are included in default installation of Perl), some additional modules are required for running the script. The list is as follows:
  • expat version 1.95.8
  • XML::Parser version 2.34
  • XML-Parser-EasyTree
  • Devel-CoreStack version 1.3

These modules are installed by default by the product. Please ensure that the above mentioned modules are installed when using a different installation of Perl.

To run the script, change to the path where the script is present and type:
./getSupportDetails.sh [--level=...] [--xmlfile=...] [--help]

Keyword/Variable

Description

--level

Specifies the level of debug to run. It can have a maximum of 3 levels. The level 3 provides the most detailed information.Default: 1

--xmlfile

Specifies the xml file name to be used for collecting the log.

Default: getSupportDetails.xml

--help

Displays the supported keywords/variables.



For example, ./getSupportDetails.sh --level=3 --xmlfile=/tmp/getSupportDetails.xml

Supported Levels:

The logs that can be collected for different levels are as follows:
  • Level 1:
    • Current status (running / not running) of the product
    • Current version of the installed product
    • Current Config files of the product
    • Output of the following commands:
      netstat -nr
      
      scstat
      
  • Level 2:
    • Logs from Level 1
    • All Log files (Including Old Logs)
    • Information of Solaris version and current patch installed
    • Output of the following commands:
      On both Solaris and Linux:
      netstat -nr
      
      ifconfig -a
      
      df -v
      
      uname -a
      
      ps -eaf
      
      On Solaris:
      showrev
      
      prstat 1 1
      
      On Linux:
      top -n 1 -b
      
      env
      
  • Level 3:
    • Logs from level 2
    • Listing of directory pointed by "essbasedirectorypath" from ess_config file
    • Output of the following commands:
      On Linux:
      rpm -q --all --queryformat
      '%-30{NAME}\t%{VERSION}\t%-60{SUMMARY}\t%{GROUP}\n'
      
      cat /proc/cpuinfo
      
      cat /proc/meminfo
      
      On Solaris:
      pkginfo
      
      prtdiag