> WEM Server Files and Operation
> Once the WEM software has been installed according to the information and instructions in the previous chapter, the information in this chapter can be used as a reference for performing further configuration of the Web Element Management Server to customize it to your needs.
> This chapter provides information on the WEM Server directory structure and important files, description for the various configuration files supported by the application, and instructions for verifying and stopping/starting component processes.
> Important: If no further configuration is needed, proceed to Preparing and Using the Client Workstation chapter of this guide for information and instructions for preparing and using WEM Clients.
> This chapter includes the following topics:
> Server Directory Structure and Important Files
> The following table provides information on the application’s directory structure after installation and identifies important files.
> WEM Server Directory Structure and Important Files

> Directory/Filename | Description
> /_jvm | Contains files pertaining to Java and Java Runtime Environment (JRE).
> /_uninst | Contains files for uninstalling the WEM application.
> | uninst | Executable file for uninstalling the WEM application.
> /apache | Contains files pertaining to the Apache Web Server used to provide access to the WEM.
> | /bin | Contains files required for starting/stopping the Apache process and HTTP daemon.
> | apachectl | Executable file for starting/stopping the Apache Web Server. Refer to the Controlling Server Component Processes section later in this chapter for more information.
> /client | Contains files pertaining to the WEM Client.
> | .java.policy | Java Policy file required by client to access the WEM Server. Refer to Preparing and Using the Client Workstation chapter of this guide for more information.
> | /xxxx | A directory containing client files for the current build, xxxx is the last four digits of the current build number.
> |  | /audio | .WAV audio files used for WEM's audio alarming functions
> |  | /mapimages | .GIF map image files that can be configured for the client(s)
> |  | /webhelp | HTML-based online help files for the application; they are accessible through the client Graphical User Interface (GUI).
> | img.html | Webpage used for accessing WEM. Refer to Preparing and Using the Client Workstation chapter of this guide for more information.
> | imgdebug.html | The same as the img.html file except that it enables application level logs in the Java Console.
> console.log | Log file containing console messages generated during installation.
> inst.log | Log file created to track installation progress.
> /perlx.x.x | Contains files pertaining to the specified version of the PERL scripting language, x.x.x is the version of PERL.
> /postgresx.x.x | Contains files pertaining to the PostgreSQL database used by the WEM, x.x.x is the version of Postgres.
> | /bin | Contains files required for starting/stopping PostgreSQL and the Postmaster daemon.
> |  | pg_ctl | Executable for starting/stopping the PostgreSQL server.
> |  | pg_start | Executable for starting Postmaster daemon. Refer to the Controlling Server Component Processes section later in this chapter for more information.
> |  | pg_stop | Executable for stopping Postmaster daemon. Refer to the Controlling Server Component Processes section later in this chapter for more information.
> |  | psql | Enters the PostgreSQL interactive terminal.
> |  | vacuumdb | Removes old and/or unused records from the various WEM databases. Refer to WEM Database Maintenance chapter of this guide for more information.
> | logfile | Log file containing information pertaining to PostgreSQL operation.
> /server | Contains files pertaining to the WEM Server.
> | /alarmscripts | A directory for storing user-defined scripts.
> | /bin | Contains WEM server and component executable files.
> | /bulkstat_archive | Contains Bulk statistics archive files (if collected).
> | /crondir | A directory for storing cron-specific files.
> | /dbscripts | Contains scripts for use with the PostgreSQL database.
> |  | migrate.tar.gz | File containing the migrate script used for migration of WEM from one version to another version used when performing a software upgrade or downgrade.
> | /data | Contains un-parsed bulk statistic data received from managed systems.
> | /etc | Contains WEM configuration files. Important: Many of these files contain user-configurable parameters for tailoring the operation of the WEM. Information on the various parameters in these files is provided within the files. Refer to the Server Configuration Files section of this chapter for more information.
> |  | alarmid.cfg | SNMP alarm identification configuration file
> |  | audio.cfg | Audio alarms configuration file
> |  | blacklist.cfg | Configuration file containing information that are used to enable blacklisting feature on WEM.
> |  | bs.cfg | Bulk statistic configuration file
> |  | bsparser.cfg | Bulk statistic parser configuration file
> |  | bsparserlogger.xml | Bulk statistic parser logger file. Refer to Enhanced WEM Logging chapter of this guide for more information.
> |  | bsserver.cfg | Bulk statistic server module configuration file
> |  | bsserverlogger.xml | Bulk statistic server logger file Refer to Enhanced WEM Logging chapter of this guide for more information.
> |  | bstca.cfg | Bulk statistic threshold configuration file
> |  | cdp.cfg | Configuration file for ad-hoc report generation
> |  | configupdate.cfg | System software update functionality configuration file
> |  | db.cfg | Postgres database configuration file Caution: Improper configuration of this file can adversely affect WEM operation.
> |  | emscert.txt | WEM RSA key and certificate Caution: Do not alter this file.
> |  | emslic.cfg | WEM license configuration file
> |  | emslogger.xml | WEM Server logger file Refer to Enhanced WEM Logging chapter of this guide for more information.
> |  | fm.cfg | Fault management configuration file containing default severity levels for product SNMP traps.
> |  | hwinv.cfg | Contains the e-mail ID used when sending notifications of hardware changes.
> |  | ism.cfg | WEM/managed system communication configuration file
> |  | mcrdbs.cfg | MCRDBS configuration file
> |  | MeasurementStat.dtd | Document Type Definition (DTD) configuration file for validating bulk statistic data.
> |  | nb.cfg | NorthBound Notification Service configuration file
> |  | nbserver.cfg | File containing configuration parameters for NorthBound Server and Notification Service processes and other configuration parameters for NorthBound interface. Important: The NorthBound Server and Notification Service processes are customer-specific and are not operational without the appropriate license. Please contact your local sales representative for additional information.
> |  | nbserverlogger.xml | NorthBound Server logger file
> |  | nms.cfg | WEM parameter configuration file
> |  | processmonitor.cfg | Process Monitor server configuration file
> |  | ps.cfg | Polling support configuration file
> |  | psmon.cfg | Process Monitor rule configuration file
> |  | pw_dict.* | All pw_dict files pertain to the password-complexity checking algorithm in support of ANSI security.
> |  | res.cfg | Screen-specific resource-bundle path configuration file Caution: Do not alter this file.
> |  | scriptlogger.xml | Script Server logger file Refer to Enhanced WEM Logging chapter of this guide for more information.
> |  | ssl-key-cert.txt | Secure Sockets Layer (SSL) RSA key and certificate configuration file Caution: Do not alter this file.
> |  | STARENT-EMS-MIB.txt | WEM Server SNMP MIB fileRefer to the SNMP MIB Reference for more information on the objects and alarms defined in the MIB.
> |  | thr.cfg | Thread pool configuration file
> |  | ua.cfg | User administration configuration file
> |  | vacuum.cfg | Database vacuuming configuration file
> |  | wblist.cfg | Configuration file storing the white black list database file paths
> | /flash | Backup directory for SPC/SMC Compact Flash memory modules
> | ./flash | Storage Server Directory where the periodic backup of database is stored.
> | getSupportDetails.pl | Script used for collecting server logs and other information that is useful for troubleshooting. Refer to Capturing WEM Server Logs using Script section of Troubleshooting the WEM appendix of this guide for more information on using this script.
> | /installdata | Contains copies of the configuration files used at the time of installation.
> | /log | Contains log files for the various components of the WEM.
> | psmon | Process Monitor executable file Refer to WEM Process Monitor chapter of this guide for more information.
> | /scripts | Contains script files that can be used in conjunction with the WEM.
> | serv | Executable file for starting/stopping the WEM Server. Refer to the Controlling Server Component Processes section later in this chapter for more information.
> | /xmldata | Contains XML-parsed bulk statistic records if the “Generate XML Files” option is enabled.
> | ems_migrate | Script used for backup and restore of EMS directories. Important: This script should be used with the same EMS releases. It will be required in case of EMS setup migration from one machine to another machine, OS change (Solaris 8 to Solaris 10), etc. There is a config file with this script, in which the list of directories for backup and restore can be specified For example: dir_list=postgres/data server/etc server/bsschema

> Server Configuration Files
> As identified in the WEM Server Direstory and Important Files table, the WEM provides a number of configuration files which can be modified to fine-tune the operation of the application. These files are located in the /<ems_dir>/server/etc directory by default.
> This section provides descriptions for each of the configuration files. Details of the parameters in each file is located within the files. The default values for these parameters are suitable for most installations. However, the values can be modified using a text editor (such as Vi Editor) if required.
> Caution: For most of these files, the WEM application process must be stopped and restarted in order for modifications to take effect. To ensure proper operation, be sure to read and understand all the information provided in the files prior to making changes.
> Important: Refer to the WEM Configuration File Parameters appendix for detailed information on the server configuration file parameters.
> The alarmid.cfg File
> This file provides parameters pertaining to the configuration of SNMP alarms received from managed devices.
> The audio.cfg File
> This file provides alarm severity associations to specific audio files. The audio files are in.WAV format and are played by the WEM when an alarm of that severity is received.
> Additionally, this file can be used to associate an audio file to a specific alarm for further customization.
> The blacklist.cfg File
> This file contains a flag that is set to enable/disable the blacklisting feature on WEM. The file provides parameters for specifying information such as blacklist backup limit, time interval to poll the blacklist directory, etc.
> The bs.cfg File
> This file provides bulk statistic configuration information pertaining to schemas, formatting, and parameters for specifying time tolerances for searching bulk statistic records.
> Caution: To ensure proper operation of the WEM, bulk statistic schema and formatting configuration in this file should not be altered.
> The bsparser.cfg File
> This file provides parameters for specifying such information as the directories in which bulk statistics records received from the chassis are stored and archived after they are processed, in addition to time intervals for polling and record transfer.
> Important: Several parameters contained in this file are configurable through the installation process.
> The bsserver.cfg File
> This file provides parameters pertaining to the XML-parsing of bulk statistic records received from the chassis. These parameters provide the ability to enable/disable XML parsing and specify such things as the directory in which parsed records are to be stored, time intervals for checking new records, and whether a single XML file should be generated for all subsystems or one XML file per subsystem in accordance with 3GPP TS 32.401 V4.1.0 and 3GPP TS 32.435 V6.2.0. Secure protocols should be used between WEM and the external system while exporting the performance data in the XML format.
> The bulkstatistics data from a managed system is received by the WEM Server in Comma Separated Value (CSV) format. The data is stored in the /server/data/<system-name> directory by default. The XML files should get generated under <server-dir>/xmldata as long as the files are getting parsed and the data is getting logged in database. This generation of XML files is done by a separate process namely bulkstatserver.
> Important: Several parameters contained in this file are configurable through the installation process.
> The bstca.cfg File
> This file contains parameters pertaining to the threshold configuration for bulk statistic counters. This file includes the threshold values, i.e. clear threshold and set threshold to notify and clear SNMP traps accordingly. These threshold values are specified based on the threshold limit.
> The cdp.cfg File
> This file contains CDP specific parameters that are configured for generation of ad-hoc reports. The parameters are also configured to export Optimized Customer Master Database (OPTCMDB) files to CDP.
> The configupdate.cfg File
> This file contains parameters pertaining to performing chassis software configuration updates using the WEM. The directory in which configuration files are stored, and the directory in which configuration update log files have to be stored can be configured.
> The db.cfg File
> This file contains parameters pertaining to the WEM’s use of the Postgres database. It identifies the various databases and their locations.
> Caution: Improper configuration of this file can adversely affect WEM operation.
> The emslic.cfg File
> This file contains license information for the WEM Server installation.
> The fm.cfg File
> This file contains parameters pertaining to the handling of received alarms. It provides parameters for configuring things such as color indicators for the various severities, E-mail server information for alert notifications, and SNMP operation.
> Important: Several parameters contained in this file are configurable through the installation process.
> The hwinv.cfg File
> This file provides the e-mail ID used when sending hardware change notifications.
> Important: The parameter in this file is configurable through the installation process.
> The ism.cfg File
> This file contains parameters pertaining to communication between the WEM and managed chassis. These parameters include the security information used to access the chassis as well as the ports over which communication takes place.
> This file also contains a parameter that provides the ability to enable/disable the SSL encryption for client-to-server and server-to-boxer communication. This parameter is configured so that the SSL encryption feature is enabled by default.
> The mcrdbs.cfg File
> This file contains parameters that are configured to support conversion of Vendor Format Master Database (VFMDB) to Starent Networks Format Master Database (SFMDB). This file also contains information on MCRDBS to handle the database files.
> The nb.cfg File
> This file contains parameters for NorthBound notifications from the WEM. The information in this file is used by the application for forwarding fault management information to other management devices on the NorthBound interface. This information includes things such as the device IP address and port number on which to forward notifications.
> The nbserver.cfg File
> This file contains parameters for the configuration of NorthBound Server and Notification Service processes, and NorthBound interface.
> Important: The NorthBound Server and Notification Service processes are customer-specific and are not operational without the appropriate license. Please contact your local sales representative for additional information.
> The nms.cfg File
> This file contains parameters which control WEM access to the PostgreSQL database, client access to the server, and other properties used by the application for proper operation.
> Important: Several parameters contained in this file are configurable through the installation process.
> The processmonitor.cfg File
> This file contains parameters used by the Process Monitor (refer to WEM Process Monitor chapter for more information) function. These include parameters such as the directories from which WEM-related processes are started, polling intervals, and maximum percentage thresholds.
> The ps.cfg File
> This file contains parameters which control WEM polling intervals for things such as managed chassis and database queries.
> The psmon.cfg File
> This file contains parameters for the operation of the PSMon (Process Monitor) function supported by the WEM. Additional details on this function and the parameters in this file are located in WEM Process Monitor chapter of this guide.
> The res.cfg File
> This file contains parameters for associating resource-bundles to specific WEM dialogs.
> Caution: To ensure proper operation of the WEM, do not edit the parameters in these files.
> The thr.cfg File
> This file contains parameters pertaining to the WEM thread pool.
> Caution: To ensure proper operation of the WEM, do not edit the parameters in these files.
> The ua.cfg File
> This file contains parameters pertaining to the WEM’s support for the ANSI T1.276 security specification. These include parameters granting and restricting access, login failures, password aging, and password complexity.
> The vacuum.cfg File
> This file contains parameters pertaining to PostgreSQL database vacuuming. Additional details on this function and the parameters in this file are located in WEM Database Maintenance chapter of this guide.
> The wblist.cfg File
> This file provides information on the white black list database file paths that are mainly used for content filtering.
> Server Log Files
> By default, log files generated by the WEM application are stored in the /<ems_dir>/server/log directory. The location is specified in the xxxxlogger.xml files stored in the /<ems_dir>/server/etc directory (refer to Enhanced WEM Logging chapter for information on these files).
> Within the log directory, subdirectories are created as needed to contain log files for each of the various components that comprise the WEM Server:

> • | EMS Server = SERVER_LOG

> • | Script Server = SCRIPT_LOG

> • | BulkStat Server = BS_SERVER_LOG

> • | BulkStat Parser = BS_PARSER_LOG

> The above directory names are appended with a datestamp (YYYYMMDD) and timestamp (HHMMSS). For example, a subdirectory containing EMS Server logs might be named SERVER_LOG_20051230_164433.
> Note that logs for the Process Monitor are written to the watchdog.log file in the log base directory.
> The WEM provides enhanced logging functionality that allows for the customization of the log file output. More detailed information on configuring and using this functionality can be found in Enhanced WEM Logging chapter of this guide.
> Log File Severities
> There are seven severity levels possible for log messages, as shown below:

> • | Disabled (value 8)

> • | Severe (value 7)

> • | Warning (value 6)

> • | Info (value 5)

> • | Config (value 4)

> • | Fine (value 3)

> • | Finer (value 2)

> • | Finest (value 1)

> Each message being generated by the server, has one of the above severity levels.
> What ever debugging level is configured, the messages with that level and above will be logged in the log file. For example, if level 3 is specified then, all messages of severity Severe, Warning, Info, Config, and Fine appear in the log, however, those messages with a severity of Finer and Finest do not.
> The logging level is configurable using the WEM Server executable, serv (located in the /<ems_dir>/server directory by default). The default level is Finest.
> Alternatively, the xxxxlogger.xml files stored in the /<ems_dir>/server/etc directory provide a mechanism for specifying the log level for the specific WEM server modules. These levels only take effect if the configured level is more granular than that configured at the server level.
> Server Scripts
> As identified in the WEM Server Direstory and Important Files table, a number of user-executable scripts are provided with the WEM. These scripts are located in the /<ems_dir>/server/scripts directory by default.
> The scripts provide extended, often CPU and memory intensive functionality which is not available through the WEM Client. Because the scripts provide a more efficient mechanism for these processing-intensive functions, WEM Server performance is optimized.
> The following scripts are available upon installation:

> • | backup.sh: Used to perform the backup/restore of the WEM databases.

> • | batchJob.sh: Uses user-provided information to telnet into a managed device, execute a command, and write its output to a log file. Refer to the README.batchJob.txt file in the same directory for more information.

> • | deleteFilesTables.sh: Deletes unnecessary information from database tables and unused files. The tableFile.cfg file is used to specify the criteria for determining which information or files are “unnecessary”. Refer to the README.deleteFilesTables.txt file in the same directory for more information.

> • | kill_ems_clnt.sh: Provides a mechanism for terminating specific WEM client sessions. Refer to the README.kill_ems_clnt.txt file in the same directory for more information.

> • | loadConfig.sh: Used by the Script Server to load configuration files on the managed systems. This is not a user-executable script.

> • | loadRootCron.sh: Used by the Script Server to load the root user’s cron entry for scheduling system configurations as part of the update feature. This is not a user-executable script.

> • | set_superuser_password.sh: This script is used to reset the “superuser” password to the default.

> • | showCommand.sh: Used by the Script Server to access and execute commands on managed devices. This is not a user-executable script.

> • | showsupportdetails.sh: Writes the show support details command output from the specified IMG to a text file for later processing. Refer to the README.ShowSupportDetails.txt file in the same directory for more information.

> • | sortIPPools.sh: Used to fetch the information of IP addresses for a given group in a sorted mail and to sort it out. Refer to the README_sortIPPools.txt file in the same directory for more information.

> The above scripts can be executed by entering the following command as the root user:
> ./script_name
> Where script_name is the name of the script as identified above.
> Controlling Server Component Processes
> This section provides information on verifying and manually stopping and starting WEM component processes.
> Manually Verifying that WEM Components are Running
> To verify that the various applications are running, follow the procedures in this section.
> Verifying that the WEM is Running
> These instructions assume that the WEM was installed in the default directory, /users/ems.

> Step 1 | Log into the server as the user root.

> Step 2 | Go to the server directory within the WEM installation directory by entering the following command:

> cd /<ems_dir>/server

> Step 3 | Verify that the application is running by entering the following command:

> ./serv status
> The output of this command is a table listing the process, its ID (PID) and its status as shown in the following example:
> PID Process Status
> ----------------------------------------------------------
> 2093 Monitor Server Running
> 1972 EMS Server Running
> 2004 Script Server Running
> 1929 BulkStat Server Running
> 2022 BulkStat Parser Running
> 2076 NorthBound Server Running
> 2057 Notification Service Running
> ----------------------------------------------------------
> Important: The NorthBound Server and Notification Service processes are customer-specific and are not operational without the appropriate license. Please contact your local sales representative for additional information.
> Verifying that the Apache Web Server is Running

> Step 1 | Log into the WEM server as the user root.

> Step 2 | Verify that the application is running by entering the following command:

> ps -aef | grep httpd
> Verifying that the Posters Database is Running

> Step 1 | Log into the WEM server as the user root.

> Step 2 | Verify that the application is running by entering the following command:

> ps -aef | grep postmaster
> Manually Stopping WEM Component Processes
> At times, it may be necessary to manually stop WEM processes. For example, if the server on which the application is installed is to be serviced, the application processes could be stopped prior to beginning.
> The WEM application and its ancillary components must be stopped in the exact order as listed below.
> Important: These instructions assume that the WEM was installed in the default directory, /users/ems.

> Step 1 | Log into the WEM server as the user root.

> Step 2 | Go to the server directory within the WEM installation directory by entering the following command:

> cd /<ems_dir>/server

> Step 3 | Stop the WEM Server processes by entering the following command:

> ./serv stop

> Step 4 | Verify that the WEM server and WEM Script server processes are no longer running by entering the following command:

> ps -ef | grep server

> Step 5 | Go to the PostgreSQL installation directory by entering the following command:

> cd /<ems_dir>/postgresx.x.x/bin/

> Step 6 | Stop the Postgres processes that are currently running on the server by entering the following command:

> ./pg_stop

> Step 7 | Verify that the Postgres processes are no longer running by entering the following command:

> ps -ef | grep postgres

> Step 8 | Go to Apache Web Server installation directory by entering the following command:

> cd /<ems_dir>/apache/bin/

> Step 9 | Stop the Apache web server process by entering the following command:

> ./apachectl stop

> Step 10 | Verify that the Apache web server processes are no longer running by entering the following command:

> ps -ef | grep apache
> Manually Starting the WEM Server Components
> At times it may be necessary to manually start WEM processes. For example, if the server on which the application is installed is rebooted, the application processes must be restarted according to the instructions in this section.
> The WEM application and its ancillary components must be started in the exact order as listed in this section.
> Important: These instructions assume that the WEM was installed in the default directory, /users/ems.
> <X-Refs (online)>step <$paranum> 1: Start Apache Web Server

> Step 1 | Log into the WEM server as the user root.

> Step 2 | Move to the Apache directory by entering the following command:

> cd /<ems_dir>/apache/bin

> Step 3 | Start the Apache web server by entering the following command:

> ./apachectl start

> Step 4 | Verify that the process is running using the instructions provided in the Manually Verifying that WEM Components are Running section of this chapter.

> <X-Refs (online)>step <$paranum> 2: Start Postgres Database

> Step 1 | Log into the WEM server as the user root.

> Step 2 | Move to the Postgres directory by entering the following command:

> cd /<ems_dir>/postgresx.x.x/bin

> Step 3 | Start the Postgres database by entering the following command:

> ./pg_start

> Step 4 | Verify that the process is running using the instructions provided in the Manually Verifying that WEM Components are Running section of this chapter.

> <X-Refs (online)>step <$paranum> 3: Start WEM Server Application

> Step 1 | Log into the WEM server as the user root.

> Step 2 | Move to the WEM Server directory by entering the following command:

> cd /<ems_dir>/server

> Step 3 | Start the WEM server application by entering the following command:

> ./serv start

> Step 4 | Verify that the process is running using the instructions provided in the Manually Verifying that WEM Components are Running section of this chapter.

Cisco Systems Inc.
Tel: 408-526-4000
Fax: 408-527-0883