Table Of Contents

Maintaining and Troubleshooting EPOM

Viewing Reports

Viewing BTS EMS Reports

Managing CORBA

Changing CORBA and SSL Values

Managing CORBA Cache

Managing EPOM

Upgrading EPOM

Backing up BTS Servers

Stopping EPOM

Uninstalling EPOM

Archiving EPOM Log Files

Troubleshooting

Maintaining and Troubleshooting EPOM

---

Viewing Reports

---

Step 1 Navigate to EMS server from a Domain window.

Step 2 Click Reports.

The BTS Component Reports window opens.

Step 3 In the Reports tree, under Performance or Billing, select the type of report.

A report query form opens.

Step 4 Specify the criteria for.

Step 5 Click OK.

Step 6 To view the reports see Viewing BTS EMS Reports.

---

Viewing BTS EMS Reports

View reports like Audit Report, System Health, and BTS Import Errors by clicking the BTS device from the Top Menu bar.

Access BTS EMS reporting parameters that include performance and billing reports (call detail records created on the system through the billing_record function).

---

Step 1 Click the BTS device in the Domains Tree and then click the device from the Top pane. A login dialog box appears.

Step 2 Enter the BTS login and password. A window with options to view the reports and help file appears.

Step 3 Select View a BTS10200 report link. A window with the list of indexes and reports appears.

Step 4 Click the required report or index link, the respective report or index gets displayed.

---

Managing CORBA

Changing CORBA and SSL Values

The EPOM server arrives with SSLvalues set. Only do the following if you change the default directory from /opt/CSCOepom .

---

Step 1 Log into the EPOM web server.

Step 2 Change user to superuser:

su - root

Step 3 Go to the main properties file:

cd /opt/CSCOepom/tomcat/webapps/ROOT/WEB-INF/classes/com/cisco/opus/
props

Step 4 Change corba.orb.configfile:

corba.orb.configfile=/opt/new_directory_name/tomcat/ssliiop/OpenORB.xml

Step 5 List SSL files:

cd /opt/myEPOM/tomcat/ssliiop

The following files appear:

ls

bts10200.cer  bts10200_ts  OpenORB.xml  pss.xml

bts10200_ks   default.xml  ots.xml      SSLIOP.xml

Step 6 Change SSLIOP.xml:

<property name="context.keyStore.URL" value = 
"/opt/new_directory_name/tomcat/ssliiop/bts10200_ks" />

<property name="context.trustStore.URL" value = 
"/opt/new_directory_name/tomcat/ssliiop/bts10200_ts" />

---

Managing CORBA Cache

The CORBA Cache reduces BTS query time and enhances EPOM performance.Using the Online Cache window an administrator can do the following with EPOM's internal CORBA cache:

•view

•delete—when a BTS server is deleted from the inventory list of the EPOM server

•renew

–when a BTS server restarts

–when someone invokes pkill cis on a BTS server

–when EPOM cannot find BTS CORBA objects

If the EPOM cannot query the BTS server, the administrator can delete or renew the CORBA Cache. This allows EPOM servers to query restarted BTS servers without being restarted themselves.

---

Step 1 From EPOM window, click Online Cache.

The Online Cache Management window opens.

Step 2 Cache information for all the configured BTS inventories can be viewed.

Step 3 Click Renew or Delete for BTS inventory to renew or delete its CORBA Cache.

Step 4 Click any BTS inventory on the left pane and the proceed with normal operations for the relevant inventory.

---

Managing EPOM

Upgrading EPOM

Backing up BTS Servers

Use the bts_export command to backup information on a BTS server. This is especially important to do before an upgrade.

bts_export looks like the following:

root@cyber229:opt 21> /opt/CSCOepom/bin/bts_export

NAME

bts_export - BTS Config Export

SYNOPSIS

bts_export -h hostname -v BTSver of hostname -o outfile [-l login -p password

]

DESCRIPTION

Export a BTS Configuration.

OPTIONS

-h Hostname

-v BTSversion of hostname (3.5,4.1,4.2,4.4.0,4.4.1. 4.5.0)

-o Output file

-l Login

-p Password

-s Site ID

EXAMPLES

1. bts_export -h bts_host -v host_bts_ver -o config.cli

2. bts_export -h bts_host -v host_bts_ver -l login -p passwd -s       siteid -o con

      fig.cli

---

Step 1 Change user to superuser:

su - root

Step 2 Create a temporary directory (epom_install2) for new EPOM image:

cp epom-5_0_20041216-1.tar /opt/epom_install2

cd /opt/epom_install2

Step 3 Extract image from tarfile:

tar -xvf epom-5_0_20041216-1.tar

Step 4 List contents:

ls

epom-5_0_20041216-1 epom-5_0_20041216-1.tar

Step 5 Move new, untarred image:

cd epom-5_0_20041216-1

Step 6 Install new image:

./setup

This message appears:

Warning:

The Extensible Provisioning and Operations Manager x.x is currently installed and must be 
uninstalled before this version is installed.

Step 7 Confirm uninstall current version.
A message appears:

Note: The CSCOepom data directory, /var/opt/CSCOepom still exists and must be removed 
manually.

Using previously installed MySQL database.

Using previously installed EPOM database.

Verifying EPOM Database.

Starting mysqld daemon with databases from /var//opt/CSCOepom/data/db

MySQL Server has started

Verifying EPOM Database Tables

Extensible Provisioning and Operations Manager installation is complete. 

Step 8 To clear the databases following a EPOM upgrade. Reinitialize the MySQL and EPOM databases by entering:

/opt/CSCOepom/mysql/install/bin/installMySQLDB -ifs

/opt/CSCOepom/mysql/install/bin/installEPOMDB -ifs

Stopping EPOM

Stopping the EPOM means stopping the tomcat server and mysql. After doing this, the browser will not show the EPOM login screen.

---

From the EPOM web server, type:

/opt/CSCOepom/bin/epom stop

The following message appears:

% /opt/CSCOepom/bin/epom stop

------------------------------------------------

Stopping EPOM

------------------------------------------------

Stopping MySQL

041216 15:26:12 mysqld ended (This appears only in the window from which EPOM was started)

MySQL server has stopped

Stopping Tomcat

Tomcat has stopped

-------------------------------------------------

EPOM Stopped

-------------------------------------------------

Uninstalling EPOM

You may need to uninstall EPOM in cases of :

•installation failure

•partial installation

•port set up assignment

Uninstalling EPOM ensures it will not conflict with other management tools like LAN Management Suite (LMS) or Call Manager.

---

Step 1 Change user to superuser:

su - root

Step 2 Ensure you are not in/opt/CSCOepom .

Step 3 Run uninstallation script:

/opt/CSCOepom/uninstall/uninstall

Step 4 Manually delete data files from /var/opt/CSCOepom.

---

Archiving EPOM Log Files

EPOM automatically generates various log files. These logs are created in the logs directory in the "/var/opt/CSCepom/logs/" path :

•User and Domain related information is generated in "audit.log"

•Stand alone tomcat information is generated in "Catalina.out"

•User HTTP request information is generated in "localhost_access_log" , this is generated date wise.

•Detail log is generated in the server "localhost_log" , this is generated date wise.

•Detail log of all debugging information is generated in "trace.log"

User access log details organized by date. Similarly "trace.log" appends information to the existing log. EPOM does not automatically clean this log information.

---

Step 1 Archive all logs:

/usr/bin/tar -cvf epom_logs.tar /var/opt/CSCOepom/logs/

Step 2 FTP or TFPT to another machine.

---

Troubleshooting

Problem 1

Recreating the EPOM database if needed (for example, if you forget the admin password and cannot access the database)

Troubleshooting Action 1

Reinitialize the EPOM databases:

/opt/CSCOepom/mysql/install/bin/install/ 
MySQLDB -ifs

/opt/CSCOepom/mysql/install/bin/install/EPOMDB -ifs

Problem 2

Problems in EPOM communicating to the EMS server, such as the one in the message below

Could not retrieve object attributes for object name. The most possible reason is failure to log into the EMS server or CORBA agent on EMS server is not working. Please make sure hostname/login/password/siteid is correct. Also check the log file

Troubleshooting Action 2

•Check the information in the EMS server definition to make sure it is correct.

•Check connectivity between EPOM and the Cisco BTS EMS (if they are not co-resident): Log in to the EPOM server and ping the EMS server.

•Verify that the correct CORBA adapter has been installed on the EMS server.

•Check the log files.

•View log files: Log files are stored in:

/var/$EPOM_INST_DIR/logs,
where $EPOM_INST_DIR is the EPOM installation directory.

By default log directory is /var/opt/CSCOepom/logs

Most important log file is trace.log, current activity can be seen by logging onto EPOM server and running the command:

tail -f /var/$EPOM_INST_DIR/logs/trace.log

•View BTS CIS log file, /opt/ems/log/CIS.log. Current activity can be seen by logging onto Cisco BTS server and running command:

tail -f /opt/ems/log/CIS.log

•Use log files for debugging EPOM problems or for supplying information to Cisco TAC.

•Check whether BTScis package is installed on EMS server. Log onto EMS server and running command:

pkginfo BTScis

•Check whether cis and ins services are running on EMS server. Log onto EMS server and invoke commands:

ps -ef | grep cis

ps-ef |grep ins

•Check whether ports 683 and 14001 are in LISTEN state on the EMS server. Log onto EMS server and invoke commands:

netstat -an |grep 683

netstat -an |grep 14001

•Check whether ports 683 and 14001 on EPOM server are in ESTABLISHED state with the EMS server. Log onto EPOM server and invoke commands:

netstat -an |grep 683

netstat -an |grep 14001

•Check connectivity between EPOM server and EMS server. Log onto EPOM server and ping EMS server from EPOM server.

Check whether EPOM server can send packets to EMS server at ports 683 and 14001. Log onto EMS server and invoke commands:

telnet bts_server 683

telnet bts_server 14001

•Check whether EPOM server can resolve the hostname of the EMS server. As a workaround insert hostname entries into the /etc/hosts file of EPOM server.

•Log onto both EPOM server and EMS server and use snoop command to verify that sizeable amount of data is exchanged between the 2 servers on port 683 and 14001.

•Verify that the hostname of the EMS server is same as the name used by the EMS server for its ins service. Log onto EMS server and invoke following command:

ps -ef | grep ins

Output would also contain the hostname to which ins service is bound.

•Use IP address of the EMS server, while adding to EPOM server.

•Use IP address of the EMS server to initialize ins and cis services on the EMS server. This makes it for EPOM CORBA client to locate the EMS server CORBA server naming context.

•If you notice exception of sort vmcid 0X200 or another hexadecimal digit, then In this case renew the EPOM CORBA Cache for the relevant BTS server.

•If this happens too often for a EMS server that was accessible some time before, it means that the particular EMS server has been restarted or the CIS services have been restarted. In this case renew the EPOM CORBA Cache for the relevant BTS server.

Problem 3

EPOM BTS Command Navigation Tree, problems. Tree appears without commands or JavaScript is rendered on the page.

Troubleshooting Action 3

EPOM has introduced a new Clear Tree button, click on this button to re-build EPOM BTS Command Navigation Tree. This tree is otherwise cached and without clicking the Clear Tree buttons, invalid tree appears.

In case you still face the issue, then delete all the cache from your web-browser. For Internet Explorer, click Tools > Internet Options > General > Temporary Internet Files > Delete Files. Or use a separate machine to do the same.

Problem 4

EPOM logs not being renewed.

Troubleshooting Action 4

•Check disk usage of /var directory on EPOM server. Log onto EPOM server and invoke command:

df -k |grep var

•The logs with older date stamps can be safely archived in some other directory.

•The logs can be moved to another directory that has more disk space than /var directory e.g /opt , but that needs restart of EPOM server. As an example if /opt/CSCOepom is the install directory, then the logs can be moved from their current default location /var/opt/CSCOepom/logs to /opt/CSCOepom/logs. Log onto EPOM Server and invoke following commands:

cd /var/opt/CSCOepom

mv logs /opt/CSCOepom/logs

ln -s /opt/CSCOepom/logs logs

/opt/CSCOepom/bin/epom stop

/opt/CSCOepom/bin/epom start

Problem 5

EPOM takes time to build its CORBA Cache.

Troubleshooting Action 5

This is default behavior of EPOM, generally EPOM should be used at least 10 minutes after it has been started. The time taken by EPOM depends on the number of EMS servers added to EPOM server.

If one of the EMS server faces CORBA communication problem, then it would have a tail effect on the EMS servers following it. Once the EPOM CORBA Cache is built, it takes very less time to communicate through CORBA with the EMS servers.

Problem 6

EPOM JVM tunings

Troubleshooting Action 6

The best Sun JVM settings have been observed as:

-server -XX:+UseParallelGC -Xms512M -Xmx512M

These settings have major impact on performance of EPOM and Cisco doesn't suggest changing these parameters.

If really essential, the tunings can be changed by modifying EPOM_OPTS variable in the file /opt/CSCOepom/tomcat/bin/epomcatalina.sh

If it is felt that the current Sun JVM settings are not fine for your installation, then enable the:

-verbosegc

option in EPOM_OPTS, as defined above and contact Cisco TAC with your installation details and logfile:

/var/opt/CSCOepom/logs/Catalina.out

Problem 7

For a Change (Edit) operation, if a particular parameter has some value and the user wants that value to be null. The simply deleting the value from EPOM GUI during a change (Edit) operation would not nullify it.

Troubleshooting Action 7

Enter NULL for the relevant parameter in EPOM GUI.

Problem 8

EPOM bts_export errors

Troubleshooting Action 8

Many errors arise while usage of bts_export, due to two main reasons:

•CORBA Communication is not established properly; try using IP address rather than hostname.

•root or other such Cisco BTS EMS logins are used, which are not associated with a BTS shell. Try using Cisco BTS EMS logins that are associated with BTS CLI shell, like optiuser.

Problem 9

CORBA Error: Exceeded maximum number of users.

Troubleshooting Action 9

•You must terminate the CIS process in BTS by logging into the EMS server and invoke the following commands:

ps -ef |grep cis

kill -9 <cis process id

Problem 10

After changing BTS CIS from non-secure to secure or secure to non-secure EPOM is unable to communicate with BTS 10200 EMS.

Troubleshooting Action 10

For changing from secure to non-secure and vice-versa, you must renew or delete the cache for that inventory.