Table Of Contents

Troubleshooting VQE Software Components

Useful Linux Tools

VQE Logging and Log Files

VQE Server and VQE Tools Logging and Log Files

VQE-S AMT and VCPT Logging and Log Files

VQE Files, Directories, and Permissions

Changing Permissions

Replacing a Lost or Corrupt VQE File

VQE Server, VQE-S AMT, and VCPT Troubleshooting

Monitoring Received Multicast Streams

Static Route for a Management Network Is Missing on CDE110 Hosting VQE-S or VQE Tools

Repeated Message on CDE110 Hosting VQE-S or VQE Tools

VQE-S AMT and VCPT Unavailable

Channels Are Not Displayed in VQE-S AMT

VQE-S AMT Channels Status Summary Has No Graphs

Internal Server Error When VCPT Tries to Send Channel Information to VQE-S

Using the vqereport Command

Stopping, Starting, and Restarting VQE-S

Stopping, Starting, and Restarting VCDS

Troubleshooting VQE Software Components

---

This chapter describes how to identify and remedy problems related to Cisco VQE Server (VQE-S), VQE-S Application Monitoring Tool (VQE-S AMT or AMT), and VQE Channel Provisioning Tool (VCPT). This chapter contains the following major topics:

•Useful Linux Tools

•VQE Logging and Log Files

•VQE Files, Directories, and Permissions

•VQE Server, VQE-S AMT, and VCPT Troubleshooting

•Using the vqereport Command

•Stopping, Starting, and Restarting VQE-S

•Stopping, Starting, and Restarting VCDS

Useful Linux Tools

Table 5-1 provides information on some Linux commands that are particularly useful for troubleshooting VQE software components.

Table 5-1 Useful Linux Commands

Command	Description	Notes
arp	Manipulates the kernel's ARP cache	—
ethtool	Displays or changes Ethernet card settings	Detailed status on an Ethernet interface
ifconfig	Configures a network interface	Traditional interface configuration and status
ip	Shows or manipulates routing, devices, policy routing and tunnels	Most useful interface to all Linux networking
netstat	Displays network connections, routing tables, interface statistics, masquerade connections, and multicast memberships	Provides traditional network state information
ping	Sends ICMP ECHO_REQUEST to network hosts	—
tcpdump	Dumps and analyzes traffic on a network	Captures packet data

VQE Logging and Log Files

VQE-related log entries can provide useful information for troubleshooting. The VQE-related log files are described in these sections:

•VQE Server and VQE Tools Logging and Log Files

•VQE-S AMT and VCPT Logging and Log Files

VQE Server and VQE Tools Logging and Log Files

VQE Server logging is based on the Berkeley syslogd utility, which is part of the inetutils package in Linux. The following is a typical VQE-S log entry:

Oct 22 08:42:03 minnie vqes_mlb: <<%VQES_MLB-3-MLB_NIC_DOWN>> Network interface eth3 went down. 

Every logged message contains five fields in this order:

•Timestamp (Oct 22 08:42:03)

•Host name (minnie)

•Name of process logging the message (vqes_mlb)

•Identity string (<<%VQES_MLB-3-MLB_NIC_DOWN>>)

•Log message (Network interface eth3 went down.)

For information on VQE system messages, see Appendix C, "VQE System Messages."

By default, VQE-S and VQE Tools system messages are written to the file /var/log/vqe/vqe.log. The location of the file is configured in the /etc/syslog.conf file. For information on configuring VQE-S logging, see the "Configuring VQE-S Logging" section on page 4-17.

VQE-S and VQE Tools log files are rotated by logrotate. The logrotate facility is configured to rotate a log file when it is larger than 20 MB. The old versions of the log files are rotated up to 20 times before being removed.

---

Note If you move the VQE log file to another directory, be sure to stop the syslogd daemon before moving the vqe.log file. If syslogd is started with no vqe.log file in the /var/log/vqe directory, syslogd creates the vqe.log file.

---

VQE-S AMT and VCPT Logging and Log Files

VQE-S AMT and VCPT are both web applications that make use of the Apache Tomcat application server. Apache Tomcat uses the log4j logging system. For AMT and VCPT, the logging output is configured by these log4j configuration files:

•For AMT, the configuration file is located at /usr/share/tomcat5/webapps/ems/WEB-INF/classes/ log4j.properties.

•For VCPT, the configuration file is located at /usr/share/tomcat5/webapps/vcpt/WEB-INF/classes/ log4j.properties.

By default, the log files for AMT and VCPT are saved in these locations:

•For AMT, the log files are saved in /usr/share/tomcat5/logs/vqe.log.

•For VCPT, the log files are saved in /usr/share/tomcat5/logs/vcpt.log.

Table 5-2 lists the most important default log4j configuration settings for AMT and VCPT logging.

Table 5-2 Log4j Default Configuration for AMT and VCPT

Configuration Property	Default Setting
Logging priority level	Warning
Logging output	Log messages are written to both the console and the log file
Maximum size of the log file before it rotates	100 KB
Maximum number of backup files	4

For information on configuring the log4j.properties file, see the log4j documentation at these URLs:

http://tomcat.apache.org/tomcat-5.5-doc/logging.html

http://logging.apache.org/log4j/1.2/manual.html

VQE Files, Directories, and Permissions

If you encounter a problem related to VQE-related files, directories, or permissions, use the information in the following tables to resolve the issues.

•Table 5-3 provides information for the CDE110 that hosts VQE-S.

•Table 5-4 provides information for the CDE110 that hosts VCPT and VQE Client Channel Configuration Delivery Server.

These sections provide additional information on resolving problems:

•Changing Permissions

•Replacing a Lost or Corrupt VQE File

Table 5-3 VQE-S Files, Directories, and Permissions  

File or Directory	Required Permissions, Owner, and Group	Description
/etc/opt/vqes (directory)	drwx------ vqes vqes	--
/etc/opt/vqes/vcdb.conf	-rw-r--r-- root root	VCDB configuration file
/etc/opt/vqes/vcdb.conf.sample	-r-------- vqes vqes	VCDB sample configuration file
/etc/opt/vqes/vqe_channels.cfg	-rw-r--r-- vqes vqes	VQE-S channel configuration file
/etc/opt/vqes/vqes_syslog.conf	-rw-r--r-- vqes vqes	VQE-S syslog configuration file
/opt/vqes (directory)	drwxr-xr-x vqes vqes	--
/opt/vqes/bin (directory)	drwxr-xr-x vqes vqes	VQE-S binary directory
/opt/vqes/bin/fbt_flush.sh	-r-xr----- root vqes	VQE-S Feedback Target flush script
/opt/vqes/bin/mlb	-r-xr-xr-x root root	VQE-S Multicast Load Balancer process
/opt/vqes/bin/mlb_ethtool	-r-sr-x--- root vqes	VQE-S Ethernet utility for Multicast Load Balancer
/opt/vqes/bin/process_monitor	-r-xr-xr-x root root	VQE-S Process Monitor process
/opt/vqes/bin/stun_server	-r-xr-xr-x root root	VQE-S STUN Server process
/opt/vqes/bin/vqe_cfgtool	-r-x------ vqes vqes	vqe_cfgtool command
/opt/vqes/bin/vqereport	-r-x------   root root	vqereport command
/opt/vqes/bin/vqes_cp	-r-x------ root root	VQE-S Control Plane process
/opt/vqes/bin/vqes_dp	-r-xr-xr-x root root	VQE-S Data Plane process
/opt/vqes/bin/vqes_bin.sha1sum	-r--r----- vqes vqes	SHA-1 checksums list
/opt/vqes/bin/vqes_syslogd	-r-xr-xr-x vqes vqes	VQE-S syslogd process
/opt/vqes/bin/xmlrpc	-r-xr-xr-x vqes vqes	XML-RPC client utility
/var/log/vqe (directory)	drwxr-xr-x vqes vqes	VQE log file directory
/var/log/vqe/vqe.log	-rw------- vqes vqes	VQE system messages log file

Table 5-4 VQE Tools Files, Directories, and Permissions  

File or Directory	Required Permissions, Owner, and Group	Description
/etc/opt/vqes (directory)	drwx------ root root	--
/etc/opt/vqes/vcdb.conf	-rw-r--r-- root root	VCDB configuration file
/etc/opt/vqes/vcdb.conf.sample	lrwxrwxrwx root root	A link to the VCDB sample configuration file
/etc/opt/vqes/vcdb.conf.tools.sample	-r-------- vqes vqes	VCDB sample configuration file
/etc/opt/vqes/VCDServer.cfg	-rw-r--r-- root root	VQE Client Channel Configuration Delivery Server (VCDS) configuration file
/etc/opt/vqes/vqec_channels.cfg	-rw--r--r-- root root	VCDS channel configuration file
/etc/opt/vcpt (directory)	drwxr-xr-x root root	--
/etc/opt/vcpt/data (directory)	drwxr-xr-x root root	Directory for VCPT configuration files
VCPT configuration files in /etc/opt/vcpt/data	-rw-r--r--  root root	VCPT configuration files (Filenames are user-defined and vary.)
/opt/vqes (directory)	drwxr-xr-x vqes vqes	--
/opt/vqes/bin (directory)	drwxr-xr-x vqes vqes	--
/opt/vqes/bin/vqe_cfgtool	-r-x------ vqes vqes	vqe_cfgtool command
/opt/vqes/bin/vqereport	-r-x------   root root	vqereport command
/var/log/vqe (directory)	drwxr-xr-x vqes vqes	VQE log directory
/var/log/vqe/vqe.log	-rw------- vqes vqes	VQE system messages log file

Changing Permissions

To change permissions on a file or directory, use the chmod command:

---

Step 1 If needed, log in as root on the CDE110 that hosts VQE-S or that hosts VCPT.

Step 2 To change permissions, issue the following command:

chmod mode file_or_dir_name

In the preceding command, the two arguments are as follows:

•mode specifies the correct permissions.

•file_or_dir_name specifies the file or directory whose permissions will be set.

For example:

[root@system]# chmod 555 /opt/vqes/bin/vqes_cp 

For information on setting permissions, see the man page for the chmod command.

---

Replacing a Lost or Corrupt VQE File

If a required VQE file that has not been backed up becomes lost or corrupt, you must reinstall the version of the VQE software that the CDE110 server is running. To preserve the current VQE configuration, the software reinstallation can be an ISO upgrade installation or an RPM installation. For information on VQE software installation, see the Release Notes for Cisco CDA Visual Quality Experience Application, Release 3.0.

If the lost or corrupt file that was not backed up is a configuration file located in the directories under /etc, you can get a copy of the factory default version of the configuration file from the /vqe-etc/etc-pristine directory. To restore the lost or corrupt file, copy the factory default configuration file to its correct location under /etc, and apply the current VCDB configuration to the /etc configuration files using the vqe_cfgtool command and the -apply option.

For information on the vqe_cfgtool command, see the "Using the VQE Configuration Tool Command-Line Options" section on page 6-16.

VQE Server, VQE-S AMT, and VCPT Troubleshooting

This section provides information on troubleshooting the following specific problems that can occur with VQE Server, VQE-S AMT, and VCPT.

•Monitoring Received Multicast Streams

•Static Route for a Management Network Is Missing on CDE110 Hosting VQE-S or VQE Tools

•Repeated Message on CDE110 Hosting VQE-S or VQE Tools

•VQE-S AMT and VCPT Unavailable

•Channels Are Not Displayed in VQE-S AMT

•VQE-S AMT Channels Status Summary Has No Graphs

•Internal Server Error When VCPT Tries to Send Channel Information to VQE-S

Monitoring Received Multicast Streams

For troubleshooting VQE-S, using VQE-S AMT to monitor received multicast streams is recommended.

We recommend that you use AMT to monitor received multicast streams. In the AMT Channel Statistics window for each channel (Figure 5-1), the Lost RTP Packets field for the Channel Input Stream is a per-channel counter that provides the number of missing RTP packets. The Lost RTP Packets counter increments when an RTP packet is sent for the channel's original source stream but is not received by the VQE Server.

Figure 5-1 Lost RTP Packets

Static Route for a Management Network Is Missing on CDE110 Hosting VQE-S or VQE Tools

Symptom: A static route for a management network, which was configured in /etc/sysconfig/static-routes-iputil, are missing, and the VQE-S host or VQE Tools host can no longer reach the management network.

Explanation: This condition can occur when CDE110 interfaces are manually shutdown by the operator using the Linux ifdown ifname command and then manually brought back up using the ifup ifname command. This condition can also occur when the CDE110 interfaces are reset for any reason.

If the next hop routers for a static route configured in /etc/sysconfig/static-routes-iputil become unreachable due to CDE110 interfaces being manually shut down in this manner, the corresponding routes are removed from the routing table and will not be reinstalled even when one or more of the CDE110 interfaces are brought back up.

(The VQE Startup Utility and the VQE Configuration Tool can be used to configure a static route for a management network. These facilities use /etc/sysconfig/static-routes-iputil for the static route to a management network.)

Remedy: To reinstall a management-network static route that has been removed from the routing table, use the route add command:

route add -net management-network-addr/prefix-length gw gateway-addr

For example:

[root@system]# route add -net 192.0.2.0/16 gw 10.86.21.1 

In the preceding, the management-network-addr/prefix-length is the IP address and prefix length for the management network. The gateway-addr is the IP address of the router interface that is directly attached to the CDE110 Ethernet port that will be used for management network traffic.

To avoid this problem in the future, instead of using the ifup ifname command to bring each of the interfaces up individually, bring them all back up at once using the service network start command. This will bring up all interfaces that were shut down, and will also reinstall all of the routes contained in the file /etc/sysconfig/static-routes-iputil.

Repeated Message on CDE110 Hosting VQE-S or VQE Tools

Symptom: During initial system configuration, a new Cisco CDE110 hosting VQE-S or VQE Tools boots successfully, the root password is set successfully, but the following message displays repeatedly:

SIOCGIFADDR: Cannot assign requested address

SIOCGIFADDR: Cannot assign requested address

SIOCGIFADDR: Cannot assign requested address

SIOCGIFADDR: Cannot assign requested address

SIOCGIFADDR: Cannot assign requested address

SIOCGIFADDR: Cannot assign requested address

Explanation: This error can occur when you configure the CDE110 server manually (rather than use the VQE Startup Configuration Utility). The message may display so frequently that it is difficult to edit a file to continue the initial system configuration.

The message can be caused by an Ethernet interface configuration issue in the ifcfg-eth# files (ifcfg-eth1, ifcfg-eth2, and so on).

Remedy: Look for Ethernet interface address or other errors in the four ifcfg-eth# files. Use a text editor to correct any errors.

Use the following command to verify that the addresses specified in the ifcfg-eth# files have been assigned to the interfaces:

ifconfig -a

Continue with the initial configuration tasks.

VQE-S AMT and VCPT Unavailable

Symptom: To browse to VQE-S AMT or VCPT, you enter a valid IP address or fully qualified domain name, but the browser displays the message "Page Cannot Be Found" (status code 404) or "Service Temporarily Unavailable" (status code 503).

Explanation: Both AMT and VCPT require the Apache HTTP server service (httpd) and the Apache Tomcat application server service (tomcat5) to be running. When you attempt to browse to AMT or VCPT and one of these services is not running, the status codes are typically as follows:

•"Page Cannot Be Found" (status code 404) indicates that httpd is not running.

•"Service Temporarily Unavailable" (status code 503) indicates that tomcat5 is not running.

To verify that httpd is not running, log in as root and issue following command:

[root@system]# ps -ef | grep httpd 

If httpd is running, the output is similar to the following. If httpd is not running, this output will not be present.

[root@system]# ps -ef | grep httpd 

root      2880     1  0 Jul18 ?        00:00:00 /usr/sbin/httpd

apache    4881  2880  0 04:03 ?        00:00:00 /usr/sbin/httpd

apache    4882  2880  0 04:03 ?        00:00:00 /usr/sbin/httpd

apache    4883  2880  0 04:03 ?        00:00:00 /usr/sbin/httpd

apache    4884  2880  0 04:03 ?        00:00:00 /usr/sbin/httpd

apache    4885  2880  0 04:03 ?        00:00:00 /usr/sbin/httpd

apache    4886  2880  0 04:03 ?        00:00:00 /usr/sbin/httpd

apache    4887  2880  0 04:03 ?        00:00:00 /usr/sbin/httpd

apache    4888  2880  0 04:03 ?        00:00:00 /usr/sbin/httpd

To verify that tomcat5 is running, log in as root and issue following command:

[root@system]# ps -ef | grep tomcat5

If tomcat5 is running, the output is similar to the following. If tomcat5 is not running, this output will not be present.

root      2915     1  0 Jul18 ?        00:00:11 /usr/java/default/bin/java 
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager 
-Djava.util.logging.config.file=/usr/share/tomcat5/conf/logging.properties 
-Djava.endorsed.dirs=/usr/share/tomcat5/common/endorsed -classpath 
:/usr/share/tomcat5/bin/bootstrap.jar:/usr/share/tomcat5/bin/commons-logging-api.jar 
-Dcatalina.base=/usr/share/tomcat5 -Dcatalina.home=/usr/share/tomcat5 
-Djava.io.tmpdir=/usr/share/tomcat5/temp org.apache.catalina.startup.Bootstrap start

Remedy: If either httpd or tomcat5 is not running, log in as root and start the services that are stopped as follows:

[root@system]# service httpd restart 

[root@system]# service tomcat5 restart

Verify that the httpd and tomcat5 services are running as shown earlier in this Explanation.

If the httpd or tomcat5 processes continue to fail, check the log files for the services to determine the cause. The tomcat5 log files are in /var/log/tomcat5/catalina.date.log. For information on the log files for each service, see the vendor documentation for the Apache HTTP server or for the Apache Tomcat application server.

Channels Are Not Displayed in VQE-S AMT

Symptom: After channels are defined in VCPT and channel information is pushed to servers, VQE-S AMT does not display the channels.

Explanation: The most likely reason why AMT does not display the channels is that the VQE Server was not associated with the channels in VCPT.

To determine whether the channels are associated with this VQE Server, follow these steps:

---

Step 1 Log in to VCPT.

Step 2 Click the Association tab

Step 3 Select the VQE Server in the Select Server pull-down menu.

The channels should be in the Selected group. Also, make sure the VQE Server is defined with the role VQE-S. If the role is wrong, correct it with VCPT from the Servers tab. For information on updating server information, see the "Viewing or Updating Server Information" section on page 3-15.

---

If the channels are correctly associated with the VQE Server, it is possible that there were problems when VQE Server attempted to process the channels. Check the VQE Server log files for errors in channel processing. For information on these log files, see the "VQE Server and VQE Tools Logging and Log Files" section.

Remedy: If the channels have not been associated with the VQE Server in VCPT, associate the channels with VCPT from the Association tab. For information on associating channels with servers, see the "Defining Channel Associations for a Server" section on page 3-16.

VQE-S AMT Channels Status Summary Has No Graphs

Symptom: In the AMT VQE-S Status window (Figure 4-2 on page 4-4), the Channels Status Summary does not have graphs for the channels.

Explanation: If one or more channels are successfully created in AMT, the channels should appear in a Channels Status Summary graph of active, inoperative, and inactive channels in the VQE-S Status window. To be displayed, the graphs require that Adobe Flash Player be installed on the computer that hosts the browser accessing AMT.

Remedy: Download and install Adobe Flash Player on the computer that hosts the browser used to access AMT. The software is free and can be found at this URL:

http://www.adobe.com/shockwave/download/download.cgi?P1_Prod_Version=ShockwaveFlash

It is recommended that you close all other open browser windows before performing the installation.

Internal Server Error When VCPT Tries to Send Channel Information to VQE-S

Symptom: When VCPT tries to send channel information to VQE Server, you get an "Internal Server Error." Java exceptions in the catalina.out file indicate that there is an address conflict or unreachable address.

Explanation: The Internal Server Error is displayed when you use VCPT to send channel information to a VQE Server. The Status of Last Send column on the VCPT Servers tab shows the error.

From the CDE110 that hosts VQE-S, you are able to successfully ping the machine running the web browser used for VQE-S AMT, and ping the CDE110 hosting VCPT.

In the /usr/share/tomcat5/logs/catalina.out file of the Apache Tomcat application server, there are Java exceptions indicating an address conflict or unreachable address. For example:

java.net.BindException: Address already in use 
at java.net.PlainSocketImpl.socketBind(Native Method) 
at java.net.PlainSocketImpl.bind(Unknown Source) 
...

java.net.SocketException: Network is unreachable 
at java.net.PlainSocketImpl.socketConnect(Native Method) 
at java.net.PlainSocketImpl.doConnect(Unknown Source)

Remedy: The Internal Server Error can be caused by one or more errors in the /etc/hosts file on the CDE110 that hosts VQE-S. Check the /etc/hosts file for typing or syntax errors, correct them, and save the file. Reboot the CDE110.

If you still receive the same Internal Server Error and Java exceptions, check the following files to see if you can find some indication for what is causing the error.

•/var/log/vqe/vqe.log

•/var/log/httpd/error_log

•/var/log/httpd/access_log

•/usr/share/tomcat5/logs/catalina.out

•/var/log/messages

Using the vqereport Command

The vqereport command can be useful for VQE troubleshooting. You can use the vqereport command to gather information on the VQE system's software configuration and hardware. The information can be used by the VQE administrator or Cisco technical support personnel to diagnose problems with VQE software or with the Cisco Content Delivery Engine 110 (CDE110) hardware.

---

Caution The vqereport command can take up to 15 minutes to complete and may cause VQE services (such as Unicast Retransmission) to be degraded for the duration of the execution.

---

The syntax for the vqereport command is as follows:

vqereport [-h | -help]

Syntax Description

-h | -help

Displays help information.

Usage Guidelines

The vqereport command generates a report file that, when the command finishes, is located in /root/hostname.dateandtime.tar.bz2. If appropriate, the report file can be attached to field issue reports for Cisco technical support.

---

Note You must log in as root to execute the vqereport command.

---

The vqereport command can be executed on the CDE110 that hosts VQE Server or on the CDE110 that hosts VCPT. The vqereport executable is located at /opt/vqes/bin/vqereport.

To have the operating system find the vqereport executable, you must set the PATH environment variable to include the location of the /opt/vqes/bin directory, or use the full path to vqereport when executing the command.

All information gathered is considered confidential, and Cisco will use this information for diagnostic purposes only.

Examples

The following example shows the execution of the vqereport command and some abbreviated output.

[root@system ~]# /opt/vqes/bin/vqereport 

This utility will go through and collect some detailed information about the hardware and 
setup of your VQE system. This information will be used to diagnose problems with your 
system and will be considered confidential information. Cisco will use this information 
for diagnostic purposes ONLY. 

Please wait while we collect information about your system. 

This process may take a while to complete.... 
No changes will be made to your system during this process. 
NOTE: You can safely ignore a failed message. This only means a file we were checking for 
did not exist. 

Press ENTER to continue, or CTRL-C to quit.

********************************************************************** 
Collect Redhat sysreport 

 
********************************************************************** 
Collect information of installed RPM packages 

... 

output omitted 

... 

Report file /root/<hostname>.20071115164635.tar.bz2 has been generated, please send it to 
Cisco support.

Stopping, Starting, and Restarting VQE-S

The VQE-S application is a service that is started with the Linux service command.

To stop, start, or restart the VQE-S application, follow these steps:

---

Step 1 Log in as root.

Step 2 Depending on what is required, issue one of the following commands:

•To stop VQE-S, issue the following command:

[root@system]# service vqes stop 

•To start VQE-S, issue the following command:

[root@system]# service vqes start 

•To restart (stop and then start) VQE-S, issue the following command:

[root@system]# service vqes restart 

---

Stopping, Starting, and Restarting VCDS

The VQE Client Channel Configuration Delivery Server (VCDS) application is a service that is started with the Linux service command.

To stop, start, or restart the VCDS application, follow these steps:

---

Step 1 Log in as root.

Step 2 Depending on what is required, issue one of the following commands:

•To stop VCDS, issue the following command:

[root@system]# service vcds stop 

•To start VCDS, issue the following command:

[root@system]# service vcds start 

•To restart (stop and then start) VCDS, issue the following command:

[root@system]# service vcds restart 

---