Troubleshooting VQE Software Components
This chapter describes how to identify and remedy problems related to the Cisco VQE-S (VQE-S), the VQE-S Application Monitoring Tool (AMT), and the VQE Channel Provisioning Tool (VCPT). This chapter contains the following major topics:
•Useful Linux Tools
•VQE Logging and Log Files
•VQE Files, Directories, and Permissions
•VQE-S and VQE Tools Server Troubleshooting
•RCC Troubleshooting
•Using the vqereport Command
•Gathering Information for a Service Request
•Stopping, Starting, and Restarting VQE-S
•Stopping, Starting, and Restarting VCDS
•Resetting the Root Password on the VQE-S or VQE Tools Server
Useful Linux Tools
Table 6-1 provides information on some Linux commands that are particularly useful for troubleshooting VQE software components.
Table 6-1 Useful Linux Commands
|
|
|
arp |
Manipulates the ARP cache of the kernel |
— |
ethtool |
Displays or changes Ethernet card settings |
Detailed status on an Ethernet interface |
ifconfig |
Configures a network interface |
Traditional interface configuration and status |
ip |
Shows or manipulates routing, devices, policy routing, and tunnels |
Most useful interface to all Linux networking |
netstat |
Displays network connections, routing tables, interface statistics, masquerade connections, and multicast memberships |
Provides traditional network state information |
ping |
Sends ICMP ECHO_REQUEST to network hosts |
— |
tcpdump |
Dumps and analyzes traffic on a network |
Captures packet data |
VQE Logging and Log Files
VQE-related log entries can provide useful information for troubleshooting. The VQE-related log files are described in these sections:
•VQE Server and VQE Tools Logging and Log Files
•VQE-S AMT, VCDS AMT, and VCPT Logging and Log Files
VQE Server and VQE Tools Logging and Log Files
The VQE Server (VQE-S) logging is based on the syslog-ng utility, which is part of the inetutils package in Linux. The following is a typical VQE-S log entry:
Oct 22 08:42:03 minnie vqes_mlb: <<%VQES_MLB-3-MLB_NIC_DOWN>> Network interface eth3 went down.
Every logged message contains the following five fields in this order:
•Timestamp (For example, Oct 22 08:42:03)
•Host name (For example, minnie)
•Name of process logging the message (For example, vqes_mlb)
•Identity string—FACILITY-SEVERITY-MNEMONIC (For example, <<%VQES_MLB-3-MLB_NIC_DOWN>>)
•Message text (For example, Network interface eth3 went down.)
For information on VQE system messages, see "VQE System Messages."
By default, the VQE-S and VQE Tools system messages are written to the file /var/log/vqe/vqe.log. The location of the file is configured in the /etc/syslog.conf file. For information on configuring VQE-S logging, see the "Change VQE-S Logging" section.
The VQE-S and VQE Tools log files are rotated by logrotate. The logrotate facility is configured to rotate a log file when it is larger than 20 MB. The old versions of the log files are rotated up to 20 times before being removed.
Note If you move the VQE log file to another directory (for example, when backing up log files), be sure to stop the syslog-ng daemon before moving the vqe.log file. If syslog-ng is started with no vqe.log file in the /var/log/vqe directory, syslog-ng creates the vqe.log file.
In addition, you can send system messages to remote servers for centralized logging. For information on configuring remote syslog servers, see the "Remote Syslog Hosts" section.
VQE-S AMT, VCDS AMT, and VCPT Logging and Log Files
The VQE-S AMT, VCDS AMT, and VCPT are all web applications that make use of the Apache Tomcat application server. Apache Tomcat uses the log4j logging system. For the VQE-S AMT, the VCDS AMT, and the VCPT, the logging output is configured by these log4j configuration files:
•For the VQE-S AMT and the VCDS AMT, the configuration file is located at /usr/share/tomcat5/webapps/ems/WEB-INF/classes/ log4j.properties.
•For the VCPT, the configuration file is located at /usr/share/tomcat5/webapps/vcpt/WEB-INF/classes/ log4j.properties.
By default, the log files for the VQE-S AMT, the VCDS AMT, and the VCPT are saved in these locations:
•For the VQE-S AMT and the VCDS AMT, the log files (vqe.log) are saved in the /usr/share/tomcat5/logs directory.
•For VCPT, the log files (vcpt.log) are saved in /usr/share/tomcat5/logs directory.
Table 6-2 lists the most important default log4j configuration settings for VQE-S AMT, VCDS AMT, and VCPT logging.
Table 6-2 Log4j Default Configuration for VQE-S AMT, VCDS AMT, and VCPT
|
|
Logging priority level |
Warning |
Logging output |
Log messages are written to both the console and the log file |
Maximum size of the log file before it rotates |
100 KB |
Maximum number of backup files |
4 |
For information on configuring the log4j.properties file, see the log4j documentation at these URLs:
http://tomcat.apache.org/tomcat-5.5-doc/logging.html
http://logging.apache.org/log4j/1.2/manual.html
VQE Files, Directories, and Permissions
If you encounter a problem related to VQE-related files, directories, or permissions, use the information in the following tables to resolve the issues:
•Table 6-3 provides information for the CDE110 that hosts the VQE-S.
•Table 6-4 provides information for the CDE110 that hosts VCPT, VCDS AMT, and VCDS.
These sections provide additional information on resolving problems:
•Changing Permissions
•Replacing a Lost or Corrupt VQE File
Table 6-3 VQE-S Files, Directories, and Permissions
|
Required Permissions, Owner,
|
|
/etc/opt/vqes (directory) |
drwx------ vqes vqes |
— |
/etc/opt/vqes/vcdb.conf |
-rw-r--r-- root root |
VCDB configuration file |
/etc/opt/vqes/vcdb.conf.sample |
-r-------- vqes vqes |
VCDB sample configuration file |
/etc/opt/vqes/vqe_channels.cfg |
-rw-r--r-- vqes vqes |
VQE-S channel configuration file |
/opt/vqes (directory) |
drwxr-xr-x vqes vqes |
— |
/opt/vqes/bin (directory) |
drwxr-xr-x vqes vqes |
VQE-S binary directory |
/opt/vqes/bin/fbt_flush.sh |
-r-xr----- root vqes |
VQE-S feedback target flush script |
/opt/vqes/bin/mlb |
-r-xr-xr-x root root |
VQE-S Multicast Load Balancer process |
/opt/vqes/bin/mlb_ethtool |
-r-sr-x--- root vqes |
VQE-S Ethernet utility for Multicast Load Balancer |
/opt/vqes/bin/process_monitor |
-r-xr-xr-x root root |
VQE-S Process Monitor process |
/opt/vqes/bin/stun_server |
-r-xr-xr-x root root |
VQE-S STUN Server process |
/opt/vqes/bin/vqe_cfgtool |
-r-x------ vqes vqes |
vqe_cfgtool command |
/opt/vqes/bin/vqereport |
-r-x------ root root |
vqereport command |
/opt/vqes/bin/vqes_cp |
-r-x------ root root |
VQE-S Control Plane process |
/opt/vqes/bin/vqes_dp |
-r-xr-xr-x root root |
VQE-S Data Plane process |
/opt/vqes/bin/vqes_bin.sha1sum |
-r--r----- vqes vqes |
SHA-1 checksums list |
/etc/syslog-ng/syslog-ng.conf |
-rw-r--r-- 1 root root |
VQE-S syslog configuration file |
/opt/syslog-ng/sbin/syslog-ng |
-rwxr-xr-x root root |
VQE-S syslog-ng process |
/opt/vqes/bin/xmlrpc |
-r-xr-xr-x vqes vqes |
XML-RPC client utility |
/var/log/vqe (directory) |
drwxr-xr-x vqes vqes |
VQE log file directory |
/var/log/vqe/vqe.log |
-rw------- vqes vqes |
VQE system messages log file |
Table 6-4 VQE Tools Files, Directories, and Permissions
|
Required Permissions, Owner,
|
|
/etc/opt/vqes (directory) |
drwx------ root root |
— |
/etc/opt/vqes/vcdb.conf |
-rw-r--r-- root root |
VCDB configuration file |
/etc/opt/vqes/vcdb.conf.sample |
lrwxrwxrwx root root |
Link to the VCDB sample configuration file |
/etc/opt/vqes/vcdb.conf.tools.sample |
-r-------- vqes vqes |
VCDB sample configuration file |
/etc/opt/vqes/vqec_channels.cfg |
-rw--r--r-- root root |
VCDS channel configuration file |
/etc/opt/vcpt (directory) |
drwxr-xr-x root root |
— |
/etc/opt/vcpt/data (directory) |
drwxr-xr-x root root |
Directory for VCPT configuration files |
VCPT configuration files in /etc/opt/vcpt/data |
-rw-r--r-- root root |
VCPT configuration files (Filenames are user-defined and vary.) |
/etc/opt/vcpt/data/sdp (directory) |
drwxr-xr-x root root |
Directory for SDP-formatted files that VCPT creates for each valid VCPT configuration |
SDP-formatted files in /etc/opt/vcpt/data/sdp |
-rw-r--r-- root root |
SDP-formatted files for each valid VCPT configuration |
/opt/vqes (directory) |
drwxr-xr-x vqes vqes |
— |
/opt/vqes/bin (directory) |
drwxr-xr-x vqes vqes |
— |
/opt/vqes/bin/vcds_send_file |
-r-x------ vqes vqes |
Command to send a client database file and group attribute file to VCDS |
/opt/vqes/bin/vqe_cfgtool |
-r-x------ vqes vqes |
vqe_cfgtool command |
/opt/vqes/bin/vqereport |
-r-x------ root root |
vqereport command |
/etc/syslog-ng/syslog-ng.conf |
-rw-r--r-- 1 root root |
VQE-S syslog configuration file |
/opt/syslog-ng/sbin/syslog-ng |
-rwxr-xr-x root root |
VQE-S syslog-ng process |
/var/log/vqe (directory) |
drwxr-xr-x vqes vqes |
VQE log directory |
/var/log/vqe/vqe.log |
-rw------- vqes vqes |
VQE system messages log file |
Changing Permissions
To change permissions on a file or directory, use the chmod command:
Step 1 If needed, log in as root on the CDE110 that hosts VQE-S or that hosts VCPT.
Step 2 To change permissions, issue the following command:
chmod mode file_or_dir_name
In the preceding command, the two arguments are as follows:
•mode specifies the correct permissions.
•file_or_dir_name specifies the file or directory whose permissions is set.
For example:
[root@system]# chmod 555 /opt/vqes/bin/vqes_cp
For information on setting permissions, see the man page for the chmod command.
Replacing a Lost or Corrupt VQE File
If a required VQE file that has not been backed up is lost or corrupted, you must reinstall the version of the VQE software that the CDE110 server is running. To preserve the current VQE configuration, the software reinstallation should be an ISO full upgrade. For information on VQE software installation, see the Release Notes for Cisco CDA Visual Quality Experience Application, Release 3.5.
If a configuration file located in the directories under /etc is lost or corrupted and was not backed up, you can do one of the following:
•Use the vqe_cfgtool command with the -fix_config option to discard the current copy of each /etc configuration file where there is a checksum mismatch and replace it with the original copy of the file that comes with the RPM package. Then apply the current VCDB configuration to the /etc configuration files using the vqe_cfgtool command and the -apply option.
•Get a copy of the factory default version of the configuration file from the /vqe-etc/etc-pristine directory. To restore the lost or corrupted file, copy the factory default configuration file to its correct location under /etc, and apply the current VCDB configuration to the /etc configuration files using the vqe_cfgtool command and the -apply option.
For information on the vqe_cfgtool command, see the "Using the VQE Configuration Tool Command-Line Options" section.
VQE-S and VQE Tools Server Troubleshooting
This section provides information on troubleshooting the following specific problems that can occur with a VQE-S or VQE Tools server. Table 6-5 lists the troubleshooting topics by the category of the problem.
* Some VCPT troubleshooting topics apply to other channel-provisioning servers, such as Cisco IPTV Services Delivery Server (ISDS).
Monitoring Received Multicast Streams
For troubleshooting the VQE-S, using the VQE-S AMT to monitor received multicast streams is recommended.
We recommend that you use the VQE-S AMT to monitor received multicast streams. In the VQE-S AMT Channel Statistics window for each channel (see Figure 6-1), the Lost RTP Packets field for the Channel Input Stream is a per-channel counter that provides the number of missing RTP packets. The Lost RTP Packets counter increments when an RTP packet is sent for the original source stream of the channel but is not received by the VQE-S.
Figure 6-1 Lost RTP Packets
Repeated Message on CDE110 Hosting VQE-S or VQE Tools
Symptom: During initial system configuration, a new Cisco CDE110 hosting the VQE-S or VQE Tools boots successfully, the root password is set successfully, but the following message displays repeatedly:
SIOCGIFADDR: Cannot assign requested address
SIOCGIFADDR: Cannot assign requested address
SIOCGIFADDR: Cannot assign requested address
SIOCGIFADDR: Cannot assign requested address
SIOCGIFADDR: Cannot assign requested address
SIOCGIFADDR: Cannot assign requested address
Explanation: This error can occur when you configure the CDE110 server manually (rather than use the VQE Startup Configuration Utility). The message may display so frequently that it is difficult to edit a file to continue the initial system configuration.
The message can be caused by an Ethernet interface configuration issue in the ifcfg-eth# files (ifcfg-eth1, ifcfg-eth2, and so on).
Remedy: Look for Ethernet interface address or other errors in the four ifcfg-eth# files. Use a text editor to correct any errors.
Use the ifconfig -a command to verify that the addresses specified in the ifcfg-eth# files have been assigned to the interfaces.
Continue with the initial configuration tasks.
Performing a Date and Time Change with NTP
When performing a date and time change with NTP, do the following:
Step 1 Log in as root.
Step 2 Stop the VQE-S services by issuing the following command:
[root@system]# service vqes stop
Step 3 Stop the ntpd service by issuing the following command:
[root@system]# service ntpd stop
Step 4 If needed, set the time zone with the vqe_cfgtool command using the -config option. Use the System Parameters menu on the Configuration Tool and the Timezone choice.
Step 5 Set the system date and time to a date and time close to the NTP server date and time by issuing the following command:
date -s "date_time_string"
For example:
[root@system]# date -s "16:55:30 July 7, 2008"
Step 6 Synchronize the clock to the configured NTP servers by issuing the following command:
If the system clock is off by a lot, the command takes considerable time to return.
Step 7 Start the ntpd service by issuing the following command:
[root@system]# service ntpd start
Step 8 Synchronize the hardware clock by issuing the following command:
[root@system]# /sbin/hwclock --systohc
Step 9 Check NTP synchronization by issuing the following command:
Step 10 Reboot the VQE-S by issuing the following command:
Performing a Date and Time Change with the Linux date Command
When performing a time and date change with the Linux date command only, perform the following commands:
Step 1 Log in as root.
Step 2 Stop the VQE-S services by issuing the following command:
[root@system]# service vqes stop
Step 3 If needed, set the time zone with the vqe_cfgtool command using the -config option. Use the System Parameters menu on the Configuration Tool and the Timezone choice.
Step 4 Set the system date and time by issuing the following command:
date -s "date_time_string"
For example:
[root@system]# date -s "16:55:30 July 7, 2008"
Step 5 Synchronize the hardware clock by issuing the following command:
[root@system]# /sbin/hwclock --systohc
Step 6 Reboot the VQE-S by issuing the following command:
VQE-S AMT, VCDS AMT, or VCPT Unavailable
Symptom: The browser displays the message Page Cannot Be Found (status code 404) or Service Temporarily Unavailable (status code 503) even if the user enters a valid IP address or fully qualified domain name to browse the VQE-S AMT, VCDS AMT, or VCPT.
Explanation: The VQE-S AMT, VCDS AMT, and VCPT require the Apache HTTP server service (httpd) and the Apache Tomcat application server service (tomcat5) to be running. When you attempt to browse to the VQE-S AMT, VCDS AMT, or VCPT and one of these services is not running, the status codes are typically as follows:
•"Page Cannot Be Found" (status code 404)—indicates that httpd is not running.
•"Service Temporarily Unavailable" (status code 503)—indicates that tomcat5 is not running.
To check whether the httpd service is running, log in as root and issue the following command:
[root@system]# /sbin/service httpd status
httpd (pid 15836 13179 2593 2592 2591 2590 2589 2588 2587 2586 2576) is running...
To check whether the tomcat5 service is running, log in as root and issue the following command:
[root@system]# service tomcat5 status
Remedy: If either httpd or tomcat5 is not running, log in as root and start the services that are stopped as follows:
[root@system]# service httpd start
[root@system]# service tomcat5 start
Verify that the httpd and tomcat5 services are running, as shown earlier in the Explanation section. If the httpd or tomcat5 processes continue to fail, check the log files for the services to determine the cause.
•httpd log file is /etc/httpd/logs/error_log.
•tomcat5 log files are /var/log/tomcat5/catalina.date.log.
For information on the log files for each service, see the vendor documentation for the Apache HTTP server or for the Apache Tomcat application server.
Channels Are Not Displayed in VQE-S AMT
Symptom: The VQE-S AMT does not display the channels after channels are defined in VCPT and channel information is pushed to the servers.
Explanation: The most likely reason why the VQE-S AMT does not display the channels is that the VQE-S was not associated with the channels in VCPT.
To determine whether the channels are associated with this VQE-S, follow these steps:
Step 1 Log in to VCPT.
Step 2 Click the Association tab.
Step 3 Select the VQE Server in the Select Server pull-down menu.
The channels should be in the Selected group. Also, make sure the VQE-S is defined with the role VQE-S. If the role is wrong, correct it with VCPT from the Servers tab. For information on updating server information, see the "Viewing or Updating Server Information" section.
If the channels are correctly associated with the VQE-S, it is possible that there were problems when the VQE-S attempted to process the channels. Check the VQE-S log files for errors in channel processing. For information on these log files, see the "VQE Server and VQE Tools Logging and Log Files" section.
Remedy: If the channels have not been associated with the VQE-S in VCPT, associate the channels with the VCPT from the Association tab. For information on associating channels with servers, see the "Defining Channel Associations for a Server" section.
VQE-S AMT Channels Status Summary Has No Graphs
Symptom: In the AMT VQE-S Status window (Figure 4-2), the Channels Status Summary does not have graphs for the channels.
Explanation: If one or more channels are successfully created in the VQE-S AMT, the channels should appear in a Channels Status Summary graph of active, inoperative, and inactive channels in the VQE-S Status window. To be displayed, the graphs require that Adobe Flash Player be installed on the computer that hosts the browser accessing the VQE-S AMT.
Remedy: Download and install Adobe Flash Player on the computer that hosts the browser used to access the VQE-S AMT. The software is free and can be found at this URL:
http://get.adobe.com/flashplayer/
It is recommended that you close all other open browser windows before performing the installation.
VCPT Fails to Initialize
Symptom: VCPT fails to initialize when the VQE Tools server starts. The following error message is displayed:
Initialization Failed. VCPT Data directory is missing, please refer to the Cisco CDA VQE
Application User Guide - Working with VCPT Configuration Files section."
Explanation: For VCPT to initialize successfully, the following directories must exist:
•/etc/opt/vcpt/data
•/etc/opt/vcpt/data/sdp
One or both of these directories are missing and may have been accidentally deleted.
Remedy: To recreate the directories and start VCPT, follow these steps:
Step 1 Login as root.
Step 2 Determine which directory or directories are missing.
Step 3 Create each missing directory using the permissions, owner, and group specified in Table 6-4 for that directory.
Step 4 If up-to-date VCPT configuration files for the /etc/opt/vcpt/data directory are available in a set of backup files, the backup files can be copied to the /etc/opt/vcpt/data directory.
Note There is no need to copy backup files for the set of SDP-formatted files in the /etc/opt/vcpt/data/sdp directory. The VCPT creates the SDP-formatted files when a request is made to send channel information to the VQE-S or VCDS servers.
Step 5 Restart the Apache Tomcat application server:
[root@system]# service tomcat5 restart
The VCPT is a web application and should initialize successfully.
Step 6 To verify that the VCPT is accessible from a web browser, enter as the URL the IP address of the Cisco CDE110 that hosts VCPT:
https://ip_address_of_VCPT_host
Log in with a Linux username and password.
If you are able to log in successfully, the VCPT is running correctly.
Channel-Provisioning Server Cannot Send Channel Information to VQE-S:
Trusted Provisioning Client Problem
Symptom: The channel-provisioning server (for example, VCPT or Cisco IPTV Services Delivery Server [ISDS]) cannot send channel information to a VQE-S.
Explanation: When the channel-provisioning server attempts to send channel information to a VQE-S, the send operation fails. On the VQE-S, the VQE Configuration Tool parameter Trusted Provisioning Client is not configured with the IP addresses for the channel-provisioning server.
If VCPT is the channel-provisioning server and the send operation fails because the Trusted Provisioning Client parameter is not configured with the IP addresses of the Ethernet interfaces on the VCPT host, VCPT displays the following in the Status of Last Send column in its VCPT Servers Summary window:
Failed - Connection refused
In addition, the following error message is written to the VCPT log file (/usr/share/tomcat5/logs/vcpt.log):
ERROR: Unable to send the SDP File java.net.ConnectException: Connection refused
Remedy: Use the VQE Configuration Tool to configure the System Parameter > Trusted Provisioning Client with the IP addresses of the channel-provisioning servers. For information on using the Configuration Tool, see the "Using the VQE Configuration Tool" section.
Note If VCPT is the channel-provisioning server, the IP addresses of all Ethernet interfaces (that have been assigned IP addresses) on the VCPT host must be configured as trusted provisioning clients on the VQE-S host.
Note If ISDS is the channel-provisioning server, for the Trusted Provisioning Clients, use the Broadcast System IP Address, which is configured on the ISDS VASP List. The VASP List can be found on the ISDS Administrative Console under ISDS > Network Element Provisioning > VASP.
Channel-Provisioning Server Cannot Send Channel Information to VQE-S: SSL Certificates Problems
Symptom: The channel-provisioning server (for example, VCPT or ISDS) cannot send channel information to a VQE-S.
Explanation: When the channel-provisioning server attempts to send channel information to a VQE-S, the send operation fails. On the VQE-S or the channel-provisioning server or both, the Secure Sockets Layer (SSL) certificates are not valid or the needed items have not been copied to the correct locations.
If VCPT is the channel-provisioning server and the send operation fails because of SSL certificate problems, VCPT displays the following in the Status of Last Send column in its VCPT Servers Summary window:
Failed - Unable to find valid certification path to requested target
In addition, the following error message is written to the VCPT log file (/usr/share/tomcat5/logs/vcpt.log):
ERROR: Unable to send the SDP File javax.net.ssl.SSLHandshakeException:
sun.security.validator.ValidatorException: PKIX path building failed:
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid
certification path to requested target
Remedy: Make sure all SSL certificate-related items have been created correctly and the needed items have been copied to the correct locations on the VQE-S and on the channel-provisioning server. For information on setting up SSL certificates, see the "Setting Up SSL Certificates" section.
After the certificate-related items have been copied to the correct locations on the VQE-S and on the channel-provisioning server, restart the httpd and tomcat5 services on both servers. For example, login as root and issue the following commands on both servers:
[root@system]# service httpd restart
[root@system]# service tomcat5 restart
Note If the tomcat5 service is not running when you try to restart it, an exception is displayed on the console when the attempt is made to stop the service. The exception can be ignored.
Unable to Connect Error When VCPT Tries to Send Channel Information
Symptom: When the VCPT tries to send channel information to a VQE-S or VCDS, the send operation fails. In VCPT, the Servers tab displays "Failed - Unable to connect with Server" in the Status of Last Send column.
Explanation: Possible causes for the failed send operation are that the Apache HTTP server service (httpd) or the Apache Tomcat application server service (tomcat5) on the target VQE-S or VQE Tools server (hosting VCDS) is not running.
To check whether the httpd service is running, log in as root and issue the following command:
[root@system]# service httpd status
httpd (pid 15836 13179 2593 2592 2591 2590 2589 2588 2587 2586 2576) is running...
To check whether the tomcat5 service is running, log in as root and issue the following command:
[root@system]# service tomcat5 status
Remedy: If either httpd or tomcat5 is not running, log in as root and start the services that are stopped as follows:
[root@system]# service httpd start
[root@system]# service tomcat5 start
Verify that the httpd and tomcat5 services are running, as shown earlier in the Explanation section. If the httpd or tomcat5 processes continue to fail, check the log files for the services to determine the cause.
•httpd log file is /etc/httpd/logs/error_log.
•tomcat5 log files are /var/log/tomcat5/catalina.date.log.
For information on the log files for each service, see the vendor documentation for the Apache HTTP server or for the Apache Tomcat application server.
Internal Server Error When VCPT Tries to Send Channel Information to VQE-S
Symptom: When VCPT tries to send channel information to the VQE-S, you get an "Internal Server Error." Java exceptions in the catalina.out file indicate that there is an address conflict or unreachable address.
Explanation: The Internal Server Error is displayed when you use the VCPT to send channel information to a VQE-S. The Status of Last Send column on the VCPT Servers tab shows the error.
From the CDE110 that hosts the VQE-S, you are able to successfully ping the machine running the web browser used for the VQE-S AMT, and ping the CDE110 hosting VCPT.
In the /usr/share/tomcat5/logs/catalina.out file of the Apache Tomcat application server, there are Java exceptions indicating an address conflict or unreachable address. For example:
java.net.BindException: Address already in use
at java.net.PlainSocketImpl.socketBind(Native Method)
at java.net.PlainSocketImpl.bind(Unknown Source)
...
java.net.SocketException: Network is unreachable
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(Unknown Source)
Remedy: The Internal Server Error can be caused by one or more errors in the /etc/hosts file on the CDE110 that hosts the VQE-S. Check the /etc/hosts file for typing or syntax errors, correct them, and save the file. Reboot the CDE110.
If you still receive the same Internal Server Error and Java exceptions, check the following files to see if you can find some indication for what is causing the error.
•/var/log/vqe/vqe.log
•/var/log/httpd/error_log
•/var/log/httpd/access_log
•/usr/share/tomcat5/logs/catalina.out
•/var/log/messages
RCC Troubleshooting
Symptom: If a rapid channel change (RCC) fails or there is degradation in the performance of an RCC, video freeze or macroblocking may be observed. The following section describes how to troubleshoot RCC problems from the VQE-S.
VQE-S RCC Troubleshooting
The section lists possible explanations for RCC problems and provides a remedy for each explanation:
•Channel is not configured on the VQE-S
•Channel is not active on the VQE-S
•RCC is disabled on the channel
•No connectivity between the VQE-C and the FBT on the VQE-S
•No connectivity between the VQE-C and the VQE-S
•RCC refused by the VQE-S control or data planes
•RCC refused by the VQE-S—invalid e-factor
•RCC refused by the VQE-S—insufficient bandwidth
•MPEG Parser is unable to locate the burst start point
•VQE-S MPEG Parser is unable to locate the ECM PID
•Burst start point not found within the backfill range
•No burst start point found—stream is invalid
To perform the troubleshooting activities described in this section you need to log in to the VQE-S AMT. For information on accessing the VQE-S AMT, see "Logging into and Navigating in the VQE-S AMT" section.
To troubleshoot RCC problems, enable debugging of RCC messages on the VQE-S and analyze the debug logs. To enable debugging, see the "Change VQE-S Debugging" section. The debug messages are written to the file /var/log/vqe/vqe.log. Alternatively, to view RCC failures, click the channel name in the navigation tree in the VQE-S AMT and click the RCC Troubleshooting tab. The RCC Troubleshooting tab displays the last twenty RCC failures on this channel.
Channel is not configured on the VQE-S
Explanation: An RCC fails if the new channel is not configured on the VQE-S. To verify that a channel is configured on the VQE-S, click the Channel node in the navigation tree, and verify that the new channel is listed in the channel lineup.
Remedy: If the channel is not displayed in the channel lineup, add the channel to the channel configuration using the VCPT. For more information on adding channels, see the "Adding a Channel" section. After the channel has been added to the channel configuration, send the updated configuration to the VQE-S and the VCDS. For information on deploying channel configurations, see the "Sending or Pushing Channel Information to Servers" section.
Channel is not active on the VQE-S
Explanation: An RCC fails if the new channel is not active on the VQE-S or if the channel is not receiving RTP packets. To verify that a channel is active on the VQE-S and receiving RTP packets, do the following:
Step 1 Click the Channel node in the navigation tree to expand the channel lineup.
Step 2 Click the channel in the lineup.
Step 3 Click the Status/Statistics tab.
Step 4 Verify that the Status field is set to active.
Step 5 Verify that the value of the Received Packets field is greater than zero.
Step 6 Take note of the value of the Average Stream Rate(Mbps) field.
Step 7 Click the Configuration tab.
Step 8 Take note of the value of the Multicast IP Address field.
Step 9 Take note of the value of the Bit Rate(Kbps) field for the original stream.
Step 10 Verify that the Average Stream Rate(Mbps) for the original stream is within 10% of the configured Bit Rate (Kbps) for the original stream.
Remedy: If there is a mismatch between the expected bit rate and the actual bit rate, ensure that the configured bit rate represents the peak bit rate expected on the channel.
If the Status of the channel is inactive or if the number of received packets is zero, check that the interface on which the channel has joined is properly configured. To verify that the interface has been configured correctly, do the following:
Step 1 Click the Interfaces node in the navigation tree.
Step 2 In the Multicast Group table, locate the multicast address of the channel and identify the name of the interface associated with the channel.
Step 3 Verify that the interface has been configured correctly.
Note 10 or 100 Mbps interfaces are not supported on the VQE-S.
If the interface is configured correctly, verify that the multicast stream is being received by the networking stack. To output a record of all TCP traffic to and from an Ethernet interface to the screen, use secure shell (SSH) to connect to the VQE-S and run the command /sbin/tcpdump -i <ethx> host <multicast_address> on the interface as user root.
RCC is disabled on the channel
Explanation: An RCC fails if RCC is disabled on the new channel. To verify that RCC is enabled on a channel, do the following:
Step 1 Click the Channel node in the navigation tree to expand the channel lineup.
Step 2 Click the new channel in the lineup.
Step 3 Click the Configuration tab.
Step 4 Verify that the Rapid Channel Change field is set to true.
Remedy: If the Rapid Channel Change parameter is set to false for the channel, update the channel configuration in VCPT by clicking the Enable Rapid Channel Change check box. For information on updating channel information, see the "Viewing or Updating Channel Information" section. After RCC has been enabled in the channel configuration, send the updated configuration to the VQE-S and the VCDS. For information on deploying channel configurations, see the "Sending or Pushing Channel Information to Servers" section.
No connectivity between the VQE-C and the FBT on the VQE-S
Explanation: An RCC fails if there is no connectivity between the VQE-C and the feedback target (FBT) of the new channel on the VQE-S.
To verify that the VQE-C can connect to the feedback target of the new channel, do the following:
Step 1 Click the Channel node in the navigation tree to expand the channel lineup.
Step 2 Click the new channel in the lineup.
Step 3 Click the Configuration tab.
Step 4 Take note of the IP address in the Feedback Target Address field.
Step 5 Ping the feedback target IP address from the VQE-C to verify that it is reachable.
Remedy: If there is no connectivity between the VQE-C and the FBT on the VQE-S, check that the channel is active. A channel that is inactive does not have a FBT that is reachable. If the channel is active and a FBT for this channel does not exist, check the VQE system messages in the /var/log/vqe/vqe.log file for a FBT failure message.
No connectivity between the VQE-C and the VQE-S
Explanation: An RCC fails if there is no connectivity between the VQE-C and the VQE-S.
To verify that the VQE-C can connect to the service interfaces of the VQE-S, do the following:
Step 1 Click the Interfaces node in the navigation tree.
Step 2 In the Interfaces Summary Table, take note of the IP address of each service interface.
Step 3 Ping the IP address of each service interface from the VQE-C to ensure it is reachable.
Remedy: If there is no connectivity between the VQE-C and the VQE-S service interfaces, fix problems in the network relating to cabling, interface configurations, and routing tables.
RCC refused by the VQE-S control or data planes
An RCC fails if the RCC is refused by the VQE-S control plane (CP) or data plane (DP). To verify that an RCC has not been refused for a channel by the VQE-S CP or DP, do the following:
Step 1 Click the Channel node in the navigation tree to expand the channel lineup.
Step 2 Click the new channel in the lineup.
Step 3 Click the Advanced button.
Step 4 Under RCC Debug Counters, verify that the values of the Refused RCC—RCC disabled on DP and Refused RCC—RCC disabled on CP counters are zero.
Explanation: If the counter Refused RCC-RCC disabled on DP counter has been incremented, the RCC has been refused by the VQE-S DP.
Remedy: If an RCC has been refused by the VQE-S DP, verify that the following line is not included in the /etc/opt/vqes/vcdb.conf file:
vqe.vqes.rcc_disable="true"
If the line is present in the file, remove the line, and use the vqe_cfgtool command with the -apply option to apply the change to the VQE-S.
Note The parameter vqe.vqes.rcc_disable is a hidden parameter and should only be used on the direction of the a Cisco technical support representative.
If the counter Refused RCC-RCC disabled on CP is incremented, the RCC has been refused by the VQE-S CP. If the Refused RCC - RCC disabled on CP counter has been incremented, contact your Cisco technical support representative.
RCC refused by the VQE-S—invalid e-factor
Explanation: An RCC fails if the VQE-S refuses the RCC request because the e-factor is invalid. To verify that a valid e-factor has been configured on the VQE-S, check the that the value of the VCDB parameter vqe.vqes.excess_bw_fraction is within the defined range of 3 to 500 in the /etc/opt/vqes/vcdb.conf file.
Remedy: If the e-factor value is outside of the permitted range, change the value of the parameter vqes.excess_bw_fraction in the /etc/opt/vqes/vcdb.conf file, and use the vqe_cfgtool command with the -apply option to apply the change to the VQE-S.
RCC refused by the VQE-S—insufficient bandwidth
Explanation: An RCC fails if the VQE-S refuses an RCC because there is not enough bandwidth available on the access link to the VQE-C to perform the RCC.
To verify that there is sufficient bandwidth available to perform an RCC, do the following:
Step 1 Click the Error Repair and RCC node in the navigation tree to expand the channel lineup.
Step 2 Click the Statistics tab.
Step 3 Click the Advanced button.
Step 4 Verify that the total bandwidth available as displayed in the System available output b/w (Mbps) counter under the section CP Output Bandwidth Stats matches the expected total bandwidth of the service interfaces. For example, if two Gigabit Ethernet interfaces are configured and used for the access link, then the value of the System available output b/w (Mbps) counter should be 2000 Mbps.
Step 5 Verify that the value of the Committed RCC output b/w (Mbps) counter is less than the value of the RCC output b/w budget (Mbps) counter.
Step 6 Click the Interfaces node in the navigation tree.
Step 7 Verify that all service interfaces have a role of service and have a status of Up. Verify that each service interface is a 1000 Mbps interface.
Remedy: If a service interface is a 10 Mbps or a 100 Mbps interface, you need to upgrade to a 1000 Mbps interface. If the committed budget for RCC is close to the maximum budget for RCC, you are close to reaching the scalability limit of the VQE-S.
MPEG Parser is unable to locate the burst start point
An RCC fails if the VQE-S MPEG parser is unable to locate the burst start point (that is, the Transport Stream Random Access Point [TSRAP]). To verify that the VQE-S is correctly identifying the burst start point, do the following:
Step 1 Click the Channel node in the navigation tree to expand the channel lineup.
Step 2 Click the new channel in the lineup.
Step 3 Click the Advanced button.
Step 4 Locate the MPEG Parser: Locating a Random Access Point section.
Step 5 Verify that the value of the Num RAPs found by Adaptation Field RAI bit counter and Num RAPs found by locating I-frame or IDR counter is zero.
Explanation: If the value of the Num RAPs found by Adaptation Field RAI counter and the Num RAPs found by locating I-frame or IDR counter has been incremented, it is likely that the VQE-S is unable to find a TSRAP. The VQE-S is optimized to skip the parsing of the Elementary Stream (ES) which contains the actual group of pictures (GOPs). Instead, the VQE-S uses the random_access_indicator (RAI) bit in the MPEG2-TS packet header. However, in some deployments, the RAI bit is not set. Without an RAI, random access points (RAPs) may be overlooked as valid RCC burst start points.
Remedy: To prevent the VQE-S from skipping the ES, add the following line to the /etc/opt/vqes/vcdb.conf file:
Use the vqe_cfgtool command with the -apply option to apply the change to the VQE-S.
Explanation: If RCCs continue to fail, it may be that the VQE-S is still unable to find a TSRAP. The VQE-S is also optimized to skip to the beginning of the next MPEG Packetized Elementary Stream (PES) once it determines the picture type. In some deployments, more than one GOP is contained in the PES header. If multiple GOPs are skipped, some RAPs may be overlooked as valid RCC burst start points.
Remedy: To prevent the VQE-S from skipping multiple GOPs in a PES header, add the following line to the /etc/opt/vqes/vcdb.conf file:
vqe.vqes.mpeg_search_limit_disable="true"
Use the vqe_cfgtool command with the -apply option to apply the change to the VQE-S.
VQE-S MPEG Parser is unable to locate the ECM PID
Explanation: When the new channel is scrambled, an RCC fails if the VQE-S MPEG parser is unable to locate the Entitlement Control Messages (ECM) Packet identifier (PID). To verify that the VQE-S is correctly locating the ECM PID, do the following:
Step 1 Click the Channel node in the navigation tree to expand the channel lineup.
Step 2 Click the new channel in the lineup.
Step 3 Click the Advanced button.
Step 4 Locate the MPEG Parser: PIDs section.
Step 5 Verify that the list of PIDs includes a PID Type (ES Video), with a Stream Type of H.262 or H.264.
Step 6 Verify that the list of PIDs includes a PID Type (ECM PID).
Step 7 Verify that none of the error counters associated with either the ES Video PID or the ECM PID have been incremented.
Remedy: If either the ES Video PID or the ECM PID are not included under the list of PIDs, or if the error counters associated with either PID have been incremented, check the stream configuration and contact your Cisco technical support representative.
Burst start point not found within the backfill range
Explanation: An RCC is rejected by the VQE-S if it is unable to find a burst start point (TSRAP) within the backfill range sent by the VQE-C. The VQE-C sends the parameters min_rcc_backfill and max_rcc_backfill in its RCC request. The VQE-S must find a TSRAP within the range provided by these two parameters. If none is found, the RCC is rejected.
The following is an example of a VQE-C PLI APP message (that is, a request for an RCC) taken from a VQE-S log file:
Feb 5 10:35:53 saturn-iptv vqes_cp: <{vqes-era}>Computed PLII APP
received=TRUE, min_rcc_backfill=200, max_rcc_backfill=2105; use
min_ref_a=1233848151597, max_age=1233848153492, idr=0
In this example, the VQE-C sends a min_rcc_backfill value of 200 ms and a max_rcc_backfill value of 2105 ms.
The following is an example of the message sent from the VQE-S in response to the RCC request:
Feb 5 10:35:53 saturn-iptv vqes_cp: <{vqes-era}>Candidate TSRAP #0
(96598729:1), ref_time=1233848153147(0.545301 secs ago), is_idr:1,
rai:0, pts:10970145, pts_pcr_offset:74430[90kHz ticks]
In this example, the VQE-S indicates that it is selecting a burst start point that is 545 ms old and therefore, within the defined range.
If several RCCs have been rejected, check the VQE-S logs to verify that the backfill range for each rejected RCC was sufficiently large enough for the VQE-S to find burst start point within the backfill range.
Remedy: It is usually not a viable option to change the size of a GOP stream. Instead, consider increasing the size of the packet pool. For more information on sizing the packet pool, see VQE-C System Integration Guide.
No burst start point found—stream is invalid
Explanation: An RCC is rejected by the VQE-S if it is unable to locate a burst start point (TSRAP) in the stream sent by the stream source for one of the following reasons:
•Stream is of a type not supported by the VQE-S— VQE-S supports streams of type MPEG-2 part 2 (H.262) and MPEG-4 part 10 (H.264). The VQE-S does not support other stream types such as MPEG-4 part 2 (H.263) or DigiCipher II (DC II).
•Stream is empty or it is sent with null-padded packets.
•Stream is using a Conditional Access System (CAS) that is not ETSI ETR 289 compliant.
•Stream does not contain Program Map Tables (PMT) or Program Association Tables (PAT)—Priming information required by the VQE-S to perform the RCC is missing.
•Details of the PMT and PAT do not match the details of the stream—Priming information required by the VQE-S to perform the RCC is incorrect.
To verify that the VQE-S is correctly locating the burst start point, do the following:
Step 1 Click the Channel node in the navigation tree to expand the channel lineup.
Step 2 Click the new channel in the lineup.
Step 3 Click the Advanced button.
Step 4 Locate the RCC Debug Counters.
Step 5 Verify that the RCC Refused—No RAP for stream counter did not increment during the RCC.
Remedy: If the RCC Refused—No RAP for stream counter has incremented, verify that the stream source is providing a valid stream for the channel and contact your Cisco technical support representative.
VQE-C RCC Troubleshooting
As a starting point, use the show counters command on the VQE-C command line interface (CLI) to identify how many previous RCC requests have been attempted by the VQE-C and how many have failed. For those RCC requests that have aborted, the output of the show counters command provides RCC failure reasons and indicates the number of failures associated with each reason. The following is a sample output from the show counters command:
-- Channel Change Counters --
server stun response response burst burst
rejects timeout timeout invalid start activity other
In this example, the RCC requests field presents the number of channel change requests made by the VQE-C. The RCC with loss field indicates the number of RCCs that did not abort but for which the post-repair loss rcc counter incremented. The number of RCCs aborted per failure reason is presented under the rcc aborts field. The following failure reason counters are presented:
•Server rejects—Number of times an RCC request was rejected by the VQE-S.
•Stun timeout—Number of times a STUN response was not received from the VQE-S.
•Response timeout—Number of times an APP packet (that is, priming information) was not received from the VQE-S.
•Response invalid—Number of times an APP packet received contained invalid contents.
•Burst start—Number of times the first repair packet was not received on time.
•Burst activity—Number of times a burst activity timed out prior to completion.
For further information on using the show counters command, see VQE-C CLI Command Reference.
The following sections describe how to troubleshoot RCC problems from the VQE-C:
•RCC not enabled on the VQE-C
•Problem with priming information sent from the VQE-S
•First packets from repair burst not received within expected time period
•No multicast stream
•Output loss during RCC
•Output packets dropped
Each section presents possible explanations for RCC problems and provides a remedy for the explanation.
RCC not enabled on the VQE-C
Explanation: RCC must be enabled on the VQE-C for RCC to operate. Use the show rcc command on the VQE-C CLI to verify that RCC is enabled. The following is a sample output from the show rcc command:
Remedy: If RCC is disabled, use the rcc enable command from the VQE-C CLI in configuration mode to enable RCC. For more information on enabling RCC, see VQE-C System Integration Guide. RCC must also be enabled on the new channel. To verify that RCC is enabled on a channel, either check the channel configuration in the channel lineup from VQE-S AMT or use the show channel command on the VQE-C CLI and verify that the RCC field is set to enabled. For more information on verifying the status of RCC from the VQE-S AMT, see the "RCC is disabled on the channel" section.
Problem with priming information sent from the VQE-S
Explanation: An RCC may be unsuccessful on the VQE-C because of a problem with the APP packets (that is, priming information) sent by the VQE-S. Possible explanations are as follows:
•APP packet was not received within the rcc_start_timeout interval set at the VQE-C.
•APP packet was received by the VQE-C , but was empty.
•APP packet was received by the VQE-S, but was invalid.
To verify from the VQE-C that an RCC has succeeded, use the show tuner <tuner name> rcc command and verify that the RCC result field has a value of success. The following is a sample output from the show tuner <tuner name> rcc command:
vqec# show tuner name tuner1 rcc
If the rcc result field displays a value of failure, check the values of the cp failure reason field and the dp failure reason field.
Remedy: The cp failure reason field displays the reason why the RCC was aborted by the control plane (CP). If the field displays a value of APP_Timeout, the APP packet was not received within the RCC start timeout period set at the VQE-C. See the "VQE-S RCC Troubleshooting" section for possible explanations why APP packets are not sent by the VQE-S within the timeout period.
The dp failure reason displays the reason why the RCC was aborted by the data plane (DP). If the field displays a value of NULL_APP, an empty APP packet was received. In this case, the VQE-S was unable to process the RCC. To identify why an RCC was refused by the VQE-S, view the Refused RCC counters on the Advanced window of the Error Repair and RCC tab of the VQE-S AMT. For more information on identifying reasons why the VQE-S rejects an RCC, see the "VQE-S RCC Troubleshooting" section. If the value of the dp failure reason is INVALID_APP, one or more of its fields in the APP packet was not set correctly by the VQE-S.
First packets from repair burst not received within expected time period
Explanation: The VQE-C aborts an RCC if it does not receive the first repair packet (specified in the APP packet) from the VQE-S within the period of time specified by the VQE-C System Configuration parameter rcc_start_timeout and presents a FIRST_REPAIR_TIMEOUT error message on the CLI.
In a deployment that includes a NAT device between the STB and the VQE-S, the VQE-C must be configured to use NAT mode, as specified by the VQE-C System Configuration parameter sig_mode.
Remedy: Use Wireshark to determine that the unicast repair burst is being sent from the VQE-S and to determine that the repair burst is reaching the STB hosting the VQE-C. If the repair burst is reaching the VQE-C, the VQE-S is operating as intended, and the problem is occurring within the STB.
Furthermore, if the round-trip time on the network is large or inconsistent, it may help to increase the value of the VQE-C System Configuration parameter rcc_start_timeout, which increases the amount of time the VQE-C waits for a repair burst to start before failing with a FIRST_REPAIR_TIMEOUT error. For more information on configuring the rcc_start_timeout parameter, see VQE Client System Configuration Guide.
No multicast stream
Explanation: When an RCC unicast burst is received but no multicast burst is received for the primary stream, the RCC appears to start and then stop. To verify that multicast bursts are received after the initial RCC unicast burst, use the show tuner name <tuner_name> rcc command on the VQE-C CLI to identify the time the primary packet was received. Verify that the value of the PRIM field under Actual Relative Times is a non-zero value. In addition, verify that the first primary sequence field under Output statistics contains a non-zero value. For further information on using the show tuner name <tuner_name> rcc command, see VQE-C CLI Command Reference. The following is a sample output from the command:
vqec> show tuner name 0 rcc
---Actual relative times (ms)---
CC Pli APP Rep Join Prim ER Join-lat
0 7 11 33 149 204 2422 30
--- Output statistics ---
first primary sequence: 13767
first repair sequence: 167261298
In the sample output, the Prim field under the Actual relative times (ms) section contains a non-zero value. The first primary sequence field under Output statistics also contains a non-zero value.
Remedy: If the multicast burst for the primary stream is not received, verify that IGMP is setup correctly in your network configuration.
Output loss during RCC
Explanation: Video freeze or macroblocking may be observed during an RCC if output loss occurs at the VQE-C. To check for output losses on a tuner, use the show tuner name <tuner_name> rcc command. The following is a sample output from this command:
vqec> show tuner name 0 rcc
--- Output statistics ---
first primary sequence: 3125
rcc output loss packets: 0
rcc duplicate packets: 26
first packet output time 34
last packet output time 2363
first packet decode time 183
Under Output Statistics, the rcc output loss packets field displays the number of missing packets on the output stream and the rcc output loss holes field displays the number of missing packet holes on the output stream. If no output loss has occurred, the value of both fields should be zero. If losses are observed in the network, check you network configuration.
The following are possible explanations why RCC output losses may occur:
•IGMP join latency
•Packet pool is not size appropriately
•Excess bandwidth fraction is larger than the available bandwidth
•Previous stream is still flowing
IGMP join latency
Explanation: Video freeze or macroblocking may be observed during an RCC if the IGMP join latency is longer than expected. IGMP join latency is the time between a request to join a multicast group and the receipt of the first byte of data for a multicast group. The actual join latency should be less than the or equal to the value of the VCDB parameter vqe.vqes.igmp_join_variability. This parameter sets the amount of variability between the fastest and slowest IGMP joins for RCC.
To identify the actual IGMP Join Latency, use the show tuner name <tuner_name> rcc command on the VQE-C CLI. For information on using this command, see VQE-C CLI Command Reference. The following is a sample output from the show tuner name <tuner_name> rcc command:
vqec> show tuner name 0 rcc
---Actual relative times (ms)---
CC Pli APP Rep Join Prim ER Join-lat
0 7 11 33 149 204 2422 30
In the sample output, the Join-lat field is displayed under Actual relative times (ms).
Remedy: If Join-lat is greater than the value of the VCDB parameter vqe.vqes.igmp_join_variability, determine the maximum igmp join latency that will be experienced on your network, and set the VCDB parameter vqe.vqes.igmp_join_variability to this value in the /etc/opt/vqes/vcdb.conf file. Use the vqe_cfgtool command with the -apply option to apply the change to the VQE-S.
Note Increasing the value of the VCDB parameter vqe.vqes.igmp_join_variability increases the VQE-C memory requirements and may lead to RCCs being refused.
A historical view of the IGMP join latencies across STBs is provided by the STB IGMP Join Latency histogram on the VQE-S AMT. To view this histogram, do the following:
Step 1 Click the System node in the navigation tree.
Step 2 Click the Histograms tab.
The Set-top Box IGMP Latency Histogram is displayed.
Step 3 Move the slider below the histogram to change the way in which the histogram is displayed.
Packet pool is not size appropriately
Explanation: If the packet pool is not sized appropriately, video freeze or macroblocking may be observed during an RCC, and may continue after an RCC has been cancelled. The packet pool is a fixed size memory buffer pool for storing packet data. To verify that the packet pool buffer is sized appropriately, use the show pak-pool command from the VQE-C CLI. The following is a sample output from the show pak-pool command:
global input pak pool stats:
The high water entries field records a high number of packets allocated in the packet pool memory. The max entries field displays the maximum number of packets that can be allocated from the packet memory pool. If the high water mark value is close in value to the max entries value, the VQE-C has reached the currently configured limit of the packet pool.
Remedy: The system configuration parameter pakpool_size should be sized appropriately for streams and for associated network characteristics. For more information on sizing the packet pool, see VQE-C System Integration Guide.
Excess bandwidth fraction is larger than the available bandwidth
Explanation: Video freeze or macroblocking may be observed during an RCC if there is not enough bandwidth available on the access link to perform an RCC. To verify that the excess bandwidth factor has not exceeded the available bandwidth, check the values of the VCDB parameter vqe.vqes.excess_bandwidth_fraction or, in the case of a high definition channel, the VCDB parameter vqe.vqes.excess_bw_fraction_high_def. Verify that the values of these parameters are appropriate for the capacity of the access link.
Note The VQE-S does not take FEC into account when calculating available bandwidth. If your deployment uses FEC, consider the bandwidth used by FEC when determining the excess bandwidth fraction.
In your deployment, if the e-factor is determined on a per-client basis, verify that the e-factor calculated based on the VQE-C parameters max_recieve_bandwidth_sd and max_receive_bandwidth_hd is appropriate for the capacity of the access link.
Remedy: If a global e-factor value is used, modify the value of the VCDB parameter vqe.vqes_excess_bandwidth_fraction or the VCDB parameter vqe.vqes.excess_bw_fraction_high_def. If the e-factor is determined on the per-client basis, modify the value of the VQE-C parameters max_receive_bandwidth_sd or max_receive_bandwidth_hd to accurately reflect the maximum receive bandwidth.
Previous stream is still flowing
Explanation: Video freeze or macroblocking may be observed during an RCC if the previous stream is still flowing into the VQE-C. If igmp_immediate_leave is not configured in the DSLAM, the combined bandwidth of the multicast stream of the previous channel and the bandwidth allocated for RCC on the new stream (the multicast bitrate of the new stream * [1 + VQE e-factor used for the RCC]) exceeds the bandwidth of the access link. In this scenario, some of the bandwdith is still being consumed by the previous stream, and there is not enough bandwidth remaining on the access link for the unicast burst required to start the RCC for the new channel. The problem disappears as soon as the VQE-C joins the multicast stream for the new channel, usually several hundred milliseconds to several seconds after the start of the unicast burst.
If you want to verify that the previous stream has stopped flowing, connect to the access link using packet capture software such as Wireshark to identify which streams are flowing.
Remedy: If the previous stream is still flowing, configure igmp immediate leave processing in the DSLAM. Some DSLAMS may not stop the previous stream when an IGMP leave message is received. Instead, the DSLAM may wait until a new multicast stream has been joined before stopping the previous stream. In Cisco VQE Release 3.5.4, a new VQE-C configuration parameter, rcc_extra_igmp_ip, allows you to specify a multicast IP address that the VQE-C can join briefly on leaving an old channel and joining a new channel as part of an RCC to stop the flow of the multicast stream of the old channel. For more information on configuring the rcc_extra_igmp_ip VCDB parameter, see VQE Client System Configuration Guide.
Output packets dropped
Explanation: Video freeze or macroblocking may be observed during an RCC if the capacity of the output packet queue is exceeded and output packets are dropped. To verify that output packets have not been dropped, use the show tuner name <tuner_name> detail command to view the value of the qdrops field under tuner status.
vqec# show tuner name 0 detail
If the qdrops field has been incremented, the set-top box (STB) decorder is not serving the VQE-C buffer fast enough to display the stream.
Remedy: If output packets are being dropped, increase the value of the output_pakq_limit parameter on the VQE-C. For more information on configuring the output_pakq_limit, see the VQE-C System Integration Guide.
Using the vqereport Command
The vqereport command can be useful for VQE troubleshooting. You can use the vqereport command to gather information on the software and hardware configuration of the VQE system. The information can be used by the VQE administrator or Cisco technical support representative to diagnose problems with VQE software or with the Cisco Content Delivery Engine 110 (CDE110) hardware.
Caution
The
vqereport command can take up to 15 minutes to complete and may cause VQE services (such as Unicast Retransmission) to be degraded for the duration of the execution.
The syntax for the vqereport command is as follows:
vqereport [-h | -help]
Syntax Description
-h | -help |
Displays help information. |
Usage Guidelines
The vqereport command generates a report file that, when the command finishes, is located in /root/hostname.dateandtime.tar.bz2. If appropriate, the report file can be attached to field issue reports for Cisco technical support representatives.
Note You must log in as root to execute the vqereport command.
The vqereport command can be executed on the CDE110 that hosts the VQE-S or on the CDE110 that hosts VCPT. The vqereport executable is located at /opt/vqes/bin/vqereport.
All information gathered is considered confidential, and Cisco uses this information for diagnostic purposes only.
Examples
The following example shows the execution of the vqereport command and some abbreviated output:
[root@system ~]# /opt/vqes/bin/vqereport
This utility will go through and collect some detailed information about the hardware and
setup of your VQE system. This information will be used to diagnose problems with your
system and will be considered confidential information. Cisco will use this information
for diagnostic purposes ONLY.
Please wait while we collect information about your system.
This process may take a while to complete....
No changes will be made to your system during this process.
NOTE: You can safely ignore a failed message. This only means a file we were checking for
did not exist.
Press ENTER to continue, or CTRL-C to quit.
**********************************************************************
Collect Redhat sysreport
**********************************************************************
Collect information of installed RPM packages
Report file /root/<hostname>.20071115164635.tar.bz2 has been generated, please send it to
Cisco support.
Gathering Information for a Service Request
If you are experiencing a problem with Cisco VQE and need to submit a service request to Cisco, please gather and provide the following information to assist Cisco technical support representatives in diagnosing your issue:
•Report file that is generated from the vqereport command for issues involving the VQE-S or VQE Tools server.
•For an issue involving VQE-C, output from the following commands:
–show tuner all detail
–show pak-pool
–show counters
–show channel
–show system
For information on the vqereport command, see the "Using the vqereport Command" section.
For information on submitting a service request, see the monthly What's New in Cisco Product Documentation at:
http://www.cisco.com/en/US/docs/general/whatsnew/whatsnew.html
Stopping, Starting, and Restarting VQE-S
The VQE-S application is a service that is started with the Linux service command.
To stop, start, or restart the VQE-S application, follow these steps:
Step 1 Log in as root.
Step 2 Depending on what is required, issue one of the following commands:
•To stop the VQE-S, issue the following command:
[root@system]# service vqes stop
•To start the VQE-S, issue the following command:
[root@system]# service vqes start
•To restart (stop and then start) the VQE-S, issue the following command:
[root@system]# service vqes restart
Stopping, Starting, and Restarting VCDS
The VCDS application is a service that is started with the Linux service command.
To stop, start, or restart the VCDS application, follow these steps:
Step 1 Log in as root.
Step 2 Depending on what is required, issue one of the following commands:
•To stop VCDS, issue the following command:
[root@system]# service vcds stop
•To start VCDS, issue the following command:
[root@system]# service vcds start
•To restart (stop and then start) VCDS, issue the following command:
[root@system]# service vcds restart
Resetting the Root Password on the VQE-S or VQE Tools Server
To reset the password on the VQE-S or the VQE Tools server, you need to boot into single-user mode from the GRUB loader and reset the password. To reset the password, do the following:
Step 1 Connect to the serial console and reboot the VQE-S or the VQE Tools server by issuing the following command:
The server reboots.
Step 2 Press any key when the following message is displayed.
Press any key to continue.
Press any key to continue.
Press any Key to continue.
Red Hat Linux is selected as the operating system and the Grub Loader screen is displayed.
Step 3 Select the entry "Red Hat Enterprise Linux Server (2.6.18-157.el5)" if not already selected, and press e.
Step 4 Using the arrow keys, move the cursor to select the following line:
kernel /boot/vmlinuz-2.6.15-1-686 root=dev/sda1 ro
Step 5 Press e to enter edit mode.
Step 6 Using the arrow keys, move the cursor to the end of the line, press the Spacebar and enter single.
For example:
grub edit > kernel /boot/vmlinuz-2.6.15-1-686 root=dev/sda1 ro single
"single" is appended to the end of the line.
Step 7 Press Enter to exit edit mode and return to the Grub Loader screen.
Step 8 Press b to boot the Linux kernel into single user mode.
The sh-3.2# prompt is displayed.
Step 9 Enter passwd and specify the new password. For example:
sh-3.2# passwd new-password
Step 10 Issue the following command to reboot the VQE-S or the VQE Tools sever:
The system reboots.
Step 11 Login to the server using the new password.
Your root password has been changed.