The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This appendix presents troubleshooting tips for the upgrade and downgrade procedures. The topics covered in this appendix include:
•Interfacing with the CDS Server During an Upgrade
•Examples of Output for the Upgrade and Downgrade Scripts
This section covers the following topics:
•Viewing Upgrade Progress with Keyboard and Monitor Interface
•Backspace Does Not Work in Terminal
•Monitoring OS Load Runtime Syslog File
If the serial and VNC Listener cannot be used to perform an upgrade, the keyboard and video monitor can be used for the upgrade. However, performing the upgrade using the keyboard and monitor provides limited visibility and is not recommended.
If the only option is to use a keyboard and monitor for the upgrade, press Enter when the upgrade script prompts for the VNC Listener. The server reboots several times. During the second stage of the operating system (OS) upgrade, the package installation progress and other steps of the upgrade can be viewed on the third virtual terminal on the keyboard/monitor interface. To access this virtual terminal press Alt-F3 and the video switches to the third terminal.
When working with different applications that provide SSH or telnet capability there are times when the terminal definition does not translate correctly to the shell opened to the server being upgraded. This causes the scripts to be unable to read the text entered and the backspace key does not function correctly.
To assist in allowing the backspace to work, determine the backspace key definition and use stty erase command to set the definition.
Note This does not always work because the application closest to the user may still not translate the key definition correctly even after determining the key definition.
To determine the backspace key definition, do the following:
Step 1 After logging in to the CDS server, press Ctrl-V followed by the Backspace or Delete key.
The key sequence for the backspace is displayed. Typically, the key sequences are ^H or ^?.
Step 2 Enter the key sequence exactly with the stty erase command; for example:
stty erase ^H
Note The ^H or ^? are specifically Shift-6 and uppercase H or question mark.
During the upgrade, when a VNC Listener is used, the stage 2 OS loading process can be monitored from the serial console window. View the /tmp/syslog file for details of the operations currently taking place. To view the syslog file use the tail -f /tmp/syslog command. After the stage 2 OS load has completed, the tail is killed and the server reboots.
Note The following messages in the syslog file are known benign messages:
•<4>end_request: I/O error, dev fd0, sector 0
•<4>end_request: I/O error, dev fd0, sector 0
•<3>Buffer I/O error on device fd0, logical block 0
This section describes the known issues you may encounter with the upgrade and downgrade procedures, and the workarounds if you do encounter them.
After Software Installation
•CSCtf84424
Symptom:
The kdump process outputs not enough space warning after install and reboot.
Conditions:
If /arroyo partition is less than the amount of physical RAM on the server (16 GB or 32 GB depending on the CDE model).
For the Streamers, Vaults and Caching Nodes, a new kernel is installed and the kdump.conf is modified. When the system is rebooted and the kdump img is processed, the following warning is provided:
Rebuilding /boot/initrd-2.6.18-53.el5.cdstv.2.2.0.b127kdump.img
Warning: There is not space enough to save a vmcore.
The size of LABEL=/arroyo should be much greater than 32897580 bytes.
The number provided is actually kilobytes, not bytes.
Workaround:
None. The warning provided includes the incorrect space requirements. As long as the file system is less then 95 percent full there is enough space to support the kdump service. Any file systems over 90 percent full should be cleaned up where and when needed irrespective of this issue.
During Operating System Upgrade
•CSCtf59075
Symptom:
Cannot break into a server shell during an OS upgrade.
Conditions:
If the VNC Listener and serial console are not used when performing an upgrade from Release 2.0.x or 2.1.x to Release 3.2.1.
Workaround:
Use a VNC Listener and serial console when performing an upgrade. This is recommended for performing the upgrades.
During Software Downgrade
•CSCtf56571
Symptom:
Kdump reports invalid partition for the defined crash location.
Conditions:
After downgrade has been performed.
Workaround:
After the OS downgrade, the configuration restore, and the cdsinstall of the TV CDS software image, but before the server reboots, change the /etc/kdump.conf file to use the disk partition /dev/sd?6 to /dev/sda6. Then reboot the server.
•CSCtf94113
Symptom:
References to "upgrade" are in the downgrade kit. The string "upgrade" is seen when performing a downgrade from Release 3.2.1 to Release 2.1.x or Release 2.0.x.
Conditions:
Always.
Workaround:
The string "upgrade" is only provided in messages presented to the user. There is no operational impact. These benign messages can be ignored.
During Shutdown
•CSCte62872
Symptom:
The avs_shutdown process outputs the following error when system is rebooted:
/sbin/avs_shutdown: line 6: /proc/calypso/test/shutdown: No such file or directory
Conditions:
When system is rebooted and the TV CDS processes are not running.
Workaround:
This is a known benign error.
During Software Installation
•CSCte39340
Symptom:
The cdsinstall script fails to run the makedumpfile command and the following error message is displayed:
Failed to run mkdumprd
Conditions:
On installation of TV CDS software.
Workaround:
This is a benign error and can be ignored. It has no service affecting impact.
•CSCtf87159
Symptom:
The kickstart file is not found on initial boot. Error downloading kickstart file.
Conditions:
When using most USB CD-ROM drives.
Workaround:
When the message is displayed, press Enter on the cdrom: line and select OK. The installation proceeds.
Additional Information:
The error provided on the console is below. Press Enter on the cdrom:/ line, then OK is highlighted.
+-------------+ Error downloading kickstart file +----------------+
| |
| |
| Unable to download the kickstart file. Please modify the |
| kickstart parameter below or press Cancel to proceed as an |
| interactive installation. |
| |
| cdrom:/x20ssd.cfg_____________________________________________ |
| |
| +----+ +--------+ |
| | OK | | Cancel | |
| +----+ +--------+ |
| |
| |
+ ----------------------------------------------------------------+
The following error messages may occur during the software upgrade:
***ERROR: Unable to copy <file> to <target location>: cp: cannot stat ...
If this error is seen, it is most likely a result of a corrupted kit. During the initial extraction of the kit, the contents are checked to ensure integrity. Verify that the initial kit extraction did not have errors. Recovery procedure for this issue is provided in the "Upgrade Abort Failure Recovery" section.
***ERROR: Failed to delete partition /dev/sda#
If this error is seen, it may be a result of an initial system configuration issue. See the"Upgrade Abort Failure Recovery" section to attempt to correct this issue.
***ERROR: /boot_cds is not mounted. Please manually mount it before running this upgrade script again.
This could be an indication that the /boot_cds directory has been manually unmounted. Mount this partition before proceeding with the upgrade.
***ERROR: You have to be root to perform the remote ISO installation!
You must be logged in as user root to run the upgrade or downgrade kit.
***ERROR: Hardware type is neither Merrimack nor Lindenhurst.
You are attempting to run the upgrade or downgrade script on a nonsupported server.
***ERROR: Unable to create directory <dir>
This may be the result of a missing parent directory or failed hardware. Run the cds_upg_report.sh script and contact Cisco Support.
***ERROR: Unable to mount <dir> on <dir>
This may be the result of a missing file or directory, or hardware failure. Run the cds_upg_report.sh and contact Cisco Support.
***ERROR: Absolute path to ISO image not given.
This indicates the ISO image file provided on the command line was not provided with the full path. Under normal operation this command line option is not needed. Rerun the command without providing the ISO option.
***ERROR: Unable to find ISO image
This may be an indication that the kit was corrupted. Verify that no errors were provided during the initial extraction and rerun the self-extracting file.
***ERROR: <isofilepath> is not a "ISO 9660 CD-ROM filesystem" format file
This indicates the ISO image file provided on the command line is not the standard ISO image file required for the upgrade. Under normal operation this command line option is not needed. Rerun without providing the ISO option.
***ERROR: Unable to unmount <mount point>
This is an indication that a running process has a file open, or a shell is open currently in the mount point. Be sure there are no users running processes (for example, the vi editor) in the location that is identified as mount point.
***ERROR: VNC Listener <vnc_ip>:<vnc_port> is not reachable.
This is an indication that the VNC Listener cannot be reached. Verify system, vnc_ip, has a running VNC Listener and that the server to be upgraded can ping the VNC system (check that your management routes are correct).
***ERROR: Partition /dev/sda7 does not exist on /dev/sda
On initial running of the script, this is an expected benign error.
***ERROR: Hard drive <disk> not detected on server.
The server has a hardware failure. Verify that the server was operating correctly and all drives defined in the hardware guide are operational.
***ERROR: unknown command line option
An unknown or unsupported upgrade command line option was provided. Verify syntax of the command entered and rerun.
***ERROR: There is no default route on this server, cannot deduce the management interface.
The upgrade script expects an operational server to perform the upgrade. In this situation a management default route should exist on the server. If a default route does not exist, log in to the CDSM GUI and verify one exists for this server (Server Setup page). If it does, re-submit and verify that a default route is configured on the server by running the netstat -rn command.
If this is a newly shipped CDS server, manually configure the management interface configuration, enable the interface and manually put in a default route. For more information, see the "Upgrading a New CDS Server to Release 3.2.1" section.
***ERROR: ISO image SHA1 checksum <string> does not match <string> in iso.sha1sum.
This means the upgrade or downgrade kit is corrupted. Redownload the kit and extract it again.
***ERROR: /arroyo/db is not mounted.
This could be an indication that the /arroyo/db directory has been manually unmounted or the upgrade exited prematurely in a previous run. Reboot the server and verify that the /arroyo/db directory is mounted.
***ERROR: restore.list|backup.list does not exist.
This indicates that the kit has been corrupted in transition to the destination server. Verify that the checksum provided the Cisco software download website is the same as the checksum as the file on the server. If the checksums match, reextract the kit and run the upgrade again.
***ERROR: scp failed. Please retry:
A failure occurred during the transfer of the backup archive to the remote server. Check the IP address, username, password, and network connectivity to the remote server.
***ERROR: Cannot create archive <backup_archive_name>.tgz.
This indicates that the creation of the backup archive failed, most likely because of limited space in the /arroyo/db directory. The upgrade gives a best effort to calculate the amount of required space before the initial reboot, however if files have changed during the initial reboot this error may occur. Remove all non-essential files in the /, /home, and /arroyo/db directories and rerun the upgrade.
***ERROR: <partition> is XX % used, please clean <partition> up before running this upgrade script again.
This means the <partition> partition does not have enough space for the upgrade to continue. Clean the partition space and rerun the upgrade.
***ERROR: There are two or more upgrade partitions on the system.
If this error is encountered, it means there is more then one upgrade partition created. There can only be one upgrade partition, the second partition must be removed before proceeding.
If the upgrade script has aborted after the final "Do you wish to continue?" question but before the OS load reboot phase, use the following procedure to restore the server to a state where the upgrade can be performed.
Note If this has happened there may be more serious issues with the current state of the server (for example, a corrupted kit was used to perform the upgrade or the downgrade, hardware issues exist, the server was not initially imaged correctly). If this procedure does not correct the state of the server, run the cds_upg_report.sh script, contact Cisco Support, and provide them the resulting archive from the cds_upg_report.sh script along with the details of the failure.
To restore the CDS server, do the following:
Step 1 Copy a known valid upgrade or downgrade kit to the server.
Step 2 Extract the kit to ensure integrity of all files and review the output of the extraction process. There should be no errors on the extraction. If there are errors, the source location where this kit was copied from is corrupt, or the method used to transfer the kit to the destination is corrupting the kit. Verify the checksums of the kit to ensure integrity.
Step 3 If /arroyo/db does not exist, create this directory.
Note If the /arroyo partition does not exist contact Cisco Support.
Step 4 If /arroyo/db is not mounted, mount it.
Step 5 Restore the backup created on the initial run of the previous kit. For more information, see the "restore_cfg.sh Details" section.
Step 6 Run the upgrade or downgrade procedure again.
The following warning messages may occur during the software upgrade:
***WARNING: Cannot create backup <type> archive <file name>
The backup of the configuration, database, or log files has failed. Cleanup the file system space and rerun the upgrade.
***WARNING: Not enough space available to perform backup.
The /arroyo/db directory does not have enough space to perform the backup. Remove all non-essential files and re-run the upgrade.
All other errors that have not been commented on should be provided to Cisco Support along with the cds_upg_report.sh archive and the output of the Linux history command (run as root).
***WARNING: /dev/sda7 does not exist, it needs to be created from /dev/sda6.
The /dev/sda6 partition has XXXXXXXXXXKB total disk space, which is more than the 8000000KB required to create new /dev/sda7 partition.
This is an expected warning the first time the cds_remote_iso_install runs. No action is required.
"WARNING!!! VNC server running with NO PASSWORD!
You can use the vncpassword=<password> boot option if you would like to secure the server. This is an expected benign warning. No action required.
WARNING: No existing version found on this server. Cannot backup existing files.
This warning is provided when running the cdsinstall after the server has been upgraded. This is an expected warning and no action is required.
"No kdump initial ramdisk found.[WARNING]" ... "Failed to run mkdumprd"
This warning is provided when running the cdsinstall after the upgrade and on first boot after cdsinstall. The kdump configuration is resolved on the first reboot after the software installation. No action is required.
In rare instances there is a failed or stalled upgrade. For example, this could occur if power is not maintained during the upgrade. The backup performed before the upgrade was started can be used with the new server image procedure to install Release 3.2.1.
Note If power is lost during an upgrade there is no guarantee the upgrade will continue when power is restored.
A first attempt of just restoring power and booting the server should be performed. If this fails, then the server must be imaged directly to Release 3.2.1 and the configuration should be restored.
To image the server directly and restore the configuration, do the following:
Step 1 Follow the instructions in the "Imaging a CDS Server with 64-Bit OS using a DVD" section.
If the server OS upgrade has not completed within 45 minutes or the upgrade is hung for a long time, then the upgrade has encountered an uncorrectable error.
If the upgrade has stalled on the rpm kernel add portion and a VNC Listener was used, tail the /tmp/syslog file from the serial port console. If VNC Listener was not used, go to the video console and press Alt-F3 to switch to the virtual terminal. If the following error is seen:
Code={SATA NCQ Fail All Commands After Error},
the loading of the kernel is failing on a disk drive. Look on the front of the server to see if one of the drives is slowly blinking. If the drive is slowly blinking, remove it. If another drive starts blinking, remove it as well. Continue removing drives until no drives are blinking. After this, the server should continue to proceed with the installation.
Step 2 Restore the backed up configuration and database files. See the "restore_cfg.sh Details" section for more information.
Step 3 Run the cdsinstall script.
# cd /root
# ./cdsinstall /CDS-TV-3.2.1.iso
Step 4 Move the /etc/rc.d/upgrade.rc.local file to /etc/rc.d/rc.local.
# mv -f /etc/rc.d/upgrade.rc.local /etc/rc.d/rc.local
The VNC Listener is an application that waits to receive input from a remote host. When upgrading or downgrading the TV CDS, the VNC Listener is used to remotely display the status of an upgrade, instead of the upgrade or downgrade output being sent to the video console.
We recommend that the software upgrade be monitored from a VNC Listener. During the stage 2 boot process of the upgrade, a window is provided to the user to see the operations taking place on the server. Using the VNC Listener provides the additional ability to triage issues through the serial console if there is a failure during the upgrade.
This section covers example outputs for the following scripts:
•Upgrade Script with the Check Option
•Backup Script—Configuration and Database
•Restore Script—Configuration and Database
The following output provides an example of the output for the upgrade script with the --check option:
# ./cds_remote_iso_install --check
Upgrade version: cdstv-3.2.1
Script mode: --check
Hardware type: Merrimack
ISO location:
Baud rate:
Script running directory: /root/cds_remote_upgrade
Command line options: --check
Doing prerequisite checking... [DONE]
CDSTV application processes are not running.
/dev/hda1 is mounted on /boot.
/boot is 22% used.
/dev/hda2 is mounted on /.
/ has 4718952 KB space available, more than 1000000 KB required.
/dev/sda2 is mounted on /home.
/dev/sda3 is mounted on /var.
/dev/sda6 is mounted on /arroyo.
/dev/sda1 is mounted on /arroyo/db.
/boot_cds has 3846340 KB space available, more than 1000000 KB required.
Stored chkconfig --list output to chkconfig-list.output.
Copied chkconfig-list.output to /root/chkconfig-list.output.
Found CDS version cdstv-2.1.3-b1 to backup
Found restore.list.
Found backup.list.
Detailed size of backup files list:
/home/isa 209252KB
/etc 32276KB
/root/chkconfig-list.output 4KB
/arroyo/test 64KB
/arroyo/image/tags 4KB
/usr/local/share/snmp/snmpd.conf 4KB
/var/spool/cron 12KB
/var/net-snmp 12KB
Total available space on /arroyo/db: 46301624 KB,
Space needed for /arroyo/db/DATADIR: 40580 KB,
Space needed for /arroyo/log: 41288 KB,
Total estimated space required for all archives is: 323496 KB,
Excluded space of preexisting backups: 0 KB,
Calculated space needed for backup: 107832 KB.
There is enough space for backup.
Cleaning up... [DONE]
Exiting remote iso install setup script...
Logfile at: /var/log/cds/cds_upgrade_host-iptv.log
The following output provides an example of the output for the cds_upg_report.sh script:
# ./cds_upg_report.sh
******************************************************************
****** WARNING WARNING WARNING ******
******************************************************************
This is the cds_upg_report.sh that is expected to be used to collect
useful information and logs from CDS remote upgrade/downgrade. The
collected information and logs will be archived for further upgrade
or downgrade assistance or review.
Do you wish to continue? [yes/no]: yes
Creating archive /boot_cds/upgrade/cds_upg_report_host_20100319-130154.tgz...
tar: Removing leading `/' from member names
/root/install.log
/root/cds_restore.log
/var/log/messages
/var/log/anaconda.log
/var/log/anaconda.syslog
/var/log/cds/cds_upgrade_host.log
/boot_cds/upgrade/CDS_UPGRADE_INFO
/boot_cds/upgrade/ks_remote_template.cfg
/boot_cds/upgrade/ks_upgrade_host_stage1.cfg
/boot_cds/upgrade/ks_upgrade_host_stage2.cfg
/boot_cds/upgrade/cds_upg_report.log
/tmp/ks-script-fQaPB-.log
Report archive /boot_cds/upgrade/cds_upg_report_host_20100319-130154.tgz completed.
Details of the report procedure are logged in:
/boot_cds/upgrade/cds_upg_report.log
The following output provides an example of the output for the backup_cfg.sh script:
# ./backup_cfg.sh
Stored chkconfig --list output to /root/chkconfig-list.output.
Doing backup checking...
Found CDS version cdstv-2.1.3-b1 to backup
Found backup.list.
Detailed size of backup files list:
/home/isa 209252KB
/etc 32276KB
/root/chkconfig-list.output 4KB
/arroyo/test 64KB
/arroyo/image/tags 4KB
/usr/local/share/snmp/snmpd.conf 4KB
/var/spool/cron 12KB
/var/net-snmp 12KB
Total available space on /arroyo/db: 46301624 KB,
Space needed for /arroyo/db/DATADIR: 40580 KB,
Space needed for /arroyo/log: 41288 KB,
Total space needed for backup, /arroyo/db/DATADIR and /arroyo/log: 323496 KB,
Excluded space of preexisting backups: 0 KB,
Calculated space needed for backup: 107832 KB.
There is enough space for backup.
Doing backup...
Successfully created backup tarball /arroyo/db/host_cdstv-2.1.3-b1_20100319-125015_backup.tgz.
Successfully created backup db archive /arroyo/db/host_cdstv-2.1.3-b1_20100319-125015_backup_db.tgz.
Successfully created backup log archive /arroyo/db/host_cdstv-2.1.3-b1_20100319-125015_backup_log.tgz.
Backup archive /arroyo/db/host_cdstv-2.1.3-b1_20100319-125015_backup.tgz is complete.
Backup db archive /arroyo/db/host_cdstv-2.1.3-b1_20100319-125015_backup_db.tgz is complete.
Backup log archive /arroyo/db/host_cdstv-2.1.3-b1_20100319-125015_backup_log.tgz is complete.
Would you like to upload backup and log archive to a remote server? [yes/no]: yes
Remote SSH server address: 1.1.1.1
Remote SSH server path: /arroyo/db
Username: root
scp /arroyo/db/host_cdstv-2.1.3-b1_20100319-125015_backup*.tgz root@1.1.1.1:/arroyo/db
root@1.1.1.1's password:
Successfully copied /arroyo/db/host_cdstv-2.1.3-b1_20100319-125015_backup.tgz to 1.1.1.1
Successfully copied /arroyo/db/host_cdstv-2.1.3-b1_20100319-125015_backup_db.tgz to 1.1.1.1
Successfully copied /arroyo/db/host_cdstv-2.1.3-b1_20100319-125015_backup_log.tgz to 1.1.1.1
Removed /root/CDS_BACKUP_INFO
Backup completed.
Details of the backup procedure are logged in:
/var/log/cds/backup_cfg.log
The following output provides an example of the output for the restpre_cfg.sh script:
./restore_cfg.sh
******************************************************************
****** WARNING WARNING WARNING ******
******************************************************************
This is the restore_cfg.sh that is expected to be used to restore
a system configuration archive that was created from the backup_cfg.sh
or from an upgrade/downgrade process for the CDS-TV systems.
If the archive being restored is not from either source defined above,
do not continue this process. Continuing without a valid archive
could render the system inoperative. This will be the only warning.
Do you wish to continue? [yes/no]: yes
Backup configuration tgz archive?: /arroyo/db/host_cdstv-2.1.3-b1_20100319-125015_backup.tgz
Found tgz archive /arroyo/db/host_cdstv-2.1.3-b1_20100319-125015_backup.tgz.
Found db tgz archive /arroyo/db/host_cdstv-2.1.3-b1_20100319-125015_backup_db.tgz.
Found /boot_cds/upgrade/restore.list.
Found restore file list in /boot_cds/upgrade/restore.list.
Changed directory to /.
Checking /arroyo/db/host_cdstv-2.1.3-b1_20100319-125015_backup.tgz...
Checking /arroyo/db/host_cdstv-2.1.3-b1_20100319-125015_backup_db.tgz...
Extracting /arroyo/db/host_cdstv-2.1.3-b1_20100319-125015_backup.tgz...
Restore the services from /root/chkconfig-list.output
chkconfig --level 0 ntpd off
chkconfig --level 1 ntpd off
chkconfig --level 2 ntpd on
chkconfig --level 3 ntpd on
chkconfig --level 4 ntpd on
chkconfig --level 5 ntpd on
chkconfig --level 6 ntpd off
Removing existing /arroyo/db/DATADIR...
Extracting /arroyo/db/host_cdstv-2.1.3-b1_20100319-125015_backup_db.tgz...
Restoration completed.
Review the details of the restoration procedure in:
/var/log/cds/restore_cfg.log