Table of Contents

Problem Scenarios

This chapter describes some typical VCO/4K system problems.

Cisco Connection Online

Scenarios in this chapter may be referenced to a Cisco DDTS issue. These issues can be viewed via the Cisco Connection Online (CCO). Cisco Connection Online (CCO) is Cisco Systems' primary, real-time support channel. Maintenance customers and partners can self-register on CCO to obtain additional information and services.

You can access CCO in the following ways:

• WWW:  http://www.cisco.com
  
  
• WWW:  http://www-europe.cisco.com
  
  
• WWW:  http://www-china.cisco.com
  
  
• Telnet:  cco.cisco.com
  
  
• Modem:  From North America, 408 526-8070; from Europe, 33 1 64 46 40 82. Use the following terminal settings: VT100 emulation; databits: 8; parity: none; stop bits: 1; and connection rates up to 28.8 kbps.
  
  

For a copy of CCO's Frequently Asked Questions (FAQ), contact cco-help@cisco.com. For additional information, contact cco-team@cisco.com.

Problems with the VCO Subsystem

API

Reporting to Host

$EA

Generic will report to the Host a flipped Call Reference Flag bit in the Call Reference Number byte of the $EA report.

For details, see DDTS issue CSCdp41872 on CCO.

If the VCO receives an ISDN RELEASE COMPLETE message with Call Reference Number from the network, then the Generic will report to the Host a dummy Call Reference Number byte of "FF FE" in the $EA report.

For details, see DDTS issue CSCdp41872 on CCO.

Receiving Commands from Host

$49

The Generic will reject a $49 command if the $49 command is used to connect two clear-channel ports that are in "CP_SETUP" state. A network status byte (NSB) error code of $20 will be returned. The associated port will have a DTMF receiver attached.

For details, see DDTS issue CSCdp72822 on CCO.

Boot

Failure to Complete Boot Cycle

"WARNING: unreliable read/write to NVRAM" message upon boot up

The VCO CPU will not finish its boot-up cycle. This indicates that the NVRAM battery on the CPU is defective or has been drained.

Replace the CPU.

Multiple Re-boots/Continual Re-boots

VCO Reboots Periodically Throughout a 24-hour Period

This usually means that a circuit card (typically other than the AAC) is failing in such a way that it functions for awhile, then fails and causes the system to switch sides and reboot. Isolate the faulty circuit card as follows:

Step 1   Unseat all circuit cards, except the NBC3 in slot 1, the A-side Combined Controller (with CPU) in slots 3 and 4, and the AAC card.

Step 2   Hard-select the A-side by moving the selector switch on the AAC card to A.

Step 3   Reset the A-side by pressing the A-reset button on the AAC card, then isolate for possible faults in A-side control hardware.

Step 4   Isolate for possible faults in B-side control hardware by selecting the B-side on the AAC card:

Step 5   Insert each card, present in slots 7 to 21, into the system, one at a time.

   a. Insert one card and ensure that it properly downloads. If it does not properly download and come back into service, then permanently remove it (this could be the culprit card). Proceed to the next circuit card.

---

Note   If a card properly downloads and comes back into service, then wait two minutes after it comes back active before proceeding to insert the next card.

---

   b. Repeat the process until the card that you insert causes the system to reboot. This will isolate the cause of the periodic rebooting to one of service circuit or network circuit cards.

Step 6   To restore the system redundant operation, hard-select the AAC card to AUTO by moving the selector switch on the AAC card to AUTO.

Installation Diskette

Booting off the Generic Diskette #1, or Optional Diskette, and the Boot does Not Complete

The boot cycle stops at "Reading boot file from disk device A:/boot/boot.sds" as follows:

This means that no diskette volume label was entered on the diskette. Do the following:

Step 1   Enter the volume label INSTALL on this diskette from a PC.

Step 2   Reinsert the diskette into the system and press the reset button on the AAC card.

The boot cycle may stop at "Please insert disk volume INSTALL... Press ENTER key when ready...": as follows:

This means either of the following two possibilities:

• The label in Generic diskette #1 does not contain a volume label that is exactly INSTALL. This also applies to the optional diskette, e.g., ETHERNET (for V5.1(2) or lower) or OPTIONAL SOFTWARE. If this is the case, then do the following:
  
  

Step 1   Enter the volume label INSTALL on this diskette.

Step 2   Reinsert the diskette into the system and press the Return key and the boot cycle will continue as follows:

• The diskette you inserted in the system is one of the following:
  
  

Generic diskette 2

Generic diskette 3

Generic diskette 4

Generic diskette 5

Generic diskette 6

These diskettes cannot be used to boot the system. Insert Generic diskette #1 into the system and press the Return key and the boot cycle will continue:

Loading a Generic Diskette Other than Diskette #1

If you attempt to boot the system from a diskette other than diskette #1, you will receive an error of "check_vol: error 0x769 Check Floppy Disk" or an error of "Wrong Volume in Floppy." You might also encounter errors due to missing volume labels or inaccurate volume labels.

• The output could look something like this:
  
  

Copyright Motorola Inc. 1988 - 1997 All Rights Reserved
 
MVME147 Monitor/Debugger Release 2.44 - 8/7/97
CPU running at 33 MHz
 
FPC  passed test
MMU  passed test
 
COLD Start
 
Onboard  RAM start = $00000000,  stop = $00FFFFFF
 
147-Bug> Searching for ROM Boot 
                                                                               
147-Bug>G FFA0002C
Effective address: FFA0002C 
 
SDS Initializing...
Hard disk mounted successfully
Hard disk dismounted successfully
Reading boot file from disk device A:/boot/boot.sds
Loading file A:/boot/install.exe
Hard disk mounted successfully
Hard disk dismounted successfully
 
I N S T A L L A T I O N   U T I L I T I E S
 
 
       1) Install/Configure Basic System Software
       2) Incremental Install of Basic System Software
       3) Disk Utilities
       4) Install Another Software Option
       5) Database Conversion
       6) License Configuration
       7) Set Extended Operational Mode
       8) Enable C-Bus Mode
       X) Terminate Installation
 
 
       Enter Selection: 1

 
Device C: Will Be Reformatted. All Information Will Be Erased.
Do You Wish To Back-Up The System Data Base (Y/N) =N?n
 
This Installation Process Reformats The Device C:
All Information Will Be Erased
Press Return to Continue
 
Formatting Device C:
Format of Device C: complete
Creating Directory Structure on Device C:
Directory Structure on Device C: Complete
Creating System Configuration File ..... Done
Copying A:/BOOT/SOFTWARE.CFG ...
1 file[s] copied
 
 
Insert disk 2 of Installation Set
Press return to continue
 
Copying Files
Copying A:/* ...
0 file[s] copied
Copying A:/BOOT/* ...
3 file[s] copied
Copying A:/DBASE/* ...
19 file[s] copied
Copying A:/LOG/* ...
0 file[s] copied
Copying A:/TRACE/* ...
0 file[s] copied
 
 
Insert disk 3 of Installation Set
Press return to continue
 
Copying Files
Copying A:/* ...
0 file[s] copied
Copying A:/BOOT/* ...
6 file[s] copied
 
 
Insert disk 4 of Installation Set
Press return to continue
 
Copying Files
Copying A:/* ...
0 file[s] copied
Copying A:/BOOT/* ...
3 file[s] copied
 
 
Insert disk 5 of Installation Set
Press return to continue
 
check_vol: error 0x769
Check Floppy Disk
 

This means that no diskette volume label was entered onto diskette #5. Do the following:

Step 1   Enter the volume label SUMMA4D on this diskette using a PC.

Step 2   Re insert the diskette into the system and press Return for the installation to resume.

• If a volume label was entered but is not correct, you might see the following output:
  
  

Wrong Volume in Floppy
 

Make sure the volume label is correct. Insert the correct diskette into the system and/or correct the volume label and reinsert the diskette and press Return for the installation to resume. See Table 2-1 for correct volume labels.

Booting off Generic Diskette #1, or Optional Diskette, and an Error Occurs

The output will likely be an error "NonBootable Floppy Exists in A: Drive Replace it to Boot from A:" .

The following message sequence means that Generic diskette #1, or an Optional diskette, does not have the "/boot" directory on the diskette (but does have the correct volume label):

The files on the given diskette in question are probably located in the root directory. No files for any installation diskette (volume label = INSTALL), including Generic diskettes 2 through 6, should be in the root directory. All files should be in the /boot directory except as noted in Table 2-2.

Note   If you received ftp-downloaded Generic software as a WinZip V7.0 self-extracting file, then use WinZip V7.0 to unzip the file. WinZip creates the required directory structure automatically. Customers using PKUnZip have reported that PKUnZip will not create the required directories. If you use PKUnZip, you will have to manually create the directories indicated in Table 2-1.

Ethernet LAN (TCP/IP)

No Connection

VCO-Host Socket Connection Lost

The following error message generally indicates that this problem has occurred:

The VCO has closed the VCO-Host socket connection because the VCO's TCP/IP transmit (tx) queue has completely filled up. There is no more free space in the VCO's TCP/IP transmit (tx) queue for the application residing on the VCO to write data to.

Here is an example VCO log file (A-MMMDD.LOG in C:/log/) foot print:

There are two causes of HST013.

Cause 1:

The Host's TCP/IP receive (rx) queue is filling up faster than the Host can empty it (read it). You can easily determine this by monitoring the VCO-Host socket connection with a LAN protocol analyzer (sniffer).

You will see in the protocol analyzer's trace that the Host is sending to the VCO a "window size = zero (0)" message. When the VCO reads this message, it then instructs itself to "store" the messages in its transmit (tx) buffer and not send them to the Host. If the application residing on the VCO is still continuing to write to its tx queue while the tx queue is in the "store" state, the tx queue will eventually completely fill up. When the tx queue completely fills up, the VCO closes the VCO-Host socket and issues "HST013: Host < host-name > Socket Write Error $A23:Operation Would Block" to the VCO log file.

Example LAN protocol analyzer trace:

In packet 3974 above, the Host, "10.10.25.156" ( = source), is sending to the VCO, "10.10.25.19" (DST = destination), a "window size = 0 bytes" message (th_win=0) implying that the Host is not reading the data out of its rx queue fast enough.

In packet 3981 above, just 4 packets later, the VCO, "10.10.25.19" ( = source), is sending to the Host, "10.10.25.156" (DST = destination), a "window size = 4096 bytes" message (th_win=4096) implying that the VCO's tx queue is not full.

Further Isolation and Possible Resolution

To improve operation:

• Ensure that the LAN network segment is not being disrupted by excessive amounts of other traffic which could keep the Host from accessing the VCO. Remove the other traffic.
  
  
• Increase the Host's rx queue size.
  
  
• Look at the task priorities that you run on your Host. Verify that there are no tasks, used to communicate with the VCO, that are running at a priority lower than other tasks that provide similar functionality. Your Host runs at "0% CPU idle", the effected task would not get enough CPU time and would start to fall behind (when your Host CPU runs at less than 0% idle, this task would get enough time to run and you would see no problem). To address this issue, try raising the priority of the task in question.
  
  

Cause 2:

The application residing on the VCO is filling up its tx queue faster than it can empty it (transmit it). You will see nothing abnormal in the LAN protocol analyzer trace, and the HST013 message will be generated in the log file.

The application residing on the VCO (generic) writes its output directly to the tx queue, which is the low-level network device driver that resides on the "vrtx ethernet controller" on the CPU, and, in this case, it is filling up its tx queue faster than it can send the data out to the Host. The "vrtx ethernet controller" will eventually close the socket and not report an error on the wire.

Further Isolation and Possible Resolution

The VCO is trying to send report messages to the host as fast as it can in response to the rate at which the host is sending command messages to the VCO.

To improve operation:

• Try reducing the rate at which the host sends call processing commands (e.g., $49 commands) to the VCO.
  
  
• If you are using the $83, $82, or $90 commands, see if you can reduce their frequency of use.
  
  

---

Note   One $83, $82, or $90 command (each a "non-callprocessing" command) has the potential to generate up to 30 VCO acknowledgement reports.

---

• Evaluate the bit settings in each "non-callprocessing" command and determine (for example) if your host application actually needs to be informed of the status of every port in the system at this particular time.
  
  

File System

Disk Utility Commands

VCO allows you to move to the root directory a log file with today's date, but will not automatically create a new log file until midnight

The system allows you to move (using the Rename Files command) a log file that is "open" (i.e., named as today's date) to the root directory (C:/) or to another C-drive directory. But, the system will not automatically create a new today's log file in C:/log until midnight.

To restart today's log file without having to wait until midnight:

Step 1   After moving and renaming the log file, make a note of the time on the system and on your watch.

Step 2   Change the time on the system to 23:59:55, from the Clock/Calendar Configuration screen, and wait one minute. A new log file, "tomorrow's" log file, will be created in C:/log and "today's" log file will close.

Step 3   Change the time on the system back to the time you need. This will create a new "today's" log file in C:/log. Today's log file operation has now been restored.

Step 4   Delete "tomorrow's" log file from C:/log.

Missing Files

VCO will not create a new log file until midnight

See the "VCO allows you to move to the root directory a log file with today's date, but will not automatically create a new log file until midnight" section.

VCO Log File Messages—Special Considerations

Live Upgrade Log File Messages (C:/log/UGmmddyy.LOG)

"Event Handler failed. Returned LUPG_EVENT_ERROR"

For details, see issue CSCdp23217 on CCO.

Log File Messages (C:/log/x-mmmdd.LOG)

ALM096: Trace File Exceeded 1MB Size

Perform the following procedure to clear ALM096:

Step 1   Disable Host Message Trace and NBC Message Trace from the System Trace Configuration screen.

Step 2   Copy the file if you need it (we recommend using FTP).

Either delete the file, or move it (using the Rename Files command) to the root directory (C:/) or to another C-drive directory.

Step 3   Turn tracing on again (creating a new trace file) and let it run for 3 minutes; and then turn it off (you can do this during a slow traffic period).

---

Note   The system writes 1000 time stamps to the new trace file and then it samples the size of the trace file. Three minutes (even during a slow traffic period) should be sufficient time to generate 1000-plus writes. When the Generic detects that 1000 writes have been made, it will then sample the trace file. When it samples the trace file, it will see approximately 40 kBytes (which is less than 1 MB) and clear ALM096. Every 1000 writes generates approximately 40 kBytes of tracing.

---

Step 4   Verify that ALM096 clears in your Host, in the Log File, and in the System Alarms Display/Major screen.

Step 5   Delete this new trace file.

ALM097: Log File Exceeded 1MB Size

Perform the following procedure to clear ALM097:

Step 1   Identify and then eliminate the root cause of the streaming log file messages through troubleshooting.

---

Note   Streaming log file messages are typically the result of a Service Circuit card or Network Interface card that is failing. In this case the corrective action would be to remove the defective card from the system.

---

Step 2   Copy the file if you need it (we recommend using FTP).

Step 3   After midnight, either delete the file, or, move it (using the Rename Files command) to the root directory (C:/) or to another C-drive directory.

---

Note   The system does not permit you to delete a log file that is open (i.e., named as today's date). The message "Error during Disk Operation" is returned. The system does allow you to move (rename) a log file that is open but the system will still not create a new log file until midnight (see CSCdm85746 on CCO). See the "VCO allows you to move to the root directory a log file with today's date, but will not automatically create a new log file until midnight" section. The system will automatically clear ALM097 when the following conditions are met: The system needs to write 1000 date/time stamps to the subsequent log file(s) before it samples the size of the new open log file. Approximately 107 kBytes of logging (29 printed pages of log file messages) will generate 1000-plus writes. When the Generic detects that 1000 writes have been made, it will then sample the log file and see less than 1 MB and clear ALM097.

---

---

Note   If the system then runs without error after the alarm incident, it will take the system approximately 50 days before generating 1000-plus log file writes (necessary to automatically clear ALM097).

---

You can also clear ALM097 within 2 hours.

You can clear ALM097 in approximately 1 hour and 15 minutes by running Test Port Card as specified below. This procedure will manually introduce log file messages. (we recommend performing this procedure during a slow traffic period.)

You can clear ALM097 in approximately 1 hour and 45 minutes by running Test Service Circuits as specified below. This procedure will manually introduce log file messages. (we recommend performing this procedure during a slow traffic period.)

The Test Port Card procedure setup:

T E S T    P O R T    C A R D
Port Card  R,L,S  1 1 15-1-2    Start Channel 1_  End Channel 24
MRC/DRC    R,L,S  1 1 11-1-3    Start Port 1_     End Port 32
Port Card Type    ____          Receiver Type    ____
(*)Print Error Messages Only (Y/N) N
(*)Continually Loop Through Test (Y/N) Y

---

Note   The network span and the service circuit engine that you select for the Test Port Card procedure must first be placed in Diagnostic Mode using the Card Maintenance screen. Spans in the diagnostic state cannot be used for call processing. Note also the settings indicated by (*).

---

The Test Test Service Circuits procedure setup:

S E R V I C E    C I R C U I T    T E S T    U T I L I T Y
Receiver R,L,S   1 1 11-1-3__      Card Type DTMF
Monitor R,L,S,P  _ _ ______ ___
Start Test at Port 1             End Test at Port 32 
(*)Print Error Messages Only (Y/N) N
(*)Continually Loop Through Test (Y/N) Y

---

Note   Port numbers start at 1. The start port must be specified. If the end port is not specified, or if the end port is less than or equal to the start port, only the start port will be tested. The service circuit engine that you select for the Service Circuit Test Utility procedure must first be placed in Diagnostic Mode from the Card Maintenance screen. Spans in the diagnostic state cannot be used for call processing. Note also the settings indicated by (*).

---

Verify that ALM097 clears in your Host, in the Log File, and in the System Alarms Display/Major screen.

FRM340: CODE Error - ethernet_rx.c,393: SOCK == EOF

For details, see issue CSCdk80666 on CCO.

If you see "FRM 340 CODE Error - ethernet_rx.c,393: SOCK == EOF" in the log file, along with "HST013: Host <host-link-name-Z> ... $A30:Addr already in use", then the Ethernet socket connection between the Host and the VCO, called <host-link-name-Z>, has become permanently unusable.

The only way to make this socket reusable is to reboot that side of the VCO.

Note   This problem is intermittent. Your system may never experience this problem.

The best defensive strategy is to define, in the VCO Host Configuration screen, all eight sockets. If you use only two socket connections between the host and a given side of the VCO, still define all eight sockets.

When a socket becomes permanently unusable, reassign the host connection to one of the other sockets currently not in use.

When all sockets become permanently unusable on one side of a VCO, that side of the VCO must be rebooted.

Problems with the SS7 Subsystem

Boot

Stack Start Error Messages

Scrolling "srv_connect" Message Upon SS7 Startup

Sometimes if the standby side of SS7 system is started without first starting SS7 on the active side, or if the standby side of SS7 is started first and then SS7 on the active side is started, EBS "tli" processes do not sync up. Redundancy does not work and following message is displayed.

Stopping and starting the SS7 on either side does not help.

The following procedure (run on both the A- and B-sides) will fix the problem and enable redundancy between SS7 sides:

Step 1   Enter the following on the A-side:

This returns information on the tli process.

Step 2   Enter the following on both sides:

This kills the tli process.

Step 3   Enter the following on the A-side:

This restarts tli on the A-side.

Step 4   Repeat on the B-side.

ECPT Modules are Not Seen Being Loaded During Execution of the ebs_start or start-ss7.sh Script

See the "Reinstalling EBS Drivers" section, Case 2.

Solaris sys-config returns error message if "none" is selected for "Name Service"

During the sys-config, when the Name Service screen is reached, there are four options to select from. If the customer is not using any Name Service, then the obvious selection is None. But when that is selected and F2 is pressed, the following message is displayed:

Resolution:

When you run sys-config, the system expects a name service to be defined. When the Name Service screen option is set to NONE, the system still tries to look for a service type and becomes stuck in a loop. If the system is in this situation, do the following:

Step 1   The terminal you are using determines how you get to the system prompt.

• If you are using a UNIX/Solaris terminal, press the Stop key followed by the 'a' key.
  
  

This places you at an OK prompt.

• If you are using a WYSE terminal / PC, press the Cntrl (control) key followed by the Pause/Break key.
  
  

(Sometimes the Alt key followed by Pause/Break works.)

This will place you at an OK prompt.

Step 2   From the OK prompt, type

This brings up the system in single-user mode.

Step 3   The system will prompt with a message:

Step 4   Enter the standard root password followed by the Return/Enter key

Step 5   Once logged in as root, change directory to /etc and create the file nsswitch.conf and nsswitch.files as dummy files.

Basically you are creating files that the system was missing.

Files could easily be created using the vi editor:

vi /etc/nsswitch.conf (this creates the file when exit from edit mode by option wq)

vi /etc/nsswitch.files (this creates the file when exit from edit mode by option wq)

Step 6   Verify if the files now exist under the directory /etc.

Step 7   Change directory to /usr/sbin.

Step 8   Run the command sys-unconfig to revert to an unconfigured system.

Step 9   Type reboot to bring the system back in configuration mode (sys-config).

Step 10   Continue with the regular sys-config procedure. This time when Name Service option is set to NONE, the system will allow you to configure the rest to the fields without any problem.

Installation/Upgrade

Stack Start Error Messages

ECPT Modules are Not Seen Being Loaded During EBS Software Install

See the "Reinstalling EBS Drivers" section, Case 1.

Solaris sys-config returns error message if "none" is selected for "Name Service"

See the "Solaris sys-config returns error message if "none" is selected for "Name Service"" section.

Protocol Errors/Violations

ISUP

COT, Report Interpretation "Continuity Failures (COT)"

To resolve SS7 COT (Continuity Test) failures, do the following:

Step 1   Find out what flavor of SS7 the customer is running (ITU or ANSI) and the version.

Step 2   Determine which of the four types (Incoming on Present Circuit, Incoming on Previous Circuit, Outgoing on Present Circuit, or Outgoing on Previous Circuit) of COT Failure they are getting.

Step 3   Obtain a copy of the CKTINT Log with Debug Flags 2, 5, 9, 11, 12, 21, 22 and 28 enabled which shows the COT Failures and a copy of the VCO Host and NBC Trace (if possible).

Step 4   Using the model identified in Step 2, analyze the CKTINT log and the VCO Host/NBC Trace to determine cause of the COT Failures. Some things to look for include missing messages and rejected commands.

MTP

LUA/LUN, SS7 Subsystem Fails to Send LUA Message in Response to the Network's LUN Message During Test "Q.782,7.9)

If your SS7 Subsystem is configured with only one link in the LinkSet (ADD-LSET:,...ACTIVE=1, LOADED=1) then you will encounter this protocol problem.

You can resolve the problem by adding alternate routes as in lines 6 and 7 in the EBS mtp.mml file. The example .mml file is given as follows: