Diagnostic Procedures
Available Languages
Table Of Contents
Failures Due To Network Problems
Failure of Cisco IOS for S/390 to Respond
Incorrect Output - DNR, EPS, etc.
Creating Formatted Dumps for GateD
Troubleshooting the CDLC driver
Diagnosing Application-Level Problems
Diagnosing Other IUCV Failures
Macro API Diagnostic Facilities
Character Formatted Socket Trace
Diagnostic Procedures
This chapter describes the problem determination and reporting procedures for Cisco IOS for S/390. The chapter organized into the following sections.
Describes problem diagnostic procedures and how to use the diagnostic commands.
Describes the documentation needed when reporting problems to customer support.
Describes how to diagnose problems with GateD.
•
Troubleshooting the CDLC driver
Describes procedures to use with the CDLC driver.
Describes procedures specific to IUCV sockets
Problem Determination
If Cisco IOS for S/390 does not respond properly, diagnostic action is required to restore proper service. This section provides a background in performing diagnostic action and gathering documentation for problem resolution.
Although there can be any number of different kinds of software failures, the majority of them fit into some general categories.
Whenever a problem is suspected, a good first step in problem determination is to check the T01LOG and JES message log (or console log) for any messages that may indicate the nature of the problem. The IFS LOGGING command can be used to enable more diagnostic messages, such as Debugging messages. For more information, read LOGGING in the chapter of this manual.
Failures That Produce Dumps
In the event of an abend, a system dump (SVC dump) will be produced. This is usually accompanied by message T00IF050E in the message log. Save the dump, logs, and the complete Cisco IOS for S/390 job output, and forward to Customer Support.
For certain abends, the dump is suppressed, for instance, if an external application passes an invalid buffer address. In this case, an abend message may appear in the log without an accompanying dump. If this happens frequently, the application should be identified and corrected.
Failures Due To Network Problems
If the problem is reproducible, use the real-time network trace facility, TCPEEP, to diagnose the problem. For intermittent problems, use the TRACE CT command to initiate a non-wrapping trace of the network interface(s) involved. For more information, read TCPEEP and TRACE.
Failure of Cisco IOS for S/390 to Respond
Certain problems may lead to a failure of Cisco IOS for S/390, or a major application, to respond. Try to identify the failing component.
Failure of Major Applications
Determine whether the TCP stack is not responding, or whether the problem involves only an application (such as Telnet or FTP). Try to ping the Cisco IOS for S/390 host from a workstation on the local area network. This will tell whether the network interface is alive and the IP layer is responding. If there is no response to the ping, check the status lights on the network device, and note what they indicate. Using a simple application such as the TSO PING command to send an outbound request will tell whether the transport provider is active. The NETSTAT and SYSTAT commands can be used from TSO or the operator console to query and cancel connections. For more information, read PING and NETSTAT/SYSSTAT.
Loops
Cisco IOS for S/390 has loop detection code in most cases, to cancel a process that is looping, and take a diagnostic dump. If Cisco IOS for S/390 is not responding, and using large amounts of CPU, it may be necessary to cancel the job. First take a dump of the address space using the Cisco IOS for S/390 SVCDUMP command, or, if that fails to respond, use the console DUMP command.
Latch Contention
Cisco IOS for S/390 uses a latch facility, called ILATCH, to serialize use of resources. If a latch or latches are not freed, other processes can be affected. Cisco IOS for S/390 has automatic hung-latch detection code that will normally issue message T00IF038W in this event. Additionally, the ILATCH DISPLAY command can be used to show any latches held over 60 seconds. Use the SVCDUMP command to take a diagnostic dump of Cisco IOS for S/390. Use the ILATCH FREE command to free any held latches, starting at the latch with the highest number, and working down. For more information, read ILATCH.
Storage shortages
If the amount of virtual storage being used by Cisco IOS for S/390 exceeds the threshold defined on the MAXSTGPCT parameter of the IFSPARM statement in the IJTCFGxx configuration member, it will stop accepting new connections. This may be seen by TOPEN error messages in the T01LOG, accompanied by TPL dumps. Check the TPL return code at offset X'10' in the TPL to see whether it indicates a resource shortage condition. (See the Cisco IOS for S/390 Unprefixed Messages and Codes for a description of the TPL return codes). This condition may also be accompanied by message T00IF092W on the console.
Use the IFS POOL command and VSM command to display virtual storage allocation. Use the IFS SVCDUMP command to take a diagnostic dump of the Cisco IOS for S/390 address space. In some cases, it may be possible to use NETSTAT to free up storage by cancelling old or inactive sessions. Read NETSTAT/SYSSTAT for more information.
If the problem is chronic and involves a slow leak of virtual storage, use the SMF statement of the IJTCFGxx configuration member to turn on INTERVAL recording and record subtype 80. Be sure your system SMF parameters and exits (such as IEFU85) allow the recording of the Cisco IOS for S/390 SMF record type. When the SMF data is collected, the SMF Report Writer Program can be used to view storage utilization and growth during the recording period. Read SMF Report Writer Program for more information.
Note
Problem Reporting
When reporting a Cisco IOS for S/390 problem to customer support, document as much information as you can to fully characterize the state of the system at the time of failure and any detectable failure symptoms. The exact documentation collected for a given problem varies, depending on the kind of failure experienced.
In general, the following items should always be provided as the initial documentation of a failure:
•
Note that optional products such as EPS and CPT have their own release levels independent of Cisco IOS for S/390.
•
T01LOG
SYSPRINT
DNRLOG
MAPLOG
SNMLOG
•
•
•
•
It is better to collect an abundance of initial documentation than wait to have Technical Response Center technicians request additional information later when the data may not be available (especially in the case of intermittent failures or hardware problems).
ABENDs
In the event of an ABEND, collect the following information:
•
•
•
•
Incorrect Output - Telnet
For incorrect output through Telnet, collect the following information:
•
•
•
•
Incorrect Output - FTP
For incorrect output through FTP, collect the following information:
•
•
•
•
•
•
Incorrect Output - DNR, EPS, etc.
For incorrect output through DNR, EPS, etc., collect the following information:
•
•
•
•
Hardware
For hardware problems, collect the following information:
•
•
•
•
•
Startup and Parameter Errors
For startup problems and parameter errors, collect the following information:
•
•
•
TCP API Trace Procedures
Use the following steps to gather detailed documentation for API failures or C-sockets interface problems.
Caution
You are advised to gather the information all at one time since separate sessions have time stamps which don't match up between components:
Step 1
MODIFY TRACE ON SIZE(256)
To use a half megabyte of above the line storage for the IFS trace table, set SIZE(128) on the MODIFY command.
Step 2
Step 3
Step 4
If the problem session is of short duration and is readily reproducible, it may be sufficient to use TCPEEP, or to write a wraparound trace to a relatively small data set, using the TRACE command. If the problem takes longer to develop, you will probably need to use the TRACE command, and write the output to a file that is sufficiently large to hold the data. Specify trace group TLI. If the problem involves network traffic, specify group NETIF. If possible, use filters (host names, port numbers, protocols) to limit the amount of data traced. Only trace the data if necessary.
Step 5
Step 6
F runtcp,SVCDUMP,JOB(jobname)
where runtcp is the name of the Cisco IOS for S/390 stack, and jobname is the job name of the API application. This will take a single SVC dump of both address spaces.
Step 7
Step 8
•
•
•
•
•
Troubleshooting GateD
This section provides information on troubleshooting GateD. For more information on GateD commands, refer to the Cisco IOS for S/390 User's Guide.
GateD Logs
There are three logs associated with GateD:
The OPSFmon and ripquery tools are provided with Cisco IOS for S/390 to help with GateD implementation. Read the OSPFMON and RIPQUERY sections in the chapter of this manual for more information.
Controlling GateD
ACTEST can be used to stop GateD. To determine the pta_address, issue the command TASK GATED and note the pta_address that is returned. Then issue the command PPOST pta_address CLOSE. GateD will stop after this command is issued, but will then restart automatically, using the configuration member specified on the IP statement in member TCPCFGxx.
Creating Formatted Dumps for GateD
GateD has a trace dump facility which formats many of the control blocks for use in debugging. You can PPOST the GateD task via ACTEST to signal it to perform special actions. These are:
For more information about RIP and OSPF, refer to RFCs 1058 and 1247, respectively.
Troubleshooting the CDLC driver
Certain types of CDLC controllers, such as the 3745, are governed by NCP generation parameters and may be subject to VTAM commands dealing with PU activation and deactivation. If the CDLC controller has no NCP dependencies, no VTAM considerations are applicable.
Prior to Cisco IOS for S/390 startup, the channel adapter PU dedicated for IP traffic should be in a PCTD2 state. After Cisco IOS for S/390 initialization, the PU should go into an ACTIV state. When Cisco IOS for S/390 is brought down, the following VTAM message appears and the PU returns to the PCTD2 state:
IST259I INOP RECEIVED FOR punameIf, upon starting Cisco IOS for S/390, you encounter this message:
T01LL106E Device dev_name: initialization sequence failed to complete (timeout). Device shutdown initiated.use these commands to reset the channel adapter PU that is dedicated to IP traffic:
v net,inact,id=punamev net,act,id=punameYou should then receive the following message:
T01LL181I Device dev_name: interface if_name is fully operational.Cisco IOS for S/390 need not be shut down to issue the vary commands.
Check the job output from the T01LOG and the MVS syslog output for CDLC driver ABENDs, device I/O errors, device allocation errors, and other such errors.
Troubleshooting IUCV Sockets
The IUCV sockets replace the corresponding IBM TCP/IP IUCV transport address space and TCP/IP facilities and let existing IUCV-based applications execute transparently. Debugging facilities that exist within those interfaces, as well as IUCV socket traces, can be used to diagnose problems that may appear when using this feature.
Potential problems fall into two broad groups:
•
•
Diagnosing Application-Level Problems
Once the IUCV and Cisco IOS for S/390 tasks have started, application programs can connect to the network via TCP/IP. At this point, problems tend to be protocol related and there are two levels to consider:
•
•
The socket libraries (IBM C socket library), and the macro socket and IUCV APIs, depend on the correct encapsulation of a socket-family call inside an IUCV parameter list, as well as a functional IUCV connection (path) between the application, the IUCV task, and the Cisco IOS for S/390 task.
IUCV Transport
The application program using IUCV sockets uses two items of information to build a connection with Cisco IOS for S/390 over IUCV (this is different than a socket connection, described in detail below).
•
•
If your configuration matches these values, then connection should proceed without any error. However, the most commonly seen error code is in response to socket function calls. For example:
RC=1 on IUCV_CONNECT to tcpip, FD = -254, Path = nn, IPRCODE = 11The IPRCODE of 11 indicates that the task name tcpip was not found or did not accept the connection. Verify that the stepname of the Cisco IOS for S/390 task matches the TCPJOBNAME of the socket library (the Macro API determines this name from a zappable constant in the Cisco IOS for S/390 version of EZASOK03).
It is more difficult to diagnose an incorrect subsystem name because many applications expect that the VMCF subsystem is always present and these applications are not prepared to handle error cases.
These return codes are set by the IUCV sockets support routines when the subsystem is not found or not initialized:
•
•
•
Note
Socket Function Calls
Once the IUCV connection has been made and a path exists to Cisco IOS for S/390, the actual socket function can proceed. The number and order of function calls depends on the application, but typically a socket() is allocated. For a server program, this is followed by a bind() and listen(); for a client program, it is followed by connect(). In this example, the function is a socket connect, which takes place over an existing IUCV connection.
Problem determination at this level depends on tracking the function requested by the application, through the socket library, through IUCV, through Cisco IOS for S/390, and out to the network, then following the return status back through the same series of components to the application. The IUCV sockets supply some of the trace tools needed; the remainder exist either in the application, in the socket library, or on the network.
Applications may have some level of socket-call-level tracing that can be enabled via parameters or environment variables.
The IBM C socket library provides the SOCKDEBUG option in hlq.TCPIP.DATA to list the IUCV arguments and reply data to the SYSPRINT DD statement.
The Cisco IOS for S/390 Macro API has a trace function that is enabled by zapping constants in the option module T02U80EZ. It then prints trace records to the designated DD statement. See Macro API Diagnostic Facilities for details of the trace.
Both the IUCV task (RUNIUCV) and the Cisco IOS for S/390 task (RUNTCP) record trace data into an internal trace buffer when the TRACE is turned on and the TEST attribute is set to ON for the following task groups:
•
•
This is accomplished via command statements in the STARTxx script executed at task startup. Read the STARTxx Configuration section in the chapter for detailed syntax of these statements.The samples provided with this feature are pre-configured with usable defaults.
Trace entries are written into the internal circular buffer once tracing has been enabled. They will be formatted to sysout when the following operator command is entered:
•
f runiucv,ijt snap all
•
f runtcp,tcp snap all
Syntax Description
The trace table being snapped exists in different task groups in the IUCV and TCP tasks. In addition to the trace table, selected control blocks will also be formatted.
Trace entries contain details about conditions within the task which include control block addresses, register locations, and internal error indicators. Trace entry format is shown in the "IFS Internal Trace Entries" chapter of "Cisco IOS for S/390 Unprefixed Messages and Codes," however complete interpretation of the trace data is intended to be performed by Customer Support.
Cisco IOS for S/390 can record the network message exchange via the TCPEEP tracing tool. For socket applications, select the TCP level of recording with the DATA option greater than 256.
When possible, collect traces at all points in the application-to-network path from a known, repeatable set of starting conditions. All traces should record the same event, with consistent time stamps. Accurate trace data is key to diagnosing protocol related errors.
Diagnosing Other IUCV Failures
Other than the problems noted above, program checks (ABENDs) can occur or performance issues may be detected during otherwise normal operation.
In the case of ABENDs, it is critical to retain all the memory dump information, along with the sysout logs from both Cisco IOS for S/390 and IUCV tasks. If the tasks remain functional, issue an IJT SNAP ALL or TCP SNAP ALL to the IUCV task and Cisco IOS for S/390 task, respectively. In many cases, an SVC dump will be taken automatically of all three address spaces (application, IUCV, Cisco IOS for S/390) and recorded into a SYS1.DUMPxx data set. This dump is intended to be analyzed via the IPCS tool, so formatted printing is not recommended. If possible, note the application(s) running at the time and any other related messages. If an application-task dump occurs, retain that also.
Performance problems are diagnosed by obtaining IUCV and Cisco IOS for S/390 internal traces, which contain very accurate time stamps, along with a TCPEEP trace to monitor the data flow.
Reporting IUCV Failures
When a failure occurs, collect documentation as appropriate (see the previous sections for details) and contact Customer Support. They may request that you send the items of documentation in machine-readable form (in other words, not on paper). More rapid diagnosis can be made if documentation can be analyzed with online tools.
Be prepared to describe the environment that existed at the time of the failure and if the failure is repeatable (and if so, under what circumstances) or if it is an isolated occurrence.
Macro API Diagnostic Facilities
The configuration and diagnostic capabilities available with the Macro API runtime support module, EZASOK03, are described in this section.
The load module T02U80EZ (alias EZASOK03) must be in an APF authorized library. Several application defaults can be set or overridden by zapping the defaults CSECT IUCVCONS in module T02UIUCV.
Output from the Macro API Trace is written to the DD name chosen (default SYSPRINT) in the client job or address space.
This table describes the values of the CSECT IUCVCONS:
Table 4-1 CSECT IUCVCONS
Note
Examples
Examples of the character formatted socket trace and the in-memory trace table are in the IUCVNOTE member of the SAMP library.
Example
This job is an example of how to zap application defaults/overrides:
//*//* ZAP FOR SETTING EZASOK03 CONFIGURATION OVERRIDES/DEFAULTS//*//STEP EXEC PGM=AMASPZAP//SYSPRINT DD SYSOUT=*//SYSLIB DD DSN=TCPICS.Vxxx.LOAD,DISP=SHR//SYSIN DD *NAME T02UIUCV IUCVCONSREP 0008 E3C3D7C9D7404040 'TCPIP ' - TCPIP ADDRESS SPACE NAMEREP 0010 C1C3E2C8 'ACSH' - DNR SUBSYSTEM IDREP 0014 C9D5D3D2 'INLK' - VMCF SUBSYSTEM IDREP 0018 80 TRACE ONREP 001A 0032 50 INCORE TRACE TABLE ENTRIESREP 001C E2E8E2E3D9C1C3C5 SYSTRACE - DD STATEMENT FOR* CHAR TRACE/*The tracing output that is made available by zapping T02UIUCV comes in two parts: an in-memory trace table that records the parameter lists, and a character formatted socket trace.
USERMOD
++ APAR (MU1IUCV) /* TCP/IP CORRECTIVE MAINTENANCE */++ VER (Z038)FMID(TCP0xxx)PRE(TPXXXXX)/*PROBLEM DESCRIPTION (PSR052177):*===============================================================================** PROBLEM SUMMARY: ** ZAP IUCV CONSTANTS FOR ALL MODULES **-------------------------------------------------------------------------------** RECOMMENDATION: APPLY THE APAR TO CHANGE THE CONSTANTS. **-------------------------------------------------------------------------------** PROBLEM DESCRIPTION: CHANGE IUCV CONSTANTS **===============================================================================** CJB *THE FOLLOWING MODULES/MACROS ARE AFFECTED BY THIS PTF:LIST BEGINT02UIUCVLIST END*/ .++ZAP(T02UIUCV) DISTLIB(ATCPLOAD) .NAME T02UIUCV IUCVCONSREP 0008 E3C3D7E5F4F1F040 TCPV410REP 0010 C1C3E2E9 ACSZREP 0014 C9E4C3E5 IUCVREP 0018 8000 X'80' TRACE FLAG ONREP 001A 0032 50 INCORE TRACE TABLE ENTRIESREP 001C E2E8E2E3D9C1C3C5 SYSTRACE - DD STATEMENT//SMPCNTL DD *SET BDY(GLOBAL) .RECEIVE S(MU1IUCV) .SET BDY(TCPTZN) .APPLY S(MU1IUCV) .//Character Formatted Socket Trace
There are two trace records written to the trace DD statement for each function being traced. The first record, , is written before the PC call, the call to IUCVMULT, or the dirsrv request (get...by... emulation). The second record, , is written after the function has completed.
Table 4-3 Format of the In-Memory Trace Table Records
The in-memory trace table has a default size of 25 entries and will wrap.
Examples of the character formatted socket trace and the in-memory trace table are in the IUCVNOTE member of the SAMP library.
IUCV Debugging Checklist
Use this checklist to help you gather information necessary for reporting failures before you call Customer Support.
•
•
•
•
•
•
FeedbackContact Cisco
- Open a Support Case
- (Requires a Cisco Service Contract)