Table Of Contents
Interpreting sys.log Log Messages
Frequently Queried Log Messages
Log Messages
This appendix contains the following information concerning log messages for the CSS 11050, CSS 11150, and CSS 11800.
•
Interpreting sys.log Log Messages
•
For more detailed information on configuring logging on the CSS, refer to the Content Services Switch Basic Configuration Guide.
Logging Messages on the CSS
The CSS generates log messages to assist in debugging and monitoring it. By default, the CSS saves boot and subsystem event log messages to log files on its hard or flash disk. The content of these files is recorded in ASCII text. You can also configure the CSS to send log messages to an active CSS session, email address, or another host system, as described in the Content Services Switch Basic Configuration Guide.
The boot log messages are the result of the boot process. The CSS saves these messages in the boot.log file.
The subsystem log messages are subsystem events that occur during the operation of the CSS. The CSS saves theses messages in the sys.log file. The CSS creates this file when the first loggable subsystem event occurs. The CSS determines which subsystem messages to log by its configured logging level. By default, the CSS logs events on all subsystems with a level of warning. The warning level designates that the CSS logs fatal, alert, critical, error, and warning messages for the subsystem. To display the current levels for all subsystems, use the show log-state command. For example:
(config)# show log-stateIf you want to log subsystem messages at a different level than the default warning level, use the logging subsystem command. The level you specify instructs the CSS to log subsystem activity that occurs at that level and the activity greater than that level. For example, if you wish to log information messages for the proximity subsystem, enter:
(config)# logging subsystem proximity level info-6In addition to informational messages, the CSS also logs notice, warning, error, critical, alert, and fatal messages.
To view a list of subsystems, enter:
(config)# logging subsystem ?To view a list of logging levels, enter:
(config)# logging subsystem subsystem_name level ?Displaying Log Messages
Whether you want to send log message to your current console session as they occur, or display the messages in a log file saved on the disk, you need to use the show log command.
Note
•
# show logDisplaying Log events.Press any key to abort...APR 14 16:28:09 5/1 2398 NETMAN-7: HTTPC:HTTPC_Open:ERROR->connect <-1,0> <192.20.1.7> <80>APR 14 16:28:15 5/1 2399 NETMAN-7: HTTPC:HTTPC_Open:ERROR->connect <-1,0> <192.20.1.7> <80>APR 14 16:28:21 5/1 2400 NETMAN-7: HTTPC:HTTPC_Open:ERROR->connect <-1,0> <192.20.1.7> <80>APR 14 16:28:27 5/1 2401 NETMAN-7: HTTPC:HTTPC_Open:ERROR->connect <-1,0> <192.20.1.7> <80>To exit out of logging messages during the session, press any key.
•
For example, to view the content of the sys.log file, enter:
(config)# show log sys.logThe CSS displays the log file starting from the beginning of the file. The top of the file list the older messages and the bottom list the most recent messages. To view the bottom and most recent portion of the file, use the tail option with the show log command. This option allows you to specify the number of lines to display, starting from the end of the log file. You can view a maximum of 1000 lines. For example, to view the most recent 500 lines in the sys.log file, enter:
(config)# show log sys.log tail 500You can also display the contents of the SNMP trap log file by using the show log traplog command. The trap log file is an ASCII file in the log directory containing generic and enterprise traps. By default, the following events generate level critical-2 messages:
•
•
•
•
•
•
All other SNMP traps generate level notice-5 messages.
Note
Interpreting sys.log Log Messages
The following is an example of a sys.log message. This section uses this example to describe the parts of a log message.
FEB 16 14:01:13 5/1 2453 VLANMGR-7: Transmit sfm STP BPDU on bPort 1, egressLp 0x1f00 VlanLpSend() ret:0A log message consists of the following components:
•
•
–
–
1/1 is slot 1 port 1.
13/5 is slot 13, port 5.
In this example, because this is a CSS 11800, the CSS displays the slot and port (5/1), slot 5 port 1.
•
•
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
The possible log levels are:
–
–
–
–
–
–
–
–
•
Transmit sfm STP BPDU on bPort 1, egressLp 0x1f00 VlanLpSend() ret:0
Note
Frequently Queried Log Messages
Table D-1 lists CSS log messages that users frequently query. This table includes information on the possible cause and corrective action, if required. The table lists messages alphabetically. The listed subsystem log messages are listed starting from the subsystem field.
Table D-1 Log Messages
The disk device driver timed out waiting for a response from the disk drive during its initialization. The message typically means that the IDM module is absent from the CSS chassis, but could mean that the disk itself is unresponsive.
Make sure that the IDM is located in the chassis. If it is, try removing and reinstalling it, and then reboot the CSS.
The power supply lost AC power from the source. If the following message appears before these messages, then you can assume that the problem is with the AC power source, not the power supply.
CHM-1: CHMGR: Cannot locate power supply: PS2.To determine if both power supplies are working, both green LEDs should be lit on the front of the active SCM.
The CSS chassis is not able to find the power supply. If you know that the power source is supplied to the chassis and correctly flowing to it, then the problem may be the power supply.
To determine if both power supplies are working, both green LEDs should be lit on the front of the active SCM.
Cause codes indicate the status of interfaces within VLANs. The logical port cause and command codes are as follows:
Cause Code
CM_CIRCUIT_CREATED 1
CM_IP_REGISTER 2
CM_IP_NOT_REGISTER 3
CM_IP_MODIFIED 4
CM_LP_STATE_CHG 5
CM_CIRCUIT_REMOVED 6
CM_LP_ADDED 7
CM_LP_REMOVED 8
CM_LP_MODIFIED 9
CM_LP_FAILOVER 10
CM_CIRCUIT_DOWN 11This log message indicates that a port is added to the VLAN. This can occur when the association to a VLAN changes as the port goes up and down.
In CSS software version 4.x or higher, use the show circuit command to list the VLANs. Check the status of the ports of the VLAN or determine whether or not the VLAN is active.
The number of domain names being sent by the peer exceeds the size of the CSS list. You can configure this parameter through the (config) dns-peer command. It is also recommended that you configure the receive and send slots with the same value. The default slot value is 255.
For a Layer 5 rule, the CSS is trying to establish a connection with the backend server. The CSS sent four SYNs to the backend server and did not get a response.
For the CSS to attempt to establish a connection with the backend server then on the front side, the CSS saw:
SYN->
<-SYN/ACK
ACK->
GET->Only after receipt of the GET, the CSS opens up the backend connection and the log message is generated.
When several of these messages occur, this may indicate a bad server. This could be from keepalives, or from regular TCP HTTP traffic. Make sure that the port 80 sockets are not full on the servers.
One of the ports received an exceeding number of invalid packets and overflowed the fastpath. The flow manager received a control message from the fastpath that was badly formatted. Either it is intermittent hardware and the fastpath is corrupting the packets, or the fastpath is getting streams of invalid packets and some are leaking up to the flow manager.
Use the show ether-errors command to display information for a port with many errors. Try disconnecting the port or changing ports and determine if the errors stop. Note that all code numbers have the same meaning.
Accessing one of the MIBII tables in rfc1213.mib causes this message.
This is an informational message. It serves no purpose and has been taken out of the log in software version 4.01 build 2 and higher.
A removed circuit is still configured to an ACL or another configuration parameter. Check the CSS configuration.
An mbuf is a data structure in BSD UNIX-based IP stacks (such as the VxWorks stack) that is used for buffering. This indicates that the CSS received a packet that was addressed to one of the CSSs IP addresses, and when attempting to send it up the VxWorks IP stack, it had no buffers left.
Note
If you receive this message, contact Cisco customer support.
An ARP TX task is running on the SFM, receiving packets from the SFM, and transmitting them to the proper egress ports. This message includes the circuit number (2001 in this example). If the circuit number for this ARP was down or inactive while the ARP was still being queued in the SFM, the message would appear in the log. An action caused the circuit to be removed while data for that circuit was still in the buffer.
Determine if either all physical interfaces in a circuit VLAN are going up and down, or a configuration change occurred on the VLAN at the time of the message.
The IPV4 receive task received a packet and the IP version is displayed in the parenthesis (). The CSS discards anything that is not Ipv4 Version 4. In this example, it was version 0. If you see many of these messages, it could be a misconfigured device or a DOS attack.
This message occurs when the CSS is configured with HTTP keepalives (HEAD or GET) and the servers are transtioning between states. The service event (kalSvcEvent) values are:
•
•
•
Check the status of the server.
EPIF 0 belongs to the first four ports on a FEM. This is usually a problem with the SFM not getting the code to the FEM or the FEM not reading the SFM properly.
In this case, there is a communications problem between the SFP 9/2 and the FEM.
JAN 5 00:31:43 arrowpoint1.com 9/2 385390 SYSSOFT-3: ONDM: Timeout downloading image to EPIF 0 from the switch.JAN 5 00:31:45 arrowpoint1.com 9/2 385407 SYSSOFT-3: ONDM: Timeout downloading image to EPIF 0 from the switch.The recommended course of action is to reseat the SFM in slot 9, and the FEM in slot 13 controlled by the SFM, and then power cycle the CSS.
An information message when the service has transationed state. Check the status of the server based on the keepalive parameters.
A user on the CLI, either through the console or Telnet, entered a wrong command. A Telnet or console session displays this message stating that the command was incorrect.
The SNMP Enterprise login failure traps are enabled and a user entered a bad username/password pair. For example, the user typed the wrong username and password three times.
The next hop IP address is no longer reachable. When you configure a static route, a internal service is created such that when the service is up, the static route is included in the routing table. If the service goes down, it is removed from the routing table.
Make sure that all of the routes in the routing table are up. The routes may have transitioned states.
The flow manager tries to reallocate a VC that was already established.
These messages occur when the port is coming up. They do not represent a problem. The messages are also most noticeable at the end of boot time if you connected through the console.
A user is trying to use SNMP to poll the CSS, but they have the wrong community string.
An invalid CLI command was issued. In this example, a user issued the dir command in debug mode, and gave an invalid directory. For example:
(debug)# dir d:A valid SNMP agent (community string matched) is trying to set an invalid object. The CSS does not know what it is.
Associated with line data moving through the stack after a line has been disconnected (Telnet application disconnect). It is at DEBUG level for developer information purposes only.
In CSS software version 3.x, these error messages do not indicate a problem. They are logged because a new connection does not match an existing portmapper. To stop the message from appearing in the log, change the portmapper logging option to informational-4 level.
In CSS software version 4.x or higher, these error messages refer to the source group running out of portmappers. Increase the portmappers on the source group so that the message does not appear, and users or services do not see a performance or NATing problem.
This message occurs when a port goes up and down. Check the autonegotiation, which could be a mismatch, bad cable, or bad hardware.
The CSS could not find the image file to load on the disk. There is something wrong with the disk or the file was deleted out of the directory.
Contact Cisco customer support.
The "SYSSOFT-4 IMM Message not Deliverable" message appears in the log when a certain queue on the CSS becomes full, overutilized, or needs to state a problem.
All messages of this type look very similar except for the values in the CE, message type, and message queue id field. The easiest way to decipher the message is to look at the CE value.
To determine which message queue is full, take bits 8-20 from the CE field and match them to the following message.
OCT 31 21:04:41 7/1 1324 SYSSOFT-4: IMM message not deliverable, message queue full, ce = 0x30202600, message type = 0x370000, message queue id = 0x8229ec60In the CE field for this example, the number is 26.
CE=0xXXXX26XXThen look at the following IMM messages for a similar message.
All the other information is CSS-specific. This field is constant throughout all versions of CSS software.
The ARP transmits queue, which is 128 entries, is full. These are ARPs that the CSS is generating at bootup or when you assume mastership; it would be all the gratuitous ARPing for the VIPs. When you see this message, that gratuitous ARP packet is lost. The packet will not be resent and the upstream device is not notified about the MAC change in the case of redundancy.
In CSS software version 3.10, most customers do not have more that 128 VIPs.
In CSS software version 4.x, you can configure VIP ranges. With ranges, you could suddenly have 1000s of VIPs and these messages would be seen all the time. The CSS automatically rate-limits the number of ARPs that can be sent. In CSS software version 4.x, the CSS also rate-limits the number of gratuitous ARP requests so that the ARP transmit queue does not overflow.
Note
This IMM message queue error refers to the VLANMGR_SFM_TID task. This is the queue on the SFP that handles spanning tree events from the SCM. The queue is 128 entries long. Each time this message appears, a spanning-tree protocol (STP) age or state change event is being lost. Three types of packets come to this queue:
•
•
•
In a CSS running normally, these events are rare. We suggest you troubleshoot spanning tree.
Note
A queue is getting overrun. This type of log entry occurs when the CSS is receiving a high volume of extraneous traffic. Because the CSS acts as both a router and a bridge, it must examine each packet. This particular queue handles all non-specific and non-IP traffic, including spanning tree BPDU, non-IP bridgeable traffic, ICMP, ARP, UDP fragments, and packets with expired TTL. This should not have any impact on user TCP traffic, as TCP is sent to a different queue.
Under load, especially due to a high number of UDP fragments or different Denial of Service (DOS) attacks, this queue would fill up and the CSS might drop some packets.
If this error appears in the log occasionally, check the show dos command to make sure the site is not under attack. Also, check the network topology to make sure the routing is solid. If the log is filling rapidly with these errors, then a packet capture may be helpful in isolating the cause.
Contact Cisco customer support.
Note
The ASYNC message task processing cannot deliver a message because the queue is full. At boot up time, you can ignore it.
If these are streaming into the log during normal operations, it would warrant more investigation.
Contact Cisco customer support.
Note
The SFM_FW_MAP_PORT_MSG cannot be delivered because the message queue is full. The message is sent when an FTP/RealAudio Control PORT, 227 Entering Passive, O (backspace) command is received and causes the CSS to allocate a port-mapper port to be used for the data channel that is about to open up.
The port uses this notification message (SFM_FW_MAP_PORT_MSG) to inform all other SFMs to allocate the same port-mapper resource. The port-mapper resource can then be found and used when the data-channel's SYN comes in any SFM.
The possible cause of this error could be a large amount of FTP/RealAudio coming into the CSS. Increase the number of portmaps in the source group.
Note
The SFM_FW_UNMAP_PORT_MSG cannot be delivered because the message queue is full. This message is similar to the previous IMM message except in this case, the CSS is trying to inform all other SFMS to unmap the same port-mapper resource. The possible cause of this error could be a large amount of FTP/RealAudio coming into the CSS.
Contact Cisco customer support.
Note
This is the IPV4 Receive message queue on the SCM. The only traffic that reaches this queue is destined to one of the circuit addresses. This traffic includes keepalives, Telnet, SSH, FTP, CAPP, SNMP, XML and so forth.
This message indicates a large number of services with individual keepalives. At times, they can overwhelm this queue. A reconfiguration using global keepalives is needed.
It also could be that someone is attacking the site. This message is of concern and should be investigated. Contact Cisco customer support.
Note
This IMM message queue error refers to the IPV4_ARP_TID. This is a 256-entry queue on the SCM that handles route changes, and MAC address events (moved, down, up).
They are also due to a large number of VIPs. Missing a few messages from this queue is not fatal. Contact Cisco customer support.
Note
The task ID 0x39 from the CE field is CSD_SCM_TID. The message type of 0x390003 is a SCM_CNT_CREATE. Each time the SFM sees a new request for content, it sends a message to the SCM. This is for hotlist statistics and the show content command. The SFMs overwhelm the SCM with these messages causing the queues to fill up and then this message occurs.
Lower the logging level for syssoft to critical. Contact Cisco customer support.
Note
The task ID 0x2c or 44 is the VLANMGR_SCM_TID. This message queue is on the SCM and receives the control STP BPDUs from the SFPs. The control message occurs when the port changes state appearing as if the port is being overrun with STP BPDUs. This might occur on a CSS 11800 in the core of a network. Because the queue is full, you would lose some BPDUs, which is probably not critical.
This message may occur during a link up/down when the bridging status changed. Contact Cisco customer support.
Note
There is a bad block on the disk. Either reformat or replace the disk. Disk replacement is recommended.
An informational message, which does not require any further action.
An informational message that an ARP came in on a different port.
