This document provides information on how to troubleshoot the WS-X6101-OC12 ATM module when Enhanced MANagement Agent Through Extensions (EMANATE) Subagent error messages are seen from its log buffer. It does not provide information on how to troubleshoot ATM modules supported technologies such as LANE and RFC1483.
For more information on document conventions, see the Cisco Technical Tips Conventions.
There are no specific prerequisites for this document.
This document was developed and tested using the software and hardware versions below.
Catalyst 6500 with Supervisor II with Multilayer Switch Feature Card 2 (MSFC2)
CatOS 6.2(6) on the Supervisor Engine II
Cisco IOS® 12.1(13)E on the ATM module
The information presented in this document was created from devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If you are working in a live network, ensure that you understand the potential impact of any command before using it.
EMANATE is a run-time extensible agent system framework developed by SNMP Research that consists of a Master agent with zero to several Subagents. These components allow for SNMP management of systems and modular devices, and they address the issue of handling multiple SNMP agents on a single platform such as the Catalyst 5000 and Catalyst 6000 family of switches.
The Master agent, which is MIB independent but protocol dependent, is what would traditionally be thought of as the SNMP agent on the device. The Master agent contains the agent protocol engine (SNMPv1 and/or SNMPv2), and is in charge of the authentication, authorization, access control and privacy mechanism. The Master agent processes all SNMP requests by a network management station and also handles the Subagent management.
A Subagent is the independent SNMP agent that resides on the Catalyst 5000 and Catalyst 6000 modules such as the WS-X6101 ATM module. Subagents are independent of the Master agent wherein it can be stop or start without affecting the Master agent. The typical Subagent simply runs in a loop and awaits requests from the Master agent whereupon it gathers the information requested and returns the response.
To learn in detail about EMANATE, read the following: SNMP Research .
The ATM module runs its own Cisco IOS and its entirely independent from the CatOS supervisor engines in terms of switching packets. To access the module, you need to administer the session <slot#> command from the supervisor engine.
Note: The WS-X6101 ATM module is not supported on Native IOS configuration.
The module comes with dual-PHY OC12 ports, but only one can be used at a time. Unlike the OC3 LANE module from the Catalyst 5000, the standby port will be in shutdown state. That means the peer device (atm switch) where the standby port is connected will not see any link signal from the WS-X6101 module.
To forward data traffic from the external OC12 port, each PHY port has a corresponding 622Mbps connection to the chassis's bus. These are configured as trunk ports and cannot be changed.
The following is a capture from the supervisor engine where the WS-6101 module is on slot 2:
6500-39 (enable) show port 2/1 Port Name Status Vlan Duplex Speed Type ----- -------------------- ---------- ---------- ------ ----- ------------ 2/1 connected trunk full 622 OC12 MMF ATM Port ifIndex ----- ------- 2/1 1627 Use 'session' command to see ATM counters. 6500-39 (enable) show port 2/2 Port Name Status Vlan Duplex Speed Type ----- -------------------- ---------- ---------- ------ ----- ------------ 2/2 connected trunk full 622 OC12 MMF ATM Port ifIndex ----- ------- 2/2 1628 Use 'session' command to see ATM counters.
The session command from the supervisor engine is made possible via a separate connection of the module to the backplane. This connection is a Fast Ethernet port that can be seen from the show version command. Here’s an example capture:
ATM# show version Cisco Internetwork Operating System Software IOS (tm) C6ATM Software (C6ATM-LC-M), Version 12.1(13)E, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1) TAC Support: http://www.cisco.com/tac Copyright (c) 1986-2002 by cisco Systems, Inc. Compiled Wed 04-Sep-02 15:16 by eaarmas Image text-base: 0x60010BD8, data-base: 0x603CC000 ROM: System Bootstrap, Version 12.0(5r)XS, RELEASE SOFTWARE (fc1) BOOTLDR: C6ATM Software (C6ATM-LC-M), Version 12.1(13)E, EARLY DEPLOYMENT RELEASE SOFTWARE (fc1) ATM uptime is 39 minutes System returned to ROM by power-on System restarted at 14:55:38 UTC Sat Dec 14 2002 System image file is "bootflash:c6atm-lc-mz.121-13.E" cisco WS-X6101-OC12 (R5K) processor with 32768K/8192K bytes of memory. RM5230 processor, Implementation 40, Revision 1.0 Last reset from backplane Authorized for ATM software set. (0x0) 1 FastEthernet/IEEE 802.3 interface(s) 1 ATM network interface(s) 506K bytes of non-volatile configuration memory. 16384K bytes of processor board Boot flash (Read/Write) Configuration register is 0x101
Note: On Catalyst 5000 ATM modules, this will display as "1 Ethernet/IEEE 802.3 interface(s)" from show version and "Ethernet0" from show interface.
On lower WS-X6101 IOS versions, this interface is displayed as EOBC0 (Ethernet Out of Band Channel) interface from the show interface command. Higher Cisco IOS versions, such as 12.1.13E, no longer display this EOBC0 from the show interface command. A show eobc command is available to display its status. This command is only introduced on Catalyst 6000. The following is a capture of it:
|On WS-X6101-OC12 Only|
ATM# show eobc EOBC0 is up, line protocol is up Hardware is AmdFE, address is 0000.0400.0000 (bia 0000.0400.0000) Internet address is 127.0.0.40/8 MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, loopback not set Keepalive not set Unknown duplex, Unknown Speed, 100BaseTX/FX ARP type: ARPA, ARP Timeout 04:00:00 Last input 00:00:00, output 00:00:00, output hang never Last clearing of "show interface" counters never Input queue: 1/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue :0/40 (size/max) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 1000 bits/sec, 2 packets/sec 184079 packets input, 9771683 bytes Received 0 broadcasts, 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog 0 input packets with dribble condition detected 183803 packets output, 11344040 bytes, 0 underruns 0 output errors, 0 collisions, 1 interface resets 0 babbles, 0 late collision, 10 deferred 0 lost carrier, 0 no carrier 0 output buffer failures, 0 output buffers swapped out Interface EOBC0 Hardware is AMD Laguna ADDR: 607BF700, FASTSEND: 0, MCI_INDEX: 0 DIST ROUTE ENABLED: 0 [snip] ....
Aside from the session command, this port is also used on all management related functions to and from the Supervisor Engine. This includes, status monitoring by the supervisor engine, SNMP messages relayed by the module for the supervisor engine to process, and TFTP image and configuration transfer to and from the module.
Whenever an SNMP client accesses the Catalyst 6500 via the atmForum (enterprises.353) object ID (OID), the supervisor engine uses EMANATE to access the ATM module. Supervisor engine is the EMANATE master while the ATM module is the subagent. The ATM module in return responds to the request and reports back to the master. The master then responds to the SNMP client.
Unfortunately, sometimes the communication is broken between the supervisor engine and the ATM module. The ATM module usually logs a "%EM_SUBAGENT:SendMasterEvent:Failure in AgentSocketWriteBlocking" error message. This message indicates a failure by the subagent to update the EMANATE master.
The supervisor engine may report an error message like the following:
2001 Feb 16 13:55:31 %SNMP-6-SUBAGENTCONN:Subagent 0 connected 2001 Feb 16 14:03:21 %SNMP-6-SUBAGENTDIS:Disconnect Subagent 0, socket for reading is not available 2001 Feb 16 14:03:21 %SNMP-6-SUBAGENTCONN:Subagent 0 disconnected 2001 Feb 16 14:03:26 %SNMP-6-SUBAGENTCONN:Subagent 0 connected
These messages are usually communication issues between the supervisor engine and the line card such as the WS-X6101.
Ensure first that the module is communicating properly with the supervisor engine by performing the following steps:
Verify if there are any errors (like CRC) reported under the show eobc command. Its possible that the module is not properly seated to the chassis. Reseat the module if you detect any errors. If any errors are detected on multiple WS-X6101 modules, reseat the supervisor engine. If a problem persists, move the module to different slots and verify that the errors disappear.
Test the continuity of the EOBC by running System Control Protocol (SCP) ping tests from the NMP to the module using test scp <slot#>. Run it several times and verify whether it passes the tests. The following is a sample capture:
6500 (enable) test scp 2 Pinging Module 2, Length 1464, Count 3 0: PASSED 1: PASSED 2: PASSED 6500 (enable) test scp 2 Pinging Module 2, Length 1464, Count 3 0: PASSED 1: PASSED 2: PASSED
Check the SCP statistics of the module using the show scp module command. Verify that the total number of SCP messages sent and received are the same. Here is a sample capture:
6500-39 (enable) sho scp module Module 1 SCP status: Total SCP Msgs Sent: 178812 SCP Retries: 0 Total SCP Msgs Rcvd: 178812 Total Missing Status Polls: 0 Consecutive Missing Status Polls: 0 Module 2 SCP status: Total SCP Msgs Sent: 316885 SCP Retries: 0 Total SCP Msgs Rcvd: 316885 Total Missing Status Polls: 0 Consecutive Missing Status Polls: 0
If the SCP ping fails or there are missed polls in step 3, ensure that the module is properly seated to the backplane. Reinsert the module first. If the problem persists, move the module to a different slot. If the module works in a different slot, your are likely to have a bad slot on the chassis. If the module does not work in different slot, replace it with a spare. If you do not have a spare, open a TAC case for further troubleshooting.
Once done with the above tests and problem still persists, checkout the following EMANATE-related issues:
|Bug ID||Title||How to Identify|
|CSCdt71307 CSCdt47870||SNMP getmany on atmForum fails on Catalyst 6500.||Other than being unable to do SNMP getmany on atmForum, the show log from the atm module displays: %EM_SUBAGENT:SendMasterEvent :Failure in AgentSocketWriteBlocking|
|CSCdk36902||ATM subagent disconnects in presence of 13% traffic; does not recover||The log on the supervisor reports the following message: SNMP-6:Disconnect Subagent 0, socket for reading is not available SNMP-6:Subagent 0 disconnected|
|CSCdt02646 CSCdt63056 CSCdr21362||WS-X6101 module is being reset constantly by supervisor engine. Approximate after nine to ten minutes.||
Module is rebooting and no SCP ping response. Supervisor engine
displays the following messages:
%SYS-5-MOD_NOSCPPINGRESPONSE:Module 5 not responding...
%SYS-5-MOD_NOSCPPINGRESPONSE:Module 5 not responding...
%SYS-5-MOD_NOSCPPINGRESPONSE:Module 5 not responding...
Note: Module 5 indicates ATM module is on slot 5.The problem is triggered by configured extended range vlan (e.g. vlan1006) on the supervisor engine.
Memory leak on emanate subagent
Note: This does not display a %EM_SUBAGENT message.
|If an SNMP client is performing constant getmany on atmForum (enterprises.353), the memory held by EMANATE Subagent is increasing. This can be seen via "show proc memory". Following is an example: ATM#sh proc m | incl EMANATE 47 0 758747008 758594180 160032 0 0 EMANATE SUBAGENT ATM#sh proc m | incl EMANATE 47 0 1234579984 1234329500 257688 0 0 EMANATE SUBAGENT ATM#sh proc m | incl EMANATE 47 0 1852131416 1851754728 383892 0 0 EMANATE SUBAGENT ATM#sh proc m | incl EMANATE 47 0 2254431960 2253972916 466248 0 0 EMANATE SUBAGENT|
Note: Future bugs may exist and may not be covered in this document. Open a TAC case for further troubleshooting.
The Cisco Support Community is a forum for you to ask and answer questions, share suggestions, and collaborate with your peers.
Refer to Cisco Technical Tips Conventions for information on conventions used in this document.