Guest

Cisco Catalyst 5500 Series Switches

Troubleshooting EMANATE Subagent Error Message on WS-X6101-OC12 ATM Module

Document ID: 43572

Updated: Dec 14, 2007

   Print

Introduction

This document provides information on how to troubleshoot the WS-X6101-OC12 ATM module when Enhanced MANagement Agent Through Extensions (EMANATE) Subagent error messages are seen from its log buffer. It does not provide information on how to troubleshoot ATM modules supported technologies such as LANE and RFC1483.

Before You Begin

Conventions

For more information on document conventions, see the Cisco Technical Tips Conventions.

Prerequisites

There are no specific prerequisites for this document.

Components Used

This document was developed and tested using the software and hardware versions below.

  • Catalyst 6500 with Supervisor II with Multilayer Switch Feature Card 2 (MSFC2)

  • WS-X6101-OC12-MMF module

  • CatOS 6.2(6) on the Supervisor Engine II

  • Cisco IOS® 12.1(13)E on the ATM module

The information presented in this document was created from devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If you are working in a live network, ensure that you understand the potential impact of any command before using it.

Before You Begin

What is EMANATE?

EMANATE is a run-time extensible agent system framework developed by SNMP Research that consists of a Master agent with zero to several Subagents. These components allow for SNMP management of systems and modular devices, and they address the issue of handling multiple SNMP agents on a single platform such as the Catalyst 5000 and Catalyst 6000 family of switches.

The Master agent, which is MIB independent but protocol dependent, is what would traditionally be thought of as the SNMP agent on the device. The Master agent contains the agent protocol engine (SNMPv1 and/or SNMPv2), and is in charge of the authentication, authorization, access control and privacy mechanism. The Master agent processes all SNMP requests by a network management station and also handles the Subagent management.

A Subagent is the independent SNMP agent that resides on the Catalyst 5000 and Catalyst 6000 modules such as the WS-X6101 ATM module. Subagents are independent of the Master agent wherein it can be stop or start without affecting the Master agent. The typical Subagent simply runs in a loop and awaits requests from the Master agent whereupon it gathers the information requested and returns the response.

To learn in detail about EMANATE, read the following: SNMP Research leavingcisco.com.

WS-X6101 Module Architecture

The ATM module runs its own Cisco IOS and its entirely independent from the CatOS supervisor engines in terms of switching packets. To access the module, you need to administer the session <slot#> command from the supervisor engine.

Note: The WS-X6101 ATM module is not supported on Native IOS configuration.

The module comes with dual-PHY OC12 ports, but only one can be used at a time. Unlike the OC3 LANE module from the Catalyst 5000, the standby port will be in shutdown state. That means the peer device (atm switch) where the standby port is connected will not see any link signal from the WS-X6101 module.

To forward data traffic from the external OC12 port, each PHY port has a corresponding 622Mbps connection to the chassis's bus. These are configured as trunk ports and cannot be changed.

The following is a capture from the supervisor engine where the WS-6101 module is on slot 2:

Catalyst 6500
6500-39 (enable) show port 2/1
Port  Name                 Status     Vlan       Duplex Speed Type
----- -------------------- ---------- ---------- ------ ----- ------------
 2/1                       connected  trunk        full   622 OC12 MMF ATM
 
Port   ifIndex
-----  -------
 2/1   1627
 
Use 'session' command to see ATM counters.
 
6500-39 (enable) show port 2/2
Port  Name                 Status     Vlan       Duplex Speed Type
----- -------------------- ---------- ---------- ------ ----- ------------
 2/2                       connected  trunk        full   622 OC12 MMF ATM
 
Port   ifIndex
-----  -------
 2/2   1628
 
Use 'session' command to see ATM counters.
 

The session command from the supervisor engine is made possible via a separate connection of the module to the backplane. This connection is a Fast Ethernet port that can be seen from the show version command. Here’s an example capture:

WS-X6101-OC12
ATM# show version
Cisco Internetwork Operating System Software
IOS (tm) C6ATM Software (C6ATM-LC-M), Version 12.1(13)E, EARLY DEPLOYMENT RELEASE 
SOFTWARE (fc1)
TAC Support: http://www.cisco.com/tac
Copyright (c) 1986-2002 by cisco Systems, Inc.
Compiled Wed 04-Sep-02 15:16 by eaarmas
Image text-base: 0x60010BD8, data-base: 0x603CC000
 
ROM: System Bootstrap, Version 12.0(5r)XS, RELEASE SOFTWARE (fc1)
BOOTLDR: C6ATM Software (C6ATM-LC-M), Version 12.1(13)E, EARLY DEPLOYMENT RELEASE 
SOFTWARE (fc1)
 
ATM uptime is 39 minutes
System returned to ROM by power-on
System restarted at 14:55:38 UTC Sat Dec 14 2002
System image file is "bootflash:c6atm-lc-mz.121-13.E"
 
cisco WS-X6101-OC12 (R5K) processor with 32768K/8192K bytes of memory.
RM5230 processor, Implementation 40, Revision 1.0
Last reset from backplane
Authorized for ATM software set. (0x0)
1 FastEthernet/IEEE 802.3 interface(s)            
1 ATM network interface(s)
506K bytes of non-volatile configuration memory.
16384K bytes of processor board Boot flash (Read/Write)
 
Configuration register is 0x101

Note: On Catalyst 5000 ATM modules, this will display as "1 Ethernet/IEEE 802.3 interface(s)" from show version and "Ethernet0" from show interface.

On lower WS-X6101 IOS versions, this interface is displayed as EOBC0 (Ethernet Out of Band Channel) interface from the show interface command. Higher Cisco IOS versions, such as 12.1.13E, no longer display this EOBC0 from the show interface command. A show eobc command is available to display its status. This command is only introduced on Catalyst 6000. The following is a capture of it:

On WS-X6101-OC12 Only
ATM# show eobc
EOBC0 is up, line protocol is up
  Hardware is AmdFE, address is 0000.0400.0000 (bia 0000.0400.0000)
  Internet address is 127.0.0.40/8
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive not set
  Unknown duplex, Unknown Speed, 100BaseTX/FX
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 1/75/0/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue :0/40 (size/max)
  5 minute input rate 0 bits/sec, 0 packets/sec
  5 minute output rate 1000 bits/sec, 2 packets/sec
     184079 packets input, 9771683 bytes
     Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
     0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
     0 watchdog
     0 input packets with dribble condition detected
     183803 packets output, 11344040 bytes, 0 underruns
     0 output errors, 0 collisions, 1 interface resets
     0 babbles, 0 late collision, 10 deferred
     0 lost carrier, 0 no carrier
     0 output buffer failures, 0 output buffers swapped out
Interface EOBC0
Hardware is AMD Laguna
ADDR: 607BF700, FASTSEND: 0, MCI_INDEX: 0
DIST ROUTE ENABLED: 0
  
[snip] ....
 

Aside from the session command, this port is also used on all management related functions to and from the Supervisor Engine. This includes, status monitoring by the supervisor engine, SNMP messages relayed by the module for the supervisor engine to process, and TFTP image and configuration transfer to and from the module.

Troubleshooting the EMANATE Subagent Error Message

Whenever an SNMP client accesses the Catalyst 6500 via the atmForum (enterprises.353) object ID (OID), the supervisor engine uses EMANATE to access the ATM module. Supervisor engine is the EMANATE master while the ATM module is the subagent. The ATM module in return responds to the request and reports back to the master. The master then responds to the SNMP client.

Unfortunately, sometimes the communication is broken between the supervisor engine and the ATM module. The ATM module usually logs a "%EM_SUBAGENT:SendMasterEvent:Failure in AgentSocketWriteBlocking" error message. This message indicates a failure by the subagent to update the EMANATE master.

The supervisor engine may report an error message like the following:

 2001 Feb 16 13:55:31 %SNMP-6-SUBAGENTCONN:Subagent 0 connected
 2001 Feb 16 14:03:21 %SNMP-6-SUBAGENTDIS:Disconnect Subagent 0, socket for reading is not available
 2001 Feb 16 14:03:21 %SNMP-6-SUBAGENTCONN:Subagent 0 disconnected
 2001 Feb 16 14:03:26 %SNMP-6-SUBAGENTCONN:Subagent 0 connected

These messages are usually communication issues between the supervisor engine and the line card such as the WS-X6101.

Step-by-Step Instructions

Ensure first that the module is communicating properly with the supervisor engine by performing the following steps:

  1. Verify if there are any errors (like CRC) reported under the show eobc command. Its possible that the module is not properly seated to the chassis. Reseat the module if you detect any errors. If any errors are detected on multiple WS-X6101 modules, reseat the supervisor engine. If a problem persists, move the module to different slots and verify that the errors disappear.

  2. Test the continuity of the EOBC by running System Control Protocol (SCP) ping tests from the NMP to the module using test scp <slot#>. Run it several times and verify whether it passes the tests. The following is a sample capture:

    6500 (enable) test scp 2
    Pinging Module 2, Length 1464, Count 3
    0: PASSED
    1: PASSED
    2: PASSED
     
    6500 (enable) test scp 2
    Pinging Module 2, Length 1464, Count 3
    0: PASSED
    1: PASSED
    2: PASSED
    
  3. Check the SCP statistics of the module using the show scp module command. Verify that the total number of SCP messages sent and received are the same. Here is a sample capture:

    6500-39 (enable) sho scp module
    Module 1 SCP status:
           Total SCP Msgs Sent:                  178812
           SCP Retries:                            0
           Total SCP Msgs Rcvd:                  178812
           Total Missing Status Polls:             0
           Consecutive Missing Status Polls:       0
     
    Module 2 SCP status:
           Total SCP Msgs Sent:                  316885
           SCP Retries:                            0
           Total SCP Msgs Rcvd:                  316885
           Total Missing Status Polls:             0
           Consecutive Missing Status Polls:       0
    
  4. If the SCP ping fails or there are missed polls in step 3, ensure that the module is properly seated to the backplane. Reinsert the module first. If the problem persists, move the module to a different slot. If the module works in a different slot, your are likely to have a bad slot on the chassis. If the module does not work in different slot, replace it with a spare. If you do not have a spare, open a TAC case for further troubleshooting.

Once done with the above tests and problem still persists, checkout the following EMANATE-related issues:

Bug ID Title How to Identify
CSCdt71307 CSCdt47870 SNMP getmany on atmForum fails on Catalyst 6500. Other than being unable to do SNMP getmany on atmForum, the show log from the atm module displays: %EM_SUBAGENT:SendMasterEvent :Failure in AgentSocketWriteBlocking
CSCdk36902 ATM subagent disconnects in presence of 13% traffic; does not recover The log on the supervisor reports the following message: SNMP-6:Disconnect Subagent 0, socket for reading is not available SNMP-6:Subagent 0 disconnected
CSCdt02646 CSCdt63056 CSCdr21362 WS-X6101 module is being reset constantly by supervisor engine. Approximate after nine to ten minutes. Module is rebooting and no SCP ping response. Supervisor engine displays the following messages: %SYS-5-MOD_NOSCPPINGRESPONSE:Module 5 not responding... resetting module %SYS-5-MOD_NOSCPPINGRESPONSE:Module 5 not responding... resetting module %SYS-5-MOD_NOSCPPINGRESPONSE:Module 5 not responding... resetting module

Note: Module 5 indicates ATM module is on slot 5.

The problem is triggered by configured extended range vlan (e.g. vlan1006) on the supervisor engine.
CSCdt86536 Memory leak on emanate subagent

Note: This does not display a %EM_SUBAGENT message.

If an SNMP client is performing constant getmany on atmForum (enterprises.353), the memory held by EMANATE Subagent is increasing. This can be seen via "show proc memory". Following is an example: ATM#sh proc m | incl EMANATE 47 0 758747008 758594180 160032 0 0 EMANATE SUBAGENT ATM#sh proc m | incl EMANATE 47 0 1234579984 1234329500 257688 0 0 EMANATE SUBAGENT ATM#sh proc m | incl EMANATE 47 0 1852131416 1851754728 383892 0 0 EMANATE SUBAGENT ATM#sh proc m | incl EMANATE 47 0 2254431960 2253972916 466248 0 0 EMANATE SUBAGENT

Note: Future bugs may exist and may not be covered in this document. Open a TAC case for further troubleshooting.

Related Information

Updated: Dec 14, 2007
Document ID: 43572