Guest

Cisco 3900 Series Integrated Services Routers

CFM Service Diagnostics Scripts for Integrated Services Routers White Paper

  • Viewing Options

  • PDF (1.4 MB)
  • Feedback

Objectives

Objective of this white paper is to validate existing CFM (Connectivity Fault Management for Ethernet) Service diagnostics Scripts for Cisco® Integrated Services Routers (ISRs) and provide step-by-step instructions for troubleshooting Ethernet problems. This testing was implemented using the EEM (Embedded Event Manager) framework, which comes with Cisco IOS® Software IPBase images on the Cisco ISRs. This document also contains testbed topology, configuration, snapshots of script output recorded during testing, syslog and email notifications, and logging information.

Introduction and Summary:

Cisco CFM Service Diagnostics offers a collection of powerful onboard diagnostic tools to identify commonly encountered network problems, provide real-time alerts, automatically collect relevant information, and conduct root cause analysis. These Connectivity Fault Management Diagnostics Scripts run over the EEM (Embedded Event Manager) Infrastructure.

EEM brings the management capability inside the Cisco devices. Using EEM you can monitor events and take informational and corrective action when the monitored events occur or when a threshold is reached. The EEM framework comprises Event Detectors and Events Policy. Event Detectors detect events and notify the EEM Server. The EEM policies are configured using the Cisco IOS Software command-line interface (CLI) or TCL script; then recovery is implemented on the basis of the current state of the system and the actions specified in the policy for the given event. EEM provides very sophisticated and distributed control on automated configuration, event detection, remote monitoring, recovery, and device availability. Components of the EEM Architecture are shown in Figure 1.

Figure 1.      EEM Architecture

With Ethernet moving toward the Service Provider Network, the requirement of a connectivity fault management tool becomes a high-demand requirement. High Industry interest in Ethernet connectivity fault management tools forced the IEEE to develop a standard 802.1ag (CFM). This standard specifies protocols, procedures, and managed objects to support transport fault management (detection and isolation of connectivity faults). CFM frames are distinguishable by Ether-Type 89-02 (and MAC Address for multicast messages). CFM employs regular Ethernet frames that travel in-band with the customer traffic. Devices that cannot interpret CFM Messages forward them as normal data frames. Connectivity Fault Management is end-to-end Fault Management, as shown in Figure 2.

Figure 2.      CFM Positioning

This standard divides a network into administrative domains in the form of hierarchy levels. This division helps define the relationships among all entities from a maintenance perspective, to allow each entity to monitor the layers under its responsibility and easily localize problems. Basic components of CFM are shown in Figure 3.

Figure 3.      CFM MEP/MIP (Maintenance Association End Points/Maintenance Association Intermediate Point)

CFM functions include Connectivity Check (for fault detection), Loopback (for fault verification), and Traceroute (for fault isolation).

Service Diagnostics tools are TCL scripts that use the capabilities of the CFM and EEM infrastructure to develop a framework that is ideal for automated fault diagnostics, analysis, and recovery. These diagnostic tools can help you increase network uptime, reduce time to repair, and improve service levels. These policies in the form of TCL scripts isolate the complex troubleshooting and manual fault diagnostics steps. TCL scripts also include an onboard menu system (EMM) to support interactive installation and deployment. Service Diagnostics Scripts cover all functions of CFM (i.e., Connectivity Check, Loopback, and Traceroute) and use EEM capabilities to use events generated by cctimeout to confirm connectivity breaks and verify the status of the network by performing a Loopback check and traceroute and notifying the service manager the status of the network through syslog/email/snmp notifications based on the configuration. The interaction between Service diagnostics with EEM and CFM is shown in Figure 4.

Figure 4.      Service Diagnostics Interaction Feature

CFM and EEM Interworking

Ethernet is rapidly gaining acceptance in carrier networks and replacing traditional technologies. This rapid growth of Ethernet is also supported by IEEE 802.1ag (CFM) standard specifications. Because of its end-to-end reachability and hierarchical structures, CFM is very popular in Carrier Ethernet, and service managers like to use it for fault detection and isolation. However, Configuring CFM domains, MIPs, MEPs, and hierarchical levels, and looking at various databases and error status require in-depth understanding and knowledge of this new technology. Network Managers also need to deploy an OAM management infrastructure to monitor the network status. Most of event monitoring and management is performed by devices external to the network that add extra costs of hardware and networking.

Prior to CFM Diagnostics scripts, the following diagnostics scripts are already deployed and being used by the user community:

   Border Gateway Protocol

   Open Shortest Path First

   Quality-of-Service Scripts

For more details, please visit http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6555/ps9424/white_paper_cisco_ios_service_design_bgp_osp_qos.html.

CFM diagnostic scripts also use EEM infrastructure. EEM Infrastructure is embedded in Cisco IOS Software and widely used in distributed and customized environments for event detection and automated recovery. CFM Fault detection capabilities combined with EEM event detections and policy infrastructure, and Cisco developed an innovative solution of automated service diagnostics scripts that will simplify service managers’ efforts. These automated scripts are deployed in Cisco Integrated Services Routers; they can be executed on demand or scheduled for any specific time. These scripts. when executed, collect all network status information and necessary logs and if necessary bounce the interfaces and send the status through email notification. These scripts also use SNMP notification and syslog messages and notify users accordingly. Figure 5 shows the CFM Service Diagnostics scripts currently employed.

Figure 5.      Service Diagnostics: Carrier Ethernet Scenarios

Currently we have limited scripts ready for use on Cisco ISRs, and they can be found at CBeyond. You can create your own scripts as per your requirement by following EEM user guidelines and using this innovative technology on Cisco routers. Following is a list of scripts available at Cbeyond.

   cfm_cctimeout

   cfm_autotrace

   cfm_ondemand

   cfm_shut_noshut

   cfm_undeploy

   cfm_deploy

   cfmod

   cfm_display

Service Diagnostics scripts (policies) must be deployed and registered before use. You can deploy them manually by copying in the router memory (Please see http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6555/ps9424/whitepaper_c11-566741.html for more details) or you can deploy them using EMM (Embedded Menu Manager) after copying the mdf file in Cisco router memory (please see http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6555/ps9424/whitepaper_c11-566741.html for more details).

Supported Platform

Supported platforms include the Cisco 3945, 3945E, 3925, 3925E, 2951, 2921, 2911, 2901, 1941, 1921, 1905, 880, and 890 Integrated Services Routers and the Cisco 3800, 2800, and 1800 Series Integrated Services Routers.

Tested Cisco IOS Software Images

   Cisco IOS Software Release 15.1(1)T

   EEM Version 3.10

Testbed Topology

Logical Topology

In this logical topology (Figure 6), CPE1 and CPE2 are connected to provider network (PE1 and PE2).

Figure 6.      Logical Topology

Physical Topology

Figure 7 shows the physical topology with the concepts of Maintenance Domain (MD), Maintenance Association (MA), Maintenance Association Endpoint (MEP), and Maintenance Association Intermediate Point (MIP). These components play a vital role in Connectivity Fault Management.

Figure 7.      Physical Topology

MEP defines the boundaries of Maintenance Domain, and it is associated with the association of Maintenance with MEPID. MIPs are configured at intermediate point, and they support the discovery of paths among MEPs and location of faults along those paths. MIPs can add, check, and respond to received CFM PDUs. In the topology shown in Figure 8, MEPs are depicted with an arrow and MIPs are shown by small circles in Figure 7. Please note that Customer Domain is from end to end (CPE1 to CPE2) and configured with the highest level 7, whereas the Provider domain is from PE1 to PE2 configured with level 5. Connectiveity Check Messages are catalogued in Connectivity Check Message Database and errors are catalogued in Error DB. MEP connectivity check database (CCDB) contains only Active entries, and MIP CCDB contains both active and archived entries. Please note that MAs are end to end and defined by a set of Maintenance Endpoints (associated with different MEPIDs at the ends). Maintenance Total of 4 maintenance associations (MAs) are configured in this testbed as shown in Figure 8.

Figure 8.      Maintenance Association Configurations

Configurations

3845 Configuration

2811 Configuration

agam-3845#show run

Building configuration...

Current configuration : 3562 bytes

version 15.1

service timestamps debug datetime msec localtime show-timezone

service timestamps log datetime msec localtime show-timezone

no service password-encryption

hostname agam-3845

boot-start-marker

boot system flash c3845-entservices-mz.151-0.26.T0.5

boot-end-marker

logging buffered 20000

no aaa new-model

ethernet cfm ieee

ethernet cfm global

ethernet cfm traceroute cache

ethernet cfm traceroute cache hold-time 120

ethernet cfm domain CUST1 level 7

 service SID_3 vlan 800 direction down

  continuity-check

 service SID_4 vlan 801 direction down

  continuity-check

 service SID_1 vlan 700 direction down

  continuity-check

 service SID_2 vlan 701 direction down

  continuity-check

!

ethernet cfm logging

ethernet cfm ais link-status global

 disable

clock timezone EST -5

!

ip source-route

ip cef

ip domain name cisco.com

no ipv6 cef

multilink bundle-name authenticated

!

voice-card 0

!

license udi pid CISCO3845-MB sn FOC10181Y2U

interface GigabitEthernet0/0

 ip address 10.0.0.22 255.255.255.0

 duplex auto

 speed auto

 media-type rj45

!

interface GigabitEthernet0/1

 no ip address

 duplex auto

 speed auto

 media-type rj45

 ethernet cfm mep domain CUST1 mpid 7000 vlan 700

 ethernet cfm mep domain CUST1 mpid 720 vlan 800

 ethernet cfm mep domain CUST1 mpid 721 vlan 801

 ethernet cfm mep domain CUST1 mpid 402 vlan 701

!

interface GigabitEthernet0/1.1000

 encapsulation dot1Q 700

!

interface GigabitEthernet0/1.1001

 encapsulation dot1Q 701

!

interface GigabitEthernet0/1.2000

 encapsulation dot1Q 800

!

interface GigabitEthernet0/1.2001

 encapsulation dot1Q 801

interface GigabitEthernet1/0

 no ip address

 shutdown

 negotiation auto

!

ip forward-protocol nd

no ip http server

ip route 0.0.0.0 0.0.0.0 10.0.0.1

snmp-server community public RO

snmp-server enable traps event-manager

snmp-server host 10.10.10.30 version 2c nmscore

snmp-server host 172.16.61.90 public

control-plane

line con 0

 exec-timeout 0 0

 speed 115200

line aux 0

line vty 0 4

 login

 length 0

 transport input all

exception data-corruption buffer truncate

scheduler allocate 20000 1000

end

 

7604-1

agam-7604-1#show run

Building configuration...

 

Current configuration : 6292 bytes

!

! Last configuration change at 23:19:27 UTC Wed Apr 7 2010

version 12.2

service timestamps debug datetime msec

service timestamps log datetime msec

service counters max age 10

!

hostname agam-7604-1

!

boot-start-marker

boot system sup-bootdisk:c7600s72033-adventerprisek9_dbg-mz.122-32.8.13.REC186

boot-end-marker

!

logging buffered 2000000

no logging console

enable password lab

!

no aaa new-model

ethernet cfm ieee

ethernet cfm global

ethernet cfm domain PROV1 level 5

 service SID_P1 evc evc1 vlan 700

  continuity-check

 service SID_P2 evc evc2 vlan 701

  continuity-check

!

ethernet evc evc1

 oam protocol cfm svlan 700 domain PROV1

!

ethernet evc evc2

!

no ip source-route

!

no ip domain lookup

!

vtp mode transparent

no mls flow ip

no mls flow ipv6

mls cef error action reset

multilink bundle-name authenticated

!

spanning-tree mode pvst

spanning-tree extend system-id

diagnostic cns publish cisco.cns.device.diag_results

diagnostic cns subscribe cisco.cns.device.diag_commands

!

redundancy

 main-cpu

  auto-sync running-config

 mode sso

!

vlan internal allocation policy ascending

vlan access-log ratelimit 2000

!

vlan 99,500,700-701,800-801,1000-1001,2000-2001

interface GigabitEthernet2/47

 switchport

 switchport trunk encapsulation dot1q

 switchport trunk allowed vlan 700,701,800,801

 switchport mode trunk

 ethernet cfm mep domain PROV1 mpid 321 vlan 701

 ethernet cfm mep domain PROV1 mpid 320 vlan 700

!

interface GigabitEthernet2/48

 switchport

 switchport trunk encapsulation dot1q

 switchport trunk allowed vlan 700,701,800,801

 switchport mode trunk

 ethernet cfm mip level 7 vlan 700-701,800-801

!

no ip http server

no ip http secure-server

ip route 172.18.0.0 255.255.0.0 172.18.192.1

control-plane

line con 0

 exec-timeout 0 0

line vty 0 4

 exec-timeout 0 0

 no login

 transport input all

exception data-corruption buffer truncate

end   

agam-2821#show run

Building configuration...

Current configuration : 2712 bytes

version 15.1

service timestamps debug datetime msec

service timestamps log datetime msec

no service password-encryption

hostname agam-2821

boot-start-marker

boot system flash:c2800nm-entservices-mz.151-0.26.T0.5

!

boot-end-marker

!

no aaa new-model

ethernet cfm ieee

ethernet cfm global

ethernet cfm traceroute cache

ethernet cfm traceroute cache hold-time 60

ethernet cfm domain CUST1 level 7

 service SID_3 vlan 800 direction down

  continuity-check

 service SID_4 vlan 801 direction down

  continuity-check

 service SID_1 vlan 700 direction down

  continuity-check

 service SID_2 vlan 701 direction down

  continuity-check

!

ethernet cfm logging

ethernet cfm ais link-status global

 disable

clock timezone EST -5

!

ip source-route

ip cef

ip domain name cisco.com

no ipv6 cef

multilink bundle-name authenticated

!

voice-card 0

!

license udi pid CISCO2821 sn FTX1303A08J

interface GigabitEthernet0/0

 ip address 10.0.0.46 255.255.255.0

 duplex auto

 speed auto

!

interface GigabitEthernet0/1

 no ip address

 duplex auto

 speed auto

 ethernet cfm mep domain CUST1 mpid 4201 vlan 801

 ethernet cfm mep domain CUST1 mpid 4200 vlan 800

 ethernet cfm mep domain CUST1 mpid 420 vlan 701

 ethernet cfm mep domain CUST1 mpid 42 vlan 700

!

interface GigabitEthernet0/1.1000

 encapsulation dot1Q 700

!

interface GigabitEthernet0/1.1001

 encapsulation dot1Q 701

!

interface GigabitEthernet0/1.2000

 encapsulation dot1Q 800

!

interface GigabitEthernet0/1.2001

 encapsulation dot1Q 801

!

interface GigabitEthernet0/0/0

 no ip address

 shutdown

 negotiation auto

!

ip forward-protocol nd

 

no ip http server

ip route 0.0.0.0 0.0.0.0 10.0.0.1

control-plane

 

mgcp fax t38 ecm

 

line con 0

 speed 115200

line aux 0

line vty 0 4

 login

 length 0

 transport input all

!

exception data-corruption buffer truncate

scheduler allocate 20000 1000

!

End

 

7604-2

agam-7604-2#            show run

Building configuration...

Current configuration : 6189 bytes

!

! Last configuration change at 23:05:47 UTC Thu Apr 15 2010

version 12.2

service timestamps debug datetime msec

service timestamps log datetime msec

service counters max age 10

!

hostname agam-7604-2

!

boot-start-marker

boot system sup-bootdisk:c7600s72033-adventerprisek9_dbg-mz.122-32.8.13.REC186

boot-end-marker

!

logging buffered 2000000

no logging console

enable password lab

no aaa new-model

ethernet cfm ieee

ethernet cfm global

ethernet cfm domain PROV1 level 5

 service SID_P1 evc evc1 vlan 700

  continuity-check

 service SID_P2 evc evc2 vlan 701

  continuity-check

!

ethernet evc evc1

 oam protocol cfm svlan 700 domain PROV1

!

ethernet evc evc2

 

ip source-route

no ip domain lookup

!

vtp mode transparent

no mls flow ip

no mls flow ipv6

mls cef error action reset

multilink bundle-name authenticated

!

!

spanning-tree mode pvst

spanning-tree extend system-id

diagnostic cns publish cisco.cns.device.diag_results

diagnostic cns subscribe cisco.cns.device.diag_commands

!

redundancy

 main-cpu

  auto-sync running-config

 mode sso

!

vlan internal allocation policy ascending

vlan access-log ratelimit 2000

!

vlan 101-109,500,700-701,800-801,1000-1001,2000-2001

!interface GigabitEthernet2/47

 switchport

 switchport trunk encapsulation dot1q

 switchport trunk allowed vlan 700,701,800,801

 switchport mode trunk

 ethernet cfm mep domain PROV1 mpid 321 vlan 701

 ethernet cfm mep domain PROV1 mpid 320 vlan 700

!

interface GigabitEthernet2/48

 switchport

 switchport trunk encapsulation dot1q

 switchport trunk allowed vlan 700,701,800,801

 switchport mode trunk

 ethernet cfm mip level 7 vlan 700-701,800-801

!

no ip http server

no ip http secure-server

ip route 172.18.0.0 255.255.0.0 172.18.192.1

control-plane

line con 0

 exec-timeout 0 0

line vty 0 4

 exec-timeout 0 0

 no login

 transport input all

exception data-corruption buffer truncate

end

Useful Show Commands

The following show command is used with Shut/No-Shut Script

agam-2821#show ethern cfm mpd domain CUST1 service SID_1

* = Can Ping/Traceroute to MEP

--------------------------------------------------------------------------------

MPID  Domain Name                                 MacAddress          Version  

Lvl   Domain ID                                   Ingress                      

Expd  MA Name                                     Type Id             SrvcInst 

      EVC Name                                                        Age      

--------------------------------------------------------------------------------

4   * CUST1                                       0017.95e4.4c71      IEEE-CFM

7     CUST1                                       Gi0/1.1000   

-     SID_1                                       Vlan 700            N/A      

      N/A                                                             3s

4000* CUST1                                       0017.95e4.4c71      IEEE-CFM

7     CUST1                                       Gi0/1.1000   

EXPD  SID_1                                       Vlan 700            N/A      

      N/A                                                             134s

400 * CUST1                                       0017.95e4.4c71      IEEE-CFM

7     CUST1                                       Gi0/1.1000   

EXPD  SID_1                                       Vlan 700            N/A      

      N/A                                                             46s

 

Total Remote MEPs: 3

 

agam-3845#show event manager policy available

No.  Type    Time Created              Name

1    system  Thu Feb 7  01:28:15 2036  ap_perf_test_base_cpu.tcl

2    user    Fri Apr 5  15:25:20 1940  cfm_autotrace.tcl

3    user    Fri Apr 5  15:25:22 1940  cfm_cctimeout.tcl

4    user    Fri Apr 5  15:25:22 1940  cfm_ondemand.tcl

5    user    Mon Apr 8  14:34:22 1940  cfm_router_shut.tcl

6    user    Fri Apr 5  15:25:24 1940  cfmod.tcl

7    system  Thu Feb 7  01:28:15 2036  cl_show_eem_tech.tcl

8    user    Fri Apr 5  15:25:24 1940  collectEmailParameters.tcl

9    system  Thu Feb 7  01:28:15 2036  no_perf_test_init.tcl

10   user    Fri Apr 5  15:25:18 1940  sdiag_router_cfm.tcl

11   user    Fri Apr 5  15:25:20 1940  sdiag_router_cfm_display.tcl

12   user    Fri Apr 5  15:25:20 1940  sdiag_router_cfm_undeploy.tcl

13   system  Thu Feb 7  01:28:15 2036  sl_intf_down.tcl

14   system  Thu Feb 7  01:28:15 2036  tm_cli_cmd.tcl

15   system  Thu Feb 7  01:28:15 2036  tm_crash_reporter.tcl

16   system  Thu Feb 7  01:28:15 2036  tm_fsys_usage.tcl

agam-3845#show event manager version

Embedded Event Manager Version 3.10

Component Versions:

eem: (v310_throttle)4.1.18

eem-gold: (v310_throttle)1.0.7

eem-call-home: (v310_throttle)1.0.6

Event Detectors:

Name                Version   Node        Type   

application         01.00     node0/0     RP     

syslog              01.00     node0/0     RP     

track               01.00     node0/0     RP     

resource            01.00     node0/0     RP     

routing             02.00     node0/0     RP     

cli                 01.00     node0/0     RP     

counter             01.00     node0/0     RP     

interface           01.00     node0/0     RP     

ioswdsysmon         01.00     node0/0     RP     

none                01.00     node0/0     RP     

oir                 01.00     node0/0     RP     

snmp                01.00     node0/0     RP     

snmp-notification   01.00     node0/0     RP     

timer               01.00     node0/0     RP     

ipsla               01.00     node0/0     RP     

test                01.00     node0/0     RP     

config              01.00     node0/0     RP     

env                 01.00     node0/0     RP     

gold                01.00     node0/0     RP     

nf                  01.00     node0/0     RP     

agam-3845#show event manager detector all

No.  Name                Version   Node        Type   

1    application         01.00     node0/0     RP     

2    syslog              01.00     node0/0     RP     

3    track               01.00     node0/0     RP     

4    resource            01.00     node0/0     RP     

5    routing             02.00     node0/0     RP      

6    cli                 01.00     node0/0     RP     

7    counter             01.00     node0/0     RP     

8    interface           01.00     node0/0     RP     

9    ioswdsysmon         01.00     node0/0     RP     

10   none                01.00     node0/0     RP     

11   oir                 01.00     node0/0     RP     

12   snmp                01.00     node0/0     RP     

13   snmp-notification   01.00     node0/0     RP     

14   timer               01.00     node0/0     RP     

15   ipsla               01.00     node0/0     RP     

16   test                01.00     node0/0     RP     

17   config              01.00     node0/0     RP     

18   env                 01.00     node0/0     RP     

19   gold                01.00     node0/0     RP     

20   nf                  01.00     node0/0     RP     

agam-3845#

Prerequisites of CFM Diagnostics Deployment

Before deployment of Service Diagnostics Scripts, verify that:

   CFM is configured in ieee mode; i.e.,

ethernet cfm ieee

ethernet cfm global

   CFM MEP, MIP, MD, and MA are configured; verify remote connectivity through show ethernet cfm maintenance-association remote command.

   CFM Traceroute cache is enabled for CFM EEM policies (cfm_autotrace.tcl, cfm_ondemand.tcl)

   CFM alarm syslogs are enabled (ethernet cfm logging) for CFM EEM policies (cfm_cctimeout.tcl, cfm_shut.tcl, cfm_unshut.tcl)

   SNMP community/manager are configured

   EEM SNMP traps are enabled

   Domain name (ip domain-name “domainname.com”) for email notification is configured; ensure that the correct email server IP or FQDN is known.

   CFM MA input file for CFM EEM policies (cfm_cctimeout.tcl, cfm_autotrace.tcl, cfm_shut.tcl) exists

CFM Service Diagnostics Deployment Considerations

Some of the known CFM SD deployment considerations follow:

   Execution time of CFM SD EEM policy is directly proportional to the number of remote MEPs and corresponding operations (e.g., ping/traceroute) performed. Ethernet ping / traceroute timeout is 5 seconds (Applicable to cfm_ondemand.tcl, cfm_cctimeout.tcl, cfm_autotrace.tcl).

   In case of concurrent syslogs triggering the same or different policies (i.e., cfm_cctimeout.tcl, cfm_shut.tcl), the execution is always done in sequence.

   Automatic Diagnostics and Shutdown Port policies (cfm_cctimeout.tcl, cfm_shut.tcl) have a common trigger (network failure event [remote mepTimeout]). This situation may cause both policies to trigger at the same time if input MA files have common entries. In this case, the order of policy deployment determines the execution order. Thus, it is recommended to always deploy first the Shutdown Port policy.

   On-demand Diagnostics policy parses out the content of CFM traceroute cache. Therefore, it is recommended to deploy and run the Autotrace policy in order to maintain the traceroute cache populated.

   Syslog and SNMP notifications are faster compared to Email. Email is sent at the end of processing.

Installing CFM SD (Service Diagnostics) Scripts in Cisco Access Router:

1.     Installation Steps Using tclsh CLI parser mode

a)     Download the CFM SD scripts zip file in local TFTP server.

b)     Unzip the CFM SD zip file.

c)     Create policy directory (i.e. svc_diag).

d)     Copy policy scripts in newly created directory (i.e. svc_dir).

e)     Create Library Directory (i.e. user_lib).

f)      Copy library scripts in newly created library directory (i.e. lib_dir).

2.     Installation Steps Using EMM

a)     Download the mdf file (cfm_router.mdf) in local TFTP server.

b)     Copy mdf file to Router disk system (flash).

c)     Execute emm mdf flash:/cfm_router.mdf.

d)     Steps follow:

agam-2821#emm mdf cfm_router.mdf

 

=======================================================

Connectivity Fault Management Diagnostics

 

Enter ? for help or ?# for item help

--------------------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Set Global Variables (email parameters)

3. Deploy CFM Scenarios

4. Display Registered Policies

5. Display Environment Variables

6. Remove Diagnostic Policies

7. Exit

Enter selection [7]: 1

Enter ? for help

Enter a directory to store the CFM diagnostic policies in the form of a URL (excluding filename, e.g.  disk0:/svc_diag

Enter value [flash:/svc_diag]: flash:/svc_diag

Enter ? for help

Enter a directory for the user library files in the form of a URL (excluding filename, e.g.  disk0:/user_lib

Enter value [flash:/user_lib]: flash:/user_lib

Writing files....Done.

Press any key to continue...

 

===========================================================================

Connectivity Fault Management Diagnostics

Enter ? for help or ?# for item help

--------------------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Set Global Variables (email parameters)

3. Deploy CFM Scenarios

4. Display Registered Policies

5. Display Environment Variables

6. Remove Diagnostic Policies

7. Exit

Enter selection [7]: 7

Policy Directory contains following policy scripts:

1. sdiag_router_cfm.tcl

2. sdiag_router_cfm_undeploy.tcl

3. sdiag_router_cfm_display.tcl

4. cfm_autotrace.tcl

5. cfm_router_shut.tcl

6. cfm_ondemand.tcl

7. cfm_cctimeout.tcl

8. cfmod.tcl

9. collectEmailParameters.tcl

Service diagnostics Library contains following Library scripts:

1. cfm_lib.tcl

2. diag_lib.tcl

3. lib.tcl

4. lib_2.tcl

5. userlib.tcl

6. userlib_2.tcl

7. tclIndex

8. email_template_cmd

Carrier Ethernet Scenarios – Diagnostics

ON-DEMAND Diagnostics

Upon customer request to troubleshoot a given service, EEM script automatically verifies and isolates faults to remote MEPs affected by service failure. It could also search traceroute-cache. A diagnostics report is sent to NOC/Service Center (Figure 9).

Figure 9.      On-Demand Diagnostics

EEM event detector: none

Script variables: CFM MD name, shortMA name, remote mep id (opt)

Working steps of ON-DEMAND Diagnostics are given as under. It is assumed that CFM DOMAIN name, MEP-ID and S_VLAN_ID is configured.

ParseCollectData

Verify CFM Domain Information

show ethernet cfm domain zzz

Inspect CFM Connectivity Check Messages Database

Examine MEP Connectivity Check Messages Database for entry logged

show ethernet cfm remote domain zzz

show ethernet cfm maintenace remote detail mpid xxx domain zzz vlan yyy

Inspect CFM ErrorDB

Look for errors logged

show ethernet cfm error domain-id zzz service www

Connectivity / Failure Verification (ping ethernet mac)

ping ethernet H.H.H domain zzz vlan xxx

ping ethernet multicast domain zzz vlan xxx

Path Discovery Isolate Failure (trace ethernet mac)

traceroute ethernet H.H.H  domain zzz vlan xxx

Inspect Traceroute cache

Show ethernet cfm traceroute-cache

Generate Report

Place together the following information in report

   Number of active MEPs in CCDB with port state == UP

   Details of active MEPs in CCDB with port state <> UP

   Details of entries in Error DB

   Number of remote MEPs with verified connectivity

   Details of remote MEPs with connectivity problems

   Traceroute results for MEPs with connectivity problems

   Results of traceroute cache examination

Automatic Diagnostics

Upon a Service failure, EEM script automatically verifies and isolates faults to the remote MEPs affected. A diagnostics report is sent to NOC/Service Center (Figure 10).

Figure 10.    Automatic Diagnostics

 

EEM event detector: CFM MEP Down (timeout) syslog

Script variables: Input file with list of shortMA names

Working steps of automatic diagnostics after even is detected are described as under.

Inspect CFM Connectivity Check Message DB

Examine MEP CCDB for entry logged against it.

 show ethernet cfm maintenace remote detail   mpid iii domain zzz vlan xxx

Inspect CFM ErrorDB

Look for errors logged against it.

 show ethernet cfm error domain-id zzz service www

Connectivity / Failure Verification (ping ethernet mac)

Issue LBM towards it Record results.

 ping ethernet H.H.H domain zzz vlan xxx

Path Discovery Isolate Failure (trace ethernet mac)

Issue LTM towards it Record results.

traceroute ethernet H.H.H   domain zzz vlan xxx

Generate Report

Place together the following information:

   Details of entry in Error DB

   Details of failure verification

   Details of failure isolation

Auto-Traceroute

EEM event detector: timer expiration

Script variables: Input file with list of shortMA names

Timer value

Working steps of automatic diagnostics after even is detected are described as under.

Verify If Cache Is Enabled

Verify if traceroute cache is enabled. Also verify the size and the hold-time of traceroute cache.

show ethernet cfm traceroute-cache

Parse CFM CCM DB

Initialize these variables for the desired MAs:

   cfm Domains

   rmep mpids

   rmep MAC addresses

   vlan number

Verify with following show commands:

show ethernet cfm domain brief

show ethernet cfm domain <domain>

show ethernet cfm maintenance-points remote

show ethernet cfm maintenance-points remote domain <domain>

Path Discovery (trace ethernet mac)

Issue LTM for each of the rmeps discovered.

traceroute ethernet H.H.H domain zzz vlan xxx

Generate Report

Report. Gather the following information:

   Total number of traceroute operations performed

   rmep details

Action Scripts

Shut/No-shut Script

Shut/No-shut script is a combination of shut and no-shut script with DOWN MEPs. In point-to-point environments with NON-ELMI capable CEs, Shut/No-shut script bounces the interface, thus reducing CE blackholing of traffic when EVC or remote UNI failures occur (Figure 11).

Figure 11.    Shut / No-shut operation

 

EEM event detector:

REMOTE_MEP_UP

REMOTE_MEP_DOWN

Script variables: Input file with list of shortMA names

Working steps of Shut/No-shut Diagnostics are given as under.

Assumptions: Single service (no multiplexing), CFM from UNI to UNI

Parse Collect Data:

Initialize CFM Domain name, S-VLAN ID, Mepid and verify with following command

show ethernet cfm maintenance-points remote

Inspect CFM CCM DB:

IF MEP UP IF state syslog

  THEN verify if IF State == DOWN

            Continue next step

Inspect CFM Local DB:

Examine the CFM maintenance point local DB to determine the interface that holds the MEP associated with the service under monitoring.

show ethernet cfm maintenance-points remote   detail mpid <mpid> domain <zzz> vlan <xxx>

SHUT/No-Shut Port:

   Shutdown port

   Record remote mpid / shortMA name / MD name / Vlan information.

   No-shut port

Generate Report:

   Verify that interface is UP.

   Send message indicating the MA name and corresponding

   Interface that was unshut by the script.

Deploying CFM SD (Service Diagnostics) Scenarios in Cisco Access Router

Step-by-step instructions are shown to deploy CFM SD policy scripts in a Cisco Access Router.

Deployment of CFM SD scenarios using tclsh CLI parser mode

The following steps are required to deploy scenarios using tclsh CLI parser mode:

Configuring Policy and Library path:

event manager directory user policy "flash:/svc_diag"

event manager directory user library "flash:/user_lib"

Create Maintenance Association file:

agam-3845#more ma_list

SID_1

SID_2

SID_3

SID_4

Verify policy and library path is set using show run output:

event manager directory user policy "flash:/svc_diag"

event manager directory user library "flash:/user_lib"

Add email parameters:

agam-3845# tclsh flash:/svc_diag/collectEmailParameters.tcl c3845@cisco.com arshadm@cisco.com arshadm@cisco.com 64.102.124.15

Displaying policy deploying policy using tclsh:

agam-3845#tclsh flash:/svc_diag/sdiag_router_cfm.tcl

ERROR: Incorrect number of arguments

CFM CC-TIMEOUT POLICY Syntax:

Usage:tclsh <disk#:>/<dir_name>/sdiag_router_cfm.tcl cfm_cctimeout.tcl <notification> <configurationHistory> <EventHistory> <CommandHistory> <prepend trigger message> <user_pol_dir> <user_lib_dir> <ma_list>

CFM ON-DEMAND POLICY syntax:

Usage:tclsh <disk#:>/<dir_name>/sdiag_router_cfm.tcl cfm_ondemand.tcl <notification> <configurationHistory> <EventHistory> <CommandHistory> <prepend trigger message> <user_pol_dir> <user_lib_dir> <domain_name> <ma_name> <optional mep-id>

CFM SHUT POLICY Syntax:

Usage:tclsh <disk#:>/<dir_name>/sdiag_router_cfm.tcl cfm_router_shut.tcl <notification> <configuration history> <event history> <command history> <prepend trigger message> <user_pol_dir> <user_lib_dir> <ma_list> <p2p>

CFM AUTO-TRACE POLICY Syntax:

Usage:tclsh <disk#:>/<dir_name>/sdiag_router_cfm.tcl cfm_autotrace.tcl <notification> <configuration history> <event history> <commandHistory> <prepend trigger message> <user_pol_dir> <user_lib_dir> <ma_list> <timer>

For policy detailed parameters, use <policy> -help

For example: tclsh <disk#:>/<dir_name>/sdiag_router_cfm.tcl cfm_router_shut.tcl -help

For policy environment variables, use <policy> -variables

For example: tclsh <disk#:>/<dir_name>/sdiag_router_cfm.tcl cfm_router_shut.tcl -variables

Deploying cfm_cctimeout.tcl policy using tclsh CLI Parser:

agam-3845# tclsh flash:/svc_diag/sdiag_router_cfm.tcl cfm_cctimeout.tcl email,syslog,snmp true true true true flash:/svc_diag flash:/user_lib ma_test

WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification

Policy cfm_cctimeout.tcl successfully registered

agam-3845#

Displaying parameters of deployed policies:

agam-3845#tclsh flash:/svc_diag/sdiag_router_cfm_display.tcl cfm_cctimeout.tcl

THE INPUTS GIVEN FOR CFM CC-TIMEOUT SCENARIO ARE:

Notification : EMAIL,SYSLOG,SNMP

Configuration history option : TRUE

Event history option : TRUE

Command history option : TRUE

Prepend option : TRUE

MA File name is : ma_test

Undeploying cfm_cctimeout.tcl policy:

agam-3845#tclsh flash:/svc_diag/sdiag_router_cfm_undeploy.tcl cfm_cctimeout.tcl

cfm_cctimeout.tcl is unregistered successfully

Deployment of CFM SD scenarios using emm

Step-by-step Instructions for deploying CFM SD scenario using EMM follow.

Setting global parameter using emm:

agam-3845#emm mdf flash:/cfm_router.mdf

--------------------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Set Global Variables (email parameters)

3. Deploy CFM Scenarios

4. Display Registered Policies

5. Display Environment Variables

6. Remove Diagnostic Policies

7. Exit

Enter selection [7]:2

Enter the email address for the _email_from field

Enter value [c3845@cisco.com]:c3845@cisco.com

 

Enter the email address for the _email_to field

Enter value [arshadm@cisco.com]:arshadm@cisco.com

Enter the email address for the _email_cc field

Enter value [arshadm@cisco.com]:arshadm@cisco.com

Enter the IP address for the email server

Enter value [64.102.124.15]:64.102.124.15

Enter the directory where the CFM diagnostic policies are located

in the form of a URL (e.g.  disk0:/svc-diag)

Enter value [flash:/svc_diag]:flash:/svc_diag

Enter the directory where the user library files are located

in the form of a URL (e.g.  disk0:/user_library)

Enter value [flash:/user_lib]:flash:/user_lib

Press any key to continue...

================================================================================

                   Connectivity Fault Management Diagnostics

                                            Enter ? for help or ?# for item help

--------------------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Set Global Variables (email parameters)

3. Deploy CFM Scenarios

4. Display Registered Policies

5. Display Environment Variables

6. Remove Diagnostic Policies

7. Exit

Enter selection [7]:3

 

Deploying cfm_cctimeout.tcl policy using emm

 

agam-3845#emm mdf flash:/cfm_router.mdf

================================================================================

                   Connectivity Fault Management Diagnostics

 

                                            Enter ? for help or ?# for item help

--------------------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Set Global Variables (email parameters)

3. Deploy CFM Scenarios

4. Display Registered Policies

5. Display Environment Variables

6. Remove Diagnostic Policies

7. Exit

Enter selection [7]: 3

================================================================================

               Connectivity Fault Management Scenario Deployment

                                                          Enter ?# for item help

--------------------------------------------------------------------------------

1. Deploy CFM CC-Timeout Script

2. Deploy CFM On-Demand Script

3. Deploy Action Shut Script

4. Deploy CFM Action Auto Trace Script

5. Return to main CFM menu

Enter selection [5]: 1

Enter the path to the Maintenance Association name file (e.g.  disk0:/)

Enter value [flash:]: <ENTER>

Please select the Maintenance Association name file or 'CREATE_FILE'

     1. cert.pem

     2. ma_list.txt

     3. new_ma

     4. ma_list

     5. ma_test

     6. CREATE_FILE

Enter choice: 5

Note:    You may use an existing file or create a new one. We have mentioned procedure for creating a new file in                 router_shut scenario.

Select the type of notification to be sent when a problem is diagnosed

     1. Email Only

     2. SNMP Only

     3. Syslog Only

     4. Email and Syslog

     5. Email and SNMP

     6. Syslog and SNMP

     7. Email, Syslog, and SNMP

Enter choice: 7

Do you want to log configuration history?

     1. Yes

     2. No

Enter choice: 1

Do you want to log event history?

     1. Yes

     2. No

Enter choice: 1

Do you want to log command history?

     1. Yes

     2. No

Enter choice: 1

Do you want to prepend trigger syslog messages to the diagnosis?

     1. Yes

     2. No

Enter choice: 1

WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification

Policy cfm_cctimeout.tcl successfully registered

Press any key to continue...

================================================================================

               Connectivity Fault Management Scenario Deployment

                                                          Enter ?# for item help

--------------------------------------------------------------------------------

1. Deploy CFM CC-Timeout Script

2. Deploy CFM On-Demand Script

3. Deploy Action Shut Script

4. Deploy CFM Action Auto Trace Script

5. Return to main CFM menu

 

Enter selection [5]: <ENTER>

================================================================================

                   Connectivity Fault Management Diagnostics

                                            Enter ? for help or ?# for item help

--------------------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Set Global Variables (email parameters)

3. Deploy CFM Scenarios

4. Display Registered Policies

5. Display Environment Variables

6. Remove Diagnostic Policies

7. Exit

Enter selection [7]: <ENTER>

agam-3845#

 

Undeploying cfm_cctimeout.tcl script:

Use following steps to undeploy cctimeout scenario

 

agam-3845#emm mdf flash:/cfm_router.mdf

 

================================================================================

                   Connectivity Fault Management Diagnostics

                                            Enter ? for help or ?# for item help

--------------------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Set Global Variables (email parameters)

3. Deploy CFM Scenarios

4. Display Registered Policies

5. Display Environment Variables

6. Remove Diagnostic Policies

7. Exit

Enter selection [7]: 6

Please select diagnostic scenario to remove from the running-configuration

     1. CFM CC-Timeout

     2. CFM On-Demand

     3. CFM Action Shut

     4. CFM Auto Trace

     5. ALL CFM Policies

Enter choice: 1

cfm_cctimeout.tcl is unregistered successfully

1

Press any key to continue...

================================================================================

                   Connectivity Fault Management Diagnostics

                                            Enter ? for help or ?# for item help

--------------------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Set Global Variables (email parameters)

3. Deploy CFM Scenarios

4. Display Registered Policies

5. Display Environment Variables

6. Remove Diagnostic Policies

7. Exit

Enter selection [7]: <ENTER>

agam-3845#

Deployment of On-Demand  Scenario

Cfm_ondemand scenario can be deployed per  ma (maintenance association) basis

Deploying cfm_ondemand.tcl policy using tclsh CLI Parser:

 

agam-3845#tclsh flash:/svc_diag/sdiag_router_cfm.tcl cfm_ondemand.tcl email,syslog true true true true flash:svc_diag flash:/user_lib CUST1 SID_1 

WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification

Policy cfm_ondemand.tcl successfully registered

agam-3845#

 

Undeploying cfm_ondemand.tcl policy using tclsh CLI Parser:

 

agam-3845#tclsh flash:/svc_diag/sdiag_router_cfm_undeploy.tcl  cfm_ondemand.tcl

cfm_ondemand.tcl is unregistered successfully

 

Deploying cfm_ondemand.tcl policy using emm:

 

Following step by step instruction shows how to deploy cfm_ondemand scenario

agam-3845#emm mdf cfm_router.mdf

================================================================================

                   Connectivity Fault Management Diagnostics

                                            Enter ? for help or ?# for item help

--------------------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Set Global Variables (email parameters)

3. Deploy CFM Scenarios

4. Display Registered Policies

5. Display Environment Variables

6. Remove Diagnostic Policies

7. Exit

Enter selection [7]: 3

===============================================================================

               Connectivity Fault Management Scenario Deployment

                                                          Enter ?# for item help

--------------------------------------------------------------------------------

1. Deploy CFM CC-Timeout Script

2. Deploy CFM On-Demand Script

3. Deploy Action Shut Script

4. Deploy CFM Action Auto Trace Script

5. Return to main CFM menu

Enter selection [5]: 2

Please select a domain for the Maintenance Association of interest

     1. CUST1

Enter choice: 1

Select the Maintenance Association Name for Domain CUST1

     1. SID_3

     2. SID_4

     3. SID_1

     4. SID_2

Enter choice: 1

Do you want to specify a Maintenance End Point (MEP)?

     1. Yes

     2. No

Enter choice: 1

Please select the Maintenance End Point for SID_3

     1. 4200

Enter choice: 1

Select the type of notification to be sent when a problem is diagnosed

     1. Email Only

     2. SNMP Only

     3. Syslog Only

     4. Email and Syslog

     5. Email and SNMP

     6. Syslog and SNMP

     7. Email, Syslog, and SNMP

Enter choice: 4

Do you want to log configuration history?

     1. Yes

     2. No

Enter choice: 1

Do you want to log event history?

     1. Yes

     2. No

Enter choice: 1

Do you want to log command history?

     1. Yes

     2. No

Enter choice: 1

Do you want to prepend trigger syslog messages to the diagnosis?

     1. Yes

     2. No

Enter choice: 1

WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification

Policy cfm_ondemand.tcl successfully registered

Press any key to continue...

================================================================================

               Connectivity Fault Management Scenario Deployment

                                                          Enter ?# for item help

--------------------------------------------------------------------------------

1. Deploy CFM CC-Timeout Script

2. Deploy CFM On-Demand Script

3. Deploy Action Shut Script

4. Deploy CFM Action Auto Trace Script

5. Return to main CFM menu

Enter selection [5]: 5

 

Undeploying cfm_ondemand.tcl script:

Use the following steps to undeploy ondemand script scenario:

 

agam-3845#emm mdf flash:/cfm_router.mdf

================================================================================

                   Connectivity Fault Management Diagnostics

                                            Enter ? for help or ?# for item help

--------------------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Set Global Variables (email parameters)

3. Deploy CFM Scenarios

4. Display Registered Policies

5. Display Environment Variables

6. Remove Diagnostic Policies

7. Exit

Enter selection [7]: 6

Please select diagnostic scenario to remove from the running-configuration

     1. CFM CC-Timeout

     2. CFM On-Demand

     3. CFM Action Shut

     4. CFM Auto Trace

     5. ALL CFM Policies

Enter choice: 2

cfm_ondemand.tcl is unregistered successfully

Press any key to continue...

 

================================================================================

                   Connectivity Fault Management Diagnostics

                                            Enter ? for help or ?# for item help

--------------------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Set Global Variables (email parameters)

3. Deploy CFM Scenarios

4. Display Registered Policies

5. Display Environment Variables

6. Remove Diagnostic Policies

7. Exit

Enter selection [7]: <ENTER>

 

Deployment of autotrace Scenario

Deploying cfm_autotrace.tcl policy using tclsh CLI Parser:

 

agam-3845#tclsh flash:/svc_diag/ sdiag_router_cfm.tcl cfm_autotrace.tcl email/syslog true true true true flash:svc_diag flash:/user_lib ma_test 60

WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification

Traceroute Cache Hold Time: 120 mins

CFM autotrace timer: 60 mins

Policy cfm_autotrace.tcl successfully registered

 

Undeploying cfm_autotrace.tcl policy using tclsh CLI Parser:

 

agam-3845# tclsh flash:/svc_diag/sdiag_router_cfm_undeploy.tcl  cfm_autotrace.tcl

cfm_autotrace.tcl is unregistered successfully

 

Deploying cfm_autotrace.tcl action policy using emm

 

Please follow following steps to deploy cfm_autotrace scenario

agam-3845#emm mdf cfm_router.mdf

================================================================================

                   Connectivity Fault Management Diagnostics

                                            Enter ? for help or ?# for item help

--------------------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Set Global Variables (email parameters)

3. Deploy CFM Scenarios

4. Display Registered Policies

5. Display Environment Variables

6. Remove Diagnostic Policies

7. Exit

Enter selection [7]: 3

================================================================================

               Connectivity Fault Management Scenario Deployment

 

                                                          Enter ?# for item help

--------------------------------------------------------------------------------

1. Deploy CFM CC-Timeout Script

2. Deploy CFM On-Demand Script

3. Deploy Action Shut Script

4. Deploy CFM Action Auto Trace Script

5. Return to main CFM menu

Enter selection [5]: 4

Enter the path to the Maintenance Association name file (e.g.  disk0:/)

Enter value [flash:]: <ENTER>

Please select the Maintenance Association name file or 'CREATE_FILE'

     1. cert.pem

     2. ma_list.txt

     3. new_ma

     4. ma_list

     5. ma_test

     6. CREATE_FILE

Enter choice: 5

Select the type of notification to be sent when a problem is diagnosed

     1. Email Only

     2. SNMP Only

     3. Syslog Only

     4. Email and Syslog

     5. Email and SNMP

     6. Syslog and SNMP

     7. Email, Syslog, and SNMP

Enter choice: 4

Do you want to log configuration history?

     1. Yes

     2. No

Enter choice: 1

Do you want to log event history?

     1. Yes

     2. No

Enter choice: 1

Do you want to log command history?

     1. Yes

     2. No

Enter choice: 1

Do you want to prepend trigger syslog messages to the diagnosis?

     1. Yes

     2. No

Enter choice: 1

Enter the timer for this policy (minutes)

Enter a value in the range  <60 - 80> [80]: 60

WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification

Traceroute Cache Hold Time: 120 mins

CFM autotrace timer: 60 mins

Policy cfm_autotrace.tcl successfully registered

1

Press any key to continue...

================================================================================

               Connectivity Fault Management Scenario Deployment

                                                          Enter ?# for item help

--------------------------------------------------------------------------------

1. Deploy CFM CC-Timeout Script

2. Deploy CFM On-Demand Script

3. Deploy Action Shut Script

4. Deploy CFM Action Auto Trace Script

5. Return to main CFM menu

Enter selection [5]: <ENTER>

================================================================================

                   Connectivity Fault Management Diagnostics

                                            Enter ? for help or ?# for item help

--------------------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Set Global Variables (email parameters)

3. Deploy CFM Scenarios

4. Display Registered Policies

5. Display Environment Variables

6. Remove Diagnostic Policies

7. Exit

Enter selection [7]: <ENTER>

agam-3845#

 

Undeploying cfm_ondemand.tcl script:

Use the following steps to undeploy the ccondemand scenario:

 

agam-3845#emm mdf flash:/cfm_router.mdf

================================================================================

                   Connectivity Fault Management Diagnostics

 

                                            Enter ? for help or ?# for item help

--------------------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Set Global Variables (email parameters)

3. Deploy CFM Scenarios

4. Display Registered Policies

5. Display Environment Variables

6. Remove Diagnostic Policies

7. Exit

Enter selection [7]: 6

Please select diagnostic scenario to remove from the running-configuration

     1. CFM CC-Timeout

     2. CFM On-Demand

     3. CFM Action Shut

     4. CFM Auto Trace

     5. ALL CFM Policies

Enter choice: 4

cfm_autotrace.tcl is unregistered successfully

1

Press any key to continue...

================================================================================

                   Connectivity Fault Management Diagnostics

                                            Enter ? for help or ?# for item help

--------------------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Set Global Variables (email parameters)

3. Deploy CFM Scenarios

4. Display Registered Policies

5. Display Environment Variables

6. Remove Diagnostic Policies

7. Exit

Enter selection [7]: <ENTER>

SHUT/NOSHUT Action Script

 

This is a Policy script to “bounce” corresponding Ethernet IF (sub-IF).

Deploying cfm_router_shut.tcl script using tclsh

 

Usage:tclsh <disk#:>/<dir_name>/sdiag_router_cfm.tcl cfm_router_shut.tcl <notification> <configuration history> <event history> <command history> <prepend trigger message> <user_pol_dir> <user_lib_dir> <ma_list> <p2p>

agam-3845# tclsh flash:/svc_diag/sdiag_router_cfm.tcl   cfm_router_shut.tcl email,syslog,sn mp true true true true flash:/svc_diag flash:/user_lib ma_list true

WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification

Policy cfm_router_shut.tcl successfully registered

WARNING: User should specify MAs associated to P2P services with Down MEPs on local interface.

 

Undeploy  cfm_router_shut.tcl policy:

 

agam-3845# tclsh flash:/svc_diag/sdiag_router_cfm_undeploy.tcl cfm_router_shut.tcl     

cfm_router_shut.tcl is unregistered successfully

agam-3845#

 

Deploying cfm_router_shut.tcl policy using emm

 

agam-3845#emm mdf flash:/cfm_router.mdf

================================================================================

                   Connectivity Fault Management Diagnostics

                                            Enter ? for help or ?# for item help

--------------------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Set Global Variables (email parameters)

3. Deploy CFM Scenarios

4. Display Registered Policies

5. Display Environment Variables

6. Remove Diagnostic Policies

7. Exit

Enter selection [7]:2

Enter the email address for the _email_from field

Enter value [c3845@cisco.com]:c3845@cisco.com

Enter the email address for the _email_to field

Enter value [arshadm@cisco.com]:arshadm@cisco.com

Enter the email address for the _email_cc field

Enter value [arshadm@cisco.com]:arshadm@cisco.com

Enter the IP address for the email server

Enter value [64.102.124.15]:64.102.124.15

Enter the directory where the CFM diagnostic policies are located

in the form of a URL (e.g.  disk0:/svc-diag)

Enter value [flash:/svc_diag]:flash:/svc_diag

Enter the directory where the user library files are located

in the form of a URL (e.g.  disk0:/user_library)

Enter value [flash:/user_lib]:flash:/user_lib

Press any key to continue...

 

================================================================================

                   Connectivity Fault Management Diagnostics

                                            Enter ? for help or ?# for item help

--------------------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Set Global Variables (email parameters)

3. Deploy CFM Scenarios

4. Display Registered Policies

5. Display Environment Variables

6. Remove Diagnostic Policies

7. Exit

Enter selection [7]:3

================================================================================

               Connectivity Fault Management Scenario Deployment

                                                          Enter ?# for item help

--------------------------------------------------------------------------------

1. Deploy CFM CC-Timeout Script

2. Deploy CFM On-Demand Script

3. Deploy Action Shut Script

4. Deploy CFM Action Auto Trace Script

5. Return to main CFM menu

 

Enter selection [5]:3

Enter the path to the Maintenance Association name file (e.g.  disk0:/)

Enter value [flash:]:<ENTER>

Please select the Maintenance Association name file or 'CREATE_FILE'

     1. cert.pem

     2. ma_list.txt

     3. new_ma

     4. ma_list

     5. CREATE_FILE

Enter choice: 5

Enter the file name for the Maintenance Association name file

Enter value [ma_list.txt]: ma_test

Please select a Maintenance Association name

     1. SID_3

     2. SID_4

     3. SID_1

     4. SID_2

Enter choice: 1

You have selected SID_3:

 Write this entry to ma_test?

     1. Yes

     2. No

Enter choice: 1

Please select next operation

 

     1. Add another record to the MA name file

     2. Display current MA names file contents

     3. Done with MA name file

Enter choice: 1

Please select a Maintenance Association name

     1. SID_4

     2. SID_1

     3. SID_2

Enter choice: 1

You have selected SID_4:

 Write this entry to ma_test?

     1. Yes

     2. No

Enter choice: 1

Please select next operation

     1. Add another record to the MA name file

     2. Display current MA names file contents

     3. Done with MA name file

Enter choice: 1

Please select a Maintenance Association name

     1. SID_1

     2. SID_2

Enter choice: 1

You have selected SID_1:

 

 Write this entry to ma_test?

     1. Yes

     2. No

Enter choice: 1

Please select next operation

     1. Add another record to the MA name file

     2. Display current MA names file contents

     3. Done with MA name file

Enter choice: 1

Please select a Maintenance Association name

     1. SID_2

Enter choice: 1

You have selected SID_2:

 Write this entry to ma_test?

     1. Yes

     2. No

Enter choice: 1

Please select next operation

     1. Add another record to the MA name file

     2. Display current MA names file contents

     3. Done with MA name file

Enter choice: 3

Select the type of notification to be sent when a problem is diagnosed

     1. Email Only

     2. SNMP Only

     3. Syslog Only

     4. Email and Syslog

     5. Email and SNMP

     6. Syslog and SNMP

     7. Email, Syslog, and SNMP

Enter choice:7

Enter choice: 7

Do you want to log configuration history?

     1. Yes

     2. No

Enter choice: 1

Do you want to log event history?

     1. Yes

     2. No

Enter choice: 1

Do you want to log command history?

     1. Yes

     2. No

Enter choice: 1

Do you want to prepend trigger syslog messages to the diagnosis?

     1. Yes

     2. No

Enter choice: 1

Are the specified MAs associated to P2P services with DOWN MEPs on local interface ?

     1. Yes

     2. No

Enter choice: 1

WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification

Policy cfm_router_shut.tcl successfully registered

WARNING: User should specify MAs associated to P2P services with Down MEPs on local interface.

 

Press any key to continue...

================================================================================

               Connectivity Fault Management Scenario Deployment

                                                          Enter ?# for item help

--------------------------------------------------------------------------------

1. Deploy CFM CC-Timeout Script

2. Deploy CFM On-Demand Script

3. Deploy Action Shut Script

4. Deploy CFM Action Auto Trace Script

5. Return to main CFM menu

 

Enter selection [5]: 5

================================================================================

                   Connectivity Fault Management Diagnostics

 

                                            Enter ? for help or ?# for item help

--------------------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Set Global Variables (email parameters)

3. Deploy CFM Scenarios

4. Display Registered Policies

5. Display Environment Variables

6. Remove Diagnostic Policies

7. Exit

Enter selection [7]: 7

agam-3845#

 

Un-deploying cfm_router_shut.tcl script using emm

 

agam-3845#emm mdf flash:/cfm_router.mdf

 

================================================================================

                   Connectivity Fault Management Diagnostics

 

                                            Enter ? for help or ?# for item help

--------------------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Set Global Variables (email parameters)

3. Deploy CFM Scenarios

4. Display Registered Policies

5. Display Environment Variables

6. Remove Diagnostic Policies

7. Exit

 

Enter selection [7]: 6

 

Please select diagnostic scenario to remove from the running-configuration

 

     1. CFM CC-Timeout

     2. CFM On-Demand

     3. CFM Action Shut

     4. CFM Auto Trace

     5. ALL CFM Policies

 

Enter choice: 3

 

cfm_router_shut.tcl is unregistered successfully

1

 

Press any key to continue...

================================================================================

                   Connectivity Fault Management Diagnostics

 

                                            Enter ? for help or ?# for item help

--------------------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Set Global Variables (email parameters)

3. Deploy CFM Scenarios

4. Display Registered Policies

5. Display Environment Variables

6. Remove Diagnostic Policies

7. Exit

 

Enter selection [7]:<ENTER>

 

Running CFM SD Scenarios

 

SHUT/NOSHUT Scenario

•  Deploy cfm_router.tcl scenario (use either emm or tclsh CLI parser)

•  Simulate fault in remote side by shutting down sub-interface .1000. Syslog and  email notification is display on the screen. (debug snmp detail on).

 

agam-3845#

*Apr 15 16:43:33.593 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 420 vlan 701 MA name SID_2 in domain CUST1 changed state to down with event code TimeOut.

*Apr 15 16:43:33.593 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 402 level 7 VLAN 701 dir D Interface Gi0/1 enters AIS defect condition

*Apr 15 16:43:34.325 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification

*Apr 15 16:43:34.361 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Domain from syslog is: CUST1

*Apr 15 16:43:34.361 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: MA from syslog is: SID_2

*Apr 15 16:43:35.265 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: 1

*Apr 15 16:43:35.265 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Shutting down: Gi0/1.1001

*Apr 15 16:43:35.841 EST: %SYS-5-CONFIG_I: Configured from console by  on vty1 (EEM:cfm_router_shut.tcl)

*Apr 15 16:43:35.953 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: No shut on Gi0/1.1001 ...

*Apr 15 16:43:36.153 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 402 vlan 701 for service MA name SID_2 with the event code DefRemoteCCM.

*Apr 15 16:43:36.513 EST: %SYS-5-CONFIG_I: Configured from console by  on vty0 (EEM:cfm_router_shut.tcl)

*Apr 15 16:43:36.633 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: TRIGGERED_BY "*Apr 15 16:43:33.593 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 420 vlan 701 MA name SID_2 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_shut: Service failure detected on vlan 701, CFM MD CUST1. Reason: Remote MEP timeout. Action: Interface Gi0/1.1001 with local MEP 420 has been shut and unshut.

*Apr 15 16:43:42.853 EST: %HA_EM-6-LOG: cfm_router_shut.tcl:

Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details

Email details:

-----Original Message-----

From: c3845@cisco.com [mailto:c3845@cisco.com]

Sent: Thursday, April 15, 2010 5:48 PM

To: Arshad Mahmood (arshadm)

Cc: arsham@cisco.com

Subject: From router agam-3845:

 

TRIGGERED_BY "*Apr 15 16:47:54.481 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 420 vlan 701 MA name SID_2 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_shut: Service failure detected on vlan 701, CFM MD CUST1. Reason: Remote MEP timeout. Action: Interface Gi0/1.1001 with local MEP 420 has been shut and unshut.

THE CONFIGURATION HISTORY is:

!Contextual Config Diffs:

+no logging buffered

+event manager environment _svcdiag_cfmShutNotif EMAIL,SYSLOG,SNMP

+event manager environment _svcdiag_cfmShutMAFile ma_list

+event manager environment _email_cc arshadm@cisco.com

-logging buffered 200000

-event manager environment _email_cc arsham@cisco.com

-event manager environment _svcdiag_cfmShutNotif ALL

-event manager environment _svcdiag_cfmShutMAFile flash:/ma_test

-event manager policy cfm_router_shut.tcl type user

agam-3845#

--------------------------------------------------------------

THE EVENT HISTORY is:

Syslog logging: enabled (0 messages dropped, 4 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

 

No Active Message Discriminator.

 

No Inactive Message Discriminator.

 

    Console logging: level debugging, 1118 messages logged, xml disabled,

                     filtering disabled

    Monitor logging: level debugging, 0 messages logged, xml disabled,

                     filtering disabled

    Buffer logging:  level debugging, 80 messages logged, xml disabled,

                    filtering disabled

    Exception Logging: size (4096 bytes)

    Count and timestamp logging messages: disabled

    Persistent logging: disabled

 

No active filter modules.

 

    Trap logging: level informational, 602 message lines logged

 

Log Buffer (200000 bytes):

 

*Apr 15 16:46:40.017 EST: %SYS-5-CONFIG_I: Configured from console by console

*Apr 15 16:47:09.309 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 402 level 7 VLAN 701 dir D Interface Gi0/1 exited AIS defect condition

*Apr 15 16:47:09.309 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 420 vlan 701 MA name SID_2 domain CUST1 interface status Up event code Returning.

*Apr 15 16:47:54.481 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 420 vlan 701 MA name SID_2 in domain CUST1 changed state to down with event code TimeOut.

*Apr 15 16:47:54.481 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 402 level 7 VLAN 701 dir D Interface Gi0/1 enters AIS defect condition

*Apr 15 16:47:55.217 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification

*Apr 15 16:47:55.241 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Domain from syslog is: CUST1

*Apr 15 16:47:55.241 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: MA from syslog is: SID_2

*Apr 15 16:47:56.145 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: 1

*Apr 15 16:47:56.145 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Shutting down: Gi0/1.1001

*Apr 15 16:47:56.721 EST: %SYS-5-CONFIG_I: Configured from console by  on vty1 (EEM:cfm_router_shut.tcl)

*Apr 15 16:47:56.833 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: No shut on Gi0/1.1001 ...

*Apr 15 16:47:57.041 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 402 vlan 701 for service MA name SID_2 with the event code DefRemoteCCM.

*Apr 15 16:47:57.393 EST: %SYS-5-CONFIG_I: Configured from console by  on vty0 (EEM:cfm_router_shut.tcl)

*Apr 15 16:47:57.513 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: TRIGGERED_BY "*Apr 15 16:47:54.481 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 420 vlan 701 MA name SID_2 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_shut: Service failure detected on vlan 701, CFM MD CUST1. Reason: Remote MEP timeout. Action: Interface Gi0/1.1001 with local MEP 420 has been shut and unshut.

agam-3845#

--------------------------------------------------------------

THE COMMAND HISTORY is:

Thu Apr 15 16:47:55 EST 2010 show event manager environ

Thu Apr 15 16:47:55 EST 2010 show ethernet cfm maintenance-points remote detail mpid 420 domain CUST1 vlan 701

Thu Apr 15 16:47:56 EST 2010 show ethernet cfm mpdb domain-id CUST1 service SID_2

Thu Apr 15 16:47:56 EST 2010 interface Gi0/1.1001

Thu Apr 15 16:47:56 EST 2010 shut

Thu Apr 15 16:47:57 EST 2010 interface Gi0/1.1001

Thu Apr 15 16:47:57 EST 2010 no shut

 

Syslogs and email notification after Shutting interface on remote 7604

 

agam-3845#

*Apr 15 17:02:35.377 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 4200 vlan 800 MA name SID_3 in domain CUST1 changed state to down with event code TimeOut.

*Apr 15 17:02:35.377 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 720 level 7 VLAN 800 dir D Interface Gi0/1 enters AIS defect condition

*Apr 15 17:02:35.377 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 420 vlan 701 MA name SID_2 in domain CUST1 changed state to down with event code TimeOut.

*Apr 15 17:02:35.377 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 402 level 7 VLAN 701 dir D Interface Gi0/1 enters AIS defect condition

*Apr 15 17:02:36.117 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification

*Apr 15 17:02:36.141 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Domain from syslog is: CUST1

*Apr 15 17:02:36.141 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: MA from syslog is: SID_3

*Apr 15 17:02:37.049 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: 0

*Apr 15 17:02:37.049 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Shutting down: Gi0/1.2000

*Apr 15 17:02:37.625 EST: %SYS-5-CONFIG_I: Configured from console by  on vty1 (EEM:cfm_router_shut.tcl)

*Apr 15 17:02:37.737 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: No shut on Gi0/1.2000 ...

*Apr 15 17:02:37.937 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 720 vlan 800 for service MA name SID_3 with the event code DefRemoteCCM.

*Apr 15 17:02:37.937 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 402 vlan 701 for service MA name SID_2 with the event code DefRemoteCCM.

*Apr 15 17:02:38.193 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut.

*Apr 15 17:02:38.193 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 7000 level 7 VLAN 700 dir D Interface Gi0/1 enters AIS defect condition

*Apr 15 17:02:38.297 EST: %SYS-5-CONFIG_I: Configured from console by  on vty0 (EEM:cfm_router_shut.tcl

*Apr 15 17:02:38.417 ES: %HA_EM-6-LOG: cfm_router_shut.tcl: TRIGGERED_BY "*Apr 15 17:02:35.377 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 4200 vlan 800 MA name SID_3 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_shut: Service failure detected on vlan 800, CFM MD CUST1. Reason: Remote MEP timeout. Action: Interface Gi0/1.2000 with local MEP 4200 has been shut and unshut.

*Apr 15 17:02:38.705 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 4201 vlan 801 MA name SID_4 in domain CUST1 changed state to down with event code TimeOut.

*Apr 15 17:02:38.705 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 721 level 7 VLAN 801 dir D Interface Gi0/1 enters AIS defect condition

*Apr 15 17:02:40.925 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 7000 vlan 700 for service MA name SID_1 with the event code DefRemoteCCM.

*Apr 15 17:02:41.477 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 721 vlan 801 for service MA name SID_4 with the event code DefRemoteCCM.

*Apr 15 17:02:47.569 EST: %HA_EM-6-LOG: cfm_router_shut.tcl:

Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details

--------------------------------------------------------------

*Apr 15 17:02:48.361 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification

*Apr 15 17:02:48.385 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Domain from syslog is: CUST1

*Apr 15 17:02:48.385 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: MA from syslog is: SID_2

*Apr 15 17:02:49.285 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: 1

*Apr 15 17:02:49.285 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Shutting down: Gi0/1.1001

*Apr 15 17:02:49.861 EST: %SYS-5-CONFIG_I: Configured from console by  on vty1 (EEM:cfm_router_shut.tcl)

*Apr 15 17:02:49.973 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: No shut on Gi0/1.1001 ...

*Apr 15 17:02:50.533 EST: %SYS-5-CONFIG_I: Configured from console by  on vty0 (EEM:cfm_router_shut.tcl)

*Apr 15 17:02:50.653 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: TRIGGERED_BY "*Apr 15 17:02:35.377 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 420 vlan 701 MA name SID_2 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_shut: Service failure detected on vlan 701, CFM MD CUST1. Reason: Remote MEP timeout. Action: Interface Gi0/1.1001 with local MEP 420 has been shut and unshut.

*Apr 15 17:02:59.809 EST: %HA_EM-6-LOG: cfm_router_shut.tcl:

Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details

--------------------------------------------------------------

*Apr 15 17:03:00.609 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification

*Apr 15 17:03:00.637 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Domain from syslog is: CUST1

*Apr 15 17:03:00.637 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: MA from syslog is: SID_1

*Apr 15 17:03:01.537 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: 1

*Apr 15 17:03:01.537 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Shutting down: Gi0/1.1000

*Apr 15 17:03:02.113 EST: %SYS-5-CONFIG_I: Configured from console by  on vty1 (EEM:cfm_router_shut.tcl)

*Apr 15 17:03:02.225 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: No shut on Gi0/1.1000 ...

*Apr 15 17:03:02.789 EST: %SYS-5-CONFIG_I: Configured from console by  on vty0 (EEM:cfm_router_shut.tcl)

*Apr 15 17:03:02.909 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: TRIGGERED_BY "*Apr 15 17:02:38.193 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_shut: Service failure detected on vlan 700, CFM MD CUST1. Reason: Remote MEP timeout. Action: Interface Gi0/1.1000 with local MEP 42 has been shut and unshut.

*Apr 15 17:03:12.069 EST: %HA_EM-6-LOG: cfm_router_shut.tcl:

Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details

--------------------------------------------------------------

*Apr 15 17:03:12.861 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification

*Apr 15 17:03:12.889 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Domain from syslog is: CUST1

*Apr 15 17:03:12.889 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: MA from syslog is: SID_4

*Apr 15 17:03:13.793 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: 0

*Apr 15 17:03:13.793 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Shutting down: Gi0/1.2001

*Apr 15 17:03:14.369 EST: %SYS-5-CONFIG_I: Configured from console by  on vty1 (EEM:cfm_router_shut.tcl)

*Apr 15 17:03:14.481 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: No shut on Gi0/1.2001 ...

*Apr 15 17:03:15.041 EST: %SYS-5-CONFIG_I: Configured from console by  on vty0 (EEM:cfm_router_shut.tcl)

*Apr 15 17:03:15.161 EST %HA_EM-6-LOG: cfm_router_shut.tcl: TRIGGERED_BY "*Apr 15 17:02:38.705 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 4201 vlan 801 MA name SID_4 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_shut: Service failure detected on vlan 801, CFM MD CUST1. Reason: Remote MEP timeout. Action: Interface Gi0/1.2001 with local MEP 4201 has been shut and unshut.

*Apr 15 17:03:23.329 EST: %HA_EM-6-LOG: cfm_router_shut.tcl:

Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details

 

Email details with logging buffer 200000

 

-----Original Message-----

From: c3845@cisco.com [mailto:c3845@cisco.com]

Sent: Thursday, April 15, 2010 6:03 PM

To: Arshad Mahmood (arshadm)

Cc: arsham@cisco.com

Subject: From router agam-3845:

 

TRIGGERED_BY "*Apr 15 17:02:38.705 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 4201 vlan 801 MA name SID_4 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_shut: Service failure detected on vlan 801, CFM MD CUST1. Reason: Remote MEP timeout. Action: Interface Gi0/1.2001 with local MEP 4201 has been shut and unshut.

 

THE CONFIGURATION HISTORY is:

!Contextual Config Diffs:

+no logging buffered

+event manager environment _svcdiag_cfmShutNotif EMAIL,SYSLOG,SNMP

+event manager environment _svcdiag_cfmShutMAFile ma_list

+event manager environment _email_cc arshadm@cisco.com

-logging buffered 200000

ethernet cfm ais link-status global

 -disable

-event manager environment _email_cc arsham@cisco.com

-event manager environment _svcdiag_cfmShutNotif ALL

-event manager environment _svcdiag_cfmShutMAFile flash:/ma_test

-event manager policy cfm_router_shut.tcl type user

 

agam-3845#

 

--------------------------------------------------------------

THE EVENT HISTORY is:

Syslog logging: enabled (0 messages dropped, 4 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

 

    Console logging: level debugging, 1659 messages logged, xml disabled,

                     filtering disabled

    Monitor logging: level debugging, 0 messages logged, xml disabled,

                     filtering disabled

    Buffer logging:  level debugging, 621 messages logged, xml disabled,

                    filtering disabled

    Exception Logging: size (4096 bytes)

    Count and timestamp logging messages: disabled

    Persistent logging: disabled

 

No active filter modules.

 

    Trap logging: level informational, 753 message lines logged

 

Log Buffer (200000 bytes):

 

*Apr 15 17:01:12.505 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 721 level 7 VLAN 801 dir D Interface Gi0/1 exited AIS defect condition

*Apr 15 17:01:12.505 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 4201 vlan 801 MA name SID_4 domain CUST1 interface status Up event code Returning.

*Apr 15 17:01:13.001 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification

*Apr 15 17:01:13.025 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Domain from syslog is: CUST1

*Apr 15 17:01:13.025 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: MA from syslog is: SID_4

*Apr 15 17:01:13.477 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Interface status associated with remote MEP with mpid 4201 vlan 801 MA name CUST1 is already Up.  Aborting CFM shut script for this instance.

*Apr 15 17:01:14.241 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification

*Apr 15 17:01:14.265 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Domain from syslog is: CUST1

*Apr 15 17:01:14.265 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: MA from syslog is: SID_3

*Apr 15 17:01:14.717 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Interface status associated with remote MEP with mpid 4200 vlan 800 MA name CUST1 is already Up.  Aborting CFM shut script for this instance.

*Apr 15 17:02:35.377 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 4200 vlan 800 MA name SID_3 in domain CUST1 changed state to down with event code TimeOut.

*Apr 15 17:02:35.377 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 720 level 7 VLAN 800 dir D Interface Gi0/1 enters AIS defect condition

*Apr 15 17:02:35.377 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 420 vlan 701 MA name SID_2 in domain CUST1 changed state to down with event code TimeOut.

*Apr 15 17:02:35.377 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 402 level 7 VLAN 701 dir D Interface Gi0/1 enters AIS defect condition

*Apr 15 17:02:36.117 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification

*Apr 15 17:02:36.141 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Domain from syslog is: CUST1

*Apr 15 17:02:36.141 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: MA from syslog is: SID_3

*Apr 15 17:02:37.049 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: 0

*Apr 15 17:02:37.049 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Shutting down: Gi0/1.2000

*Apr 15 17:02:37.625 EST: %SYS-5-CONFIG_I: Configured from console by  on vty1 (EEM:cfm_router_shut.tcl)

*Apr 15 17:02:37.737 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: No shut on Gi0/1.2000 ...

*Apr 15 17:02:37.937 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 720 vlan 800 for service MA name SID_3 with the event code DefRemoteCCM.

*Apr 15 17:02:37.937 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 402 vlan 701 for service MA name SID_2 with the event code DefRemoteCCM.

*Apr 15 17:02:38.193 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut.

*Apr 15 17:02:38.193 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 7000 level 7 VLAN 700 dir D Interface Gi0/1 enters AIS defect condition

*Apr 15 17:02:38.297 EST: %SYS-5-CONFIG_I: Configured from console by  on vty0 (EEM:cfm_router_shut.tcl)

*pr 15 17:02:38.417 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: TRIGGERED_BY "*Apr 15 17:02:35.377 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 4200 vlan 800 MA name SID_3 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_shut: Service failure detected on vlan 800, CFM MD CUST1. Reason: Remote MEP timeout. Action: Interface Gi0/1.2000 with local MEP 4200 has been shut and unshut.

*Apr 15 17:02:38.705 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 4201 vlan 801 MA name SID_4 in domain CUST1 changed state to down with event code TimeOut.

*Apr 15 17:02:38.705 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 721 level 7 VLAN 801 dir D Interface Gi0/1 enters AIS defect condition

*Apr 15 17:02:40.925 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 7000 vlan 700 for service MA name SID_1 with the event code DefRemoteCCM.

*Apr 15 17:02:41.477 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 721 vlan 801 for service MA name SID_4 with the event code DefRemoteCCM.

*Apr 15 17:02:47.569 EST: %HA_EM-6-LOG: cfm_router_shut.tcl:

Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details

--------------------------------------------------------------

*Apr 15 17:02:48.361 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification

*Apr 15 17:02:48.385 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Domain from syslog is: CUST1

*Apr 15 17:02:48.385 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: MA from syslog is: SID_2

*Apr 15 17:02:49.285 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: 1

*Apr 15 17:02:49.285 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: Shutting down: Gi0/1.1001

*Apr 15 17:02:49.861 EST: %SYS-5-CONFIG_I: Configured from console by  on vty1 (EEM:cfm_router_shut.tcl)

*Apr 15 17:02:49.973 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: No shut on Gi0/1.1001 ...

*Apr 15 17:02:50.533 EST: %SYS-5-CONFIG_I: Configured from console by  on vty0 (EEM:cfm_router_shut.tcl)

*Apr 15 17:02:50.653 EST: %HA_EM-6-LOG: cfm_router_shut.tcl: TRIGGERED_BY "*Apr 15 17:02:35.377 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 420 vlan 701 MA name SID_2 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_shut: Service failure detected on vlan 701, CFM MD CUST1. Reason: Remote MEP timeout. Action: Interface Gi0/1.1001 with local MEP 420 has been shut and unshut.

*Apr 15 17:02:59.809 EST: %HA_EM-6-LOG: cfm_router_shut.tcl:

Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details

-----------------------------------------------------------------------------------

THE COMMAND HISTORY is:

Thu Apr 15 17:03:12 EST 2010 show event manager environ

Thu Apr 15 17:03:13 EST 2010 show ethernet cfm maintenance-points remote detail mpid 4201 domain CUST1 vlan 801

Thu Apr 15 17:03:13 EST 2010 show ethernet cfm mpdb domain-id CUST1 service SID_4

Thu Apr 15 17:03:14 EST 2010 interface Gi0/1.2001

Thu Apr 15 17:03:14 EST 2010 shut

Thu Apr 15 17:03:14 EST 2010 interface Gi0/1.2001

Thu Apr 15 17:03:15 EST 2010 no shut

 

Executing cctimeout scenario:

 

Create a fault on the remote side to trigger cctimeout event. Following syslogs and email  notification is displayed  (debug SNMP detail enabled)

agam-3845#

*Apr 15 17:22:22.017 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut.

*Apr 15 17:22:22.017 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 7000 level 7 VLAN 700 dir D Interface Gi0/1 enters AIS defect condition

*Apr 15 17:22:22.749 EST: %HA_EM-6-LOG: cfm_cctimeout.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification

*Apr 15 17:22:24.577 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 7000 vlan 700 for service MA name SID_1 with the event code DefRemoteCCM.

*Apr 15 17:22:56.765 EST: %HA_EM-6-LOG: cfm_cctimeout.tcl: TRIGGERED_BY "*Apr 15 17:22:22.017 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_timeout: Diagnostics for rmep 0024.14f6.80c1 on vlan 700, CFM MD CUST1.CCDB: 0 found; ErrorDB: 1 found; Reasons: Timeout; Ping: fail; Trace: fail

*Apr 15 17:23:04.917 EST: %HA_EM-6-LOG: cfm_cctimeout.tcl:

Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details

--------------------------------------------------------------

agam-3845#

User will receive following email:

-----Original Message-----

From: c3845@cisco.com [mailto:c3845@cisco.com]

Sent: Thursday, April 15, 2010 6:23 PM

To: Arshad Mahmood (arshadm)

Cc: arsham@cisco.com

Subject: From router agam-3845:

TRIGGERED_BY "*Apr 15 17:22:22.017 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_timeout: Diagnostics for rmep 0024.14f6.80c1 on vlan 700, CFM MD CUST1.CCDB: 0 found; ErrorDB: 1 found; Reasons: Timeout; Ping: fail; Trace: fail

 

DETAILED MESSAGE:

 Diagnostics for rmep 0024.14f6.80c1 on vlan 700, CFM MD CUST1, level =  7 ;

(1) Details of remote MEP was not found in CCDB.

(2) Details of remote MEP collected from Error DB

    (a) MEP ID - 42

    (b) MAC Address - 0024.14f6.80c1

    (c) Reason - "Lifetime Timer Expired"

(3) Output of loopback operation performed to remote MEP :    

Ping to "0024.14f6.80c1" failed.

(4) Output from Traceroute operation to remote MEP : Traceroute failed.

(5) TRACEROUTE OUTPUT:

Type escape sequence to abort. TTL 64. Linktrace Timeout is 5 seconds

Tracing the route to 0024.14f6.80c1 on Domain CUST1, Level 7, vlan 700

Traceroute sent via Gi0/1

 

B = Intermediary Bridge

! = Target Destination

* = Per hop Timeout

--------------------------------------------------------------------------------

                             MAC      Ingress      Ingr Action  Relay Action   

  Hops   Host             Forwarded   Egress       Egr Action   Previous Hop    

--------------------------------------------------------------------------------

B 1                    0017.0fad.9a50                           RlyMPDB       

                       Forwarded      Gi2/48       EgrOK        0017.95e4.4c71

B 2                    0017.0fad.9290 Gi2/48       IngOk        RlyMPDB       

                       Forwarded                                0017.0fad.9a50

*

*

agam-3845#

THE CONFIGURATION HISTORY is:

!Contextual Config Diffs:

+no logging buffered

+event manager environment _svcdiag_cfmShutNotif EMAIL,SYSLOG,SNMP

+event manager environment _svcdiag_cfmShutConfig TRUE

+event manager environment _svcdiag_cfmShutEvent TRUE

+event manager environment _svcdiag_cfmShutCommand TRUE

+event manager environment _svcdiag_cfmShutPrepend TRUE

+event manager environment _svcdiag_cfmShutMAFile ma_list

+event manager environment _svcdiag_cfmShutP2P TRUE

+event manager environment _email_cc arshadm@cisco.com

-logging buffered 20000

ethernet cfm ais link-status global

 -disable

-event manager environment _email_cc arsham@cisco.com

-event manager environment _svcdiag_cfmCctimeoutNotif ALL

-event manager environment _svcdiag_cfmCctimeoutConfig TRUE

-event manager environment _svcdiag_cfmCctimeoutEvent TRUE

-event manager environment _svcdiag_cfmCctimeoutCommand TRUE

-event manager environment _svcdiag_cfmCctimeoutPrepend TRUE

-event manager environment _svcdiag_cfmCctimeoutMAFile flash:/ma_test

-event manager policy cfm_cctimeout.tcl type user

agam-3845#

--------------------------------------------------------------

THE EVENT HISTORY is:

Syslog logging: enabled (0 messages dropped, 4 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

    Console logging: level debugging, 1741 messages logged, xml disabled,

                     filtering disabled

    Monitor logging: level debugging, 0 messages logged, xml disabled,

                     filtering disabled

    Buffer logging:  level debugging, 79 messages logged, xml disabled,

                    filtering disabled

    Exception Logging: size (4096 bytes)

    Count and timestamp logging messages: disabled

    Persistent logging: disabled

No active filter modules.

    Trap logging: level informational, 770 message lines logged

Log Buffer (20000 bytes):

*Apr 15 17:15:31.913 EST: %SYS-5-CONFIG_I: Configured from console by console

*Apr 15 17:20:55.697 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 721 level 7 VLAN 801 dir D Interface Gi0/1 exited AIS defect condition

*Apr 15 17:20:55.697 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 4201 vlan 801 MA name SID_4 domain CUST1 interface status Up event code Returning.

*Apr 15 17:20:55.953 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 7000 level 7 VLAN 700 dir D Interface Gi0/1 exited AIS defect condition

*Apr 15 17:20:55.953 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 42 vlan 700 MA name SID_1 domain CUST1 interface status Up event code Returning.

*Apr 15 17:20:58.513 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 720 level 7 VLAN 800 dir D Interface Gi0/1 exited AIS defect condition

*Apr 15 17:20:58.513 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 4200 vlan 800 MA name SID_3 domain CUST1 interface status Up event code Returning.

*Apr 15 17:20:59.281 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 402 level 7 VLAN 701 dir D Interface Gi0/1 exited AIS defect condition

*Apr 15 17:20:59.281 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 420 vlan 701 MA name SID_2 domain CUST1 interface status Up event code Returning.

*Apr 15 17:22:22.017 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut.

*Apr 15 17:22:22.017 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 7000 level 7 VLAN 700 dir D Interface Gi0/1 enters AIS defect condition

*Apr 15 17:22:22.749 EST: %HA_EM-6-LOG: cfm_cctimeout.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification

*Apr 15 17:22:24.577 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 7000 vlan 700 for service MA name SID_1 with the event code DefRemoteCCM.

*Apr 15 17:22:56.765 EST: %HA_EM-6-LOG: cfm_cctimeout.tcl: TRIGGERED_BY "*Apr 15 17:22:22.017 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_timeout: Diagnostics for rmep 0024.14f6.80c1 on vlan 700, CFM MD CUST1.CCDB: 0 found; ErrorDB: 1 found; Reasons: Timeout; Ping: fail; Trace: fail

agam-3845#

--------------------------------------------------------------

THE COMMAND HISTORY is:

Thu Apr 15 17:22:22 EST 2010 show event manager environ

Thu Apr 15 17:22:23 EST 2010 show ethernet cfm domain CUST1

Thu Apr 15 17:22:23 EST 2010 show ethernet cfm maintenance-points remote detail mpid 42 domain CUST1 vlan 700

Thu Apr 15 17:22:24 EST 2010 show ethernet cfm errors domain-id CUST1 service SID_1

Thu Apr 15 17:22:50 EST 2010 ping ethernet 0024.14f6.80c1 domain CUST1 vlan 700

Thu Apr 15 17:22:56 EST 2010 traceroute ethernet 0024.14f6.80c1 domain CUST1 vlan 700

 

Executing cfm_ondemand scenario:

 

After cfm_ondemand policy is deployed, you can use following command to trigger cfm_ondemand script. Syslog and email notification follows:

 

agam-3845#tclsh flash:/svc_diag/cfmod.tcl SID_1 CUST1 -n email,syslog

 

*Apr 15 18:08:03.785 EST: %HA_EM-6-LOG: cfm_ondemand.tcl: TRIGGERED_BY "Ondemand script run manually":: SERV-DIAG_CFM_ondemand: Diagnostics for vlan 700, CFM MD CUST1, rmep:all. CCDB: 1 found, 1 with UP IF state; ErrorDB: 0 found; Reasons: Ping: 1/0 total/fail;

agam-3845#

*Apr 15 18:08:12.025 EST: %HA_EM-6-LOG: cfm_ondemand.tcl:

Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details

Email detail:

 

-----Original Message-----

From: c3845@cisco.com [mailto:c3845@cisco.com]

Sent: Thursday, April 15, 2010 7:08 PM

To: Arshad Mahmood (arshadm)

Cc: arsham@cisco.com

Subject: From router agam-3845:

 

TRIGGERED_BY "Ondemand script run manually":: SERV-DIAG_CFM_ondemand: Diagnostics for vlan 700, CFM MD CUST1, rmep:all. CCDB: 1 found, 1 with UP IF state; ErrorDB: 0 found; Reasons: Ping: 1/0 total/fail;

 

DETAILED MESSAGE:

Diagnostics for vlan 700, CFM MD CUST1, rmep:all, level =  7 .

(1) Details of remote MEPs collected from CCDB:

    (a) MEP ID - 42

    (b) MAC Address - 0024.14f6.80c1

    (c) Interface status - Up

    (d) Port status - Up

    (e) RDI state - FALSE

    (f) CC packet statistics - 1139/0(Received/Error)

(2) Details of remote MEP collected from Error DB

(3) Output of loopback operation performed:

    Ping to "0024.14f6.80c1" was successful.

    Number of successful pings: 1

    Number of failed pings: 0

(4) No MEPs with "Lifetime Timer Expired" reason is found in Error DB. Hence no traceroute operation is performed.

 

THE CONFIGURATION HISTORY is:

 

!Contextual Config Diffs:

+no logging buffered

+ethernet cfm traceroute cache hold-time 60

+event manager environment _svcdiag_cfmShutNotif EMAIL,SYSLOG,SNMP

+event manager environment _svcdiag_cfmShutConfig TRUE

+event manager environment _svcdiag_cfmShutEvent TRUE

+event manager environment _svcdiag_cfmShutCommand TRUE

+event manager environment _svcdiag_cfmShutPrepend TRUE

+event manager environment _svcdiag_cfmShutMAFile ma_list

+event manager environment _svcdiag_cfmShutP2P TRUE

+event manager environment _email_cc arshadm@cisco.com

-logging buffered 20000

-ethernet cfm traceroute cache hold-time 120

ethernet cfm ais link-status global

 -disable

-event manager environment _email_cc arsham@cisco.com

-event manager environment _svcdiag_cfmOndemandConfig TRUE

-event manager environment _svcdiag_cfmOndemandEvent TRUE

-event manager environment _svcdiag_cfmOndemandCommand TRUE

-event manager environment _svcdiag_cfmOndemandPrepend TRUE

-event manager environment _svcdiag_cfmAutotraceNotif EMAIL,SYSLOG

-event manager environment _svcdiag_cfmAutotraceConfig TRUE

-event manager environment _svcdiag_cfmAutotraceEvent TRUE

-event manager environment _svcdiag_cfmAutotraceCommand TRUE

-event manager environment _svcdiag_cfmAutotracePrepend TRUE

-event manager environment _svcdiag_cfmAutotraceMAFile ma_test

-event manager environment _svcdiag_cfmAutotraceTimer 3600

-event manager environment _svcdiag_cfmOndemandMAname SID_1

-event manager environment _svcdiag_cfmOndemandDomain CUST1

-event manager environment _svcdiag_cfmOndemandMpid NULL

-event manager environment _svcdiag_cfmOndemandNotif email,syslog

-event manager policy cfm_ondemand.tcl type user

-event manager policy cfm_autotrace.tcl type user

 

agam-3845#

 

--------------------------------------------------------------

THE EVENT HISTORY is:

Syslog logging: enabled (0 messages dropped, 4 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

 

No Active Message Discriminator.

 

No Inactive Message Discriminator.

 

    Console logging: level debugging, 1748 messages logged, xml disabled,

                     filtering disabled

    Monitor logging: level debugging, 0 messages logged, xml disabled,

                     filtering disabled

    Buffer logging:  level debugging, 86 messages logged, xml disabled,

                    filtering disabled

    Exception Logging: size (4096 bytes)

    Count and timestamp logging messages: disabled

    Persistent logging: disabled

 

No active filter modules.

 

    Trap logging: level informational, 777 message lines logged

 

Log Buffer (20000 bytes):

 

*Apr 15 17:15:31.913 EST: %SYS-5-CONFIG_I: Configured from console by console

*Apr 15 17:20:55.697 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 721 level 7 VLAN 801 dir D Interface Gi0/1 exited AIS defect condition

*Apr 15 17:20:55.697 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 4201 vlan 801 MA name SID_4 domain CUST1 interface status Up event code Returning.

*Apr 15 17:20:55.953 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 7000 level 7 VLAN 700 dir D Interface Gi0/1 exited AIS defect condition

*Apr 15 17:20:55.953 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 42 vlan 700 MA name SID_1 domain CUST1 interface status Up event code Returning.

*Apr 15 17:20:58.513 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 720 level 7 VLAN 800 dir D Interface Gi0/1 exited AIS defect condition

*Apr 15 17:20:58.513 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 4200 vlan 800 MA name SID_3 domain CUST1 interface status Up event code Returning.

*Apr 15 17:20:59.281 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 402 level 7 VLAN 701 dir D Interface Gi0/1 exited AIS defect condition

*Apr 15 17:20:59.281 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 420 vlan 701 MA name SID_2 domain CUST1 interface status Up event code Returning.

*Apr 15 17:22:22.017 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut.

*Apr 15 17:22:22.017 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 7000 level 7 VLAN 700 dir D Interface Gi0/1 enters AIS defect condition

*Apr 15 17:22:22.749 EST: %HA_EM-6-LOG: cfm_cctimeout.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification

*Apr 15 17:22:24.577 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 7000 vlan 700 for service MA name SID_1 with the event code DefRemoteCCM.

*Apr 15 17:22:56.765 EST: %HA_EM-6-LOG: cfm_cctimeout.tcl: TRIGGERED_BY "*Apr 15 17:22:22.017 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_timeout: Diagnostics for rmep 0024.14f6.80c1 on vlan 700, CFM MD CUST1.CCDB: 0 found; ErrorDB: 1 found; Reasons: Timeout; Ping: fail; Trace: fail

*Apr 15 17:23:04.917 EST: %HA_EM-6-LOG: cfm_cctimeout.tcl:

Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details

--------------------------------------------------------------

*Apr 15 17:23:13.937 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 7000 level 7 VLAN 700 dir D Interface Gi0/1 exited AIS defect condition

*Apr 15 17:23:13.941 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 42 vlan 700 MA name SID_1 domain CUST1 interface status Up event code Returning.

*Apr 15 18:04:13.657 EST: %SYS-5-CONFIG_I: Configured from console by console

*Apr 15 18:08:03.785 EST: %HA_EM-6-LOG: cfm_ondemand.tcl: TRIGGERED_BY "Ondemand script run manually":: SERV-DIAG_CFM_ondemand: Diagnostics for vlan 700, CFM MD CUST1, rmep:all. CCDB: 1 found, 1 with UP IF state; ErrorDB: 0 found; Reasons: Ping: 1/0 total/fail;

agam-3845#

--------------------------------------------------------------

THE COMMAND HISTORY is:

 

Thu Apr 15 18:07:53 EST 2010 show event manager environ

Thu Apr 15 18:07:55 EST 2010 show ethernet cfm domain CUST1

Thu Apr 15 18:07:55 EST 2010 show ethernet cfm domain CUST1

Thu Apr 15 18:07:56 EST 2010 show ethernet cfm maintenance-points remote domain CUST1

Thu Apr 15 18:07:57 EST 2010 show ethernet cfm errors domain-id CUST1 service SID_1

Thu Apr 15 18:07:58 EST 2010 ping ethernet 0024.14f6.80c1 domain CUST1 vlan 700

Thu Apr 15 18:08:03 EST 2010 ping ethernet multicast domain CUST1 vlan 700

--------------------------------------------------------------

Autotrace Scenario

 

Autotrace scenario automatically triggers the cfm_autotrace policy after a specified interval. This action is very useful and it does not require any human intervention. You are notified through syslog or email notification after regular time intervals. Following are details that are sent as an email notification every hour (60 Min):

 

-----Original Message-----

From: c3845@cisco.com [mailto:c3845@cisco.com]

Sent: Friday, April 16, 2010 1:05 PM

To: Arshad Mahmood (arshadm)

Cc: arsham@cisco.com

Subject: From router agam-3845:

 

TRIGGERED_BY "auto-trace watchdog timer every 60 minutes.":: SERV-DIAG_CFM_autotrace: A total of 4 MEPS for valid MAs specified in ma_test have been automatically tracerouted.  Results stored in the CFM traceroute cache.

 

CFM Domain CUST1

   MA SID_3 - vlan 800

      mpid 4200, mac address 0024.14f6.80c1

   MA SID_4 - vlan 801

      mpid 4201, mac address 0024.14f6.80c1

   MA SID_1 - vlan 700

      mpid 42, mac address 0024.14f6.80c1

   MA SID_2 - vlan 701

      mpid 420, mac address 0024.14f6.80c1

 

THE CONFIGURATION HISTORY is:

!Contextual Config Diffs:

+no logging buffered

+ethernet cfm traceroute cache hold-time 60

+event manager environment _svcdiag_cfmShutNotif EMAIL,SYSLOG,SNMP

+event manager environment _svcdiag_cfmShutConfig TRUE

+event manager environment _svcdiag_cfmShutEvent TRUE

+event manager environment _svcdiag_cfmShutCommand TRUE

+event manager environment _svcdiag_cfmShutPrepend TRUE

+event manager environment _svcdiag_cfmShutMAFile ma_list

+event manager environment _svcdiag_cfmShutP2P TRUE

+event manager environment _email_cc arshadm@cisco.com

-logging buffered 20000

-ethernet cfm traceroute cache hold-time 120

ethernet cfm ais link-status global

 -disable

-event manager environment _email_cc arsham@cisco.com

-event manager environment _svcdiag_cfmAutotraceNotif EMAIL,SYSLOG

-event manager environment _svcdiag_cfmAutotraceConfig TRUE

-event manager environment _svcdiag_cfmAutotraceEvent TRUE

-event manager environment _svcdiag_cfmAutotraceCommand TRUE

-event manager environment _svcdiag_cfmAutotracePrepend TRUE

-event manager environment _svcdiag_cfmAutotraceMAFile ma_test

-event manager environment _svcdiag_cfmAutotraceTimer 3600

-event manager policy cfm_autotrace.tcl type user

 

agam-3845#

--------------------------------------------------------------

THE EVENT HISTORY is:

Syslog logging: enabled (0 messages dropped, 4 messages rate-limited, 0 flushes, 0 overruns, xml disabled, filtering disabled)

 

    Console logging: level debugging, 1856 messages logged, xml disabled,

                     filtering disabled

    Monitor logging: level debugging, 0 messages logged, xml disabled,

                     filtering disabled

    Buffer logging:  level debugging, 194 messages logged, xml disabled,

                    filtering disabled

    Exception Logging: size (4096 bytes)

    Count and timestamp logging messages: disabled

    Persistent logging: disabled

 

No active filter modules.

 

    Trap logging: level informational, 885 message lines logged

 

Log Buffer (20000 bytes):

*Apr 15 17:15:31.913 EST: %SYS-5-CONFIG_I: Configured from console by console

*Apr 15 17:20:55.697 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 721 level 7 VLAN 801 dir D Interface Gi0/1 exited AIS defect condition

*Apr 15 17:20:55.697 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 4201 vlan 801 MA name SID_4 domain CUST1 interface status Up event code Returning.

*Apr 15 17:20:55.953 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 7000 level 7 VLAN 700 dir D Interface Gi0/1 exited AIS defect condition

*Apr 15 17:20:55.953 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 42 vlan 700 MA name SID_1 domain CUST1 interface status Up event code Returning.

*Apr 15 17:20:58.513 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 720 level 7 VLAN 800 dir D Interface Gi0/1 exited AIS defect condition

*Apr 15 17:20:58.513 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 4200 vlan 800 MA name SID_3 domain CUST1 interface status Up event code Returning.

*Apr 15 17:20:59.281 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 402 level 7 VLAN 701 dir D Interface Gi0/1 exited AIS defect condition

*Apr 15 17:20:59.281 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 420 vlan 701 MA name SID_2 domain CUST1 interface status Up event code Returning.

*Apr 15 17:22:22.017 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut.

*Apr 15 17:22:22.017 EST: %E_CFM-6-ENTER_AIS: local mep with mpid 7000 level 7 VLAN 700 dir D Interface Gi0/1 enters AIS defect condition

*Apr 15 17:22:22.749 EST: %HA_EM-6-LOG: cfm_cctimeout.tcl: WARNING: Configuration, Event, Command history cannot be sent along with syslog/snmp notification

*Apr 15 17:22:24.577 EST: %E_CFM-3-FAULT_ALARM: A fault has occurred in the network for the local MEP having mpid 7000 vlan 700 for service MA name SID_1 with the event code DefRemoteCCM.

*Apr 15 17:22:56.765 EST: %HA_EM-6-LOG: cfm_cctimeout.tcl: TRIGGERED_BY "*Apr 15 17:22:22.017 EST: %E_CFM-3-REMOTE_MEP_DOWN: Remote MEP mpid 42 vlan 700 MA name SID_1 in domain CUST1 changed state to down with event code TimeOut. ":: SERV-DIAG_CFM_timeout: Diagnostics for rmep 0024.14f6.80c1 on vlan 700, CFM MD CUST1.CCDB: 0 found; ErrorDB: 1 found; Reasons: Timeout; Ping: fail; Trace: fail

*Apr 15 17:23:04.917 EST: %HA_EM-6-LOG: cfm_cctimeout.tcl:

Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details

--------------------------------------------------------------

*Apr 15 17:23:13.937 EST: %E_CFM-6-EXIT_AIS: local mep with mpid 7000 level 7 VLAN 700 dir D Interface Gi0/1 exited AIS defect condition

*Apr 15 17:23:13.941 EST: %E_CFM-6-REMOTE_MEP_UP: Continuity Check message is received from a remote MEP with mpid 42 vlan 700 MA name SID_1 domain CUST1 interface status Up event code Returning.

*Apr 15 18:04:13.657 EST: %SYS-5-CONFIG_I: Configured from console by console

*Apr 15 18:08:03.785 EST: %HA_EM-6-LOG: cfm_ondemand.tcl: TRIGGERED_BY "Ondemand script run manually":: SERV-DIAG_CFM_ondemand: Diagnostics for vlan 700, CFM MD CUST1, rmep:all. CCDB: 1 found, 1 with UP IF state; ErrorDB: 0 found; Reasons: Ping: 1/0 total/fail;

*Apr 15 18:08:12.025 EST: %HA_EM-6-LOG: cfm_ondemand.tcl:

Email has been sent to arshadm@cisco.com, please check your email box for diagnostic details

THE COMMAND HISTORY is:

 

Fri Apr 16 12:04:41 EST 2010 show event manager environ

 

Fri Apr 16 12:04:41 EST 2010 show ethernet cfm traceroute-cache

References

Cisco Embedded Event Manager Overview: http://www.cisco.com/en/US/docs/ios/netmgmt/configuration/guide/nm_eem_overview_ps6441_TSD_Products_Configuration_Guide_Chapter.html

Cisco EEM Cisco.com webpage: http://www.cisco.com/en/US/products/ps6815/products_ios_protocol_group_home.html

Cisco Service Diagnostics Cisco.com webpage : http://www.cisco.com/en/US/products/ps9424/products_ios_protocol_group_home.html

Cisco Service Diagnostics Q&A: http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6555/ps9424/qa_cisco_ios_service_diagnostics.html

Cisco Border Gateway Protocol, OSPF, and QoS Diagnostics Scripts: http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6555/ps9424/white_paper_cisco_ios_service_design_bgp_osp_qos.html

Cisco Service Diagnostics CFM 2.0 Scripts: http://www.cisco.com/en/US/prod/collateral/iosswrel/ps6537/ps6555/ps9424/whitepaper_c11-566741.html

CFM in Service Provider Network Configuration Guide: http://www.cisco.com/en/US/docs/ios/cether/configuration/guide/ce_cfm-ieee.html

Service Diagnostics Documentation and Scripts: http://www.cisco.com/go/iossd