Cisco IOS Service Diagnostics: Border Gateway Protocol, Open Shortest Path First and Quality of Service Scripts [Cisco IOS Service Diagnostics]

Last Updated: July 2008.

The Service Diagnostics feature provides a bundled set of Tool Command Language (Tcl) scripts and Embedded Event Manager (EEM) policies written and tested by subject matter experts to facilitate diagnosing common networking issues in the areas of Border Gateway Protocol (BGP), Open Shortest Path First (OSPF), and Quality of Service (QoS). A new feature called Embedded Menu Manager (EMM)-available in Cisco IOS Software Release 12.4(20)T and later Cisco IOS images-may be used to guide the user in installing and deploying these scripts and policies.

Definitions

Feature	Description
API	Application Programming Interface
CLI	Command Line Interface
EEM	Embedded Event Manager
EMM	Embedded Menu Manager
ERM	Embedded Resource Manager
ESM	Embedded Syslog Manager
IFS	Cisco IOS File System
MDF	Menu Definition File
NA	Not Applicable
IOS	Internetwork Operating System
TBD	To Be Determined
TBS	To Be Supplied (at a later time)
TCL	Tool Command Language

Introduction to Service Diagnostics

The concept behind Service Diagnostics is to automate some of the vast troubleshooting experiences of Cisco engineers by using the existing scripting capabilities and embedded management tools in Cisco IOS. Cisco has been adding and enhancing such tools as EEM, Embedded Syslog Manager (ESM), and Embedded Resource Manager (ERM) to Cisco IOS over the past few years. This feature is meant to be the "glue" that combines one or more of these tools to automate common diagnostic scenarios.

The goal is to isolate the end user from the rigors of Tcl scripting, and/or EEM policy writing, and provide a simple interface for deploying and receiving feedback from scenario-specific troubleshooting scripts. Service Diagnostics provides CLI Tcl shell user interfaces as well as EMM Menu Definition Files (MDF's) for deploying troubleshooting scenarios. The scripts are posted on the Cisco Beyond Website under the Diagnostic category http://www.cisco.com/go/ciscobeyond.

Deploying Service Diagnostic Scenarios

Installing Scripts

If your image has the EMM feature, it is much easier to use the MDF vs. the ZIP file. If not, you must use the tclsh helper scripts.

Installation Using tclsh CLI parser mode

1. Download the diagnostic scripts from Cisco Beyond Website: http://www.cisco.com/go/ciscobeyond

Scripts will be organized into diagnostic scenarios, ie: sdiag_bgp_1.0.zip

2. Unzip the scenario archive.

3. On the router's disk create a directory to store all the scripts.

4. Example:

Router# mkdir svc-diag

5. The files to be copied to the dir svc-diag (per scenario) are:

BGP

· sdiag_bgp.tcl

· bgp_neighbor_loss.tcl

· bgp_neighbor_formation.tcl

· bgp_neighbor_route.tcl

· sdiag_bgp_display.tcl

· sdiag_bgp_undeploy.tcl

· collectEmailParameters.tcl

OSPF

· ospf_adj_duplicate_rid.tcl

· ospf_intf_down_detached.tcl

· ospf_miss_area_id.tcl

· ospf_stuck_cases.tcl

· ospf_timer_expired.tcl

· sdiag_ospf.tcl

· sdiag_ospf_display.tcl

· sdiag_ospf_undeploy.tcl

QoS

· sdiag_qos.tcl

· sdiag_qos_display.tcl

· sdiag_qos_undeploy.tcl

· qos_drop_packet.tcl

Resource (CPU, memory, buffer monitoring)

· sdiag_resource.tcl

· sdiag_resource_display.tcl

· sdiag_resource_undeploy.tcl

· collectEmailParameters.tcl

6. copy tftp disk#:/svc-diag

7. The router will prompt for :-

a. Address or name of remote host []?

b. Source filename []? Give the full path where the script has to be copied from

c. Destination filename []?Give the path as disk#:/svc-diag/filename.tcl

8. Create a directory to store the user libraries and the template to send email

Example:Router# mkdir user_library

9. Copy the following support files from tftp to disk following the same procedure as above to the user_library:

BGP

· email_template

· userlib.tcl

· diag_lib.tcl

OSPF

· email_template_cmd

· email_setup.tcl

· diag_lib.tcl

· tclIndex

· lib.tcl

· ospf_lib.tcl

· acl_lib.tcl

QOS

· email_template_cmd

· email_setup.tcl

· diag_lib.tcl

· tclIndex

· lib.tcl

Resource (CPU, memory, buffer monitoring)

· diag_lib.tcl

· userlib.tcl

· email_template

Note: The Diagnostic Tcl scripts posted on Cisco Beyond will be digitally signed. If your Cisco IOS router image supports Tcl script signing, you may use this feature to verify the integrity of the downloaded scripts. For more information please see the topic "Signed Tcl Scripts" here http://www.cisco.com/en/US/products/ps6441/products_feature_guide09186a00808d65fe.html.

Installation Using EMM

1. Download the MDF containing the diagnostic scripts from Cisco Beyond Website: http://www.cisco.com/go/ciscobeyond

e.g., sdiag_bfp.mdf

Note: The MDF contains all the scripts and policy files needed for a scenario. It is like a ZIP, but also contains menu definitions to guide the user.

2. Copy the MDF to a file system that the router has access to, e.g. local disk:, flash:, or tftp: server.

3. From privileged exec-mode, launch EMM

Router# emm mdf tftp://my_tftp_server/sdiag_bgp.mdf

You will be presented with a main menu as follows:

=====================================================================

BGP Diagnostics

Enter ? for help or ?# for item help

---------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Setup EEM Environment

3. Deploy BGP Neighbor Loss Diagnostic Script

4. Deploy BGP Neighbor Formation Problem Diagnostic Script

5. Deploy BGP Route Problem Diagnostic Script

6. Deploy All BGP Scripts

7. Remove Diagnostic Policies

8. Display Diagnostic Policy Configuration

9. Exit

Enter selection [9]:

4. Press the number "1" (no Enter key is needed). You will be prompted for directories for the EEM user library and user policies as follows:

Enter ? for help

Enter a directory to store the BGP diagnostic policies in the form of a URL

(excluding filename, e.g. disk0:/svc_diag

Enter value [disk0:/svc_diag]:

Enter ? for help

Enter a directory for the user library files in the form of a URL

(excluding filename, e.g. disk0:/user_lib

Enter value [disk0:/user_lib]:

Note: The MDF will query the router's available file systems and present a default directory that had sufficient free space to contain the diagnostic scripts and policies. Press the "Enter" key to accept the default.

Deploying Scenarios

Deploy the script via tclsh with parameters "notification" "configuration history option" "event history option" "user policy directory" "user library directory" where:

The value for notification can be "email or syslog or all"

The value for configuration history option, event history option can be "TRUE or FALSE"

The value for user policy and user library directories is the respective full path where the scripts and library files are stored

For each scenario, the following sections document an example command line using tclsh.

BGP Neighbor Loss

Router#tclsh disk#:/sdiag_bgp.tcl bgp_neighbor_loss.tcl email TRUE TRUE disk2:/ disk2:/user_lib

where

sdiag_bgp.tcl is the registration script

bgp_neighbor_loss.tcl is the EEM policy script

the notification is email

the configuration history option is set to TRUE

the event history option is set to TRUE

the user policy directory is disk2:

the user library directory is disk2:/user_lib

BGP Neighbor Formation:

Router#tclsh disk#:/sdiag_bgp.tcl bgp_neighbor_formation.tcl email TRUE TRUE disk2:/ disk2:/user_lib 120

where

sdiag_bgp.tcl is the registration script

bgp_neighbor_formation.tcl is the EEM policy script

the notification is email

the configuration history option is set to TRUE

the event history option is set to TRUE

the user policy directory is disk2:

the user library directory is disk2:/user_lib

the timer value is 120.

BGP Neighbor Route:

Router#tclsh disk#:sdiag_bgp.tcl bgp_neighbor_route.tcl email TRUE TRUE disk#: disk#:/user_lib 120

where

sdiag_bgp.tcl is the registration script

bgp_neighbor_route.tcl is the EEM policy script

the notification is email

the configuration history option is set to TRUE

the event history option is set to TRUE

the user policy directory is disk2:

the user library directory is disk2:/user_lib

the timer value is 120.

QoS

tclsh disk0:sdiag_qos.tcl qos_drop_packet.tcl syslog FALSE FALSE disk0: inputfile disk0:/drop_file disk0:/svc_diag disk0:/user_lib

OSPF Stuck Cases

tclsh disk0:sdiag_ospf.tcl ospf_stuck_cases.tcl email TRUE TRUE disk0:/svc_diag disk0:/user_lib

OSPF Timer Expired

tclsh disk0:sdiag_ospf.tcl ospf_timer_expired.tcl email TRUE TRUE disk0:/svc_diag disk0:/user_lib

CPU Resource

tclsh <disk#:>/<dir_name>/sdiag_resource.tcl <cpu> <notification> <configurationHistory> <EventHistory> <user_pol_dir> <user_lib_dir> <process_name or interrupt> <high_threshold> <low_threshold> <time_interval>

Memory Resource

tclsh <disk#:>/<dir_name>/sdiag_resource.tcl <memory> <notification> <configurationHistory> <EventHistory> <user_pol_dir> <user_lib_dir> <process_name> <high_threshold> <low_threshold> <time_interval> <resource_allocation>

Buffer Resource

tclsh <disk#:>/<dir_name>/sdiag_resource.tcl <buffer> <notification> <configurationHistory> <EventHistory> <user_pol_dir> <user_lib_dir> <process_name> <high_threshold> <low_threshold> <time_interval> <resource_allocation>

Deployment Using EMM

1. After installation, select menu item "2" and answer the questions regarding EEM environment variables, e.g. _email_to, _email_server, etc.

=====================================================================

BGP Diagnostics

Enter ? for help or ?# for item help

---------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Setup EEM Environment

3. Deploy BGP Neighbor Loss Diagnostic Script

4. Deploy BGP Neighbor Formation Problem Diagnostic Script

5. Deploy BGP Route Problem Diagnostic Script

6. Deploy All BGP Scripts

7. Remove Diagnostic Policies

8. Display Diagnostic Policy Configuration

9. Exit

Enter selection [9]:

2. After setting up the EEM environment, select the menu item corresponding to the desired diagnostic. You will be prompted to select the type of reporting desired:

Select the type of notification to be sent when a problem is diagnosed

1. email

2. syslog

Enter choice:

If you select email reporting, you may add more information to the report, such as configuration change history:

Do you want to log configuration history?

1. Yes

2. No

Enter choice:

Displaying Inputs

To display the inputs that have been given, invoke the file:

BGP:

· Router#tclsh disk#:/svc-diag/sdiag_bgp_display.tcl bgp_neighbor_loss.tcl (OR)

where

sdiag_bgp_display.tcl is the script that displays the BGP inputs on the router related to the BGP- EEM policy script passed as the argument

bgp_neighbor_loss.tcl is the BGP neighbor loss policy script

· Router#tclsh disk#:/svc-diag/sdiag_bgp_display.tcl bgp_neighbor_formation.tcl (OR)

where

sdiag_bgp_display.tcl is the script that displays the BGP inputs on the router related to the BGP- EEM policy script passed as the argument

bgp_neighbor_formation.tcl is the BGP neighbor formation policy script

· Router#tclsh disk#:/svc-diag/sdiag_bgp_display.tcl bgp_neighbor_route.tcl

where

sdiag_bgp_display.tcl is the script that displays the BGP inputs on the router related to the BGP- EEM policy script passed as the argument

bgp_neighbor_route.tcl is the BGP route policy script

Resource

· tclsh disk#:/dir_name/sdiag_resource_display.tcl <cpu>

· tclsh disk#:/dir_name/sdiag_resource_display.tcl <memory>

· tclsh disk#:/dir_name/sdiag_resource_display.tcl <buffer>

Display Status Using EMM

Simply select the menu item entitled "Display ..." In the sample BGP MDF below, type the number "8".

====================================================================

BGP Diagnostics

Enter ? for help or ?# for item help

---------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Setup EEM Environment

3. Deploy BGP Neighbor Loss Diagnostic Script

4. Deploy BGP Neighbor Formation Problem Diagnostic Script

5. Deploy BGP Route Problem Diagnostic Script

6. Deploy All BGP Scripts

7. Remove Diagnostic Policies

8. Display Diagnostic Policy Configuration

9. Exit

Enter selection [9]:

Removing Scenarios

To completely remove all the inputs related to a scenario, invoke the file:

BGP:

· Router#tclsh disk#:/svc-diag/sdiag_bgp_undeploy.tcl bgp_neighbor_loss.tcl (OR)

where

sdiag_bgp_undeploy.tcl is the script that removes the BGP inputs on the router related to the BGP- EEM policy script passed as the argument

bgp_neighbor_loss.tcl is the BGP neighbor loss policy script

· Router#tclsh disk#:/svc-diag/sdiag_bgp_undeploy.tcl bgp_neighbor_formation.tcl (OR)

where

sdiag_bgp_undeploy.tcl is the script that removes the BGP inputs on the router related to the BGP- EEM policy script passed as the argument

bgp_neighbor_formation.tcl is the BGP neighbor formation policy script

· Router#tclsh disk#:/svc-diag/sdiag_bgp_undeploy.tcl bgp_neighbor_route.tcl

where

sdiag_bgp_undeploy.tcl is the script that removes the BGP inputs on the router related to the BGP- EEM policy script passed as the argument

bgp_neighbor_route.tcl is the BGP neighbor formation policy script

QoS

Router#tclsh disk#:/svc-diag/sdiag_qos_undeploy.tcl qos_drop_packet.tcl

OSPF

Router#tclsh disk#:/svc-diag/sdiag_ospf_undeploy.tcl ospf_stuck_cases.tcl

Resource:

FOR CPU

tclsh <disk#:>/<dir_name>/ <sdiag_resource_undeploy.tcl> <cpu> <process name or interrupt>

FOR MEMORY

tclsh <disk#:>/<dir_name>/ <sdiag_resource_undeploy.tcl> <memory> <process name>

FOR BUFFER

tclsh <disk#:>/<dir_name>/ <sdiag_resource_undeploy.tcl> <buffer> <process name>

Removal Using EMM

Simply select the menu item entitled "Remove Diagnostic Policies". In the sample BGP MDF below, type the number "7".

=====================================================================

BGP Diagnostics

Enter ? for help or ?# for item help

---------------------------------------------------------------------

1. Install Diagnostic Scripts

2. Setup EEM Environment

3. Deploy BGP Neighbor Loss Diagnostic Script

4. Deploy BGP Neighbor Formation Problem Diagnostic Script

5. Deploy BGP Route Problem Diagnostic Script

6. Deploy All BGP Scripts

7. Remove Diagnostic Policies

8. Display Diagnostic Policy Configuration

9. Exit

Enter selection [9]:

BGP Diagnostics

BGP Neighbor Loss

Triggers/Symptoms

Syslog message on the router indicating that the BGP neighbor x.x.x.x going from UP to DOWN.

The reasons could be:

1. The interface on the neighbor or the local router is shut.

2. Access-list is configured on the local or the neighbor router that is explicitly or implicitly denying the tcp, ip or udp packets from the other.

3. Any BGP configuration on the neighbor router is removed.

Actions/Outputs

We will see one of the following messages:

*Oct 16 09:34:09.904: %HA_EM-6-LOG: tmpsys:/eem_policy/bgp_neighbor_loss.tcl:

THE SERVICE DIAGNOSTIC MESSAGE FOR BGP NEIGHBOR LOSS IS:

Neighbor x.x.x.x has gone down and does not seem to be reachable through ping. Check network connectivity.

*Oct 16 09:34:09.904: %HA_EM-6-LOG: tmpsys:/eem_policy/bgp_neighbor_loss.tcl:

THE SERVICE DIAGNOSTIC MESSAGE FOR BGP NEIGHBOR LOSS IS:

Neighbor x.x.x.x has gone down. However, configured neighbor is reachable through ping. Check BGP configuration at the peer and any access-list restrictions between the peers.

*Oct 16 09:34:09.904: %HA_EM-6-LOG: tmpsys:/eem_policy/bgp_neighbor_loss.tcl:

THE SERVICE DIAGNOSTIC MESSAGE FOR BGP NEIGHBOR LOSS IS:

Neighbor x.x.x.x has gone down and does not seem to be reachable through ping. Check network connectivity, BGP configuration at peer and any access-list restrictions between the peers

BGP Neighbor Formation Problem

Triggers/Symptoms

"Show ip bgp neighbor" reflects BGP neighbor remains at ACTIVE, IDLE or CONNECT. The reasons could be:

1. The interface on the local or neighbor router is down.

2. Access-list is configured on the local or the neighbor router that is explicitly or implicitly denying the tcp, ip or udp packets from the other.

3. For iBGP neighbor "update-source Loopback" is not configured OR is misconfigured on the local or neighbor router.

4. For eBGP neighbor "update-source Loopback" or "ebgp-multihop is not configured OR is misconfigured on the local or neighbor router.

5. For indirectly connected eBGP or iBGP neighbors the static route or ip routing between the peers may be missing.

Actions/Outputs

We will see one of the following messages: