Intelligent Services Gateway Configuration Guide, Cisco IOS XE Release 3S
Configuring ISG Troubleshooting Enhancements
Downloads: This chapterpdf (PDF - 1.32MB) The complete bookPDF (PDF - 5.22MB) | The complete bookePub (ePub - 1.6MB) | Feedback

Configuring ISG Troubleshooting Enhancements

Configuring ISG Troubleshooting Enhancements

The Intelligent Services Gateway (ISG) debugging enhancements enable you to more easily isolate issues with ISG subscriber sessions in a production network, such as a session getting stuck in a dangling state (never reaches the established state). The troubleshooting enhancements described in this module allow you to diagnose these issues by introducing expanded statistics collection and event tracing.

Restrictions for ISG Troubleshooting Enhancements

Enabling the subscribertracehistory command does not mean that all event traces for subscriber sessions are stored in the history buffer. Event traces for the DPM and PM are written to their respective history buffers only if a session has an issue (such as a session that becomes a dangling session).

Information About ISG Troubleshooting Enhancements

DPM Statistics

The DHCP policy module (DPM) has many complex interactions with other components such as the policy manager (PM) module, which can make it difficult to troubleshoot issues and find the root cause. For example, dangling IP sessions are often caused by an error in the interactions between DHCP, the DPM, and the PM, resulting in DPM sessions getting stuck in a startup state.

The DPM enhancements enable more efficient debugging of issues with DPM and its interactions in customer networks. This includes improved statistics collection at both the system and session level to show failure data, more comprehensive error messages, and event tracing.

DPM contexts could previously be displayed only by selecting the MAC address. These enhancements add the ability to search and display all DPM contexts on the router, DPM contexts for a particular client IP address, or DPM contexts in a particular state. All debug output is now prefixed with the DPM context identifier and MAC address so that in a large-scale scenario you can identify debugs for a particular transaction.

Event Tracing for Subscriber Sessions

When trying to reproduce or capture customer issues, collecting debug output is not always practical or even possible. Network administrators often do not detect an error until long after the event that caused the error has occurred. By the time a fault is detected, it is usually too late to enable debug commands because the session is already in an error state, or the session was terminated because of an error.

Event tracing allows you to capture traces for existing sessions on the router and to retain the history of any past sessions that were marked as interesting, such as a session that became stuck in a dangling state. This enables you to look at existing sessions, as well as past sessions, and review the data after the session gets into an unexpected state or never comes up.

If a session is marked as interesting, its event trace information is sent to a history log, if history logging is enabled. A session is considered interesting if it becomes stuck in a state, enters an error state, or terminates without transitioning into a target state, because of a programming error, end-user action, packet drop, or other reason. The decision whether to log an event trace is determined by the after-the-fact status of the object. Event traces for uninteresting sessions are removed to free up space in the history log buffer.

Event tracing is supported by the DPM and PM modules. Each module logs event traces for each of its session contexts independently. The event trace data for each subscriber session is attached to its session context. Previously, this data was purged when the session was terminated. These enhancements preserve the event trace data even after the sessions are gone.

Each session context that supports event trace creates a new event trace log to hold the event traces for that session context. The new event log is created at session startup or teardown, and is destroyed after the session reaches the established or destroyed state. The event trace logs can be displayed independently through show commands.

How to Enable ISG Troubleshooting Enhancements

Enabling Event Tracing for ISG Sessions

Perform the following steps to enable event tracing for ISG subscriber sessions.

SUMMARY STEPS

    1.    enable

    2.    configure terminal

    3.    subscriber trace event {dpm | pm} [retain]

    4.    subscriber trace history {dpm | pm} [sizemax-records]

    5.    exit


DETAILED STEPS
     Command or ActionPurpose
    Step 1 enable


    Example:
    Router> enable
     

    Enables privileged EXEC mode.

    • Enter your password if prompted.
     
    Step 2 configure terminal


    Example:
    Router# configure terminal
     

    Enters global configuration mode.

     
    Step 3 subscriber trace event {dpm | pm} [retain]


    Example:
    Router(config)# subscriber trace event dpm retain
     

    Enables event tracing for ISG subscriber sessions.

    • This command, without the retain keyword, is enabled by default for the DPM and PM.
     
    Step 4 subscriber trace history {dpm | pm} [sizemax-records]


    Example:
    Router(config)# subscriber trace history dpm size 200
     

    Enables saving the event traces for ISG subscriber sessions to a history log.

    • This command, without the sizekeyword, is enabled by default for the DPM and PM. Default log size is 100 records.
     
    Step 5 exit


    Example:
    Router(config)# exit
     

    Exits global configuration mode and returns to privileged EXEC mode.

     

    Displaying Event Traces for ISG Sessions

    Use the following commands to display information about the event traces that are saved to the history log for the specified module. You can use these commands in any order.

    SUMMARY STEPS

      1.    show subscriber trace statistics

      2.    show subscriber trace history {all | dpm | pm} [all | client-ip-addressip-address | mac-addressmac-address | reasonnumber | uidsession-id]

      3.    clear subscriber trace history {dpm | pm}


    DETAILED STEPS
      Step 1   show subscriber trace statistics

      Use this command to display statistics about the event traces that were saved to the history log.



      Example:
      Router# show subscriber trace statistics
      Event Trace History Statistics: DPM
      Logging enabled
      All time max records: 5
      Max records: 5
      Current records: 5
      Current log size: 200
      Proposed log size 200
      Oldest, newest index: 0 : 4
      Event Trace History Statistics: Policy Manager
      Logging enabled
      All time max records: 4
      Max records: 4
      Current records: 4
      Current log size: 64
      Proposed log size 64
      Oldest, newest index: 0 : 3
      
      Step 2   show subscriber trace history {all | dpm | pm} [all | client-ip-addressip-address | mac-addressmac-address | reasonnumber | uidsession-id]

      Use this command to display a summary of all session traces stored in the trace history log, or to display a complete trace for a specific session by using one of the optional keywords. The following example shows output for the client with IP address 10.0.0.2.



      Example:
      Router# show subscriber trace history dpm client-ip-address 10.0.0.2
      DPM session info: 5CB2A60
      MAC: aaaa.2222.cccc IP: 10.0.0.2 
      UID: 3 reason: PM callback to clear
      =========================
      ET 11:33:48.767 PST Mon Aug 30 2010 dhcp get class
             rc no c-aware cfg
      ET 11:34:07.327 PST Mon Aug 30 2010 i-if change
             ,MAC ok,ignore: same i/f
      ET 11:34:07.327 PST Mon Aug 30 2010 dhcp discover
             rc OK,proc prev req
      ET 11:34:07.327 PST Mon Aug 30 2010 dhcp get class
             rc no c-aware cfg
      ET 11:34:10.835 PST Mon Aug 30 2010 i-if change
             ,MAC ok,ignore: same i/f
      ET 11:34:10.835 PST Mon Aug 30 2010 dhcp discover
             rc OK,proc prev req
      ET 11:34:10.835 PST Mon Aug 30 2010 dhcp get class
             rc no c-aware cfg
      ET 11:34:14.843 PST Mon Aug 30 2010 i-if change
             ,MAC ok,ignore: same i/f
      ET 11:34:14.843 PST Mon Aug 30 2010 dhcp discover
             rc OK,proc prev req
      ET 11:34:14.843 PST Mon Aug 30 2010 dhcp get class
             rc no c-aware cfg
      ET 11:34:38.391 PST Mon Aug 30 2010 i-if change
             ,MAC ok,ignore: same i/f
      ET 11:34:38.391 PST Mon Aug 30 2010 dhcp discover
             rc OK,proc prev req
      ET 11:34:38.391 PST Mon Aug 30 2010 dhcp get class
             rc no c-aware cfg
      ET 11:34:41.923 PST Mon Aug 30 2010 i-if change
             ,MAC ok,ignore: same i/f
      ET 11:34:41.923 PST Mon Aug 30 2010 dhcp discover
             rc OK,proc prev req
      ET 11:34:41.923 PST Mon Aug 30 2010 dhcp get class
             rc no c-aware cfg
      ET 11:34:45.931 PST Mon Aug 30 2010 i-if change
             ,MAC ok,ignore: same i/f
      ET 11:34:45.931 PST Mon Aug 30 2010 dhcp discover
             rc OK,proc prev req
      ET 11:34:45.931 PST Mon Aug 30 2010 dhcp get class
             rc no c-aware cfg
      ET 11:35:13.591 PST Mon Aug 30 2010 PM callback
             Terminate, rc end sess,Case: REQ_TERMINATE
      
      Step 3   clear subscriber trace history {dpm | pm}

      Use this command to clear the event trace history log for the specified module.



      Example:
      Router# clear subscriber trace history dpm
      

      Displaying DPM Statistics

      Use the following commands to display information about event traces for DPM session contexts.

      SUMMARY STEPS

        1.    debug subscriber policy dpm timestamps

        2.    show subscriber policy dpm context [ip-addressip-address | mac-addressmac-address | statestate] [extended]

        3.    show subscriber policy dpm statistics

        4.    clear subscriber policy dpm statistics


      DETAILED STEPS
        Step 1   debug subscriber policy dpm timestamps

        Use this command to include timestamp information for DPM messages in debugging output.



        Example:
        Router# debug subscriber policy dpm timestamps
        SG dhcp message timestamps debugging is on
        
        Step 2   show subscriber policy dpm context [ip-addressip-address | mac-addressmac-address | statestate] [extended]

        Use this command to display event traces for DPM session contexts. The following example shows output for the client with the IP address 10.0.0.2.



        Example:
        Router# show subscriber policy dpm context ip-address 10.0.0.2
        ==================== DHCP Context Summary ==================
        Total DHCP Session Count: 1
        -------------------------------------------------------------
        DHCP Context Entry #1, 0x45FD538
             dhcp handle: 12000002
             sss handle: 0
             ip address: 10.0.0.2, subnet: 255.255.255.0
             sdb handle: 3877F6A
             policy handle: D7000002
             callback: 0, started: 1, terminate: 1, class: 0
             default config: 1, influence vrf: 0
             classname: NULL, default classname: NULL
             default vrf id: NULL, vrf id: NULL
             mac address: aaaa.2222.cccc
             dpm state: session-configured
             dpm sub-state: classname-received
             remote id: NULL, circuit id: NULL
             ip sess hdl: 81000002
             restart: 0, update: 0
             shdb hdl: 0
             standby: 0, sync ready: 1
             req for classname received time: 11:43:55.403 PST Mon Aug 30 2010
             offer notif received time: 11:46:05.959 PST Mon Aug 30 2010
             DHCP session setup time: 2s20ms
        ET 11:46:03.959 PST Mon Aug 30 2010 PM invoke
               rc OK, Session-Start
        ET 11:46:03.959 PST Mon Aug 30 2010 dhcp discover
               rc OK,No Sess,sess alloc,sess-start OK
        ET 11:46:03.959 PST Mon Aug 30 2010 dhcp discover
               rc OK,proc prev req
        ET 11:46:03.959 PST Mon Aug 30 2010 dhcp get class
               rc no c-aware cfg
        ET 11:46:03.975 PST Mon Aug 30 2010 PM callback
               Got Keys, rc dhcp wait no cb,upd msi vrf=0,Case: GOT_KEYS
        ET 11:46:05.959 PST Mon Aug 30 2010 PM invoke
               rc OK, Session-Update
        ET 11:46:05.959 PST Mon Aug 30 2010 dhcp offer
               rc OK w delay,acc.if ret
        ET 11:46:05.983 PST Mon Aug 30 2010 PM callback
               Session Update Succes, rc offer cb no-err,notify stdby,Case: UPDATE_SUCCESS
        ET 11:46:05.987 PST Mon Aug 30 2010 dhcp discover
               rc OK,proc prev req
        ET 11:46:05.991 PST Mon Aug 30 2010 i-if change
               ,MAC ok,ignore: same i/f
        ET 11:46:05.995 PST Mon Aug 30 2010 dhcp assign OK
               rc same IP
        ------------------------------------------------------
        Minimum DHCP set up time among all contexts : 2s20ms
        Maximum DHCP set up time among all contexts : 2s20ms
        MAC address for Max Setup Time among all contexts: aaaa.2222.cccc 
        
        Step 3   show subscriber policy dpm statistics

        Use this command to display statistics for DPM session contexts.



        Example:
        Router# show subscriber policy dpm statistics
                 Message Received     Duplicate    Ignored      Total
            Discover Notification  :        284          0        291
               Offer Notification  :          0          0          2
         Address Assignment Notif  :          2          0          2
           DHCP Classname request  :          0        290        290
              Input Intf Override  :          0         10        293
          Lease Termination Notif  :          0          0          2
          Session Restart Request  :          0          0          0
        Response to DHCP request for classname
        Average Time : Max Time : 
        MAC address for Max Time : 
        Response to DHCP Offer Notification
        Average Time : 30ms Max Time : 36ms
        MAC address for Max Time : aaaa.2222.cccc
        Overall since last clear
        Total Discover Init Sessions : 2
        Total Restarted Sessions : 0
        Average set up time for Discover initiated sessions : 2s26ms
        Min set up time among Discover initiated sessions : 2s20ms
        Max set up time among Discover initiated sessions : 2s32ms
        Current active Sessions
        Total Discover Init Sessions : 0
        Total Restarted Sessions : 0
        Average set up time for Discover initiated sessions : 
        Min set up time among Discover initiated sessions: 2s20ms
        Max set up time among Discover initiated sessions : 
        MAC of session with Max DHCP Setup Time : aaaa.2222.cccc
        Total number of DPM contexts allocated : 7
        Total number of DPM contexts freed : 6
        Total number of DPM contexts currently without session : 1
          Elapsed time since counters last cleared : 2h15m20s  
        
        Step 4   clear subscriber policy dpm statistics

        Use this command to clear the statistics for DPM contexts.



        Example:
        Router# clear subscriber policy dpm statistics
        

        Configuration Examples for ISG Troubleshooting Enhancements

        Example Enabling Event Tracing for ISG Sessions

        The following example shows a configuration with the DPM and PM event tracing enabled and retained. Trace history logging is enabled for the DPM and PM so traces for sessions that are marked as interesting are stored in their respective history log. Up to 100 sessions (default) can be stored in the PM history log, and up to 200 sessions can be stored in the DPM history log.

        subscriber trace event dpm retain
        subscriber trace event pm retain
        subscriber trace history pm
        subscriber trace history dpm size 200

        Additional References

        Related Documents

        Related Topic

        Document Title

        Cisco IOS commands

        Cisco IOS Master Commands List, All Releases

        Debug commands

        Cisco IOS Debug Command Reference.

        DHCP Configuration

        Part 3, “DHCP,” IP Addressing Configuration Guide.

        ISG commands

        Cisco IOS Intelligent Services Gateway Command Reference

        ISG subscriber sessions

        "Configuring ISG Access for IP Subscriber Sessions" module in this guide

        Standards

        Standard

        Title

        No new or modified standards are supported by this feature, and support for existing standards has not been modified.

        --

        MIBs

        MIB

        MIBs Link

        No new or modified MIBs are supported by this feature, and support for existing MIBs has not been modified.

        To locate and download MIBs for selected platforms, Cisco software releases, and feature sets, use Cisco MIB Locator found at the following URL:

        http:/​/​www.cisco.com/​go/​mibs

        RFCs

        RFC

        Title

        No new or modified RFCs are supported by this feature, and support for existing RFCs has not been modified.

        --

        Technical Assistance

        Description

        Link

        The Cisco Support and Documentation website provides online resources to download documentation, software, and tools. Use these resources to install and configure the software and to troubleshoot and resolve technical issues with Cisco products and technologies. Access to most tools on the Cisco Support and Documentation website requires a Cisco.com user ID and password.

        http:/​/​www.cisco.com/​cisco/​web/​support/​index.html

        Feature Information for ISG Troubleshooting Enhancements

        The following table provides release information about the feature or features described in this module. This table lists only the software release that introduced support for a given feature in a given software release train. Unless noted otherwise, subsequent releases of that software release train also support that feature.

        Use Cisco Feature Navigator to find information about platform support and Cisco software image support. To access Cisco Feature Navigator, go to www.cisco.com/​go/​cfn. An account on Cisco.com is not required.

        Table 1 Feature Information for ISG Troubleshooting Enhancements

        Feature Name

        Releases

        Feature Information

        DPM/PM Debuggability Enhancements

        12.2(33)SB9

        15.1(2)S

        Cisco IOS XE Release 3.3S

        This feature enhances debugging for ISG subscriber sessions enabling you to isolate issues through expanded statistics collection and event tracing.

        The following commands were introduced or modified: clear subscriber policy dpm statistics, clear subscriber trace history, debug subscriber policy dpm timestamps, show subscriber policy dpm context, show subscriber policy dpm statistics, show subscriber trace history, show subscriber trace statistics, subscriber trace event, subscriber trace history.