The Intelligent Services Gateway (ISG) debugging enhancements enable you to more easily isolate issues with ISG subscriber sessions in a production network, such as a session getting stuck in a dangling state (never reaches the established state). The troubleshooting enhancements described in this module allow you to diagnose these issues by introducing expanded statistics collection and event tracing.
Your software release may not support all the features documented in this module. For the latest feature information and caveats, see the release notes for your platform and software release. To find information about the features documented in this module, and to see a list of the releases in which each feature is supported, see the Feature Information Table at the end of this document.
Use Cisco Feature Navigator to find information about platform support and Cisco software image support. To access Cisco Feature Navigator, go to
www.cisco.com/go/cfn. An account on Cisco.com is not required.
Restrictions for ISG Troubleshooting Enhancements
Enabling the subscribertracehistory command does not mean that all event traces for subscriber sessions are stored in the history buffer. Event traces for the DPM and PM are written to their respective history buffers only if a session has an issue (such as a session that becomes a dangling session).
Information About ISG Troubleshooting Enhancements
The DHCP policy module (DPM) has many complex interactions with other components such as the policy manager (PM) module, which can make it difficult to troubleshoot issues and find the root cause. For example, dangling IP sessions are often caused by an error in the interactions between DHCP, the DPM, and the PM, resulting in DPM sessions getting stuck in a startup state.
The DPM enhancements enable more efficient debugging of issues with DPM and its interactions in customer networks. This includes improved statistics collection at both the system and session level to show failure data, more comprehensive error messages, and event tracing.
DPM contexts could previously be displayed only by selecting the MAC address. These enhancements add the ability to search and display all DPM contexts on the router, DPM contexts for a particular client IP address, or DPM contexts in a particular state. All debug output is now prefixed with the DPM context identifier and MAC address so that in a large-scale scenario you can identify debugs for a particular transaction.
Event Tracing for Subscriber Sessions
When trying to reproduce or capture customer issues, collecting debug output is not always practical or even possible. Network administrators often do not detect an error until long after the event that caused the error has occurred. By the time a fault is detected, it is usually too late to enable debug commands because the session is already in an error state, or the session was terminated because of an error.
Event tracing allows you to capture traces for existing sessions on the router and to retain the history of any past sessions that were marked as interesting, such as a session that became stuck in a dangling state. This enables you to look at existing sessions, as well as past sessions, and review the data after the session gets into an unexpected state or never comes up.
If a session is marked as interesting, its event trace information is sent to a history log, if history logging is enabled. A session is considered interesting if it becomes stuck in a state, enters an error state, or terminates without transitioning into a target state, because of a programming error, end-user action, packet drop, or other reason. The decision whether to log an event trace is determined by the after-the-fact status of the object. Event traces for uninteresting sessions are removed to free up space in the history log buffer.
Event tracing is supported by the DPM and PM modules. Each module logs event traces for each of its session contexts independently. The event trace data for each subscriber session is attached to its session context. Previously, this data was purged when the session was terminated. These enhancements preserve the event trace data even after the sessions are gone.
Each session context that supports event trace creates a new event trace log to hold the event traces for that session context. The new event log is created at session startup or teardown, and is destroyed after the session reaches the established or destroyed state. The event trace logs can be displayed independently through show commands.
Router(config)# subscriber trace history dpm size 200
Enables saving the event traces for ISG subscriber sessions to a history log.
This command, without the sizekeyword, is enabled by default for the DPM and PM. Default log size is 100 records.
Step 5
exit
Example:
Router(config)# exit
Exits global configuration mode and returns to privileged EXEC mode.
Displaying Event Traces for ISG Sessions
Use the following commands to display information about the event traces that are saved to the history log for the specified module. You can use these commands in any order.
Use this command to display statistics about the event traces that were saved to the history log.
Example:
Router# show subscriber trace statistics
Event Trace History Statistics: DPM
Logging enabled
All time max records: 5
Max records: 5
Current records: 5
Current log size: 200
Proposed log size 200
Oldest, newest index: 0 : 4
Event Trace History Statistics: Policy Manager
Logging enabled
All time max records: 4
Max records: 4
Current records: 4
Current log size: 64
Proposed log size 64
Oldest, newest index: 0 : 3
Use this command to display a summary of all session traces stored in the trace history log, or to display a complete trace for a specific session by using one of the optional keywords. The following example shows output for the client with IP address 10.0.0.2.
Example:
Router# show subscriber trace history dpm client-ip-address 10.0.0.2
DPM session info: 5CB2A60
MAC: aaaa.2222.cccc IP: 10.0.0.2
UID: 3 reason: PM callback to clear
=========================
ET 11:33:48.767 PST Mon Aug 30 2010 dhcp get class
rc no c-aware cfg
ET 11:34:07.327 PST Mon Aug 30 2010 i-if change
,MAC ok,ignore: same i/f
ET 11:34:07.327 PST Mon Aug 30 2010 dhcp discover
rc OK,proc prev req
ET 11:34:07.327 PST Mon Aug 30 2010 dhcp get class
rc no c-aware cfg
ET 11:34:10.835 PST Mon Aug 30 2010 i-if change
,MAC ok,ignore: same i/f
ET 11:34:10.835 PST Mon Aug 30 2010 dhcp discover
rc OK,proc prev req
ET 11:34:10.835 PST Mon Aug 30 2010 dhcp get class
rc no c-aware cfg
ET 11:34:14.843 PST Mon Aug 30 2010 i-if change
,MAC ok,ignore: same i/f
ET 11:34:14.843 PST Mon Aug 30 2010 dhcp discover
rc OK,proc prev req
ET 11:34:14.843 PST Mon Aug 30 2010 dhcp get class
rc no c-aware cfg
ET 11:34:38.391 PST Mon Aug 30 2010 i-if change
,MAC ok,ignore: same i/f
ET 11:34:38.391 PST Mon Aug 30 2010 dhcp discover
rc OK,proc prev req
ET 11:34:38.391 PST Mon Aug 30 2010 dhcp get class
rc no c-aware cfg
ET 11:34:41.923 PST Mon Aug 30 2010 i-if change
,MAC ok,ignore: same i/f
ET 11:34:41.923 PST Mon Aug 30 2010 dhcp discover
rc OK,proc prev req
ET 11:34:41.923 PST Mon Aug 30 2010 dhcp get class
rc no c-aware cfg
ET 11:34:45.931 PST Mon Aug 30 2010 i-if change
,MAC ok,ignore: same i/f
ET 11:34:45.931 PST Mon Aug 30 2010 dhcp discover
rc OK,proc prev req
ET 11:34:45.931 PST Mon Aug 30 2010 dhcp get class
rc no c-aware cfg
ET 11:35:13.591 PST Mon Aug 30 2010 PM callback
Terminate, rc end sess,Case: REQ_TERMINATE
Step 3
clearsubscribertracehistory{dpm | pm}
Use this command to clear the event trace history log for the specified module.
Example:
Router# clear subscriber trace history dpm
Displaying DPM Statistics
Use the following commands to display information about event traces for DPM session contexts.
Use this command to display event traces for DPM session contexts. The following example shows output for the client with the IP address 10.0.0.2.
Example:
Router# show subscriber policy dpm context ip-address 10.0.0.2
==================== DHCP Context Summary ==================
Total DHCP Session Count: 1
-------------------------------------------------------------
DHCP Context Entry #1, 0x45FD538
dhcp handle: 12000002
sss handle: 0
ip address: 10.0.0.2, subnet: 255.255.255.0
sdb handle: 3877F6A
policy handle: D7000002
callback: 0, started: 1, terminate: 1, class: 0
default config: 1, influence vrf: 0
classname: NULL, default classname: NULL
default vrf id: NULL, vrf id: NULL
mac address: aaaa.2222.cccc
dpm state: session-configured
dpm sub-state: classname-received
remote id: NULL, circuit id: NULL
ip sess hdl: 81000002
restart: 0, update: 0
shdb hdl: 0
standby: 0, sync ready: 1
req for classname received time: 11:43:55.403 PST Mon Aug 30 2010
offer notif received time: 11:46:05.959 PST Mon Aug 30 2010
DHCP session setup time: 2s20ms
ET 11:46:03.959 PST Mon Aug 30 2010 PM invoke
rc OK, Session-Start
ET 11:46:03.959 PST Mon Aug 30 2010 dhcp discover
rc OK,No Sess,sess alloc,sess-start OK
ET 11:46:03.959 PST Mon Aug 30 2010 dhcp discover
rc OK,proc prev req
ET 11:46:03.959 PST Mon Aug 30 2010 dhcp get class
rc no c-aware cfg
ET 11:46:03.975 PST Mon Aug 30 2010 PM callback
Got Keys, rc dhcp wait no cb,upd msi vrf=0,Case: GOT_KEYS
ET 11:46:05.959 PST Mon Aug 30 2010 PM invoke
rc OK, Session-Update
ET 11:46:05.959 PST Mon Aug 30 2010 dhcp offer
rc OK w delay,acc.if ret
ET 11:46:05.983 PST Mon Aug 30 2010 PM callback
Session Update Succes, rc offer cb no-err,notify stdby,Case: UPDATE_SUCCESS
ET 11:46:05.987 PST Mon Aug 30 2010 dhcp discover
rc OK,proc prev req
ET 11:46:05.991 PST Mon Aug 30 2010 i-if change
,MAC ok,ignore: same i/f
ET 11:46:05.995 PST Mon Aug 30 2010 dhcp assign OK
rc same IP
------------------------------------------------------
Minimum DHCP set up time among all contexts : 2s20ms
Maximum DHCP set up time among all contexts : 2s20ms
MAC address for Max Setup Time among all contexts: aaaa.2222.cccc
Step 3
showsubscriberpolicydpmstatistics
Use this command to display statistics for DPM session contexts.
Example:
Router# show subscriber policy dpm statistics
Message Received Duplicate Ignored Total
Discover Notification : 284 0 291
Offer Notification : 0 0 2
Address Assignment Notif : 2 0 2
DHCP Classname request : 0 290 290
Input Intf Override : 0 10 293
Lease Termination Notif : 0 0 2
Session Restart Request : 0 0 0
Response to DHCP request for classname
Average Time : Max Time :
MAC address for Max Time :
Response to DHCP Offer Notification
Average Time : 30ms Max Time : 36ms
MAC address for Max Time : aaaa.2222.cccc
Overall since last clear
Total Discover Init Sessions : 2
Total Restarted Sessions : 0
Average set up time for Discover initiated sessions : 2s26ms
Min set up time among Discover initiated sessions : 2s20ms
Max set up time among Discover initiated sessions : 2s32ms
Current active Sessions
Total Discover Init Sessions : 0
Total Restarted Sessions : 0
Average set up time for Discover initiated sessions :
Min set up time among Discover initiated sessions: 2s20ms
Max set up time among Discover initiated sessions :
MAC of session with Max DHCP Setup Time : aaaa.2222.cccc
Total number of DPM contexts allocated : 7
Total number of DPM contexts freed : 6
Total number of DPM contexts currently without session : 1
Elapsed time since counters last cleared : 2h15m20s
Step 4
clearsubscriberpolicydpmstatistics
Use this command to clear the statistics for DPM contexts.
Example:
Router# clear subscriber policy dpm statistics
Configuration Examples for ISG Troubleshooting Enhancements
The following example shows a configuration with the DPM and PM event tracing enabled and retained. Trace history logging is enabled for the DPM and PM so traces for sessions that are marked as interesting are stored in their respective history log. Up to 100 sessions (default) can be stored in the PM history log, and up to 200 sessions can be stored in the DPM history log.
subscriber trace event dpm retain
subscriber trace event pm retain
subscriber trace history pm
subscriber trace history dpm size 200
No new or modified RFCs are supported by this feature, and support for existing RFCs has not been modified.
--
Technical Assistance
Description
Link
The Cisco Support and Documentation website provides online resources to download documentation, software, and tools. Use these resources to install and configure the software and to troubleshoot and resolve technical issues with Cisco products and technologies. Access to most tools on the Cisco Support and Documentation website requires a Cisco.com user ID and password.
Feature Information for ISG Troubleshooting Enhancements
The following table provides release information about the feature or features described in this module. This table lists only the software release that introduced support for a given feature in a given software release train. Unless noted otherwise, subsequent releases of that software release train also support that feature.
Use Cisco Feature Navigator to find information about platform support and Cisco software image support. To access Cisco Feature Navigator, go to
www.cisco.com/go/cfn. An account on Cisco.com is not required.
Table 1
Feature Information for ISG Troubleshooting Enhancements
Feature Name
Releases
Feature Information
DPM/PM Debuggability Enhancements
12.2(33)SB9
15.1(2)S
Cisco IOS XE Release 3.3S
This feature enhances debugging for ISG subscriber sessions enabling you to isolate issues through expanded statistics collection and event tracing.
The following commands were introduced or modified:
clearsubscriberpolicydpmstatistics,
clearsubscribertracehistory,
debugsubscriberpolicydpmtimestamps,
showsubscriberpolicydpmcontext,
showsubscriberpolicydpmstatistics,
showsubscribertracehistory,
showsubscribertracestatistics,
subscribertraceevent,
subscribertracehistory.
Cisco and the Cisco Logo are trademarks of Cisco Systems, Inc. and/or its affiliates in the U.S. and other countries. A listing of Cisco's trademarks can be found at
www.cisco.com/go/trademarks. Third party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1005R)
Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.