PDF(168.7 KB) View with Adobe Reader on a variety of devices
ePub(199.4 KB) View in various apps on iPhone, iPad, Android, Sony Reader, or Windows Phone
Mobi (Kindle)(151.7 KB) View on Kindle device or Kindle app on multiple devices
Updated:August 13, 2021
The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This document describes some of the best practices in order to collect Voice Debugs in an IOS/IOS-XE Voice Router. The process of Debug collection in these platforms could be confusing and could potentially impact the performance of the device. The challenges and risks increase when there are multiple active calls established in a Voice Router. In some scenarios if the debugs are not collected correctly, it might lead to high CPU that could detriment the capacity of the Router and even cause a software crash. This document will start by describing the difference between a Cisco Unified Border Element (CUBE) and a TDM/Analog Gateway.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Basic knowledge in IOS/IOS-XE inside Integrated Services Routers (ISR).
Priviliged Access in order to execute commands in the ISR Routers.
Prior experience with Voice-over-IP (VoIP) Protocols is desired.
For VoIP Trace minimum IOS-XE 17.4.1 or 17.3.2 is required.
For the purpose of this document the components used are:
Cisco ISR 3925
Cisco ISR 4451
TDM Voice Gateways vs CUBE
TDM Voice gateways are mainly use to interconnect an internal phone system with another Private Branch Exchange (PBX) or the Public Switched Telephony Network (PSTN). The type of connections that are used in TDM gateways are T1/E1 Controllers (ISDN or CAS) and Analog circuits such as FXS and FXO ports. A Digital Signal Processor (DSP) will convert the audio from its raw form into RTP packets. In a similar way, RTP packets get converted to raw audio after the DSP has processed the RTP packets and sends the audio on the specific circuit. These gateways can interoperate with H323, MGCP or SCCP on the VoIP side, and on the TDM side its either ISDN PRI circuits or Analog as the most common connections to the PSTN or endpoints.
As shown in the image the TDM Gateways provide a bridge between your internal VoIP infrastructure and the Analog or ISDN Service Providers.
With the introduction of VoIP, customers started to rapidly change their legacy systems to a modern VoIP infrastructure. The same occurred on the Service Provider side, where they now use connections to interconnect On-Premises Telephony services with the Service Provider VoIP infrastructure and expand their capabilities in order to provide better services. The most common VoIP protocol used today is the Session Initiation Protocol (SIP) and is currently widely used by customers and Internet Telephony Service Providers (ITSP) across the world.
CUBE was introduced to provide a way to internconnect those internal VoIP systems with the external world through the ITSPs with SIP as the primary VoIP Protocol. CUBE is simply an IP-IP Gateway where it no longer needs any TDM type of connection like T1/E1 Controllers or analog ports. CUBE runs on the same platforms as TDM Gateways.
Most common VoIP Protocol used is SIP, for call establishement and teardown of the calls, and RTP for media transport. In CUBE there is no need of a DSP unless a transcoder is required. The RTP traffic flows end to end from the ITSP to the endpoint, and CUBE acts as the middle-man with address hiding as one of the many features it offers.
As shown in the image, CUBE provides a division between your internal VoIP infrastructure and the SIP ITSP:
Collection of IOS/IOS-XE Voice Debugs
Voice features run on a different list of platforms, like ISR, ASRs, CAT8Ks, amongst others, however they use a common software which is either IOS or IOS-XE (the differences between IOS and IOS-XE are not covered in this article). Lets start with the basics on how to access the IOS Router.
How to Access an IOS/IOS-XE Router via Command Line Interface (CLI)
Routers, like any other CLI based devices, require a terminal monitor to gain access to run the commands through Secure Shell (SSH) or Telnet. SSH is the most common protocol used nowadays to access the devices given it provides a secure and encrypted connection to the device. Some of the common terminal monitors used to access the CLI of the Routers are:
How to set the Terminal Monitor to Collect show commands or debugs
There are different ways to collect the output from the CLI. The recommendation is to export the information from the CLI off the Router to a separate file. This makes it easier in order to share the information to external parties.
A couple of ways to collect the outputs from the device are:
Dump all the output in the terminal, for this you need to ensure there are enough lines of scrollback, otherwise the scrollback misses the first sections of the output and the data might be incomplete. In order to increase the scrollback lines in Putty, navigate to Putty Configuration > Window > Lines of Scrollback. Normally this is set to a very high value in order to have enough scrollback output:
Later you can collect the information from the terminal monitor with the Copy All to Clipboard option and paste the output into a text file:
Another option is to log the entire session output to a .txt file. With this option all commands entered, and outputs collected, are immediately logged to the text file. This is a common practice in order to log all output in a Session. In order to log all session output to a file in Putty navigate to Putty Configuration > Session > Logging and then select All Session Output as follows:
Note: The default Log file name is used if no other name is specified, click the Browse button in order to know exactly where the file is saved in order to find it later. Also ensure you do not overwrite another putty.log file in that same file path.
Collect basic show command output from the CLI
Show commands are needed to collect basic information from the Router before any debug collection takes place. Show commands are fast to collect, and for the most part, do not have any impact in performance on the Router. Isolation of the problem could start immediately with just a show command output.
Once connected to the Router, the terminal length can be set to 0. This can make the collection faster in order to display all the output at once, and avoid the use of the space bar. The one command that collects detailed information about the Router is 'show tech', and alternatively you can collect show tech voice which shows data more specific to the voice features enabled in the Router:
Router# terminal length 0 Router# show tech !or Router# show tech voice Router# terminal default length !This cmd restores the terminal length to default
Collect Debug Output from the CLI
Debug output collection in IOS/IOS-XE can sometimes be a challenge since there is risk of a Router crash. Some of the best practices are explained in the next sections to avoid any issues though.
Before you enable any debugs, you need to ensure there is enough memory in order to store the output in the buffer.
Run the command show process memory to find out how much memory you can allocate in order to log all output in the buffer:
Tip: Use the command terminal lengh default or terminal length <num_lines> to go back to a limited amout of lines displayed in the terminal.
Router# show process memory Processor Pool Total: 8122836952 Used: 456568400 Free: 7666268552 lsmpi_io Pool Total: 6295128 Used: 6294296 Free: 832
In the example, there is 7666268552 bytes (7.6GB) free to be used by the Router. This memory is shared by the Router amongst all the System Processes, this means you cannot use the entire free memory to log the output in the buffer, but you can use a good amount of system memory as needed.
Most of the scenarios require at least 10MB to collect enough debug output before the output is lost or overwritten. In rare occasions a larger amount of data is required to be collected, in those specific scenarios you can get 50MB to 100MB worth of output in the buffer or you can go higher as long as there is memory available.
If the Free Memory is low, then there is potentially a memory leak problem, if this is the case, please engage the Architecture TAC team to revise what could be the cause of such low memory.
Central Processing Unit (CPU) Check
The CPU is affected by the amount of Processes, features and calls active in the system. The more features or calls active in the system, the busier the CPU.
A good benchmark is to ensure the Router has the CPU at 30% or less, which means you can safely enable debugs from basic to advanced (always keep an eye on the CPU when advanced debugs are used). If the Router CPU is at around 50% then basic debugs can be run and carefully monitor the CPU. If the CPU reaches higher than 80% immediately stop the debugs (shown later in this article) and engage TAC for assistance.
Use the show process cpu sorted | exclude 0.00 command to check the last 5s, 60s and 5mins CPU values along with the top Processes actively utilizing the CPU.
Router# show processes cpu sorted | exclude 0.00 CPU utilization for five seconds: 1%/0%; one minute: 0%; five minutes: 0% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 211 4852758 228862580 21 0.15% 0.06% 0.07% 0 IPAM Manager 84 3410372 32046994 106 0.07% 0.04% 0.05% 0 IOSD ipc task 202 3856334 114790390 33 0.07% 0.05% 0.05% 0 VRRS Main thread
In the output, the Router does not have much activity, CPU is low, and debugs can safely be enabled.
Caution: Pay extra attention to the top CPU Processes active, if the CPU is at 50% or higher and the top process is a Voice process only basic debugs can be enabled. Continuously monitor the CPU with the command to ensure the overall performance of the Router is not affected.
Current Active Calls Check
Each Router has different capacity thresholds. Its important to check how many calls are active in the Router to ensure it is not close to max capacity. The Cisco Unified Border Element Version 12 Data Sheet provides information about each platform capacity for reference.
Use the show call active total-calls command to get an idea on how many calls are active in the system:
Router# show call active total-calls Total Number of Active Calls : 0
Use the show call active voice summary command to get a more detailed information of the specific call types that are active:
Telephony call-legs: TDM Gateway calls, this includes Analog and PRI/ISDN calls.
SIP call-legs: Total SIP Calls. If this is a CUBE Router, then this shows 2 call legs per call. Divide the total calls shown here by 2 to get an accurate number.
H323 call-legs: Total H323 Calls.
SCCP call-legs: CUCM Controlled Media Resources used in the Router such as Transcoder and MTPs.
Logging Buffer Settings
To configure the Router to store debug output in the buffer, the configure terminal mode is entered to manually tweak the settings in the CLI. This configuration has no impact to the Router, however as shown in previous sections, show tech or show running-config command from the Router is needed in the event the configuration needs to be rolled back.
A configuration example can be seen next, which is a common baseline used by TAC Engineers. The example allocates a 10MB of buffer memory but it can be increased as needed:
# configure terminal service timestamps debug datetime msec localtime show-timezone year service timestamps log datetime msec localtime show-timezone year service sequence-numbers logging buffered 10000000 no logging console no logging monitor logging queue-limit 10000 logging rate-limit 10000 voice iec syslog
The commands accomplish these tasks:
service timestamps debug or log: Ensures that local router time is written on every logged message, with millisecond accuracy. This is useful in order to find calls based on time. Millisecond timestamps allow for you to group debug lines into logical related events when two lines occur within the same millisecond.
service sequence-numbers: Writes the sequence number of the debug in the line. This is useful (essentially required) when logs are forwarded to a syslog server. This very useful in order to identify if any debug messages to the syslog server have been dropped in the network. The sequence number is the first item in the debug, before the timestamp and actual log message. Note that this is different from the timestamp/sequence number syslog servers can write locally in their files.
logging buffer: Tells the Router to send debugs to its local buffer memory. The buffer size is set in bytes. In the configuration the buffer size was set to 10MB.
no logging console and no logging monitor: No log messages are printed in the console or terminal monitor. If these commands are not configured, they could be detrimental to the Router's performance and debug output accuracy.
Sometimes issues can be random and require a way to continuously collect debugs until the event happens. When you store the debugs in the buffer, it collects them continuously. Note that it is limited to the amount of memory you can allocate and once it reaches that amount of memory, the buffer circles around and drops the oldest messages, which leads to incomplete valuable information require to isolate the issue.
With Syslog the Router can send all debug messages out to an external server, where the Syslog Server software stores it in text files. Although its a good way to collect the debug output, is not the preferred method for log collection. Syslog Servers tend to skip or drop lines from the received output due to congestion in the Server, since debug output can overwhelm the server, or packets might get dropped due to network conditions. However in some scenarios Syslog is the only way to make progress on an issue.
If possible use a reliable transport method such as TCP to avoid any loss of information and as a suggestion connect the Syslog server to the same switch where the Router is connected or as close as possible to the Router. It still doesn't guarantee all data is stored in the files, but reduces the chances of data loss.
By default syslog servers use UDP as the transport protocol on port 514.
#configure terminal service timestamps debug datetime msec localtime show-timezone year service timestamps log datetime msec localtime show-timezone year service sequence-numbers
!Optional in case you still want to store debug output in the buffer. logging buffered 10000000
no logging console no logging monitor
logging trap debugging
!Replace the 126.96.36.199 with the actual Syslog Server IP Address logging host 188.8.131.52 transport [tcp|udp] port <port>
As soon as the commands are configured, the Router immediately forwards the messages to the Syslog server IP Address.
Once the debugs have been enabled, the buffer has to be cleared before the issue is reproduced. This is done to ensure output is as clean as possible and avoid any extra data that is not needed for analysis. Run the command clear log, this ensures that the buffer gets cleared. If there are other calls active in the Router and the debugs are enabled, output immediately prints in the buffer.
After the issue is reproduced, disable the debugs immediately to stop more output from printing in the buffer. Then collect the logs. You can dump all the output in the terminal with the commands:
Router# undebug all Router# terminal length 0 Router# show log
Sometimes PuTTY closes and won’t be able to handle all the output at once, this is normal and it doesn’t mean something wrong happened to the Router, if this happens reopen the session again and continue normally. In scenarios where the logging buffer is too large or the terminal monitor crashes due the amount of data that needs to get printed, copy the buffer output to an external device directly with the command show log | redirect:
Router# show log | redirect ftp://username:email@example.com/debugs.txt
The command copies the entire buffer output to a ftp with IP Address 184.108.40.206 with file name debug.txt. File name has to always be specified. Other destinations available to export that data are:
Each call flow and type of features (TDM, CUBE or SCCP (Media Resources)) are different and there are specific debugs you can enable. All debugs required have to be enabled at the same time. Capturing one debug at a time is not effective and provides more confusion when the data is analyzed.
Debugs are enabled inside the CLI exec prompt level Router# which requires you to have privileged execution mode permissions.
There are basic and advanced debugs. Basic debugs are used to gather signaling information either in SIP, H323, or MGCP, which shows the conversations the Router has with its peer devices.
Advanced debugs are very detailed and are normally used to gather more information in the event of internal stack errors that the basic debugs cannot show. These debugs are normally CPU intensive.
Tip: After the debugs are enabled, remember to run the command clear logging. This command ensures the buffer is cleared for a cleaner capture of the debugs.
Internal Call Control API (CCAPI) Debug
Inside each IOS/IOS-XE Router there is a call control API which is in charge of the communication between different VoIP applications, or protocols, and the Data Plane components, such as RTP, DSP, Voice Cards, amongst others. In order to capture data from this layer there is one specific debug that can be used:
debug voip ccapi inout
There are other options for this debug, however debug voip ccapi inout covers all basic dial-plan and call establishment information which is normally more than enough to understand what are the states of this layer.
Tip: debug voip ccapi inout usually has minimal impact to the CPU of the Router and is recommended to be enabled along with any signaling debugs in order to provide a complete set of logs with information of the call(s) and its different states.
SIP Call flows
These debugs are the most commonly used for SIP call flows and they can be enabled inside CUBE and TDM Gateways with a SIP Leg between the Router and CUCM or any other SIP Server/Proxy.
Basic SIP Debugs
debug ccsip messages debug ccsip error debug ccsip non-call !Optional, applies for SIP OPTIONS and SIP REGISTER Messages.
Advanced SIP Debugs
debug ccsip all debug ccsip verbose debug voice ccapi inout
Digital (PRI, BRI) Call Flows
These debugs apply for Primary Rate Interfances (PRI) T1/E1 or Basic Rate Interfaces (BRI):
Basic Digital Debug
debug isdn q931
Advanced Digital Debug
debug isdn q921
Analog Call Flows
These debugs are used when there are Analog circuits involved like Foreigh eXchange Subscriber (FXS) or Foreign eXchange Office (FXO) ports:
debug vpm signal debug voip vtsp all
MGCP Call Flows
These debugs are used when MGCP is used as the Voice Protocol between a Voice Gateway and CUCM.
debug mgcp packets debug mgcp errors
The debugs ccm-manager is used to track the configuration download, MoH and PRI/BRI backhaul messages between CUCM and the Voice Gateway. These debugs are used on as needed basis and is dependent on the scenario being troubleshoot.
debug ccm-manager backhaul !For PRI and BRI Deployments debug ccm-manager errors debug ccm-manager events debug ccm-manager config-download !Troubleshoot Configuration download issues from CUCM TFTP debug ccm-mananger music-on-hold !Troubleshoot internal MoH Process
Advanced MGCP Debugs
debug mgcp all
H323 Call Flows
Although H323 is not widely used, there are still some deployments with H323 configured:
debug cch323 h225 debug cch323 h245 debug cch323 all
SCCP Media Resources
These debugs are used to troubleshoot Skinny Call Control Protocol (SCCP) Media Resources issues which involve Media Termination Point (MTP) or Transcoders registered to a Cisco Unified Communications Manager (CUCM) server:
With the introduction of IOS-XE 17.4.1 and 17.3.2 there is a new option to capture Voice logs inside the Cisco Unifiied Border Element (CUBE). This new feature is called VoIP Trace. This is a new serviceability framework created to log SIP signaling and events without the need to enable any debugs.
VoIP Trace is enabled by default and can be disabled at any time as needed. VoIP Trace will capture specific information for SIP Calls only:
SIP Messages for SIP Trunk to Trunk calls
Events and API calls from SIP Layer to other layers in CUBE
Call Control (Unified Communication call flows processed by CUBE)
Finite State Machines (FSM) states and events
Dial Peer matched
RTP ports Allocated
IEC errors correlation with SIP Signaling
VoIP Trace will not log information related to Out-of-Dialog SIP Messages:
VoIP Trace in HA is supported, however the following caveats apply:
Standby Router will have VoIP Trace enabled by default. Only applicable traces for Standby process will be presented until it becomes active
Once Standby is Active it will NOT contain full traces from existing (checkpointed) calls and only new calls
show voip trace <key> will still work on the Standby Router and will display cover buffer and media stream data for calls
How to Enable VoIP Trace
As mentioned, this feature is enabled by default. The command to enable this feature is:
Router# configuration terminal Router(config)# voice service voip Router(conf-voi-serv)# trace Router(conf-serv-trace)#
How to Disable VoIP Trace
To disable this feature, the commands are:
Router(conf-serv-trace)# no trace !or Router(conf-serv-trace)# shutdown
Caution: After VoIP Trace is disabled, all memory is cleared and information lost.
The commands available inside the trace configuration mode are:
Router(conf-serv-trace)#? default Set a command to its defaults exit Exit from voice service voip trace mode memory-limit Set limit based on memory used no Negate a command or set its defaults shutdown Shut Voip Trace debugging
Configure Memory Limit
The memory-limit will determine how much memory will be used by VoIP Trace to store the data. By default is 10% of the available memory in the platform, but this can be changed to a max of 1GB and a min of 10MB. The memory will be allocated dynamically, which means, the feature will only use memory as needed depending on call volume. Once it reaches the max memory available, it circles around and deletes older entries.
When the memory limit is modified to be greater than the 10% available memory, there will be a warning message shown in the Command Line Interface:
Router(conf-serv-trace)#memory-limit 1000 Warning: Setting memory limit more than 10% of available platform memory (166 MB) will affect system performance.
To set the default of 10% memory usage, the command memory-limit platform can be used:
Router(conf-serv-trace)# memory-limit platform Reducing the memory-limit clears all VoIP Trace statistics and data. If you wish to copy this data first, enter 'no' to cancel, otherwise enter 'yes' to proceed. Continue? [no]:
Caution: When memory limit is reduced, all VoIP Trace data will be lost. A backup of the data has to be collected before the memory is reduced.
How to Display VoIP Trace Data
To display the data from VoIP Trace we need to use specific show commands. The data can be displayed in the same terminal session or could also be sent via Syslog to an off- box syslog server.
Note: Traces are dumped after 32 seconds from the time a BYE is received for a call.
Note: The SIP Signaling is displayed per leg and is not combined like regular debugs. Regular debugs like debug ccsip messages will display the SIP signaling of a call in the exact order the events happened. In VoIP Trace each leg will be separate. To determine the right order, the timestamps are used.
The commands available to show the data are:
Router#show voip trace ? all Display all VoIP Traces call-id Filter traces based on Internal Call Id correlator Filter traces based on FPI Correlator cover-buffers Display the summary of all cover buffers session-id Filter traces based on SIP Session ID sip-call-id Filter traces based on SIP Call Id statistics Display statistics for VoIP Trace
show voice trace all
This command will display all VoIP Trace data available in the buffer. The use of this command will impact the performance of the Router. Once the command is entered there will be a warning message to alert about the risk and confirm to continue:
Router#sh voip trace all Displaying 11858 cover buffers This may severely impact system performance. Continue? [yes/no] no
show voip trace cover-buffers
This command displays an overview of call details for all calls reported under VoIP Trace. Each call leg will have a cover buffer created which contains a summary of the call logged.
For more information about each field, please refer to the next table:
Contains a combination of calling, called number and call-id
Creation time of cover buffer
Buffer ID of the cover buffer
Call-id of the respective call leg of to the cover buffer
Call-id of the peer leg
FPI correlator of the call
Called number of the respective call-leg of the cover buffer
Calling number of the respective call-leg of the cover buffer
Sip call-id of the respective call leg of the cover buffer
Sip session ID
SIP session id of the respective call leg of the cover buffer
GUID of the respective call of the cover buffer
Anchor leg is set to yes if the respective call-leg is an anchor leg in the call forking flow or media proxy deployment
Forked Leg is set to yes if the respective call-leg is an anchor leg in the call forking flow or media proxy deployment
Associated CalI ID’s
Call-id of the associated forked legs
In order to filter the cover buffers we can use the include and section commands:
Router# show voip trace cover-buffers | include Search-key | 8845 | 3002 Search-key = 8845:3002:661 !or Router# show voip trace cover-buffers | section Search-key | 8845 | 3002 Search-key = 8845:3002:661
show voip trace call-id
In combination with the previous command, show voip trace call-id can be used to find the calls. After the call-id has been identified, this command can be used to display all the information about the specific call leg:
Router# show voip trace cover-buffers | include Search-key | 8845 | 3002 Search-key = 8845:3002:661 Router# show voip trace call-id 661
show voip trace statistics
This show command will display detailed output about status, memory consumption, errored or failure calls, successful calls, timestamps of newest and oldest entries and more.
Router#show voip trace statistics VoIP Trace Statistics Tracing status : ENABLED at *Sep 12 06:44:02.349 Memory limit configured : 803209216 bytes Memory consumed : 254550928 bytes (31%) Total call legs dumped : 2 Oldest trace dumped : *Sep 12 07:29:21.077 Search-key: 9898:30000:64 Latest trace dumped : *Sep 12 07:29:21.010 Search-key: 9898:30000:63 Total call legs captured : 11858 Total call legs available : 11858 Oldest trace available : *Sep 12 06:57:23.923, Search-key: 5250001:4720001:11 Latest trace available : *Sep 13 05:08:25.353, Search-key: 19074502232:30000:13177 Total traces missed : 0
For more information about each field, please reference the next table:
Displays tracing status, including time and date VoIP trace was enabled.
Memory limit configured
Displays the configured memory limit. This is 10% of the processor pool memory size
Displays amount of memory dynamically consumed for VoIP Trace
Total call legs dumped
Displays the number of failed call legs dumped into logging buffer. Dumped calls refers to call legs associated with IEC errors
Oldest trace dumped
Displays timestamps and search key of the oldest failed call since VoIP Trace was enabled
Latest trace dumped
Displays timestamps and search key of the latest failed call since VoIP Trace was enabled
Total call legs captured
Displays total legs captured after VoIP Trace is enabled
Total call legs available
Displays total call legs available in the history. This can be same or different compared to Total call legs captured, depending on the memory limit.
Oldest trace available
Displays timestamp and search key of the oldest cover buffer available in the memory
Latest trace available
Displays timestamp and search key of the latest cover buffer available in the memory
Total traces missed
Displays number of call legs missed due to memory limit.
Additional show commands
show voip trace correlator <correlator>
show voip trace correlator 4
Filters and displays VOIP Trace for a specific call-id starting from the cover buffer
show voip trace session-id <session-id>
show voip trace session-id
Filters and displays VOIP Trace for a call based on the SIP Session ID. Either local or remote UUID from the session ID header of the sip message can be used to display both the legs of the call.
show voip trace sip-call-id <call-id>
show voip trace sip-call-id 01e60dfa9d8442848336d79e3155a8a1
Filters and displays VOIP Trace based on SIP Call-ID