This document describes some of the best practices in order to collect Voice Debugs in an IOS/IOS-XE Voice Router. The process of Debug collection in these platforms could be confusing and could potentially impact the performance of the device. The challenges and risks increase when there are multiple active calls established in a Voice Router. In some scenarios if the debugs are not collected correctly, it might lead to high CPU that could detriment the capacity of the Router and even cause a software crash. This document will start by describing the difference between a Cisco Unified Border Element (CUBE) and a TDM/Analog Gateway.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Basic knowledge in IOS/IOS-XE inside Integrated Services Routers (ISR).
Priviliged Access in order to execute commands in the ISR Routers.
Prior experience with Voice-over-IP (VoIP) Protocols is desired.
For the purpose of this document the components used are:
Cisco ISR 3925
Cisco ISR 4451
TDM Voice Gateways vs CUBE
TDM Voice gateways are mainly use to interconnect an internal phone system with another Private Branch Exchange (PBX) or the Public Switched Telephony Network (PSTN). The type of connections that are used in TDM gateways are T1/E1 Controllers (ISDN or CAS) and Analog circuits such as FXS and FXO ports. A Digital Signal Processor (DSP) will convert the audio from its raw form into RTP packets. In a similar way, RTP packets get converted to raw audio after the DSP has processed the RTP packets and sends the audio on the specific circuit. These gateways can interoperate with H323, MGCP or SCCP on the VoIP side, and on the TDM side its either ISDN PRI circuits or Analog as the most common connections to the PSTN or endpoints.
As shown in the image the TDM Gateways provide a bridge between your internal VoIP infrastructure and the Analog or ISDN Service Providers.
With the introduction of VoIP, customers started to rapidly change their legacy systems to a modern VoIP infrastructure. The same occurred on the Service Provider side, where they now use connections to interconnect On-Premises Telephony services with the Service Provider VoIP infrastructure and expand their capabilities in order to provide better services. The most common VoIP protocol used today is the Session Initiation Protocol (SIP) and is currently widely used by customers and Internet Telephony Service Providers (ITSP) across the world.
CUBE was introduced to provide a way to internconnect those internal VoIP systems with the external world through the ITSPs with SIP as the primary VoIP Protocol. CUBE is simply an IP-IP Gateway where it no longer needs any TDM type of connection like T1/E1 Controllers or analog ports. CUBE runs on the same platforms as TDM Gateways.
Most common VoIP Protocol used is SIP, for call establishement and teardown of the calls, and RTP for media transport. In CUBE there is no need of a DSP unless a transcoder is required. The RTP traffic flows end to end from the ITSP to the endpoint, and CUBE acts as the middle-man with address hiding as one of the many features it offers.
As shown in the image, CUBE provides a division between your internal VoIP infrastructure and the SIP ITSP:
Collection of IOS/IOS-XE Voice Debugs
Voice features run on a different list of platforms, like ISR or ASR Routers, etc, however they use a common software which is either IOS or IOS-XE (the differences between IOS and IOS-XE are not covered in this article). Lets start with the basics on how to access the IOS Router.
How to Access an IOS/IOS-XE Router via Command Line Interface (CLI)
Routers, like any other CLI based devices, require a terminal monitor to gain access to run the commands through Secure Shell (SSH) or Telnet. SSH is the most common protocol used nowadays to access the devices given it provides a secure and encrypted connection to the device. Some of the common terminal monitors used to access the CLI of the Routers are:
How to set the Terminal Monitor to Collect show commands or debugs
There are different ways to collect the output from the CLI. The recommendation is to export the information from the CLI off the Router to a separate file. This makes it easier in order to share the information to external parties.
A couple of ways to collect the outputs from the device are:
Dump all the output in the terminal, for this you need to ensure there are enough lines of scrollback, otherwise the scrollback misses the first sections of the output and the data might be incomplete. In order to increase the scrollback lines in Putty, navigate to Putty Configuration > Window > Lines of Scrollback. Normally this is set to a very high value in order to have enough scrollback output:
Later you can collect the information from the terminal monitor with the Copy All to Clipboard option and paste the output into a text file:
Another option is to log the entire session output to a .txt file. With this option all commands entered, and outputs collected, are immediately logged to the text file. This is a common practice in order to log all output in a Session. In order to log all session output to a file in Putty navigate to Putty Configuration > Session > Logging and then select All Session Output as follows:
Note: The default Log file name is used if no other name is specified, click the Browse button in order to know exactly where the file is saved in order to find it later. Also ensure you do not overwrite another putty.log file in that same file path.
Collect basic show command output from the CLI
Show commands are needed to collect basic information from the Router before any debug collection takes place. Show commands are fast to collect, and for the most part, do not have any impact in performance on the Router. Isolation of the problem could start immediately with just a show command output.
Once connected to the Router, the terminal length can be set to 0. This can make the collection faster in order to display all the output at once, and avoid the use of the space bar. The one command that collects detailed information about the Router is 'show tech', and alternatively you can collect show tech voice which shows data more specific to the voice features enabled in the Router:
Router# terminal length 0 Router# show tech !or Router# show tech voice Router# terminal default length !This cmd restores the terminal length to default
Collect Debug Output from the CLI
Debug output collection in IOS/IOS-XE can sometimes be a challenge since there is risk of a Router crash. Some of the best practices are explained in the next sections to avoid any issues though.
Before you enable any debugs, you need to ensure there is enough memory in order to store the output in the buffer.
Run the command show process memory to find out how much memory you can allocate in order to log all output in the buffer:
Tip: Use the command terminal lengh default or terminal length <num_lines> to go back to a limited amout of lines displayed in the terminal.
Router# show process memory Processor Pool Total: 8122836952 Used: 456568400 Free: 7666268552 lsmpi_io Pool Total: 6295128 Used: 6294296 Free: 832
In the example, there is 7666268552 bytes (7.6GB) free to be used by the Router. This memory is shared by the Router amongst all the System Processes, this means you cannot use the entire free memory to log the output in the buffer, but you can use a good amount of system memory as needed.
Most of the scenarios require at least 10MB to collect enough debug output before the output is lost or overwritten. In rare occasions a larger amount of data is required to be collected, in those specific scenarios you can get 50MB to 100MB worth of output in the buffer or you can go higher as long as there is memory available.
If the Free Memory is low, then there is potentially a memory leak problem, if this is the case, please engage the Architecture TAC team to revise what could be the cause of such low memory.
Central Processing Unit (CPU) Check
The CPU is affected by the amount of Processes, features and calls active in the system. The more features or calls active in the system, the busier the CPU.
A good benchmark is to ensure the Router has the CPU at 30% or less, which means you can safely enable debugs from basic to advanced (always keep an eye on the CPU when advanced debugs are used). If the Router CPU is at around 50% then basic debugs can be run and carefully monitor the CPU. If the CPU reaches higher than 80% immediately stop the debugs (shown later in this article) and engage TAC for assistance.
Use the show process cpu sorted | exclude 0.00 command to check the last 5s, 60s and 5mins CPU values along with the top Processes actively utilizing the CPU.
Router# show processes cpu sorted | exclude 0.00 CPU utilization for five seconds: 1%/0%; one minute: 0%; five minutes: 0% PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process 211 4852758 228862580 21 0.15% 0.06% 0.07% 0 IPAM Manager 84 3410372 32046994 106 0.07% 0.04% 0.05% 0 IOSD ipc task 202 3856334 114790390 33 0.07% 0.05% 0.05% 0 VRRS Main thread
In the output, the Router does not have much activity, CPU is low, and debugs can safely be enabled.
Caution: Pay extra attention to the top CPU Processes active, if the CPU is at 50% or higher and the top process is a Voice process only basic debugs can be enabled. Continuously monitor the CPU with the command to ensure the overall performance of the Router is not affected.
Current Active Calls Check
Each Router has different capacity thresholds. Its important to check how many calls are active in the Router to ensure it is not close to max capacity. The Cisco Unified Border Element Version 12 Data Sheet provides information about each platform capacity for reference.
Use the show call active total-calls command to get an idea on how many calls are active in the system:
Router# show call active total-calls Total Number of Active Calls : 0
Use the show call active voice summary command to get a more detailed information of the specific call types that are active:
Telephony call-legs: TDM Gateway calls, this includes Analog and PRI/ISDN calls.
SIP call-legs: Total SIP Calls. If this is a CUBE Router, then this shows 2 call legs per call. Divide the total calls shown here by 2 to get an accurate number.
H323 call-legs: Total H323 Calls.
SCCP call-legs: CUCM Controlled Media Resources used in the Router such as Transcoder and MTPs.
Logging Buffer Settings
To configure the Router to store debug output in the buffer, the configure terminal mode is entered to manually tweak the settings in the CLI. This configuration has no impact to the Router, however as shown in previous sections, show tech or show running-config command from the Router is needed in the event the configuration needs to be rolled back.
A configuration example can be seen next, which is a common baseline used by TAC Engineers. The example allocates a 10MB of buffer memory but it can be increased as needed:
# configure terminal service timestamps debug datetime msec localtime show-timezone year service timestamps log datetime msec localtime show-timezone year service sequence-numbers logging buffered 10000000 no logging console no logging monitor logging queue-limit 10000 logging rate-limit 10000 voice iec syslog
The commands accomplish these tasks:
service timestamps debug or log: Ensures that local router time is written on every logged message, with millisecond accuracy. This is useful in order to find calls based on time. Millisecond timestamps allow for you to group debug lines into logical related events when two lines occur within the same millisecond.
service sequence-numbers: Writes the sequence number of the debug in the line. This is useful (essentially required) when logs are forwarded to a syslog server. This very useful in order to identify if any debug messages to the syslog server have been dropped in the network. The sequence number is the first item in the debug, before the timestamp and actual log message. Note that this is different from the timestamp/sequence number syslog servers can write locally in their files.
logging buffer: Tells the Router to send debugs to its local buffer memory. The buffer size is set in bytes. In the configuration the buffer size was set to 10MB.
no logging console and no logging monitor: No log messages are printed in the console or terminal monitor. If these commands are not configured, they could be detrimental to the Router's performance and debug output accuracy.
Sometimes issues can be random and require a way to continuously collect debugs until the event happens. When you store the debugs in the buffer, it collects them continuously. Note that it is limited to the amount of memory you can allocate and once it reaches that amount of memory, the buffer circles around and drops the oldest messages, which leads to incomplete valuable information require to isolate the issue.
With Syslog the Router can send all debug messages out to an external server, where the Syslog Server software stores it in text files. Although its a good way to collect the debug output, is not the preferred method for log collection. Syslog Servers tend to skip or drop lines from the received output due to congestion in the Server, since debug output can overwhelm the server, or packets might get dropped due to network conditions. However in some scenarios Syslog is the only way to make progress on an issue.
If possible use a reliable transport method such as TCP to avoid any loss of information and as a suggestion connect the Syslog server to the same switch where the Router is connected or as close as possible to the Router. It still doesn't guarantee all data is stored in the files, but reduces the chances of data loss.
By default syslog servers use UDP as the transport protocol on port 514.
#configure terminal service timestamps debug datetime msec localtime show-timezone year service timestamps log datetime msec localtime show-timezone year service sequence-numbers
!Optional in case you still want to store debug output in the buffer. logging buffered 10000000
no logging console no logging monitor
logging trap debugging
!Replace the 18.104.22.168 with the actual Syslog Server IP Address logging host 22.214.171.124 transport [tcp|udp] port <port>
As soon as the commands are configured, the Router immediately forwards the messages to the Syslog server IP Address.
Once the debugs have been enabled, the buffer has to be cleared before the issue is reproduced. This is done to ensure output is as clean as possible and avoid any extra data that is not needed for analysis. Run the command clear log, this ensures that the buffer gets cleared. If there are other calls active in the Router and the debugs are enabled, output immediately prints in the buffer.
After the issue is reproduced, disable the debugs immediately to stop more output from printing in the buffer. Then collect the logs. You can dump all the output in the terminal with the commands:
Router# undebug all Router# terminal length 0 Router# show log
Sometimes PuTTY closes and won’t be able to handle all the output at once, this is normal and it doesn’t mean something wrong happened to the Router, if this happens reopen the session again and continue normally. In scenarios where the logging buffer is too large or the terminal monitor crashes due the amount of data that needs to get printed, copy the buffer output to an external device directly with the command show log | redirect:
Router# show log | redirect ftp://username:email@example.com/debugs.txt
The command copies the entire buffer output to a ftp with IP Address 126.96.36.199 with file name debug.txt. File name has to always be specified. Other destinations available to export that data are:
Each call flow and type of features (TDM, CUBE or SCCP (Media Resources)) are different and there are specific debugs you can enable. All debugs required have to be enabled at the same time. Capturing one debug at a time is not effective and provides more confusion when the data is analyzed.
Debugs are enabled inside the CLI exec prompt level Router# which requires you to have privileged execution mode permissions.
There are basic and advanced debugs. Basic debugs are used to gather signaling information either in SIP, H323, or MGCP, which shows the conversations the Router has with its peer devices.
Advanced debugs are very detailed and are normally used to gather more information in the event of internal stack errors that the basic debugs cannot show. These debugs are normally CPU intensive.
Tip: After the debugs are enabled, remember to run the command clear logging. This command ensures the buffer is cleared for a cleaner capture of the debugs.
Internal Call Control API (CCAPI) Debug
Inside each IOS/IOS-XE Router there is a call control API which is in charge of the communication between different VoIP applications, or protocols, and the Data Plane components, such as RTP, DSP, Voice Cards, etc. In order to capture data from this layer there is one specific debug that can be used:
debug voip ccapi inout
There are other options for this debug, however debug voip ccapi inout covers all basic dial-plan and call establishment information which is normally more than enough to understand what are the states of this layer.
Tip: debug voip ccapi inout usually has minimal impact to the CPU of the Router and is recommended to be enabled along with any signaling debugs in order to provide a complete set of logs with information of the call(s) and its different states.
SIP Call flows
These debugs are the most commonly used for SIP call flows and they can be enabled inside CUBE and TDM Gateways with a SIP Leg between the Router and CUCM or any other SIP Server/Proxy.
Basic SIP Debugs
debug ccsip messages debug ccsip error debug ccsip non-call !Optional, applies for SIP OPTIONS and SIP REGISTER Messages.
Advanced SIP Debugs
debug ccsip all debug ccsip verbose debug voice ccapi inout
Digital (PRI, BRI) Call Flows
These debugs apply for Primary Rate Interfances (PRI) T1/E1 or Basic Rate Interfaces (BRI):
Basic Digital Debug
debug isdn q931
Advanced Digital Debug
debug isdn q921
Analog Call Flows
These debugs are used when there are Analog circuits involved like Foreigh eXchange Subscriber (FXS) or Foreign eXchange Office (FXO) ports:
debug vpm signal debug voip vtsp all
H323 Call Flows
Although H323 is not widely used, there are still some deployments with H323 configured:
debug cch323 h225 debug cch323 h245 debug cch323 all
SCCP Media Resources
These debugs are used to troubleshoot Skinny Call Control Protocol (SCCP) Media Resources issues which involve Media Termination Point (MTP) or Transcoders registered to a Cisco Unified Communications Manager (CUCM) server: