# InTracer Overview

This chapter provides an overview of the Cisco InTracer application and describes its architecture.

It includes the following sections:

- Introduction
- InTracer Architecture
- InTracer System Requirements
- Supported Features

Introduction

Cisco InTracer is a high-performance subscriber troubleshooting and monitoring solution. It performs call tracing, control data acquisition, processing and analysis of both active and historical subscriber sessions. This provides a framework for operators to analyze and investigate call flows and call events for subscriber sessions in near real time.

The InTracer solution consists of 2 basic parts:

- The InTracer Client part runs on the Cisco gateway(s) that needs to be enabled and configured to start sending subscriber traces (control plane information for sessions) to InTracer Server.
- The InTracer Server part runs on an external box sitting close to the gateway to process and store these subscriber traces.

IMPORTANT:

External application is now supported by the Cisco MITG RHEL v5.5 OS on selected Cisco UCS servers. The Cisco MITG RHEL v5.5 OS is a custom image that contains only those software packages required to support compatible Cisco MITG external software applications. Users must not install any other applications on servers running the Cisco MITG RHEL v5.5 OS. For detailed software compatibility information, refer to the “Cisco MITG RHEL v5.5 OS Application Note.”

InTracer Architecture

This section describes the InTracer Architecture.

Nodal Trace

The following figure shows a high level view of architecture of InTracer with other components in a deployment scenario.
Figure 1. Nodal Trace

For Nodal trace, the communication between the Gateway and the R-InTracer / R-TCE happens over a proprietary protocol on top of UDP.

3GPP Trace

In 3GPP Trace communication happens through XML files being FTP pushed from the gateway to the R-TCE. R-TCE receives the FTP pushed files from the gateway and parses them to store their data into the database.
Figure 2. 3GPP Trace

InTracer Components

This section provides the physical / logical components of InTracer solution.

Component | Nodal Trace | 3GPP Trace
--- | --- | ---
InTracer client | When configured sends a copy of all signaling packets for all sessions to the R-InTracer over UDP. | When configured, sends a copy of all signaling packets for activated trace sessions to the R-TCE by pushing the generated XML trace files via FTP/SFTP.
R-InTracer (Regional InTracer) | Receives packets over UDP from configured InTracer client, processes and stores them for the configured storage period. | N/A
R-TCE(Regional Trace Collection Entity) | N/A | Receives XML trace files via FTP/SFTP from configured InTracer client, parses the files and stores the information for the configured storage period.
C-InTracer (Central InTracer) | This component provides configuration and management in the InTracer system. It is capable of fetching data from all configured R-InTracers and displaying coalesced information to the end-user. | This component provides configuration and management in the InTracer system. It is capable of fetching data from all configured R-TCEs and displaying coalesced information to the end-user. In addition to this, it also provides an interface for activating/de-activating subscriber traces.
InTracer Web User Interface | A unified web-interface of the product providing configuration, management and data-querying options. | A unified web-interface of the product providing configuration, management and data-querying options, along with Activation / De-activation of subscriber traces.

InTracer Client Nodal Trace

Generic Features

The following generic features are supported by the InTracer Client for Nodal Trace:

- To send a copy of all signaling packets for each subscriber session to configured R-InTracer
- To check the health of configured R-InTracer (up and running)

Following are the configuration items for the InTracer Client for Nodal Trace:

- Enabling InTracer
- R-InTracer configuration
- UDP heartbeat configuration parameters

IMPORTANT:

Refer to the appropriate System and Product Administration Guides for detailed information.

Specific Features

Following are the additional configuration items to support the above features:

- License to export SA-keys (on the gateway)
- Enabling SA-keys in InTracer configuration

IMPORTANT:

Refer to the appropriate System and Product Administration Guides for detailed information.

InTracer Client 3GPP Trace

The following generic features are supported by the InTracer Client for 3GPP Trace :

- To send a copy of all transmitted signaling packets for activated trace sessions to the configured R-TCE (3GPP TCE - Trace Collection Entity)
- To send trace activation/deactivation failure notifications to the configured R-TCEFollowing are the configuration items for the InTracer Client for 3GPP Trace: Configuration of of global trace parameters Configuration of TCE profiles (R-TCE configuration) Management-based Activation / Deactivation of Trace sessions

IMPORTANT:

Refer to the appropriate System and Product Administration Guides for detailed information

R-InTracer / R-TCE

The basic functionality of the R-InTracer / R-TCE is to receive signaling packets from the gateways and process them. It stores all the packets along with additional correlation information either received along with the packet or derived after processing the packet.
Following are the features and functions supported by the R-InTracer / R-TCE:

Feature | Configurable | Default | Nodal Trace | 3GPP Trace
--- | --- | --- | --- | ---
Event Processing | Yes |  | Yes | Yes
Event Storage | Yes | 30 Days | Yes | Yes
Packet-loss detection | Yes | Enabled | Yes | Yes
Health Monitoring | Yes | Enabled | Yes | Yes
Resource Monitoring | Yes | Enabled (Default threshold values) | Yes | Yes
Maintenance Activities | Yes | Enabled (Daily midnight) | Yes | Yes
Statistical Counters | No | Always | Yes | Yes

Event Processing

The basic task of the R-InTracer / R-TCE is to receive signaling packets from the gateways and process them. It then persistently stores all these packets along with additional correlation information either received along with the packet or derived after processing the packet.

Event Storage

The R-InTracer / R-TCE stores processed packets along with other correlation information to persistent storage. The storage period is a configurable value between 2 to 236 days, default value is 30 days.

IMPORTANT:

If event information exceeds the storage period, it will be automatically deleted by the system even if the system has sufficient additional storage available.

When the R-InTracer / R-TCE receives packets from the gateway (either through UDP sockets in case of Nodal trace or through XML files in case of 3GPP trace), it requires some time before it persistently stores these packets and makes them available for query. The possible reasons for this delay are:

- It needs to take care of out-of-order packet delivery over UDP in case of Nodal trace
- It needs to detect packet-losses and take appropriate actions in case of Nodal trace
- It needs to perform event correlation

For better search, each query specified by the operator should be bounded by time intervals. The shorter the interval specified, the lesser the amount of data to be searched which results in faster query response.

IMPORTANT:

InTracer is a best-effort solution and there can be possible losses of packet data even after successfully processed by the R-InTracer / R-TCE. Such scenarios occur since some packet information is cached and not persistently stored immediately. All efforts are made internally to minimize such scenarios.

Event Filtering

Event filtering allows operators to configure which packets to be processed/ignored by the R-InTracer / R-TCE. Filtering can be done based on the event type and/or protocol type.The R-InTracer / R-TCE maintains statistical counters about the filtered packets, which can later be used to verify this behaviour.

IMPORTANT:

Events that are filtered out are not stored on persistent storage, hence cannot be queried later.

Packet-loss detection

Signaling packets may be received by the R-InTracer / R-TCE over UDP. Since UDP does not guarantee reliable packet delivery, such network losses are detected by the R-InTracer / R-TCE at application level and reported by generating alarms based on configuration.

Health Monitoring

Server heartbeat process is part of Health Monitoring of InTracer components. This is not only responsible for starting/stopping of all the processes, but also has a watchdog function that ensures that:

- All the processes are up
- Every process is functioning normally

In an event where any of the process is not up or not functioning normally as expected, it can stop and restart the process to restore the system to processing state.

It also has a function to guard the system and notify the operator using logs and alarm against persistent errors which need operator intervention.

Resource Monitoring

Resource Monitoring has below two parts to monitor:

- CPU and Memory utilization of the system and other individual processes
- Disk utilization for Export, Data, Log, Crash Directories and for Disk itself

CPU and Memory utilization of the system and other individual processes: Helps monitor complete resource utilization of InTracer.
Below table describes system resources of InTracer which can be monitored.

Process | Description
--- | ---
CPU Usage (%) | CPU utilization in percentage for specific InTracer process.
Memory Usage (%) | System memory utilization in percentage for specific InTracer process.

Disk Utilization Monitoring: Disk utilization monitoring helps monitor Disk Utilization of various components of InTracer.

Disk Utilization monitoring provides for selected device name can be performed on the below components of InTracer:

- Mount Point: The name of the mount point of storage device.
- Filesystem ID: The identifier for Filesystem on storage device.

Maintenance Activities

The R-InTracer / R-TCE performs various routine activities for the system maintenance and for persistent storage. These activities include the following:

- Call event information and SA Key information is stored for a configurable period configured at install time
- Running DB verification to ensure data is in proper and valid form
- Backup of InTracer system state information, if enabled
- Compression of processing log data, removal of older logs

These activities are performed in background by UNIX cron jobs that are invoked periodically by the system. Not all these activities are done in a single run. These are shared across various invocations of the jobs at different times of the day when the processing load on the R-InTracer / R-TCE is assumed to be less. The R-InTracer / R-TCE facilitates operators to specify configurations for all these activities. These configurations can be performed through the InTracer clients.

Statistical Counters

The R-InTracer / R-TCE process maintains current statistical information for each of its functions. The InTracer clients provide an interface to view these statistics. These statistics can be used to:

- Perform analysis of call events
- Measure performance of the AGW and the R-InTracer / R-TCE and health of the link between them

C-InTracer Features and Functions

The following are the features and functions supported by the C-InTracer

Feature | Configurable | Default | Nodal Trace | 3GPP Trace
--- | --- | --- | --- | ---
Query Processing | Yes | - | Yes | Yes
User Management | Yes | Enabled | Yes | Yes
System Configuration | Yes | Admin only | Yes | Yes
Gateway Configuration | Yes | Admin only | No | Yes
Query Management | Yes | All users | Yes | Yes
Audit Trail | Yes | 30 days | Yes | Yes
Health Monitoring | Yes | Enabled | Yes | Yes
Resource Monitoring | Yes | Enabled (default threshold values) | Yes | Yes
Maintenance Activities | Yes | Enabled (daily midnight) | Yes | Yes
Statistical Counters | No | Always | Yes | Yes

Query Processing

For C-InTracer(Nodal installation), Session related queries can be based on IMSI, NAI, Call-Id, Disconnect reasons and so on, limited by a time window.

For R-TCE (3gpp installation), Session related queries can be based on IMSI, IMEI/IMEISV, Trace Session reference, and so on, limited by a time window.

System Configuration

Use the system setup to manage and configure In Tracer components such as the R-InTracer / R-TCE Server, the C-InTracer and the Event Source.

Audit Trail

This keeps track of various system activities such as User session login/logout and configuration change activities. This audit trail is helpful for troubleshooting and monitoring the system activities.

Health Monitoring

Server Hearth beat process is part of Health Monitoring of InTracer components. This is not only responsible for starting/stopping of all the processes, but also has a watchdog function that ensures that:

- All the processes are up
- Every process is functioning normally

In an event where any of the process is not up or not functioning normally as expected, it can stop and restart the process to restore the system to processing state.

It also has a function to guard the system and notify the operator using logs and alarm against persistent errors which need operator intervention.

Resource Monitoring

Resource Monitoring has below two parts to monitor:

- CPU and Memory utilization of the system and other individual processes
- Disk utilization for Export, Data, Log, Crash Directories and for Disk itself

CPU and Memory utilization of the system and other individual processes: Helps monitor complete resource utilization of InTracer. Below table describes system resources of InTracer which can be monitored.

Process | Description
--- | ---
CPU Usage (%) | CPU utilization in percentage for specific InTracer process.
Memory Usage (%) | System memory utilization in percentage for specific InTracer process.

Disk Utilization Monitoring Helps monitor Disk Utilization of various components of InTracer. Disk Utilization monitoring Provides for selected device name can be performed on the below components of InTracer:

- Mount Point: The name of the mount point of storage device.
- Filesystem ID: The identifier for Filesystem on storage device.

Maintenance Activities

The C-InTracer performs various routine activities for the system maintenance and for persistent storage. These activities include the following:

- Call event information and SA Key information is stored for a max period of 30 days and information exceeding this storage period is automatically deleted by the server
- Running DB verification to ensure data is in proper and valid form
- Backup of InTracer system state information, if enabled
- Compression of processing log data, removal of older logs

These activities are performed in the background by UNIX cron jobs that are invoked periodically by the system. Not all of these activities are done in a single run. These are shared across various invocations of the jobs at different times of the day when the processing load on the R-InTracer / R-TCE is assumed to be less. The R-InTracer / R-TCE allows operators to specify configurations for all of these activities. These configurations can be performed through the InTracer clients.

Statistical Counters

The C-InTracer process maintains current statistical information for each of its functions. Cumulative statistics are collected on a daily basis and stored in the server statistical database. The InTracer clients provide an interface to view these statistics. These statistics can be used to:

- Perform analysis of call events
- Measure performance of the AGW and the R-InTracer / R-TCE and health of the link between them

InTracer Web User Interface

A unified Web-based user interface for performing configuration and querying in InTracer.

InTracer System Requirements

This section provides InTracer System Requirements.

InTracer Client Pre-requisites

The InTracer Nodal Trace solution is for Cisco ASR5xxx based gateway services. The InTracer Client in this case requires an ASR5xxx based gateway.

The InTracer 3GPP Trace solution is for Cisco 7600 based gateway services. The InTracer Client in this case requires a 7600 SAMI based gateway.

Note:

Intracer supports addition/configuration of services on ASR5000 Nodes. Intracer can be used to configure Tracing Profiles on ASR5000 boxes, activate Traces on them, Parse 3gpp XML files received from ASR boxes and display appropriate search results under 3gpp Trace searches.

R-InTracer / R-TCE Pre-requisites

Following are the hardware pre-requisites for the R-InTracer / R-TCE:

Hardware Parameters | Server option 1 | Server option 2
--- | --- | ---
Server | UCS C460 | UCS C210
CPU |  |
Memory | 32GB | 32GB
Internal Disk | 2 * 300 GB SCSI | 2 * 300 GB SCSI
HBA Card | 2 * 4 Gbps FC ports | 2 * 4 Gbps FC ports
Brocade Switch |  |
External Disk | Sun StorageTek 2540 | Sun StorageTek 2540
Network Interfaces | 2 physical interfaces >= 100 Mbps | 2 physical interfaces >= 100 Mbps

Following are the software pre-requisites for R-InTracer / R-TCE:

Software Parameters | Description
--- | ---
Operating System | Custom RHEL 64-bit distribution (with all required packages and patches)
File System | XFS

C-InTracer Pre-requisites

Following are the hardware pre-requisites for the C-InTracer:

Hardware Parameters | Server option 1 | Server option 2
--- | --- | ---
Server | UCS C460 | UCS C210
CPU |  |
Memory | 32 GB | 32 GB
Internal Disk | 2 * 300 GB SCSI | 2 * 300 GB SCSI
HBA Card | 2 * 4 Gbps FC ports | 2 * 4 Gbps FC ports
Brocade switch |  |
External Disk | Sun StorageTek 2540 | Sun StorageTek 2540
Network Interfaces | 1 physical interfaces >= 100 Mbps | 1 physical interfaces >= 100 Mbps

IMPORTANT:

The HBA card, Brocade switch and External disk are required for 3GPP Trace only.

Following are the software pre-requisites for C-InTracer:

Software Parameters | Description
--- | ---
Operating System | Custom RHEL 64-bit distribution (with all required packages and patches)
File System | Ext3,Ext4,XFS

InTracer User Work Station Pre-requisites

IMPORTANT:

There are no specific hardware pre-requisites for the InTracer User Work-station
Following are the software pre-requisites for User work station:

Software Parameters | OS Option1 | OS Option2
--- | --- | ---
OS | Windows 7, Windows XP | Mac OS
Browsers | Firefox 3.6Internet Explorer | Firefox 3.6

Supported Features

The InTracer solution currently supports two kinds of tracing functionalities.

Nodal Trace

A tracing solution for Cisco ASR5xxx based gateway services. In this tracing solution, once tracing is activated on the gateway, it sends a copy of all signaling packets (transmitted and received) for each subscriber session as part of normal processing to an external server.

- Protocol packets are sent from the lowest layer where subscriber or session information is available.
- Additional call correlation information is also sent along with the protocol packets.
- Correlation of packets is limited to a single gateway.
- Additional Cisco ASR5xxx specific system events like Card / CPU / Task crash etc are sent for hardware / software failures to associate call failures (if any) to these events.

3GPP Trace

A 3GPP standard based tracing solution. In this tracing solution, the gateway sends a copy of all signaling packets (transmitted and received) for only the trace sessions that are explicitly activated (either through management-based activation or signaling-based activation).

- All packets are GTPv2 packets.
- Additional correlation information like the Trace Session Reference is sent along with the protocol packets.
- Correlation of packets using Trace Session Reference is across all gateways on which the same trace is activated.
- Support available for PGW and SGW.