Feedback
Table Of Contents
Collection Manager Software Package
Priority Queues and Persistent Buffers
Topper/Aggregator (TA) Adapter
TA Adapter Memory Requirements
RAG Adapter Aggregation Buckets
RAG Adapter Process for HTTP Transaction Usage RDR
RAG Adapter Process for Video Transaction Usage RDR
RAG Adapter Mobile Based Reports
RAG Adapter Process for Subscriber Usage RDR
Managing OSFP support for Subscriber Usage RDR
Collection Manager Overview
Introduction
This chapter is an overview of the Cisco Service Control Management Suite (SCMS) Collection Manager (CM) operating with the Cisco Service Control Engine (SCE). The Cisco Collection Manager uses the hardware capabilities of the SCE platform and identifies the combination of Cisco-specific applications that create the Cisco Service Control solution.
The Collection Manager also interoperates with the Cisco Service Control solution in combination with the Cisco ASR 1000 Series Aggregation Services Routers and Cisco IOS Flexible NetFlow. Refer to the Cisco Service Control Management Suite Collection Manager NetFlow User Guide for information on this solution.
This chapter describes the following:
•
How the CM works
•
•
•
The chapter includes the following sections:
•
Data Collection Process
Cisco SCE platforms generate RDRs. The application running on the SCE platform, such as the Cisco Service Control Application for Broadband (SCA BB), specifies and defines the RDRs.
1.
2.
3.
•
•
•
You can use preinstalled utility scripts to customize many of the parameters that influence the behavior of the Collection Manager.
Note
Raw Data Records
The SCE platforms generate Raw Data Records (RDRs). The RDRs, their fields, and their semantics depend on the specific service control protocol (SCP) application. Each RDR type has a unique ID known as an RDR tag.
Table 2-1 presents examples of RDRs produced by SCP applications:
Collection Manager Software Package
The Collection Manager software package contains a group of processing and sorting modules. These modules include the following components:
•
Raw Data Record Server
As each incoming raw data record (RDR) arrives from an SCE platform, the RDR server adds an arrival timestamp and the ID of the source SCE platform, and then sends the RDR to the categorizer.
Categorizer
A categorizer classifies each RDR according to its RDR tag. It determines the destination adapter for the RDR and the priority queue through which the RDR is sent.
An RDR can be mapped to more than one adapter. A qualified technician defines the flow in a configuration file, based on user requirements.
Priority Queues and Persistent Buffers
Each adapter has one or more priority queues; a persistent buffer is assigned to each priority queue.
A priority queue queues each RDR according to its priority level and stores it in a persistent buffer until the adapter processes it.
A persistent buffer is a nonvolatile storage area that ensures that the system processes RDRs even in cases of hardware, software, or power failures.
Adapter Description
Adapters are software modules that transform RDRs to match the requirements of the target system. The adapter distributes the RDRs upon request. Currently, the following adapters are shipped with the system:
•
•
•
Some of the adapters send data to the database or write data to CSV files. The structures of the database tables, and the location and structures of these CSV files are described in Cisco Service Control Application for Broadband Reference Guide.
Each adapter has its own configuration file. All the configuration files are similar in structure. For a sample RAG adapter configuration file, see the "ragadapter.conf File" section.
JDBC Adapter
The Java Database Connectivity (JDBC) adapter receives RDRs, processes them, and stores the records in a database.
This adapter is compatible with any database server that is JDBC-compliant, and transforms the records accordingly. You can configure the JDBC adapter to use a database operating on a remote machine.
The JDBC adapter is preconfigured to support the following databases:
•
•
Note
recyclebin=off•
Comma Separated Value Adapter
The comma-separated-value (CSV) adapter receives RDRs, processes them, and writes the records to files on the disk in the CSV format. Using standard mechanisms such as FTP, a service provider OSS or a third-party billing system can retrieve these records to generate enhanced accounting and network traffic analysis records.
Custom Adapter
The Custom Adapter allows you to configure the RDR format. The Custom Adapter receives records with specified fields from the RDR.
You can configure only a few fields of TUR. Some of the configurable fields are Sub-ID, Upstream + Downstream and Application. The corresponding RDR is processed with those specific fields and stored in a CSV file. The Custom Adapter also supports basic arithmetic operations (add, sub, maul) on a few of the RDR fields.
The Custom Adapter process is disabled by default. Edit the ~scmscm/cm/config/cm.conf file and restart the Cisco Collection Manager to enable the Custom Adapter. For more information on enabling Custom Adaptor, see "Enabling Adapters" section.
Topper/Aggregator (TA) Adapter
The topper/aggregator (TA) Adapter receives subscriber usage RDRs, aggregates the data that they contain, and outputs Top Reports to the database and aggregated daily statistics of all the subscribers (not just the top consumers) to the CSV files. Top Reports are lists of the top subscribers for different metrics (for example, the top 500 volume or session consumers in the last hour).
The TA Adapter maintains a persistent saved state (saved to disk) to minimize any data loss in case of failure.
You can configure the TA Adapter, which uses the JDBC adapter infrastructure, to use any JDBC-compliant database, either locally or remotely.
For Release 3.7.2 and later, Package based aggregation has been added to the TA Adapter along with the existing service based aggregation. Now, the TA Adapter aggregates the records and lists the top subscribers for each service/package combination with different metric. The package based aggregation is disabled by default. To enable the package based aggregation in the TA Adapter the user has to update the agg_pkg_level property as true in the ~scmscm/cm/config/taadapter.conf file and restart the Cisco Collection Manager
Note
Note
•
TA Adapter Cycles
The TA Adapter works in three cycles: short, long, and peak hours (specific hour range). Cycles are fixed intervals. This data is temporily stored in a persistent buffer directory with a default location of ~scmscm/cm/adapters/TAAdapter. At the end of an interval, the adapter outputs its aggregated information to the database and to a CSV file. The default interval for the short cycle is 1 hour and the long cycle is 24 hours (every day at midnight). You can configure an interval (defined in minutes) and its start and end time. Configure the interval for the peak hours cycle. At the end of the peak hours, the adapter aggregates and outputs details of the top subscribers to the database.
Note
The activities in each cycle differ slightly, as follows:
•
–
–
•
–
–
–
•
–
–
Note
TA Adapter Memory Requirements
The TA Adapter functions correctly only if you dedicate a sufficient amount of memory to the TA Adapter. Configure the value in the cm.conf file in the following location:
[adapter_mem] com.cisco.scmscm.adapters.topper.TAAdapter=<Memory for TA Adapter>To calculate the recommended amount of memory for the TA Adapter, use the following formula:
Memory (MB) = (3.5 * TOTAL_SUBSCRIBERS * (AVG_SUBS_ID_LENGTH + 32 * ACTIVE_SERVICES) / (1024 * 1024)) * MAX_NO_OF_PKGS.
Where:
•
•
In most cases, the average character length is approximately 20.
•
The default value is 10.•
Note
Note
For Linux JRE 64 bit, you can set higher values for the configured memory.
Note
For Solaris JRE 64 bit, you can set higher values for the configured memory. To configure the TA or RAG adapters to run with the JRE 64 bit, see the "[adapter_mem] Section" section.
Real-Time Aggregating Adapter
The RAG adapter processes RDRs of one or more types and aggregates the data from predesignated field positions into buckets. The contents of the buckets are written to CSV files.
•
•
•
•
•
RAG Adapter Aggregation Buckets
A RAG adapter aggregation bucket is indexed by combining values from fields in the RDR. The indexing relation can be one-to-one or many-to-one.
The values in the bucket-identifying fields are processed using closures (equivalence classes), which are configured per type of RDR.
Example:
Bucket-identifying field = field number 3 Closures: 4 = 4,5,6; 10 = 8,10,11 Value in field 3 = 4, 5, or 6; field reported as 4 Value in field 3 = 8, 10, or 11; field reported as 10You can configure the adapter to monitor the values in certain fields for change relative to the values in the first RDR that entered the bucket. For each monitored field, an action is performed when a value change is detected. The supported actions are:
•
•
Buckets, closures, triggers, and trigger actions are defined in an XML file. For a sample XML file, see the "ragadapter.xml File" section.
Flushing a Bucket
When a bucket is flushed, it is written as one line to a CSV file.
The trigger for flushing a bucket (a checkpoint) is the earliest occurrence of any of the following:
•
•
•
•
The trigger to close a CSV file is the earliest occurrence of one of the following:
•
•
•
RAG Adapter Process for HTTP Transaction Usage RDR
If a corresponding RDR TAG is configured under the RAG adapter section in the queue.conf file before the Collection Manager is started, the RAG adapter processes HTTP_TUR RDRs.
An XML file specific to HTTP_TUR is included in the Collection Manager distribution for the RAG adapter to manage HTTP_TUR RDR fields. The aggregation period for processing the RDRs is specified in the http_TURs.xml file.
Aggregation is based on the domain, package, and service for the corresponding HTTP Transaction Usage RDR. At the end of the aggregation period, the adapter adds the aggregated Top Reports for hosts to the RPT_TOP_HTTP_HOSTS database table. The adapter adds the aggregated Top Reports for domains to the RPT_TOP_HTTP_DOMAINS database table.
RAG Adapter Process for Video Transaction Usage RDR
If you configure a corresponding RDR TAG under the RAG adapter section in queue.conf file before you start the Collection Manager, the RAG adapter processes VIDEO_TUR RDRs.
The Collection Manager distribution includes an XML file specifically for VIDEP_TUR, which enables the RAG adapter to manage VIDEO_TUR RDR fields. The aggregation period for processing the RDRs is specified in the video_TURs.xml file.
Aggregation is based on the domain, package, and service for the corresponding VIDEO Transaction Usage RDR. At the end of the aggregation period, the adapter adds the aggregated Top Reports for hosts to the RPT_TOP_VIDEO_HOSTS database table. Also, the adapter adds the aggregated Top Reports for domains to the RPT_TOP_VIDEO_DOMAINS database tables.
RAG Adapter Mobile Based Reports
The Cisco Collection Manager supports either GSM or CDMA mobile based reports. GSM reports are enabled by default.
The ~scmscm/cm/config/ragadapter.conf
file property (vsa_type=gsm or cdma) configures the mobile report type processed.
The user has to place the related xml files in ~scmscm/cm/config/ragadapter for the Cisco Collection Manager process. By default all the XML files are available in the ~scmscm/cm/config/ragadapter/repository directory.
RAG Adapter Process for Subscriber Usage RDR
If a corresponding RDR TAG is configured under the RAG adapter section in queue.conf before the Collection Manager is started, the RAG adapter processes NUR RDRs. The Subscriber Usage RDRs are processed in two ways:
•
Aggregation Based on VSA Fields
•
•
The Collection Manager supports any one of the Mobile based Reports (either GSM or CDMA). By default, GSM reports are enabled. In ~scmscm/cm/config/ragadapter.conf file a new property (VSA_Type = gsm or cdma) is added to configure which mobile report the Cisco Collection Manager needs to process.
During restart, based on the vsa_type configuration the Cisco Collection Manager will copy the corresponding xml config file to the ~scmscm/cm/config/ragadapter for processing.
Aggregation Based on GSM specific VSA Fields
The vsa_SURs.xml file manages GSM specific VSA fields in Subscriber Usage RDRs. The aggregation period for processing the RDRs is specified in this file. Aggregation is based on the GSM specific VSA fields (APN, SGSN, NETWORK_TYPE, DEVICE_TYPE, and USER_LOCATION). At the end of the aggregation period, the adapter adds the aggregated Top Reports for the GSM specific VSA fields to their corresponding database tables:
•
•
•
•
•
Aggregation Based on CDMA specific VSA Fields
The cdma_SURs.xml file manages CDMA specific VSA fields in Subscriber Usage RDRs. The aggregation period for processing the RDRs is specified in this file. Aggregation is based on the CDMA specific VSA fields (PCF ,HOME AGENT, MEID). At the end of the aggregation period, the adapter adds the aggregated Top Reports for the CDMA specific VSA fields to their corresponding database tables:
•
•
•
Aggregation Based on Package
The vlink_BW_per_pkg.xml file manages upstream and downstream VLINKs in Subscriber Usage RDRs. The aggregation period for processing the RDRs is specified in this file.
Aggregation is based on the up and down VLINK fields. At the end of the aggregation period, the adapter adds the aggregated Top Reports for the up VLINKS to the RPT_UVLINK database table. Also, the adapter adds the aggregated Top Reports for the down VLINKs to the RPT_DVLINK database table.
Managing OSFP support for Subscriber Usage RDR
OSFP (Operating System Finger Printing ) is supported in the Cisco Collection Manager software release 3.7.5 and later. New fields are added to RPT_NUR.
The ~scmscm/script/updateVSAindex.sh script supports backwards compatibilty.
The following two properties were added in the ~scmscm/cm/config/ragadpater.conf file:
•
•
Check the VSA Attribute Indicator field index value for the current Cisco Service Control Operating System (SCOS version). This is based on the SCOS version update attr_index , attr_shift_pos properties in ~scmscm/cm/config/ragadpater.conf file. The "RAG Adapter" section details the RAG adapter configuration file ragadapter.conf. Then execute the script ~scmscm/script/updateVSAindex.sh script. The script updates the corresponding XML file (vsa_SURs.xml/cdma_SURs.xml).
Table 2-1 shows the VSA Attribute Indicator field index values.
Table 2-1 VSA Attribute Indicator Field Index Value
3.7.5
17
0
3.7.0
14
3
Using Databases
The Collection Manager can use either a bundled database or an external database to store RDRs supplied by the SCE platforms.
Using the Bundled Database
In bundled mode, the Collection Manager uses the Sybase Adaptive Server Enterprise database. This database enables you to do the following:
•
•
•
The Sybase database is located on the same server as the other Collection Manager components. It uses a simple schema that includes a group of small, simple tables.
1.
2.
You maintain the database by using operating system commands and scripts. The Collection Manager supports automatic purging of old records from the bundled database. By default, the Collection Manager automatically purges the report tables of every record that is more than two weeks old. The Collection Manager polls the records once every hour. You can configure database maintenance by using the dbperiodic.sh utility script. For more information, see the "Managing the CM:RDR Periodic Deletion of Old Records" section.
Using an External Database
With the JDBC adapter, you can use any JDBC-compliant database (for example, Oracle or MySQL) with the Collection Manager. In this case, the database can be local or remote. To use an external database, perform the following tasks:
•
•
•
For details about configuring the Collection Manager to work with an external database, see Chapter 5 "Managing Databases and the Comma Separated Value Repository".
