Cisco Data Collection Manager

• Overview of Cisco Data Collection Manager
• Configuration and Deployment
• Data Collection
• Data Processing
• Data Export and Retrieval
• Performance Management Solutions

Two challenges that network providers usually face are data gathering and data analysis. Network providers need to gather large volumes of data to analyze the performance of the network and to have operational control over their network. Large service providers are strengthening their data gathering and analysis infrastructure. Traditionally, Simple Network Management Protocol (SNMP) agents are used to expose management data on managed systems. But, SNMP is not well suited for gathering large volumes of data, especially over short time intervals.

For example, service providers charge customers depending on the network usage. Also this data must be available on customer request. Accounting applications based on SNMP polling models consume significant network bandwidth because they poll large volumes of data frequently. The SNMP protocol data unit (PDU) is a complex data type specific to SNMP and is expensive to process because the SNMP objects and tables must be sorted in a lexicographic order. All the entries in SNMP MIB tables are lexicographically ordered by their object identifiers, because there is an implied ordering in the MIB based on the order of the object identifiers.

In such cases, the need to continuously poll large or bulk SNMP statistics can be avoided by using applications known as collectors to retrieve data.

The Bulkstat application is one such collector that uses the services of the Data Collection Manager (DCM) to provide the following functions:

• Collecting SNMP MIB object values, the output of show commands, and the results of expression data.
• Processing the collected data to create summary, percentiles, and auto-baselined values.
• Exporting collected data through simple file transfers or through NetFlow exports.
• Scheduling calendar events for data collection and export.

The Bulkstat application provides command-line access through a set of new configuration commands and exclusive MIB access through CISCO-DATA-COLLECTION-MIB to collect SNMP data.

You can configure Bulkstat for the following functions:

• Specify the way Bulkstat retrieves bulk statistics.
• Specify the time interval in seconds at which Bulkstat transfers data to receivers.
• Specify the maximum size of the bulk statistics file.
• Specify the context, instance, and period at which the system retrieves bulk statistics.
• Configure file-related parameters.
• Configure the interface type on which you want to collect statistics.
• View the parameters that Bulkstat uses to collect statistics by using the show bulkstat commands.

The following list shows the elements that you can configure using the Bulkstat interface:

• Data Set
• Instance Set
• Filter Set
• Process Set
• Data Group
• Data Profile
• Calendar Scheduling

You can configure the Data Collection Manager (DCM) module to collect show command output through the Bulkstat CLI. The DCM captures the raw show command output and periodically exports the data through the ASCII file format.

The Data Collection Manager (DCM) can be configured to evaluate expressions as defined in CISCO-EXPRESSION-MIB and collect the results on a periodic basis. The DCM is also capable of periodically transferring the collected data in a file format.

Data processing allows users to derive information from raw SNMP data, by calculating summaries and percentiles.

Service providers rely on monitored SNMP data to alert network management systems (NMSs) of changing network conditions. By periodically monitoring the device data and comparing it against a set of thresholds, the network can automatically alert the operators, thereby allowing efficient operations.

The file data export feature on the Data Collection Manager (DCM) exports the collected data based on the transfer configurations. Data can be exported in various formats, and Bulkstat files are one such format to collect data. The format in which the data is inserted into the file conforms to the schemaAscii format described in CISCO-DATA-COLLECTION-MIB and CISCO-BULK-FILE-MIB. The data sequence in which the data is stored is determined based on the sequence in which the data is received.

The Cisco File Transfer module is responsible for transferring the files as per the transfer configuration. This module interfaces with the Cisco IOS IFS module to transfer the file to the specified URL. A file can be retained in the device whether the transfer was a success or a failure.

File names are created using the following format:

• Raw data file name: <profile-name>_<host-name>_raw_<time when the file is created in "%y%m%d_%H%M%S%k">
• Processed data file name: <profile-name>_<host-name>_proc_<time when the file is created in "%y%m%d_%H%M%S%k">

The NetFlow protocol is used extensively in network and application performance management to collect IP traffic information. Many enterprises today rely on NetFlow along with traditional Simple Network Management Protocol (SNMP) data. The benefits of having the Data Collection Manager (DCM) collect and export the SNMP data over NetFlow are as follows:

1. NetFlow export allows management vendors that are NetFlow centric to use the functionalities of the SNMP collection MIB without using the SNMP stack. This allows vendors to provide better visibility into customer networks without huge development efforts.
2. NetFlow export provides an efficient output. NetFlow export can eliminate the need to store data in the device for a long period of time. Also, continuous NetFlow export allows for real-time troubleshooting, where a network management system (NMS) can turn on high-granularity monitoring without a storage concern.

The first step in configuring the Simple Network Management Protocol (SNMP) periodic data collection and transfer mechanism is to configure one or more data sets. A data set is used to group objects of similar types, based on the data source. The data set is defined outside of the data group. This external definition gives the user the flexibility to use the same data set across multiple data groups and to collect the output for different instances and different contexts.

Note	All objects in an SNMP data set must be indexed by the same MIB index. However, the objects in the data set must not belong to the same MIB or the MIB table.

Perform this task to configure the SNMP Bulkstat data set.

1.    enable

2.    configure terminal

3.    bulkstat data data-set-name type snmp

4.    object oid[alias alias-name]

5.    end

1.    enable

2.    configure terminal

3.    bulkstat instance instance-set-name type snmp

4.   exact oid oid

5.   exact interface interface-id

6.   wildcard

7.   wildcard oid oid

8.   wildcard interface interface-id

9.   repetition oid oid max value

10.   range start oid end oid

11.    end

1.    enable

2.    configure terminal

3.    bulkstat filter filter-set-name

4.    match object-name {eq line | start line | not {eq line | start line }}

5.    end

1.    enable

2.    configure terminal

3.    bulkstat data data-set-name type command

4.   add cmd command-line

5.    end

1.    enable

2.    configure terminal

3.    bulkstat data data-set-name type expression

4.   expression expression-line

5.   value type {counter32 | counter64 | integer32 | ipaddress | objectid | unsigned32 | timeticks | octetstring}

6.   object object-number

7.   id oid

8.   wildcard

9.   sample {absolute | changed | delta}

10.   conditional object conditional-object-id [wildcard]

11.   discontinuity object discontinuity-object-id [type {date-and-time | timestamp | timeticks} | wildcard [type {date-and-time | timestamp | timeticks}]]

12.   end

1.    enable

2.    configure terminal

3.    bulkstat data-group data-group-name

4.   collect type {{command | expression} data data-set-name filter filter-set-name | snmp {data data-set-name instance instance-set-name filter filter-set-name }

5.   context context-name

6.   interval polling polling-interval

7.   discard

8.   process

9.   end

1.   enable

2.    configure terminal

3.    bulkstat profile profile-name

4.   data-group data-group-name

5.   interval transfer {process | raw }{seconds}}

6.   file format schemaASCII

7.   file retain {disk url | memory seconds}

8.   file size bytes

9.   file transfer {retry number | url {primary url | secondary url }}

10.   flow exporter fnf-exporter-name

11.   enable

12.   end

1.    enable

2.    configure terminal

3.    bulkstat schedule schedule-name at time-of-day { date | week | month date {oneshot | recurring} | oneshot | recurring}

4.    profile profile-name start {oneshot | recurring number}

5.    profile profile-name stop

6.    end

The following shows the configuration steps that you need to perform to collect SNMP interface statistics for the specified interface, Ethernet. The polling interval is set at 30 seconds, while the interval at which the raw data is transferred is set at 60 seconds.

Device> enable
Device# configure terminal
Device(config)# bulkstat data if-mib type snmp
Device(config-bs-ds-snmp)# object ifDescr
Device(config-bs-ds-snmp)# object ifInOctets
Device(config-bs-ds-snmp)# object ifOutOctets
Device(config-bs-ds-snmp)# exit
Device(config)# bulkstat instance if-mib type snmp
Device(config-bs-is-snmp)# wildcard
Device(config-bs-ds-snmp)# exit
Device(config)# bulkstat filter if-mib
Device(config-bs-fs)# match ifDescr start "Ethernet"
Device(config-bs-fs)# exit
Device(config)# bulkstat data-group if-group
Device(config-bs-dg)# interval polling 30
Device(config-bs-dg)# collect type snmp data if-mib instance if-mib filter if-mib
Device(config-bs-dg)# exit
Device(config)# bulkstat profile snmp_profile
Device(config-bs-profile)# file transfer url primary tftp://10.64.68.12/dcm_data/
Device(config-bs-profile)# interval transfer raw 60
Device(config-bs-profile)# data-group if-group
Device(config-bs-profile)# enable
Device(config-bs-profile)# end
Device#