A collection job describes the following:
A single collection job can contain either CLI commands, SNMP MIB requests, or MDT subscriptions. Crosswork Data Gateway routes
the request to the appropriate collector to fetch the requested data.
A collection job has three main parts:
// Device Groups(s) -- identifies the different device groups from which data is
// to be collected and their authentication credentials.
// Source(s) -- identifies the different sensors and their collection details
// like sensor path , devices, etc
// Sink(s) -- identifies the different output destination the final CDG data
// is sent to
//
//
A typical collection request contains at least the following information:
-
Collection Job ID
-
Method of collection
-
Device Groups
-
Sensor Path (CLI, SNMP, or MDT)
-
Cadence or Collection Interval
-
Output Target
Shown below is a sample collection job payload:
{
"collection-job-id": "<sample-collection-job>",
"collectorType": "CLI",
"deviceGroups": [
{
"id": "deviceGroup1",
"devices": [
{
"name": "vm1-xrv9000",
"host": "<device_IP_address>"
},
{
"name": "vm2-xrv9000",
"host": "<device_IP_address>"
},
{
"name": "vm3-xrv9000",
"host": "<device_IP_address>"
}
],
"deviceParams": [
{
"key": "software",
"value": "IOS"
},
{
"key": "transport",
"value": "SSH2"
},
{
"key": "port",
"value": "22"
},
{
"key": "username",
"value": "admin"
},
{
"key": "password",
"value": "<password>"
},
{
"key": "vendor",
"value": "cisco"
}
]
},
{
"id": "deviceGroup2",
"devices": [
{
"name": "vm1-xrv9000",
"host": "<device_IP_address>"
},
{
"name": "vm2-xrv9000",
"host": "<device_IP_address>"
}
],
"deviceParams": [
{
"key": "software",
"value": "IOS"
},
{
"key": "transport",
"value": "SSH2"
},
{
"key": "port",
"value": "22"
},
{
"key": "username",
"value": "admin"
},
{
"key": "password",
"value": "<password>"
},
{
"key": "vendor",
"value": "cisco"
}
]
}
],
"sources": [
{
"name": "source1",
"deviceGroupIds": [
"deviceGroup1",
"deviceGroup2"
],
"sensorConfigs": [
{
"sensorConfigId": "sensor-id-1",
"path": "show interfaces",
"cadence": 90,
}
]
}
],
"sinks": [
{
"name": "sink2",
"parentIds": [
"source1"
],
"outputTarget": {
"messageBusTarget": {
"brokerAddresses": [
{
"host": "kafka-operational",
"port": 9092
}
],
"username": "kafka-admin",
"password": "password",
"destinationTopic": "externalOutputTopic",
"certificateKey": "BasicOne"
}
}
} ,
{
"name": "sink2",
"parentIds": [
"source"
],
"outputTarget": {
"grpcTarget": {
"password": "<password>",
"username": "admin",
"certificateKey": "some certificate",
"grpcAddress": {
"port": 60061,
"host": "<gRPC_server_IP_address>"
}
}
}
}
],
"secretKey": "skey"
The following table explains the fields of the above payload:
|
Field
|
Type
|
(M)andatory or (O)ptional
|
Description
|
|
collection-job-id
|
string
|
M
|
String for unique identification of the collection job.
|
|
collectorType
|
string
|
M
|
Acceptable values are MDT, CLI or SNMP to indicate the type of job.
|
|
deviceGroups
|
json array
|
M
|
An array of information on device details. Each device group contains the details of one device type.
|
|
deviceGroups[x] > id
|
string
|
M
|
String for the unique identification of the device group.
|
|
deviceGroups[x] > devices
|
json array
|
M
|
An array of device names and IP addresses.
|
|
deviceGroups[x] > devices[y] > name
|
string
|
M
|
Name of the device.
|
|
deviceGroups[x] > devices[y] > host
|
string
|
M
|
IP address of the device.
|
|
deviceGroups[x] > devicesParams
|
json array
|
M
|
An array of information on devices' authentication credentials.
|
|
deviceGroups[x] > devicesParams[y] > key
|
string
|
M
|
Key required for device authentication.
|
|
deviceGroups[x] > devicesParams[y] > value
|
string
|
M
|
Value for the key to authenticate the device.
|
|
sources
|
json array
|
M
|
An array of information on source details that can have multiple source info(JSON objects).
|
|
sources[x] > name
|
string
|
M
|
A name for the source.
|
|
sources[x] > deviceGroupIds
|
json array
|
M
|
An array containing names of the device groups from which data has to be collected.
|
|
sources[x] > sensorConfigs
|
json array
|
M
|
An array of Sensor Configs (JSON Obects). Each Sensor Config (JSON object) element contains config details of one collection.
At least one sensor config per collection job is a must.
| Note
|
For a registered application, each source of sensor config should have a unique string name.
|
|
|
sources[x] > sensorConfigs[y] > sensorConfigId
|
string
|
M
|
Unique ID of the sensor config.
|
|
sources[x] > sensorConfigs[y] > path
|
string
|
M
|
Mandatory path of sensor config containing the CLI command in case of CLI collector OR SNMP OID in case of SNMP collector
OR YANG PATH in case of MDT collection. In case of MDT collection, path may also contain *:* indicating to send all sensors
output for this collection job.
|
|
sources[x] > sensorConfigs[y] > cadence
|
int
|
O
|
Optional cadence value of this sensor config in seconds. If not passed the sensor config collection is done once.
|
|
sinks
|
json array
|
M
|
This contains an array of destination sink information(JSON objects).
This can provide details on one more sinks of same/different types. Currently supported sink types are Kafka and gRPC.
|
|
sink[x] > name
|
string
|
M
|
A unique name for the sink.
|
|
sink[x] > parentIds
|
string array
|
M
|
An array of parent IDs from which the sink should get its input.
|
|
sink[x] > outputTarget
|
json object
|
M
|
A JSON object that holds the final transport information. It can hold one or more type of target information. Currently supported
target types are kafka and gRPC
|
|
sercretKey
|
string
|
O
|
A key that is needed for validating the collection job.
|
|
Sink with Kafka output target
|
|
sinks[x] > outputTarget > messageBusTarget
|
json object
|
M
|
This contains connection information of the target Kafka server.
|
|
sinks[x] > outputTarget > messageBusTarget > brokerAddresses
|
json object
|
M
|
Array containing host+port information.
|
|
sinks[x] > outputTarget > messageBusTarget > brokerAddresses > host
|
string
|
M
|
An IP address or reachable name of the Kafka server.
|
|
sinks[x] > outputTarget > messageBusTarget > brokerAddresses > port
|
int
|
M
|
Port number based on transport protocol to be used to connect to the Kafka server.
|
|
sinks[x] > outputTarget > messageBusTarget > username
|
string
|
M
|
Credential username to connect to the Kafka server.
| Note
|
Username and password are not supported in Crosswork Data Gateway 1.0. Any value passed is ignored.
|
|
|
sinks[x] > outputTarget > messageBusTarget > password
|
string
|
M
|
Credential password to connect to the Kafka server.
| Note
|
Username and password are not supported in Crosswork Data Gateway 1.0. Any value passed is ignored.
|
|
|
sinks[x] > outputTarget > messageBusTarget > destinationTopic
|
string
|
M
|
Name of the destination Kafka topic.
|
|
sinks[x] > outputTarget > messageBusTarget > certificateKey
|
bytes
|
M
|
Certificate Blob to connect to the Kafka server.
|
|
Sink with gRPC output target
|
|
sink[x] > outputTarget > grpctarget
|
json object
|
M
|
This contains connection information of the target gRPC.
|
|
sink[x] > outputTarget > grpctarget > grpcaddress
|
json object
|
M
|
Array containing host+port information.
|
|
sink[x] > outputTarget > grpctarget > grpcaddress > host
|
string
|
M
|
An IP address or reachable name of the gRPC server.
|
|
sink[x] > outputTarget > grpctarget > grpcaddress > port
|
string
|
M
|
Port number based on transport protocol to be used to connect to the gRPC server.
|
|
sink[x] > outputTarget > grpctarget > username
|
string
|
M
|
Credential userid to connect to the gRPC server.
| Note
|
Username and password are not supported in Crosswork Data Gateway 1.0. Any value passed is ignored.
|
|
|
sink[x] > outputTarget > grpctarget > password
|
string
|
M
|
Credential password to connect to the gRPC server.
| Note
|
Username and password are not supported in Crosswork Data Gateway 1.0. Any value passed is ignored.
|
|
|
sink[x] > outputTarget > grpctarget > certificateKey
|
bytes
|
M
|
Certificate Blob to connect to the gRPC server.
|
Feedback