Embedded Collectors for Single VM Deployment

The scope of this chapter is limited to Embedded Collectors used in the Crosswork Network Controller deployment on a single VM.

This section contains the following topics:

Embedded Collectors

For the single VM deployment, Crosswork Network Controller consists of the Cisco Crosswork Infrastructure, Embedded Collectors, and the Element Management Functions application bundled together in a package. For information about the single VM deployment, see Cisco Crosswork Network Controller 7.0 Installation Guide.

Embedded Collectors is a solution that collects network data through the collector services. The collector transfers the data to Cisco Crosswork or an external destination, or both, using either Kafka or gRPC.

You can deploy these lightweight data collectors through the Embedded Collector and Offload Component CAPP file on the Crosswork Network Controller UI. The deployment of the CAPP file includes collectors for SNMP, gNMI, syslog, and CLI.

For information on how collectors are deployed, see Cisco Crosswork Network Controller 7.0 Installation Guide.

How to access the Embedded Collector UI?

To open the Embedded Collector management view, log in to Cisco Crosswork and choose Administration > Data Collector(s) Global Settings from the left navigation bar.

Figure 1. Data Collector(s) Global Settings Window

The Data Collector(s) Global Settings page lets you perform the administrative operations through the following menus on the left pane:

  • Data destinations: After collecting the telemetry data, the collectors deposit it to an internal or external data destination. By default, Crosswork_Kafka is an internal data destination. You can define the external destinations using the Cisco Crosswork UI or APIs.

  • Device packages: By using device packages, the collectors can extend the data collection capabilities for both Cisco applications and third-party devices. The collectors support system and custom packages.

    • System packages: The system device package includes several installation files that are delivered via an application-specific manifest file. Usually, the manifest file is in JSON format.

    • Custom packages: The collectors UI allows you upload or download CLI, MIB, SNMP, and Aggregate device packages based on the type of data that you want to collect and the device.

  • Global parameters: The collectors UI allows enables you to configure the port numbers of the collector pods, which affect the data collection services. From this window, you can also enable the resync operation that automatically syncs the USM details whenever a change occurs.

Set Up Embedded Collectors to Collect Data

When does Embedded Collectors start the data collection?

The Device Lifecycle Management and Element Management Functions applications assign collection jobs to devices onboarded to the Crosswork Network Controller. This facilitates gathering essential network information and telemetry data, ensuring accurate device health reporting, and enabling the execution of other designated tasks.

Setup Workflow of the Embedded Collectors

This workflow assumes that you have already installed Embedded Collectors as explained in the Install Cisco Crosswork Network Controller on a Single VM chapter in Cisco Crosswork Network Controller 7.0 Installation Guide.

The following tasks are listed according to the default configuration that Crosswork supports for Cisco devices. Optional tasks are only required if you wish to use the advanced features.

Table 1. SetUp Workflow of the Embedded Collectors

Task

Follow the steps in...

1. Verify that the default collection jobs are created and running successfully.

Monitor Embedded Collectors Application Health

2. (Optional) Extend device coverage to collect data from currently unsupported devices or third-party devices.

Device Packages

3. (Optional) Forward data to external data destinations.

Add or Edit a Data Destination

4. (Optional) Create custom collection jobs.

Manage Collection Jobs

Data Collector Global Configuration

This section outlines the global settings that must be configured for Embedded Collectors.

Licensing Requirements for External Collection Jobs

To set up collection jobs that send data to the external destinations, you need an extra license. We recommend installing the license before configuring Crosswork to use an external destination. If you don't install the license first, you can still use the feature for 90 days under the trial license before it gets disabled.

You will not be able to create new external collection jobs but can view and delete existing ones when:

  • you don't register with CSSM after the evaluation period ends, or

  • you exceed the device count for external collection jobs, and the License Authorization Status shows "Out of Compliance".

Viewing the license status

Use these steps to view the status of your license.

  1. From the main menu, go to Administration > Application Management > Smart License.

  2. Select Crosswork Network Collector in the application field.

  3. Ensure that the status is as follows:

    • Registration Status - Registered

      Indicates that you have registered with Cisco Smart Software Manager (CSSM) and are authorized to use the reserved licensed features.

    • License Authorization Status - Authorized (In Compliance).

      Indicates that you have not exceeded the device count in the external collection jobs.

    • Under Smart Licensing Usage, CW_EXTERNAL_COLLECT(1.0) has status as In Compliance.

Manage External Data Destinations

Cisco Crosswork enables the creation of external data destinations, such as Kafka or external gRPC, which are utilized by the collection jobs to deposit the telemetry data.

To manage the data destinations, you can navigate to Administration > Data Gateway Global Settings > Data Destinations. From there, you have the options to add or modify a data destination, delete any unused destinations, and view all the configured destinations.


Note

The Crosswork_Kafka and cd-astack-pipeline are internal data destinations and cannot be updated or deleted.
Figure 2. Data Destinations Window
Data Destinations Window

The UUID is the Unique identifier for the data destination. Cisco Crosswork automatically generates this ID when an external data destination is created. When creating collection jobs using the Cisco Crosswork UI the destination for the data is selected using a drop-down list of the configured destinations. When creating a collection job via the API, you need to know the UUID of the destination where the collector is to send the data it collects.

To view details of a data destination, in the Data Destinations pane, click the icon next to the data destination name whose details you want to view. See View Data Destination Details for more information.

Add or Edit a Data Destination

Before you begin

When configuring an external Kafka server for data collection, ensure the following:

  • The following properties are configured on the external Kafka server:


    Note

    Refer to Kafka documentation for description and usage of these properties as this explanation is out of the scope of this document.

    • num.io.threads = 8

    • num.network.threads = 3

    • message.max.bytes= 30000000

  • The necessary Kafka topics for data collection are created according to your preferences.

  • The Kafka destination is configured with the 'reachability-topic' prior to initiating a new collection job. To monitor the health of the Kafka destination, this configuration is necessary.

  • You can add multiple data destinations.

  • If you reinstall an already existing external Kafka data destination with the same IP address, the collectors must be restarted for changes to take effect.

  • You can secure the communication channel between Crosswork and the specified data destination that is, either Crosswork Kafka or external Kafka. (See Step 6 in this procedure). However, enabling security can impact performance.

  • If your external data destination requires a TLS connection, keep the public certificate ready or if it requires client authentication, keep the client certificate and key files ready. The client key might be password-encrypted which needs to be configured as part of the data destination provisioning. Currently, Embedded Collectors support IP-based certificates only.

  • Ensure that the certificates are PEM encoded and the key file is in PKCS#8 format when generating them with your Certificate Authority.

  • Ensure that you create the Kafka topics before you submit the job to Crosswork. Depending on the external Kafka and how topics are managed in that external Kafka, Crosswork logs may show the following exception if the topic does not exist at the time of dispatching the collected data to that specific external Kafka/topic. This could be because the topic is not created yet or the topic was deleted before the collection job was complete. 

    destinationContext: topicmdt4
org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This server does not host this topic-partition.

  • Check and validate the port connectivity for the data destination. If the port is unreachable in the destination, it leads to a failed collection.

  • Embedded Collectors allows you to configure custom values in the destination properties for a Kafka destination (see Step 4 in this procedure).


    Note

    This feature is not supported on a gRPC destination.

  • Global properties entered in the Destination Details pane are mandatory and will be applied to the Kafka destination by default unless there are custom values specified at the individual collector level. Custom values that you specify for a collector apply only to that collector.

Follow these steps to add a new or modify a data destination. The Embedded Collectors will send the collected data to this destination.

Procedure

Step 1

From the main menu, choose Administration > Data Collector(s) Global Settings > Data Destinations.

Step 2

In the Data Destinations page, click the Add icon button. The Data Destination page opens.

If you want to edit an existing destination, select a destination and click the Edit icon button to open the Edit Destination page and edit the parameters.

Note

 

When you update a data destination, the collector using it will need to establish a new session with that destination. The collection of data will pause and continue once the session is restored.

Step 3

Enter or update these values as per the requirements from your external data destination. If values are not provided, consider using the defaults as a starting point.

Field Value Available in

gRPC

 Available in

Kafka

Destination Name

Enter a descriptive data destination name. The name can contain a maximum of 128 alphanumeric characters, plus underscores ("_") or hyphens ("-"). No other special characters are allowed.

If you have many data destinations, make the name as informative as possible to be able to distinguish later.

Yes

Yes

Server Type

From the drop-down, select the server type of your data destination.

Yes. Select gRPC

Yes. Select Kafka

Encoding

From the drop-down, select the encoding (json or gpbkv).

Yes

Yes

Compression Type

From the drop-down, select the compression type.

Kafka supports snappy, gzip, lz4, zstd, and none. The zstd compression type is supported only for Kafka 2.0 or higher.

gRPC supports snappy, gzip, and deflate.

Yes

Yes

Maximum Message Size (bytes)

Enter the maximum message size in bytes.

  • Default Value: 100000000 bytes/100 MB

  • Min: 1000000 bytes/1 MB

  • Max: 100000000 bytes/100 MB

No

Yes

Buffer Memory

Enter the required buffer memory in bytes.

  • Default Value: 52428800 bytes/52.4288 MB

  • Min: 52428800/52.4288 MB bytes

  • Max: 314572800 bytes/314.5728 MB

No

Yes

Batch Size (bytes)

Enter the required batch size in bytes.

  • Default Value: 1048576 bytes/1.048576 MB

  • Min: 16384 bytes/16.38 KB

  • Max: 314572800 bytes/314572.8 KB

No

Yes

Linger (milliseconds)

Enter the required linger time in milliseconds.

  • Default Value: 2000 ms

  • Min: 0 ms

  • Max: 5000 ms

No

Yes

Request Timeout

Enter the duration that the request waits for a response. After the configured duration is met, the request expires.

  • Default Value: 30 seconds

  • Min: 30 seconds

  • Max: 60 seconds

No

Yes

For telemetry-based collection, it is recommended to use the destination settings of Batch size as 16,384 bytes and Linger as 500 ms, for optimal results.

Step 4

(Optional) To configure custom values that are different from global properties for a Kafka destination, in the Collectors - Custom Properties pane:

  1. Select a Collector.

  2. Enter values for the following fields:

    • Custom Buffer Memory

    • Custom Batch Size

      Note

       
      The Custom Batch Size cannot exceed the value of the Custom Buffer Memory at run time. In the case, you do not provide a value in the Custom Buffer Memory field, the Custom Batch Size will be validated against the value in the Buffer Memory field.

    • Custom Linger

    • Custom Request Timeout

    Figure 3. Add Destination Window
    Add Destination Window

  3. Click + Add Another to repeat this step and add custom settings for another collector.

Note

 
Properties entered here for individual collectors take precedence over the global settings entered in Step 3. If you do not enter values in any field here, the values for the same will be taken from the Global properties entered in Step 3.

Step 5

Select a TCP/IP stack from the Connection Details options. The supported protocols are IPv4, IPv6, and FQDN.

Note

 

The FQDN addresses are supported only for the Kafka destinations.

Step 6

Complete the fields in the Connection Details pane as described in the following table. The fields displayed vary with the connectivity type you chose. The values you enter must match the values configured on the external Kafka or gRPC server.

Note

 

You can modify the port numbers for only user-defined destinations and not for system-created destinations.
Connectivity Type Fields

Available in gRPC

Available in Kafka

IPv4

Enter the required IPv4 Address/Subnet Mask, and Port. You can add multiple IPv4 addresses by clicking + Add Another

IPv4 subnet mask ranges from 1 to 32 and port range from 1024 to 65535.

Yes

Yes

IPv6

Enter the required IPv6 Address/Subnet Mask, and Port. You can add multiple IPv6 addresses by clicking + Add Another.

IPv6 subnet mask ranges from 1 to 128 and ports range from 1024 to 65535.

Yes

Yes

FQDN

Enter the required Host Name, Domain Name, and Port. The supported port range is from 1024 to 65535.

You can add multiple FQDN addresses by clicking + Add Another.

Make sure the firewall does not block the chosen port.

Yes

Yes

If the IP and port (or FQDN and port) connectivity details match an existing destination, you'll be prompted with a confirmation message to confirm creating a duplicate destination.

Step 7

(Optional) To connect securely to the Kafka or gRPC-based data destination, enable the Enable Secure Communication option by moving the slider under Security Details.

Step 8

For Kafka or gRPC-based data destinations, select the type of authentication process by choosing one of the following:

  • Mutual-Auth: Authenticates external server and Embedded Collectors after the CA certificate, and Intermediate certificate or Key is uploaded to the Crosswork UI. Mutual-Auth is the default authentication process.

  • Server-Auth: Authenticates external server and Embedded Collectors after the CA certificate is uploaded to the Crosswork UI.

Note

 

The authentication options are available only when Enable Secure Communication is enabled.

Step 9

Click Save.

What to do next

If you have enabled the Enable Secure Communication option, navigate to the Certificate Management page in the Cisco Crosswork UI (Administration > Certificate Management) and add the relevant certificate for the newly added data destination. This step is mandatory to establish a secure communication to the device. See Manage Certificates for more information.


Important

If certificate is not added or the certificate is incomplete for the data destination after enabling the Enable Secure Communication option, Cisco Crosswork sets the destination to an error state. When the destination is in an error state, the collection job status will be degraded.

View Data Destination Details

To view details of a data destination, in the Data Destinations pane, click the icon next to the data destination whose details you want to review.

View Destination Window

Delete a Data Destination

Follow the steps to delete a data destination:

Before you begin
A data destination can only be deleted if it’s not associated with any collection job. We recommend checking in the Collection Jobs view to see if any collection jobs are using the data destination.
Procedure

Step 1

From the main menu, choose Administration > Data Collector(s) Global Settings > Data Destinations.

Step 2

Select one or more data destinations that you want to delete from the list of destinations that is displayed and click the Delete icon button.

Step 3

In the Delete Data Destination(s) pop-up, click Delete to confirm.

Device Packages

Device management enables the embedded collectors to extend the data collection capabilities to the Cisco applications and third-party devices through the device packages. The Embedded Collectors supports system and custom device packages.

The system device and MIB packages are bundled in the Crosswork software and are automatically downloaded to the system instances. You cannot modify the system device and MIB packages.

Custom device package extends device coverage and collection capabilities to third-party devices.

How to access the device packages

Access the Packages pane from Administration > Data Collector(s) Global Settings. Choose System packages or Custom packages.

System Packages

A system package contains one or more separate installables. Each file set in a package belongs to the same application.

The system packages are supplied through the application-specific manifest file as a simple JSON file. System packages are added or updated whenever the applications are installed or updated. Applications can install multiple device packages.


Important

Administrators cannot modify the system device packages. Only applications can modify these files. To modify the system device packages, contact the Cisco Customer Experience team.

Figure 4. System Packages Window
System Packages Window
Download System Packages

To download a system package, from the main menu, choose Administration > Data Collector(s) Global Settings > System Packages. Click on the Download icon button next to the package name in the File Name column.

Custom Packages

You can upload the following types of custom packages to Cisco Crosswork:

  1. CLI Device Package: The CLI-based KPIs are used to monitor device health for third-party devices. All custom CLI device packages along with their corresponding YANG models should be included in the custom-cli-device-packages.tar.xz file. Multiple files are not supported. However, you can use the aggregate package if you want to bundle different files for different devices in a single package.

  2. Custom MIB Packages: Custom MIBs and device packages can be customized to third-party devices. They are used to filter the collected data or format it differently for Cisco devices. These packages can be edited. All custom SNMP MIB packages along with YANG models should be included in file custom-mib-packages.tar.xz. Crosswork does not support multiple Custom MIB package.


    Note

    The Embedded Collectors enable SNMP polling on third-party devices for standard MIBs that are already included in the system. Proprietary MIBs are required only if the collection request references MIB TABLE names or SCALAR names from a proprietary MIB. However, if the requests are OID-based, the MIBs are not required.

  3. SNMP Device Package: The Embedded Collectors allow you to extend the SNMP coverage by uploading custom SNMP device packages in the .xar format.

  4. Aggregate Package: With the aggregate package, you have the option to include multiple supported file extensions in a single package. The Crosswork UI allows you to either upload or download these packages. Each package may contain one or multiple files with the following extensions:

    Collector files

    • YANG (.yang)

    • MIB (.mib, .my)

    • Definition (.def)

    • Device Packages (.xar)

    Application files

    • Device-metadata (.yaml, .yml)

    • Zips (.zip)

    • SDU bundle (.sdu)

Upload Custom Packages

The process of adding custom packages involves bundling multiple files into a single tar.gz package format and then uploading it. This step ensures that the packages are optimized and contain only the necessary files, such as supported file extensions, and specific collector types like snmp and cli.

Follow these steps to upload a custom software package:

Before you begin

Before uploading custom software packages, ensure that you meet the prerequisites:

  • MIB package dependencies: When you upload new MIBs, ensure they can coexist with existing system MIB files and that you resolve all dependencies correctly.

  • Supported file extensions: Ensure that the package contains only supported file extensions. For a complete list of supported extensions, refer to the relevant documentation. For a full list of supported extensions, see Custom Packages.

  • Package format: Bundle the files in the .tar.gz format before uploading.

  • Collector types: The top-level directory of the package must include at least one of the following collector types:

    • snmp

    • cli, and

    • common

  • Package accessibility: When you upload the aggregate package, the files in the cli/ and snmp/ directories are accessible to their respective CLI and SNMP collectors. Files in the common/ directory will be accessible to both.

    The following is a sample directory structure for an aggregate package:

    
├── cli
│   ├── defs
│   │   └── cli-def1.def
│   ├── device-metadata
│   │   ├── cli.yml
│   │   └── cli-device-metadata.yaml
│   ├── zips
│   │   └── cli-zip.zip
│   ├── sdus
|   │   └── cli-sdu.sdu
│   ├── xars
│   │   ├── cli-xar1.xar
│   │   └── cli-xar2.xar
│   └── yangs
│       ├── cli-yang1.yang
│       └── cli-yang2.yang
├── common
│   ├── defs
│   │   └── common-def1.def
│   ├── device-metadata
│   │   ├── common.yml
│   │   └── common-device-metadata.yaml
│   ├── zips
│   │   └── common-zip.zip
│   ├── mibs
│   │   ├── common-mib1.mib
│   │   └── common-mib2.my
│   ├── sdus
|   │   └── common-sdu.sdu
│   ├── xars
│   │   ├── common-xar1.xar
│   │   └── common-xar2.xar
│   └── yangs
│       ├── common-yang1.yang
│       └── common-yang2.yang
└── snmp
    ├── defs
    │   └── snmp-def1.def
    ├── device-metadata
    │   ├── snmp.yml
    │   └── snmp-device-metadata.yaml
    ├── mibs
    │   ├── snmp-mib1.mib
    │   └── snmp-mib2.my
    ├── sdus
    │   └── snmp-sdu.sdu
    ├── zips
    │   └── snmp-zip.zip
    ├── xars
    │   ├── snmp-xar1.xar
    │   └── snmp-xar2.xar
    └── yangs
        ├── snmp-yang1.yang
        └── snmp-yang2.yang

  • When you upload the aggregate package, the files located in the cli/ and snmp/ directories is accessible to the CLI and SNMP collectors. Also, the files in the common/ directory is accessible to both the CLI and SNMP.

  • Updating a custom CLI device package: To update a custom CLI device package, click the Upload icon next to the filename on the Custom Packages page. Updating a package replaces the existing file.

  • Uploading multiple files: To upload multiple .xar files, combine them into a single .tar.gz archive before uploading.

  • Restrictions for custom MIB packages: Crosswork Network Controller does not support overwriting system MIB package files with custom MIB files. Any attempt to do so will result in a failed upload.

  • TAR file structure: Ensure that the .tar.gz archive contains only the package folders. Avoid including parent or hierarchy folders, as incorrect file structure may cause exceptions during job execution in Crosswork Network Controller.

  • File validation: Crosswork Network Controller validates only the file extension during upload. It does not verify the contents of the uploaded files.

The performance of collection jobs using custom packages depends on the optimization of those packages. Ensure that the packages are optimized for the scale of deployment before uploading them to Cisco Crosswork. For information on how to validate custom MIBs and YANGs that can be uploaded to Crosswork Network Controller, see Use Custom MIBs and Yangs on Cisco DevNet.

Procedure

Step 1

From the main menu, choose Administration > Data Collector(s) Global Settings > Custom packages.

Step 2

In the Custom packages page, click Add icon.

Step 3

In the Add custom packages window that appears, select the type of package you want to import from the Type drop-down.

Step 4

Click in the blank field of File name to open the file browser window and select the package to import and click Open.

Step 5

Add a description of the package in the Notes field. We recommend including a unique description for each package to easily distinguish between them.

Step 6

Click Upload.

Step 7

Restart all the impacted services to get the latest custom MIB package updates.
Delete Custom Package

When you remove a custom package from Cisco Crosswork, all YANG and XAR files are automatically deleted. Removing custom packages affects all collection tasks that rely on the custom package.

Follow the steps to delete a custom package:

Procedure

Step 1

From the main menu, choose Administration > Data Collector(s) Global Settings > Custom Packages.

Step 2

From the list displayed in the Custom Packages pane, select the package you want to delete and click Delete icon.

Step 3

In the Delete Custom Package window that appears, click Delete to confirm.

Configure Global Parameters

The Global Parameters window allows you to update the port across all the data collectors in the network.


Note
These settings can only be accessed by an admin user.

Procedure

Step 1

From the main menu, navigate to Administration > Data Collector(s) Global Settings > Global Parameters.

Figure 5. Global Parameters Window
Global Parameters Window

Step 2

Change one of more of the following parameters.

Note

 
Ensure that the port values that you wish to update with are valid ports and do not conflict with the existing port values. The same port values must be configured on the device.

Parameter Name

Description

Default value for single VM deployment
Number of CLI sessions

Maximum number of CLI sessions that can be set up between an embedded data collector and devices.

Note

 
This value overrides any internal configuration set for the same parameter.

3

Accepted range is 1-50
SSH Session Timeout

The session timeout (in seconds) is the duration for which a CLI connection can remain idle in the CLI and SNMP collectors.

120

Accepted range is 1-120 seconds
SNMP Trap Port Modify this value as per your deployment environment and configuration requirements.

31062

Accepted range is 30160–31560
Syslog UDP Port Modify this value as per your deployment environment and configuration requirements.

30514

Accepted range is 30160–31560
Syslog TCP Port

-

30898

Accepted range is 30160–31560
Syslog TLS Port

-

30614

Accepted range of ports is 30160–31560
Re-Sync SNMPv3 Details

USM details change whenever a device is rebooted or reimaged. SNMPV3 collections stop working whenever there is a change in any of the USM details.

Enable the option to synchronize the USM details automatically whenever there is a change following the initial collection failure. By default, the option is disabled.

Disable

Step 3

If you are updating ports, select Yes in the Global Parameters window that appears to confirm that collectors can be restarted. Updating ports causes the collectors to restart and pause any collection jobs that are running. The jobs resume automatically once the restart is complete.

Step 4

Click Save to apply your changes.

A window appears indicating if the parameters update on embedded data collectors in the network was successful or not.

  1. If all the embedded data collectors were updated successfully, a success message appears in the UI indicating that the update was successful.

  2. If any of the embedded data collectors in the network could not be updated, an Error window appears in the UI. The embedded data collectors will automatically try to update the parameters on the failed embedded data collectors during recovery. Some of the collectors might be restarted as part of the recovery.


    Note

    One of the reasons the global parameters fail to update on a embedded data collector could be that the OAM channel is down. After the OAM channel is reestablished, embedded data collectors try sending these parameters to the embedded data collectors again (that is not in sync) and updates the values after comparison with the existing values.

Manage Collection Jobs

Applications request data collection via the collection jobs. Subsequently, Cisco Crosswork assigns these collection jobs to a collector who handles the request. The collector starts the data collection depending on the type of data to be collected.

Embedded Collectors are capable of collecting data using protocols such as CLI, SNMP, gNMI (dial-in), and syslog. The collectors can collect any type of data as long as they can forward it over one of the supported protocols.

There are two types of data collection requests in Cisco Crosswork:

  1. Data collection request to forward data for internal processes within Cisco Crosswork. Cisco Crosswork creates system jobs for this purpose. If you want the data gateway to collect specific information from non-Cisco devices, you must use custom device packages. For more information on custom device packages, see Custom Packages.

    To learn how to build a model that enables an Crosswork to communicate with non-Crosswork devices, see Cisco Devnet.

  2. Data collection request to forward data to an external data destination. For more information on configuring the external data destinations (Kafka or gRPC), see Manage External Data Destinations.

You can forward collected data to an external data destination and Crosswork Network Controller in a single collection request.

You can view collection jobs currently active from the Collection Jobs page.

In the Cisco Crosswork UI, from the left navigation bar, choose Administration > Collection Jobs.

The left pane in the Collection Jobs page has two tabs, Bulk Jobs and Parametrized Jobs. Bulk Jobs list all the collection jobs that are created by the system, or from the UI and API here. The Parametrized Jobs pane lists all active jobs that Crosswork Network Controller has created.

Types of Collection Jobs

You can create the following types of collection jobs from the Cisco Crosswork UI (CLI) or using APIs to request data.

For each collection job that you create, Embedded Collectors execute the collection request and forwards the collected data to the specified data destination.

This chapter describes how to create collection jobs from the Cisco Crosswork UI. To create collection jobs using APIs, see Crosswork Data Gateway APIs on Cisco Devnet.

The initial status for all the collection jobs in the Crosswork UI is Unknown. Upon receiving a collection job, Embedded Collectors perform basic validations on it. If the collection job is valid, its status changes to Successful, else it changes to Failed.

The Cadence value determines the frequency at which Embedded Collectors collect the data from the device. You can set the frequency between 10 and 604800000 milliseconds. We recommend a cadence of minimum 60 milliseconds.

When setting the cadence, consider how often the data in the device is subject to change and if the data is operationally significant. We recommend a higher cadence for consistent data like memory consumption or CPU utilization. For more dynamic data points, set a shorter cadence. If Embedded Collectors have to collect a lot of telemetry and more extensive data sets with a short cadence, there is an extra load on the devices and Crosswork Network Controller. As it is difficult to model these loads, we recommend that you experiment to find the values that provide the best operational insight and, most importantly, actionable information.


Note

When collection from a device is skipped due to a previous execution still in progress, Embedded Collectors raise a warning log. No alert is generated for this scenario.

CLI Collection Job

Embedded Collectors support CLI-based data collection from the network devices. Following commands are supported for this type of collection job:

  • show and the short version sh

  • traceroute

  • dir

Devices should not have any banner configuration for CLI collection to work properly. Please refer to device documentation on how to turn this off.

You can create a CLI collection job from the Cisco Crosswork UI or using APIs. For information on creating jobs through the UI, see Create a Collection Job from Cisco Crosswork UIand from API, see Cisco DevNet for more information.

Sample payload of CLI collection job

In this example, Crosswork sends the data to an external Kafka destination, including the UUID used by the device:

  1. The device is identified with a UUID rather than an IP address.

  2. The destination is also referenced by a UUID. For collections jobs built using the UI, Cisco Crosswork looks up the UUIDs. When you create your own collection jobs, you will need to look up these values. 

{
  "collection_job": {
    "application_context": {
      "context_id": "collection-job1",
      "application_id": "APP1"
    },
    "collection_mode": {
      "lifetime_type": "APPLICATION_MANAGED",
      "collector_type": "CLI_COLLECTOR"
    },
    "job_device_set": {
      "device_set": {
        "devices": {
          "device_ids": [
            "658adb03-cc61-448d-972f-4fcec32cbfe8"
          ]
        }
      }
    },
    "sensor_input_configs": [
      {
        "sensor_data": {
          "cli_sensor": {
            "command": "show platform"
          }
        },
        "cadence_in_millisec": "60000"
      }
    ],
    "sensor_output_configs": [
     {
        "sensor_data": {
          "cli_sensor": {
            "command": "show platform"
          }
        },
        "destination": {
          "destination_id": "1e71f2fb-ea65-4242-8efa-e33cec71b369",
          "context_id": "topic1"
        }
      }
    ]
  }
}

SNMP Collection Job

Embedded Collectors support SNMP-based data collection based on the OIDs supported on the devices. The SNMP OID-based collection jobs can be created from the Cisco Crosswork UI or using the API, and SNMP-traps using the API.

The SNMP collector makes a poll request to Crosswork Network Controller to get its configuration profile (a list of MIB objects to collect and a list of devices to fetch from). It determines the corresponding OIDs by looking up the prepackaged list of MIB modules or the custom list of MIB modules.


Note
Embedded Collectors enable SNMP polling on third-party devices for standard MIBs already included in the system. Proprietary MIBs are required only if the collection request references MIB TABLE names or SCALAR names from a proprietary MIB. However, if the requests are OID-based, then MIBs are not required.

After the OIDs are resolved, they are provided as input to the SNMP collectors.

The device packages can be imported into the Embedded Collectors instance as described in Section Upload Custom Packages.

Supported SNMP versions for data polling and traps are:

  • Polling Data

    • SNMP V2

    • SNMP V3 (no auth nopriv, auth no priv, authpriv)

    • Supported auth protocols – SHA-1, MD5

    • Supported priv protocols – AES-128, AES-192, AES-256, CiscoAES192, CiscoAES256, DES, and 3-DES.

  • Traps

    • SNMP V2

    • SNMP V3 (no auth nopriv, auth no priv, authpriv)

Sample Configurations on Device:

The following table lists sample commands to enable various SNMP functions. For more information, refer to the platform-specific documentation.

Table 2. Sample configuration to enable SNMP on device

Version

Command

To...

V2c

snmp-server group <group_name> v2c

snmp-server user <user_name> <group_name> v2c

Define the SNMP version, user/user group details.

snmp-server host <host_ip> traps SNMP version <community_string> udp-port 31062

snmp-server host a.b.c.d traps version 2c v2test udp-port 31062

Define the destination to which trap data must be forwarded.

Note

 

The IP address mentioned here must be the virtual IP address of Embedded Collectors.

snmp-server traps snmp linkup

snmp-server traps snmp linkdown

Enable traps to notify link status.

V3

Note

 

Password for a SNMPv3 user must be at least 8 bytes.

snmp-server host <host_IP> traps version 3 priv <user_name> udp-port 31062

Define the destination to which trap data must be forwarded.

Note

 

The IP address mentioned here must be the virtual IP address of Embedded Collectors.

snmp-server user <user_name> <group_name> v3 auth md5 <password> priv aes 128 <password>

Configures the SNMP server group to enable authentication for members of a specified named access list.

snmp-server view <user_name> < MIB > included

Define what must be reported.

snmp-server group <group_name> v3 auth notify <user_name> read <user_name> write <user_name>

Define the SNMP version, user/user group details.

snmp-server enable traps snmp [authentication ] [linkup ] [linkdown ] [warmstart ] [coldstart ]

  • When used without any of the optional keywords, enables authenticationFailure, linkUp, linkDown, warmStart, and coldStart traps.

  • When used with keywords, enables only the trap types specified. For example, to globally enable only linkUp and linkDown SNMP traps for all interfaces, use the snmp-server enable traps snmp linkup linkdown form of this command.

The SNMP Collector supports the following operations:

  • SCALAR


    Note

    If a single collection requests for multiple scalar OIDs, you can pack multiple SNMP GET requests in a single getbulkrequestquery to the device.

  • TABLE

  • WALK

  • COLUMN

These operations are defined in the sensor config (see payload sample below).


Note

There is an optional deviceParams attribute snmpRequestTimeoutMillis (not shown in the sample payloads) that should be used if the device response time is more than 1500 milliseconds. It’s not recommended to use snmpRequestTimeoutMillis unless you are certain that your device response time is high.

The value for snmpRequestTimeoutMillis should be specified in milliseconds:

The default and minimum value is 1500 milliseconds. However, there is no limitation on the maximum value of this attribute.

The following is an SNMP collection job sample:

{
  "collection_job": {
    "application_context": {
      "context_id": "collection-job1",
      "application_id": "APP1"
    },
    "collection_mode": {
      "lifetime_type": "APPLICATION_MANAGED",
      "collector_type": "SNMP_COLLECTOR"
    },
    "job_device_set": {
      "device_set": {
        "devices": {
          "device_ids": [
            "c70fc034-0cbd-443f-ad3d-a30d4319f937",
            "8627c130-9127-4ed7-ace5-93d3b4321d5e",
            "c0067069-c8f6-4183-9e67-1f2e9bf56f58"
          ]
        }
      }
    },
    "sensor_input_configs": [
      {
        "sensor_data": {
          "snmp_sensor": {
            "snmp_mib": {
              "oid": "1.3.6.1.2.1.1.3.0",
              "snmp_operation": "SCALAR"
            }
          }
        },
        "cadence_in_millisec": "60000"
      },
      {
        "sensor_data": {
          "snmp_sensor": {
            "snmp_mib": {
              "oid": "1.3.6.1.2.1.31.1.1",
              "snmp_operation": "TABLE"
            }
          }
        },
        "cadence_in_millisec": "60000"
      }
    ],
    "sensor_output_configs": [
      {
        "sensor_data": {
          "snmp_sensor": {
            "snmp_mib": {
              "oid": "1.3.6.1.2.1.1.3.0",
              "snmp_operation": "SCALAR"
            }
          }
        },
        "destination": {
          "destination_id": "4c2ab662-2670-4b3c-b7d3-b94acba98c56",
          "context_id": "topic1_461cb8aa-a16a-44b8-b79f-c3daf3ea925f"
        }
      },
      {
        "sensor_data": {
          "snmp_sensor": {
            "snmp_mib": {
              "oid": "1.3.6.1.2.1.31.1.1",
              "snmp_operation": "TABLE"
            }
          }
        },
        "destination": {
          "destination_id": "4c2ab662-2670-4b3c-b7d3-b94acba98c56",
          "context_id": "topic2_e7ed6300-fc8c-47ee-8445-70e543057f8a"
        }
      }
    ]
  }
}
SNMP Traps Collection Job

SNMP Traps Collection jobs can be created only via API. Trap listeners listen on a port and dispatch data to recipients (based on their topic of interest).


Important

Before starting the SNMP trap collection, install the Common EMS Services application and configure the host information for SNMP.

Embedded Collectors listen on UDP port 31062 for Traps.


Note

Before submitting SNMP Trap collection jobs, SNMP TRAPS must be properly configured on the device to be sent to the virtual IP address of Embedded Collectors.

SNMP Trap Collection Job Workflow

On receiving an SNMP trap, :

  1. Checks if any collection job is created for the device.

  2. Checks the trap version and community string.


    Note

    To prevent Embedded Collectors from checking the community string for SNMP traps, select the SNMP Disable Trap Check check box when adding a device through the Crosswork UI. For more information about this option, see Add Devices Through the User Interface.

  3. For SNMP v3, also validates for user auth and priv protocol and credentials.


    Note
    SNMPV3 auth-priv traps are dependent on the engineId of the device or router to maintain local USM user tables. Therefore, there will be an interruption in receiving traps whenever the engineId of the device or router changes. Please detach and attach the respective device to start receiving traps again.

filter the traps based on the trap OID mentioned in the sensor path and sends only those requested.

If the collection job is invalid, there is a missing configuration on the device, or no trap is received, the status of the job remains "Unknown". For list of supported Traps and MIBs, see List of Pre-loaded Traps and MIBs for SNMP Collection.

supports three types of non-yang/OID based traps:

Table 3. List of Supported Non-Yang/OID based Traps
sensor path purpose
* To get all the traps pushed from the device without any filter.
MIB level traps

OID of one MIB notification

(Ex: 1.3.6.1.2.1.138.0 to get all the isis-mib level traps)
Specific trap

OID of the specific trap

(Ex: 1.3.6.1.6.3.1.1.5.4 to get the linkUp trap)

Following is an SNMP-Trap collection job sample:

{
  "collection_job": {
    "application_context": {
      "context_id": "collection-job1",
      "application_id": "APP1"
    },
    "collection_mode": {
      "lifetime_type": "APPLICATION_MANAGED",
      "collector_type": "TRAP_COLLECTOR"
    },
    "job_device_set": {
      "device_set": {
        "devices": {
          "device_ids": [
            "a9b8f43d-130b-4866-a26a-4d0f9e07562a",
            "8c4431a0-f21d-452d-95a8-84323a19e0d6",
            "eaab2647-2351-40ae-bf94-6e4a3d79af3a"
          ]
        }
      }
    },
    "sensor_input_configs": [
      {
        "sensor_data": {
          "trap_sensor": {
            "path": "1.3.6.1.6.3.1.1.4"
          }
        },
        "cadence_in_millisec": "60000"
      }
    ],
    "sensor_output_configs": [
      {
        "sensor_data": {
          "trap_sensor": {
            "path": "1.3.6.1.6.3.1.1.4"
          }
        },
        "destination": {
          "destination_id": "4c2ab662-2670-4b3c-b7d3-b94acba98c56",
          "context_id": "topic1_696600ae-80ee-4a02-96cb-3a01a2415324"
        }
      }
    ]
  }
}
Enabling Traps forwarding to external applications

We recommended selectively enabling only those traps that are needed by Crosswork on the device.

To identify the type of trap from the data received on the destination, look for oid (OBJECT_IDENTIFIER, for example, 1.3.6.1.6.3.1.1.4.1.0 ) and strValue associated to the oid in the OidRecords (application can match the OID of interest to determine the kind of trap).

Following are the sample values and a sample payload to forward traps to external applications:

  • Link up

    1.3.6.1.6.3.1.1.4.1.0 = 1.3.6.1.6.3.1.1.5.4

  • Link Down

    1.3.6.1.6.3.1.1.4.1.0 = 1.3.6.1.6.3.1.1.5.3

  • Syslog

    1.3.6.1.6.3.1.1.4.1.0 = 1.3.6.1.4.1.9.9.41.2.0.1

  • Cold Start

    1.3.6.1.6.3.1.1.4.1.0 = 1.3.6.1.6.3.1.1.5.1

{
  "nodeIdStr": "BF5-XRV9K1.tr3.es",
  "nodeIdUuid": "C9tZ5lJoSJKf5OZ67+U5JQ==",
  "collectionId": "133",
  "collectionStartTime": "1580931985267",
  "msgTimestamp": "1580931985267",
  "dataGpbkv": [
    {
      "timestamp": "1580931985267",
      "name": "trapsensor.path",
      "snmpTrap": {
        "version": "V2c",
        "pduType": "TRAP",
        "v2v3Data": {
          "agentAddress": "172.70.39.227",
          "oidRecords": [
            {
              "oid": "1.3.6.1.2.1.1.3.0",
              "strValue": "7 days, 2:15:17.02"
            },
            {
              "oid": "1.3.6.1.6.3.1.1.4.1.0",  // This oid is the Object Identifier.
              "strValue": "1.3.6.1.6.3.1.1.5.3" // This is the value that determines the kind of trap.
            },
            {
              "oid": "1.3.6.1.2.1.2.2.1.1.8",
              "strValue": "8"
            },
            {
              "oid": "1.3.6.1.2.1.2.2.1.2.8",
              "strValue": "GigabitEthernet0/0/0/2"
            },
            {
              "oid": "1.3.6.1.2.1.2.2.1.3.8",
              "strValue": "6"
            },
            {
              "oid": "1.3.6.1.4.1.9.9.276.1.1.2.1.3.8",
              "strValue": "down"
            }
          ]
        }
      }
    }
  ],
  "collectionEndTime": "1580931985267",
  "collectorUuid": "YmNjZjEzMTktZjFlOS00NTE5LWI4OTgtY2Y1ZmQxZDFjNWExOlRSQVBfQ09MTEVDVE9S",
  "status": {
    "status": "SUCCESS"
  },
  "modelData": {},
  "sensorData": {
    "trapSensor": {
      "path": "1.3.6.1.6.3.1.1.5.4"
    }
  },
  "applicationContexts": [
    {
      "applicationId": "APP1",
      "contextId": "collection-job-snmp-traps"
    }
  ]
}

Syslog Collection Job

Embedded Collectors supports syslog-based events collection from devices.


Important

Before starting the syslog trap collection, install the Common EMS Services application and configure the host information for syslog.

The following syslog formats are supported:

  • RFC5424 syslog format

  • RFC3164 syslog format


Note

To gather syslog data from Embedded Collectors in the network, when adding a device, select the YANG_CLI capability and configure other parameters to receive syslog data from Embedded Collectors. Refer to the platform-specific documentation.

While the order of the configuration steps does not matter, you must complete both the steps, or no data will be sent or collected. For sample device configuration, see Configure Non-Secure Syslog on Device. Cisco Crosswork also allows you to set up secure syslog communication to the device. For more information, see Configure Secure Syslog on Device.

Sample Syslog collection payload
This is a sample syslog collection payload:
{
  "collection_job": {
      "job_device_set": {
      "device_set": {
        "devices": {
          "device_ids": [
            "c6f25a33-92e6-468a-ba0d-15490f1ce787"
          ]
        }
      }
    },
    "sensor_output_configs": [
      {
        "sensor_data": {
          "syslog_sensor": {
            "pris": {
                "facilities": [0, 1, 3, 23,4],
                "severities": [0, 4, 5, 6, 7]
            }
        }
        },
        "destination": {
          "context_id": "syslogtopic",
          "destination_id": "c2a8fba8-8363-3d22-b0c2-a9e449693fae"
        }
      }
    ],
    "sensor_input_configs": [
      {
        "sensor_data": {
          "syslog_sensor": {
            "pris": {
                "facilities": [0,1, 3, 23,4],
                "severities": [0,4, 5, 6, 7]
            }
        }
        },
        "cadence_in_millisec": "60000"
      }
    ],
    "application_context": {
      "context_id": "demomilesstone2syslog",
      "application_id": "SyslogDemo2"
    },
    "collection_mode": {
      "lifetime_type": "APPLICATION_MANAGED",
      "collector_type": "SYSLOG_COLLECTOR"
    }
  }
}

  • You can filter the output of syslog data collection by specifying either PRI-based SyslogSensor OR Filters-based SyslogSensor. The syslog events matching the facilities and severities mentioned in the payload are sent to the specified destination. All other nonmatching syslog events are dropped. You can specify the filter based on regEx, severity, or facility.

  • If you have specified values for severity and facility, then both the conditions are combined based on the logical operator specified at Filters level.

  • You can specify a maximum of three filters combinations using the logical operator AND or OR. By default, the AND operator is applied if do not specify an operator.

Syslog Collection Job Output

When you onboard a device from Crosswork Network Controller UI (Device Management > Network Devices > Device Details), the value you choose in the Syslog Format field configures the format in which syslog events received from the device should be parsed by the syslog collector. You can choose either UNKNOWN, RFC5424 or RFC3164.

Following is the sample output for each of the options:

  1. UNKNOWN - Syslog Collection Job output contains syslog events as received from device.


    Note

    If the device is configured to generate syslog events in RFC5424/RFC3164 format but no format is specified in the Syslog Format field, this is considered as UNKNOWN by default.

    Sample output:
    node_id_str: "xrv9k-VM8"
node_id_uuid: ":i\300\216>\366BM\262\270@\337\225\2723&"
collection_id: 1056
collection_start_time: 1616711596200
msg_timestamp: 1616711596201
data_gpbkv {
  timestamp: 1616711596201
  name: "syslogsensor.path"
  fields {
    name: "RAW"
    string_value: "<6>1 Mar 25 15:34:41.321 PDT - SSHD_ 69570 - - 98949: RP/0/RP0/CPU0:SSHD_[69570]: %SECURITY-SSHD-6-INFO_SUCCESS : Successfully authenticated user \'admin\' from \'40.40.40.116\' on \'vty0\'(cipher \'aes128-ctr\', mac \'hmac-sha1\') \n"
  }
  fields {
    name: "DEVICE_IP"
    string_value: "40.40.40.30"
  }
}
collection_end_time: 1616711596200
collector_uuid: "17328736-b726-4fe3-b922-231a4a30a54f:SYSLOG_COLLECTOR"
status {
  status: SUCCESS
}
model_data {
}
sensor_data {
  syslog_sensor {
    pris {
      facilities: 0
      facilities: 3
      facilities: 4
      facilities: 23
      severities: 0
      severities: 5
      severities: 6
      severities: 7
    }
  }
}
application_contexts {
  application_id: "SyslogApp-xr-8-job1"
  context_id: "xr-8-job1"
}
version: "1"

  2. RFC5424 - If the device is configured to generate syslog events in RFC5424 format and the RFC5424 format is selected in the Syslog Format field, the Syslog Job Collection output contains syslog events as received from device (RAW) and the RFC5424 best-effort parsed syslog events from the device.


    Note

    The syslog collector will parse the syslog event on best efforts as per the following Java RegEx pattern:

    Sample output:

    ....
....
 
 
collection_start_time: 1596307542398
msg_timestamp: 1596307542405
data_gpbkv {
  timestamp: 1596307542405
  name: "syslogsensor.path"
  fields {
    name: "RAW"
    string_value: "<13>1 2020 Aug  1 12:03:32.461 UTC:  iosxr254node config 65910 - - 2782: RP/0/RSP0/CPU0:2020 Aug  1 12:03:32.461 UTC: config[65910]: %MGBL-SYS-5-CONFIG_I : Configured from console by admin on vty0 (10.24.88.215) \n"
  }
  fields {
    name: "RFC5424"
    string_value: "pri=13,  severity=5,  facility=1,  version=1,  date=2020-08-01T12:03:32.461,  remoteAddress=/172.28.122.254,  host=\'iosxr254node\',  message=\'2782: RP/0/RSP0/CPU0:2020 Aug  1 12:03:32.461 UTC: config[65910]: %MGBL-SYS-5-CONFIG_I : Configured from console by admin on vty0 (10.24.88.215) \', messageId=null, processName=config, structuredDataList=null"
  }
  fields {
    name: "DEVICE_IP"
    string_value: "172.28.122.254"
  }
}
collection_end_time: 1596307542404
collector_uuid: "ac961b09-8f67-4c93-a99a-31eef50f7fa9:SYSLOG_COLLECTOR"
status {
  status: SUCCESS
}
...
...

  3. RFC3164 - If the device is configured to generate syslog events in RFC3164 format and the RFC3164 format is selected in Syslog Format field, the Syslog Job Collection output contains both RAW (as received from device) syslog events and the RFC3164 best-effort parsed syslog events from the device.


    Note

    The syslog collector will parse the syslog event on best efforts as per the following Java RegEx pattern:
    Sample output:
    ....
.....
collection_id: 20
collection_start_time: 1596306752737
msg_timestamp: 1596306752743
data_gpbkv {
  timestamp: 1596306752743
  name: "syslogsensor.path"
  fields {
    name: "RAW"
    string_value: "<14>2020 Aug  1 11:50:22.799 UTC:  iosxr254node 2756: RP/0/RSP0/CPU0:2020 Aug  1 11:50:22.799 UTC: config[65910]: %MGBL-CONFIG-6-DB_COMMIT : Configuration committed by user \'admin\'. Use \'show configuration commit changes 1000000580\' to view the changes. \n"
  }
  fields {
    name: "RFC3164"
    string_value: "pri=14,  severity=6,  facility=1,  version=null,  date=2020-08-01T11:50:22.799,  remoteAddress=/172.28.122.254,  host=\'iosxr254node\',  message=\'RP/0/RSP0/CPU0:2020 Aug  1 11:50:22.799 UTC: config[65910]: %MGBL-CONFIG-6-DB_COMMIT : Configuration committed by user \'admin\'. Use \'show configuration commit changes 1000000580\' to view the changes. \', tag=2756"
  }
  fields {
    name: "DEVICE_IP"
    string_value: "172.28.122.254"
  }
}
collection_end_time: 1596306752742
collector_uuid: "ac961b09-8f67-4c93-a99a-31eef50f7fa9:SYSLOG_COLLECTOR"
status {
  status: SUCCESS
}
....
....

If the syslog collector is unable to parse the syslog events according to the format specified in the Syslog Format field, then the Syslog Collection Job output contains syslog events as received from device (RAW).

Configure Non-Secure Syslog on Device

This section lists sample configuration to configure syslog in the RFC3164 or RFC5424 format on the device.


Note

The syslog format that you configure for the device must match the format that you specified when the device was added through the Crosswork UI. See Add Devices Through the User Interface for more information.

Configure RFC3164 Syslog format

The configuration highlighted in the code below is required to avoid formatting issues in the parsed output.

For IOS XR: 

logging <CDG IP> port 30514 OR logging <CDG IP> vrf <vrfname> port 30514 
logging trap [severity]
logging facility [facility value]
logging suppress duplicates
service timestamps log datetime msec show-timezone year
logging hostnameprefix <some host related prefix e.g.iosxrhost2>

For IOS XE: 

no logging message-counter syslog 
logging trap <severity>
logging facility <facility>
logging host <CDG IP> transport tcp port 309898 session-id string <sessionidstring> --> To use TCP channel
OR
logging host <CDG IP> transport udp port 30514 session-id string <sessionidstring> ---> To use UDP channel
OR
logging host <CDG IP> vrf Mgmt-intf transport udp port 30514 session-id string <sessionidstring> --> To use UDP via vrf 
service timestamps log datetime msec year show-timezone

Configure RFC5424 Syslog format

For IOS XR: 

logging <CDG IP> port 30514 OR logging <server 1> vrf <vrfname> port 30514 
logging trap [severity]
logging facility [facility value]
logging suppress duplicates
service timestamps log datetime msec show-timezone year
logging hostnameprefix <some host related prefix e.g.iosxrhost2>
logging format rfc5424

For IOS XE: 

no logging message-counter syslog
logging trap <severity>
logging facility <facility>
logging host <CDG IP> transport tcp port 309898 session-id string <sessionidstring> --> To use TCP channel
OR
logging host <CDG IP> transport udp port 30514 session-id string <sessionidstring> ---> To use UDP channel
OR
logging host <CDG IP> vrf Mgmt-intf transport udp port 30514 session-id string <sessionidstring> --> To use UDP via vrf 
service timestamps log datetime msec year show-timezone
logging trap syslog-format 5424 --> if applicable
Configure Secure Syslog on Device

Use the steps to establish a secured syslog communication with the device.

  1. Download the Cisco Crosswork trust chain from the Certificate Management UI page in Cisco Crosswork.

  2. Configure the device with the Cisco Crosswork trustchain.

Download Syslog Certificates

  1. In the Cisco Crosswork UI, go to Administration > Certificate Management.

  2. Click i in the crosswork-device-syslog row.

  3. Click Export All to download the certificates.

    The following files are downloaded to your system.

Configure Cisco Crosswork Trustpoint on Device
Sample IOS XR device configuration to enable TLS 
RP/0/RSP0/CPU0:ASR9k(config)#crypto ca trustpoint syslog-root
RP/0/RSP0/CPU0:ASR9k(config-trustp)#enrollment terminal
RP/0/RSP0/CPU0:ASR9k(config-trustp)#crl optional
RP/0/RSP0/CPU0:ASR9k(config-trustp)#commit
RP/0/RSP0/CPU0:ASR9k(config-trustp)#end
RP/0/RSP0/CPU0:ASR9k#
RP/0/RSP0/CPU0:ASR9k#crypto ca authenticate syslog-root
Fri Jan 22 11:07:41.880 GMT
  
  
Enter the base 64 encoded certificate.
End with a blank line or the word "quit" on a line by itself
  
-----BEGIN CERTIFICATE-----
MIIGKzCCBBOgAwIBAgIRAKfyU89yjmrXVDRKBWuSGPgwDQYJKoZIhvcNAQELBQAw
bDELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMREwDwYDVQQHEwhTYW4gSm9zZTEa
................................................................
................................................................
jPQ/UrO8N3sC1gGJX7CIIh5cE+KIJ51ep8i1eKSJ5wHWRTmv342MnG2StgOTtaFF
vrkWHD02o6jRuYXDWEUptDOg8oEritZb+SNPXWUc/2mbYog6ks6EeMC69VjkZPo=
-----END CERTIFICATE-----
  
Read 1583 bytes as CA certificate
  Serial Number  : A7:F2:53:CF:72:8E:6A:D7:54:34:4A:05:6B:92:18:F8
  Subject:
                CN=Crosswork Device Root CA,O=CISCO SYSTEMS INC,L=San Jose,ST=CA,C=US
  Issued By      :
                CN=Crosswork Device Root CA,O=CISCO SYSTEMS INC,L=San Jose,ST=CA,C=US
  Validity Start : 02:37:09 UTC Sat Jan 16 2021
  Validity End   : 02:37:09 UTC Thu Jan 15 2026
  SHA1 Fingerprint:
                209B3815271C22ADF78CB906F6A32DD9D97BBDBA
  
Fingerprint: 2FF85849EBAAB9B059ACB9F5363D5C9CDo you accept this certificate? [yes/no]: yes
RP/0/RSP0/CPU0:ASR9k#config
RP/0/RSP0/CPU0:ASR9k(config)#crypto ca trustpoint syslog-inter
RP/0/RSP0/CPU0:ASR9k(config-trustp)#enrollment terminal
RP/0/RSP0/CPU0:ASR9k(config-trustp)#crl optional
RP/0/RSP0/CPU0:ASR9k(config-trustp)#commit
RP/0/RSP0/CPU0:ASR9k#crypto ca authenticate syslog-inter
Fri Jan 22 11:10:30.090 GMT
  
  
Enter the base 64 encoded certificate.
End with a blank line or the word "quit" on a line by itself
  
-----BEGIN CERTIFICATE-----
MIIGFDCCA/ygAwIBAgIRAkhqHQXcJzQzeQK6U2wn8PIwDQYJKoZIhvcNAQELBQAw
bDELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAkNBMREwDwYDVQQHEwhTYW4gSm9zZTEa
................................................................
................................................................
5lBk617z6cxFER5c+/PmJFhcreisTxXg1aJbFdnB5C8f+0uUIdLghykQ/zaZGuBn
AAB70c9r9OeKGJWzvv1e2U8HH1pdQ/nd
-----END CERTIFICATE-----
  
Read 1560 bytes as CA certificate
  Serial Number  : 02:48:6A:1D:05:DC:27:34:33:79:02:BA:53:6C:27:F0:F2
  Subject:
                CN=device-syslog,O=CISCO SYSTEMS INC,L=San Jose,ST=CA,C=US
  Issued By      :
                CN=Crosswork Device Root CA,O=CISCO SYSTEMS INC,L=San Jose,ST=CA,C=US
  Validity Start : 02:37:11 UTC Sat Jan 16 2021
  Validity End   : 02:37:11 UTC Mon Jan 16 2023
  SHA1 Fingerprint:
                B06F2BFDE95413A8D08A01EE3511BC3D42F01E59
  
CA Certificate validated using issuer certificate.
RP/0/RSP0/CPU0:ASR9k#show crypto ca certificates
Fri Jan 22 15:45:17.196 GMT
 
 
Trustpoint       : syslog-root
==================================================
CA certificate
  Serial Number  : A7:F2:53:CF:72:8E:6A:D7:54:34:4A:05:6B:92:18:F8
  Subject:
        CN=Crosswork Device Root CA,O=CISCO SYSTEMS INC,L=San Jose,ST=CA,C=US
  Issued By      :
        CN=Crosswork Device Root CA,O=CISCO SYSTEMS INC,L=San Jose,ST=CA,C=US
  Validity Start : 02:37:09 UTC Sat Jan 16 2021
  Validity End   : 02:37:09 UTC Thu Jan 15 2026
  SHA1 Fingerprint:
         209B3815271C22ADF78CB906F6A32DD9D97BBDBA
 
 
Trustpoint       : syslog-inter
==================================================
CA certificate
  Serial Number  : 02:48:6A:1D:05:DC:27:34:33:79:02:BA:53:6C:27:F0:F2
  Subject:
        CN=device-syslog,O=CISCO SYSTEMS INC,L=San Jose,ST=CA,C=US
  Issued By      :
        CN=Crosswork Device Root CA,O=CISCO SYSTEMS INC,L=San Jose,ST=CA,C=US
  Validity Start : 02:37:11 UTC Sat Jan 16 2021
  Validity End   : 02:37:11 UTC Mon Jan 16 2023
  SHA1 Fingerprint:
         B06F2BFDE95413A8D08A01EE3511BC3D42F01E59
RP/0/RSP0/CPU0:ASR9k(config)#logging tls-server syslog-tb131
RP/0/RSP0/CPU0:ASR9k(config-logging-tls-peer)#tls-hostname 10.13.0.159
RP/0/RSP0/CPU0:ASR9k(config-logging-tls-peer)#trustpoint syslog-inter
RP/0/RSP0/CPU0:ASR9k(config-logging-tls-peer)#severity debugging
RP/0/RSP0/CPU0:ASR9k(config-logging-tls-peer)#vrf default
RP/0/RSP0/CPU0:ASR9k(config-logging-tls-peer)#commit
RP/0/RSP0/CPU0:ASR9k(config-logging-tls-peer)#exit
RP/0/RSP0/CPU0:ASR9k(config)#exit
RP/0/RSP0/CPU0:ASR9k#exit
RP/0/RSP0/CPU0:ASR9k#show running-config logging
Fri Jan 22 11:17:19.385 GMT
logging tls-server syslog-tb131
vrf default
severity debugging
trustpoint syslog-inter
tls-hostname <CDG Southbound IP>
!
logging trap debugging
logging format rfc5424
logging facility user
logging hostnameprefix ASR9k
logging suppress duplicates
  
RP/0/RSP0/CPU0:ASR9k#

Sample IOS XE device configuration to enable TLS

csr8kv(config)#crypto pki trustpoint syslog-root
csr8kv(ca-trustpoint)#enrollment terminal
csr8kv(ca-trustpoint)#revocation-check none
csr8kv(ca-trustpoint)#chain-validation stop
csr8kv(ca-trustpoint)#end
csr8kv(config)#crypto pki authenticate syslog-root
 
Enter the base 64 encoded CA certificate.
End with a blank line or the word "quit" on a line by itself
 
-----BEGIN CERTIFICATE-----
MIIFPjCCAyYCCQCO6pK5AOGYdjANBgkqhkiG9w0BAQsFADBhMQswCQYDVQQGEwJV
UzELMAkGA1UECAwCQ0ExETAPBgNVBAcMCE1pbHBpdGFzMQ4wDAYDVQQKDAVDaXNj
................................................................
................................................................
JbimOpXAncoBLo14DXOJLvMVRjn1EULE9AXXCNfnrnBx7jL4CV+qHgEtF6oqclFW
JEA=
-----END CERTIFICATE-----
 
Certificate has the following attributes:
       Fingerprint MD5: D88D6D8F E53750D4 B36EB498 0A435DA1
      Fingerprint SHA1: 649DE822 1C222C1F 5101BEB8 B29CDF12 5CEE463B
 
% Do you accept this certificate? [yes/no]: yes
Trustpoint CA certificate accepted.
% Certificate successfully imported
 
 
csr8kv(config)#crypto pki trustpoint syslog-intermediate
csr8kv(ca-trustpoint)#enrollment terminal
csr8kv(ca-trustpoint)#revocation-check none
csr8kv(ca-trustpoint)#chain-validation continue syslog-root
csr8kv(ca-trustpoint)#end
csr8kv(config)#crypto pki authenticate syslog-intermediate
 
Enter the base 64 encoded CA certificate.
End with a blank line or the word "quit" on a line by itself
 
-----BEGIN CERTIFICATE-----
MIIFfTCCA2WgAwIBAgICEAAwDQYJKoZIhvcNAQELBQAwXDELMAkGA1UEBhMCVVMx
EzARBgNVBAgMCkNhbGlmb3JuaWExDjAMBgNVBAoMBUNpc2NvMQ4wDAYDVQQLDAVT
................................................................
................................................................
Nmz6NQynD7bxdQa9Xq9kyPuY3ZVKXkf312IRH0MEy2yFX/tAen9JqOeZ1g8canmw
TxsWA5TLzy1RmxqQh88f0CM=
-----END CERTIFICATE-----
Trustpoint 'syslog-intermediate' is a subordinate CA.
but certificate is not a CA certificate.
Manual verification required
Certificate has the following attributes:
       Fingerprint MD5: FE27BDBE 9265208A 681670AC F59A2BF1
      Fingerprint SHA1: 03F513BD 4BEB689F A4F4E001 57EC210E 88C7BD19
 
csr8kv(config)#logging host <CDG Southbound IP> transport tls port 30614
csr8kv(config)#logging trap informational syslog-format rfc5424
csr8kv(config)#logging facility user
csr8kv(config)#service timestamps log datetime msec year show-timezone

csr8kv(config)#logging tls-profile tlsv12

Syslog configuration to support FQDN

Run the following commands in addition to the sample device configuration to enable TLS to support FQDN.

  1. Configure the domain name and DNS IP on the device.

    For IOS XR: 

    RP/0/RSP0/CPU0:ASR9k#config
RP/0/RSP0/CPU0:ASR9k(config)#domain name <DNS domain name>
RP/0/RSP0/CPU0:ASR9k(config)#domain name-server <DNS server IP>

    For IOS XE: 

    Device(config)# ip name-server <IP of DNS>
Device(config)# ip domain name <domain name>

  2. Configure Embedded Collectors VIP FQDN for tls-hostname.

    For IOS XR: 

    RP/0/RSP0/CPU0:ASR9k(config)#logging tls-server syslog-tb131
RP/0/RSP0/CPU0:ASR9k(config-logging-tls-peer)#tls-hostname <CDG VIP FQDN>

    For IOS XE: 

    Device(config)# logging host fqdn ipv4 <hostname> transport tls port 30614

gNMI Collection Job

Crosswork Network Controller supports gRPC Network Management Interface (gNMI) based telemetry data collection via Embedded Collectors. It supports only gNMI Dial-In (gRPC Dial-In) streaming telemetry data based on subscription and relaying subsequent subscription response (notifications) to the requested destinations.


Note

gNMI collection is supported as long as the models are supported by the target device platform. gNMI must be configured on devices before you can submit gNMI collection jobs. Check platform-specific documentation.

In gNMI, both secure and insecure modes can coexist on the device. Crosswork Network Controller gives preference to secure mode over nonsecure mode based on the information passed in the inventory.

If a device reloads, gNMI collector ensures that the existing subscriptions are resubscribed to the device.

The gNMI specification does not have a way to mark the end of a message. Hence, Destination and Dispatch cadence is not supported in gNMI collector.

Embedded Collectors support the following types of subscribe options for gNMI:

Table 4. gNMI Subscription Options

Type

Subtype

Description

Once

Collects and sends the current snapshot of the system configuration only once for all specified paths

Stream

SAMPLE

Cadence-based collection.

ON_CHANGE

First response includes the state of all the elements for the subscribed path, followed by subsequent updates to the changed leaf values.

TARGET_DEFINED

Router/Device chooses the mode of subscription on a per-leaf basis based on the subscribed path (i.e. one of SAMPLE or ON_CHANGE)

Embedded Collectors supports the ability to subscribe to multiple subscription paths in a single subscription list to the device. For example, you can specify a combination of ON_CHANGE and subscription mode ONCE collection jobs. ON_CHANGE mode collects data only on change of any particular element for the specified path, while subscription mode ONCE collects and sends current system data only once for the specified path.


Note

  • Embedded Collectors rely on the device to declare the support of one or more modes.

  • gNMI sensor path with default values does not appear in the payload. This is a known Protocol Buffers (protobuf ) behavior.

    For boolean the default value is false. For enum, it is gnmi.proto specified.

    Example 1:

    message GNMIDeviceSetting {
bool suppress_redundant = 1;
bool allow_aggregation = 4;
bool updates_only = 6;
}

    Example 2:

    enum SubscriptionMode {
TARGET_DEFINED = 0; //default value will not be printed
ON_CHANGE = 1;
SAMPLE = 2;
}
The following is a sample gNMI collection payload. In this sample you see two collections for the device group "milpitas". The first collects interface statistics, every 60 seconds using the "mode" = "SAMPLE". The second job captures any changes to the interface state (up/down). If this is detected, it is simply sent "mode" = "STREAM" to the collector.
{
    "collection_job": {
        "job_device_set": {
            "device_set": {
                "device_group": "milpitas"
            }
        },
        "sensor_output_configs": [{
            "sensor_data": {
                "gnmi_standard_sensor": {
                    "Subscribe_request": {
                        "subscribe": {
                            "subscription": [{
                                "path": {
                                    "origin": "openconfig-interfaces",
                                    "elem": [{
                                        "name": "interfaces/interface/state/ifindex"
                                    }]
                                },
                                "mode": "SAMPLE",
                                "sample_interval": 10000000000
                            }, {
                                "path": {
                                    "origin": "openconfig-interfaces",
                                    "elem": [{
                                        "name": "interfaces/interfaces/state/counters/out-octets"
                                    }]
                                },
                                "mode": "ON_CHANGE",
                                "sample_interval": 10000000000
                            }],
                            "mode": "STREAM",
                            "encoding": "JSON"
                        }
                    }
                }
            },
            "destination": {
                "context_id": "hukarz",
                "destination_id": "c2a8fba8-8363-3d22-b0c2-a9e449693fae"
            }
        }],
        "sensor_input_configs": [{
            "sensor_data": {
                "gnmi_standard_sensor": {
                    "Subscribe_request": {
                        "subscribe": {
                            "subscription": [{
                                "path": {
                                    "origin": "openconfig-interfaces",
                                    "elem": [{
                                        "name": "interfaces/interface/state/ifindex"
                                    }]
                                },
                                "mode": "SAMPLE",
                                "sample_interval": 10000000000
                            }, {
                                "path": {
                                    "origin": "openconfig-interfaces",
                                    "elem": [{
                                        "name": "interfaces/interfaces/state/counters/out-octets"
                                    }]
                                },
                                "mode": "ON_CHANGE",
                                "sample_interval": 10000000000
                            }],
                            "mode": "STREAM",
                            "encoding": "JSON"
                        }
                    }
                }
            },
            "cadence_in_millisec": "60000"
        }],
        "application_context": {
            "context_id": "testing.group.gnmi.subscription.onchange",
            "application_id": "testing.postman.gnmi.standard.persistent"
        },
        "collection_mode": {
            "lifetime_type": "APPLICATION_MANAGED",
            "collector_type": "GNMI_COLLECTOR"
        }
    }
}
Enable Secure gNMI communication between Device and Crosswork

Cisco Crosswork can only use one rootCA certificate (self-signed or signed by a trusted root CA) which means all device certificates must be signed by same CA.

If you have certificates signed by a different a trusted root CA, you can skip the first step and start from Step 2 to import the rootCA certificate in Cisco Crosswork.

Follow these steps to enable secure gNMI between Cisco Crosswork and the devices:

  1. Generate the certificates. See Generate Device Certificates.

  2. Upload the certificates to the Crosswork Certificate Management UI in Cisco Crosswork. See Configure gNMI Certificate.

  3. Update device configuration with secure gNMI port details from Cisco Crosswork UI. See Update Protocol on Device from Cisco Crosswork.

  4. Enable gNMI on the device. See Device Configuration for gNMI.

  5. Enable gNMI bundling on the device. See Configuring gNMI Bundling for IOS XR.

  6. Configure the certificates and device key on the device. See Import and Install Certificates on Devices.

Generate Device Certificates

This section explains how to create certificates with OpenSSL.

Steps to generate certificates have been validated with Open SSL and Microsoft. For the purpose of these instructions, we have explained the steps to generate device certificates with Open SSL.


Note
To generate device certificates with a utility other than Open SSL or Microsoft, consult the Cisco Support Team.

  1. Create the rootCA certificate

    # openssl genrsa -out rootCA.key
# openssl req -subj /C=/ST=/L=/O=/CN=CrossworkCA -x509 -new -nodes -key rootCA.key -sha256 -out rootCA.pem -days 1024

    In the above command, the days attribute determines the how long the certificate is valid. The minimum value is 30 days which means you will need to update the certificates every 30 days. We recommend setting the value to 365 days.

  2. Create device key and certificate

    
# openssl genrsa -out device.key
# openssl req -subj /C=/ST=/L=/O=/CN=Crosswork -new -key device.key -out device.csr
# openssl x509 -req -extfile <(printf "subjectAltName=IP.0: 10.58.56.18") -in device.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -sha256 -out device.crt -days 1024

    If you have multiple devices, instead of creating multiple device certificates, you can specify multiple device IP addresses separated by a comma in the subjectAltName. 

    # openssl x509 -req -extfile <(printf "subjectAltName=IP.0: 10.58.56.18, IP.1: 10.58.56.19, IP.2: 10.58.56.20 ..... ") -in device.csr -CA rootCA.pem -CAkey rootCA.key -CAcreateserial -sha256 -out device.crt -days 1024

  3. Verify if the certificate is created and contains the expected SAN details

    # openssl x509 -in device.crt -text -noout

    The following is a sample output:

    Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            66:38:0c:59:36:59:da:8c:5f:82:3b:b8:a7:47:8f:b6:17:1f:6a:0f
        Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN = rootCA
        Validity
            Not Before: Oct 28 17:44:28 2021 GMT
            Not After : Aug 17 17:44:28 2024 GMT
        Subject: CN = Crosswork
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                RSA Public-Key: (2048 bit)
                Modulus:
                    00:c6:25:8a:e8:37:7f:8d:1a:7f:fa:e2:d6:10:0d:
                    b8:e6:2b:b0:b0:7e:ab:c9:f9:14:a3:4f:2e:e6:30:
                    97:f4:cd:d6:11:7d:c0:a6:9b:43:83:3e:26:0f:73:
                    42:89:3c:d7:62:7b:04:af:0b:16:67:4c:8e:60:05:
                    cc:dd:99:37:3f:a4:17:ed:ff:28:21:20:50:6f:d9:
                    be:23:78:07:dc:1e:31:5e:5f:ca:54:27:e0:64:80:
                    03:33:f1:cd:09:52:07:6f:13:81:1b:e1:77:e2:08:
                    9f:b4:c5:97:a3:71:e8:c4:c8:60:18:fc:f3:be:5f:
                    d5:37:c6:05:6e:9e:1f:65:5b:67:46:a6:d3:94:1f:
                    38:36:54:be:23:28:cc:7b:a1:86:ae:bd:0d:19:1e:
                    77:b7:bd:db:5a:43:1f:8b:06:4e:cd:89:88:e6:45:
                    0e:e3:17:b3:0d:ba:c8:25:9f:fc:40:08:87:32:26:
                    69:62:c9:57:72:8a:c2:a1:37:3f:9d:37:e9:69:33:
                    a5:68:0f:8f:f4:31:a8:bc:34:93:a3:81:b9:38:87:
                    2a:87:a3:4c:e0:d6:aa:ad:a7:5c:fb:98:a2:71:15:
                    68:e7:8d:0f:71:9a:a1:ca:10:81:f8:f6:85:86:c1:
                    06:cc:a2:47:16:89:ee:d1:90:c9:51:e1:0d:a3:2f:
                    9f:0b
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Alternative Name:
                IP Address:10.58.56.18
    Signature Algorithm: sha256WithRSAEncryption
         01:41:2c:91:0b:a1:10:8a:11:1a:95:36:99:2c:27:31:d3:7d:
         e9:4b:29:56:c3:b7:00:8c:f4:39:d2:8c:50:a4:da:d4:96:93:
         eb:bb:71:e3:70:d3:fe:1f:97:b2:bc:5c:f8:f4:65:ed:83:f7:
         67:56:db:0f:67:c2:3d:0c:e7:f8:37:65:1d:11:09:9a:e3:42:
         bc:c6:a0:31:7c:1f:d7:5e:c6:86:72:43:a8:c1:0c:70:33:60:
         dc:14:5b:9d:f3:ab:3d:d5:d2:94:90:1c:ba:fd:80:4d:22:e3:
         31:93:c7:16:5f:85:20:38:ad:36:b9:1a:e0:89:8e:06:8c:f8:
         cd:55:cc:a1:89:d3:91:7f:66:61:a3:40:71:c2:1e:ee:3b:80:
         37:af:73:5e:8e:0d:db:4b:49:da:a6:bd:7d:0a:aa:9e:9a:9e:
         fa:ed:05:25:08:f2:4d:cd:2f:63:55:cf:be:b1:5d:03:c2:b3:
         32:bf:f4:7b:1a:10:b9:5e:69:ac:77:5e:4a:4f:85:e3:7f:fe:
         04:df:ce:3e:bb:28:8f:e3:bf:1a:f9:0f:94:18:08:86:7d:59:
         57:71:0a:97:0d:86:9c:63:e7:0e:48:7d:f0:0e:1d:67:ff:9b:
         1d:1b:05:25:c8:c3:1f:f4:52:0f:e1:bf:86:d7:ec:47:10:bd:
         94:cf:ca:e2
Configure gNMI Certificate

Embedded Collectors act as the gNMI client while the device acts as the gNMI server. The collectors validate the device using a trust chain. It’s expected that you have a global trust chain for all the devices. If you have multiple trust chains, add all the device trust chains (single or multiple vendors) in a single .pem file and upload this .pem file to the Crosswork Certificate Management UI.


Note
You can upload only one gNMI certificate to Crosswork.

To add the gNMI certificate.

Procedure

Step 1

From the Cisco Crosswork UI, go to Administration > Certificate Management.

Step 2

Click the + icon to add the certificate.

Step 3

In the Add Certificate window, enter the following details:

  • Device Certificate Name - Enter a name for the certificate.

  • Certificate Role - Select Device gNMI Communication from the drop-down list.

  • Device Trust Chain - Browse your local file system to the location of the rootCA file and upload it.

Step 4

Click Save.

The gNMI certificate gets listed in the configured certificates list when it is added.

Import and Install Certificates on Devices

This section describes how to import and install certificates on the IOS XR and XE devices. Certificates and trustpoint are only required for secure gNMI servers.

Certificates on a Cisco IOS XR Device

To install certificates on a Cisco IOS XR device.

  1. Copy rootCA.pem, device.key, and device.crt to the device under /tmp folder.

  2. Log in into the IOS XR device.

  3. Use the run command to enter the VM shell.

    RP/0/RP0/CPU0:xrvr-7.2.1#run

  4. Navigate to the following directory:

    cd /misc/config/grpc

  5. Create or replace the content of the following files:


    Note
    If TLS was previously enabled on your device, the following files will already be present in which case replace the content of these files as explained below. If this is the first time, you are enabling TLS on the device, copy the files from the /tmp folder to this folder.

    • ems.pem with device.crt

    • ems.key with device.key

    • ca.cert with rootCA.pem

  6. Restart TLS on the device for changes to take an effect. This step involves disabling TLS with "no-tls" command and re-enabling it with "no no-tls" configuration command on the device.

Certificates on a Cisco IOS XE Device

The following example shows how to install a certificate on a Cisco IOS XE device:

# Send:
Device# configure terminal
Device(config)# crypto pki import trustpoint1 pem terminal password password1 
 
# Receive:
% Enter PEM-formatted CA certificate.
% End with a blank line or "quit" on a line by itself.
 
# Send:
# Contents of rootCA.pem, followed by newline + 'quit' + newline:
-----BEGIN CERTIFICATE-----
<snip>
-----END CERTIFICATE-----
quit
 
# Receive:
% Enter PEM-formatted encrypted private General Purpose key.
% End with "quit" on a line by itself.
 
# Send:
# Contents of device.des3.key, followed by newline + 'quit' + newline:
-----BEGIN RSA PRIVATE KEY-----
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,D954FF9E43F1BA20
<snip>
-----END RSA PRIVATE KEY-----
quit
 
# Receive:
% Enter PEM-formatted General Purpose certificate.
% End with a blank line or "quit" on a line by itself.
 
# Send:
# Contents of device.crt, followed by newline + 'quit' + newline:
-----BEGIN CERTIFICATE-----
<snip>
-----END CERTIFICATE-----
quit
 
# Receive:
% PEM files import succeeded.
Device(config)#
 
# Send:
Device(config)# crypto pki trustpoint trustpoint1
Device(ca-trustpoint)# revocation-check none
Device(ca-trustpoint)# end
Device#
Update Protocol on Device from Cisco Crosswork

After you have configured the gNMI certificate in the Cisco Crosswork, update the device with secure protocol details either from the Cisco Crosswork UI (Device Management > Network Devices) or by specifying the protocol details as GNMI_SECURE Port in the .csv file.

The following image shows the updated secure protocol details for a device.

Figure 6. Edit Device Details Window
Edit Device Details Window
Device Configuration for gNMI

This section describes the steps to configure the IOS XR and IOS XE devices to support gNMI-based telemetry data collection.

Cisco IOS XR devices

  1. Enable gRPC over an HTTP/2 connection.

    Router#configure
Router(config)#grpc
Router(config-grpc)#port <port-number>

    The port number ranges 57344–57999. If a port number is unavailable, an error is displayed.

  2. Set the session parameters.

    Router(config)#grpc{ address-family | dscp | max-request-per-user | max-request-total | max-streams | 
max-streams-per-user | no-tls | service-layer | tls-cipher | tls-mutual | tls-trustpoint | vrf }

    where:

    • address-family: Set the address family identifier type.

    • dscp: Set QoS marking DSCP on transmitted gRPC.

    • max-request-per-user: Set the maximum concurrent requests per user.

    • max-request-total: Set the maximum concurrent requests in total.

    • max-streams: Set the maximum number of concurrent gRPC requests. The maximum subscription limit is 128 requests. The default is 32 requests.

    • max-streams-per-user: Set the maximum concurrent gRPC requests for each user. The maximum subscription limit is 128 requests. The default is 32 requests.

    • no-tls: Disable transport layer security (TLS). The TLS is enabled by default.

    • service-layer: Enable the grpc service layer configuration.

    • tls-cipher: Enable the gRPC TLS cipher suites.

    • tls-mutual: Set the mutual authentication.

    • tls-trustpoint: Configure trustpoint.

    • server-vrf: Enable the server vrf.

  3. Enable Traffic Protection for Third-Party Applications (TPA).

    tpa
vrf default
  address-family ipv4
   default-route mgmt
   update-source dataports MgmtEth0/RP0/CPU0/0
Cisco IOS XE Devices

The following example shows how to enable the gNMI server in insecure mode:

Device# configure terminal
Device(config)# gnmi-yang
Device(config)# gnmi-yang server
Device(config)# gnmi-yang port 50000 <The default port is 50052.>
Device(config)# end
Device

The following example shows how to enable the gNMI server in secure mode:

Device# configure terminal
Device(config)# gnmi-yang server
Device(config)# gnmi-yang secure-server
Device(config)# gnmi-yang secure-trustpoint trustpoint1
Device(config)# gnmi-yang secure-client-auth
Device(config)# gnmi-yang secure-port 50001 <The default port is 50051.>
Device(config)# end
Device
Configuring gNMI Bundling for IOS XR

In IOS XR, gNMI bundling is implemented to stitch together several Update messages that are included in the Notification message of a SubscribeResponse message. These messages are sent to the IOS XR device. To bundle the Update messages, you must enable bundling and specify the size of the message in the IOS XR device.

Before you begin

Make sure that you are aware of the following:

Procedure

Step 1

Enable the bundling feature using the following command:

telemetry model-driven
 gnmi
  bundling
The gNMI bundling capability is disabled by default.

Step 2

Specify the gNMI bundling size using the following command:

telemetry model-driven
 gnmi
  bundling
   size  <1024-65536>
The default bundling size is 32768 bytes.

Important

 

After processing the (N - 1) instance, if the message size is less than the bundling size, it may allow for one more instance, which results in exceeding the bundling size.

What to do next
Verify that the bundling capability is configured using the following: 
RP/0/RP0/CPU0:R0(config)#telemetry model-driven
RP/0/RP0/CPU0:R0(config-model-driven)#gnmi ?
  bundling   gNMI bundling of telemetry updates
  heartbeat  gNMI heartbeat
  <cr>
RP/0/RP0/CPU0:R0(config-model-driven)#gnmi bundling ?
  size  gNMI bundling size (default: 32768)
  <cr>
RP/0/RP0/CPU0:R0(config-model-driven)#gnmi bundling
RP/0/RP0/CPU0:R0(config-gnmi-bdl)#size ?
  <1024-65536>  gNMI bundling size (bytes)

Create a Collection Job from Cisco Crosswork UI

Follow the steps to create a collection job:


Note

Collection jobs created through the Crosswork Network Controller UI page can only be published once.

Before you begin

Ensure that a data destination is created (and active) to deposit the collected data. Also, have details of the sensor path and MIB that you plan to collect data from.

Procedure

Step 1

From the main menu, go to Administration > Collection Jobs > Bulk Jobs

Step 2

In the left pane, click Add icon button.

Step 3

In the Job details page, enter values for the following fields:

Figure 7. Job Details Window
Job Details Window

  • Application ID: A unique identifier for the application.

  • Context: A unique identifier to identify your application subscription across all collection jobs.

  • Collector Type: Select the type of collection - CLI or SNMP.

Click Next.

Step 4

Select the devices from which the data is to be collected. You can either select based on device tag or manually. Click Next.

Figure 8. Select Devices Window
Select Devices Window

Step 5

(Applicable only for CLI collection) Enter the following sensor details:

Figure 9. Sensor Details Window for CLI Path
Sensor Details Window

  • Select data destination from the Select Data Destination drop-down list.

  • Select sensor type from Sensor Types pane on the left.

If you selected CLI PATH, Click Add icon button and enter the following paramters in the Add CLI Path dialog box:

Figure 10. Add CLI Path Dialog Box
Add CLI Path Dialog Box

  • Collection Cadence: Push or poll cadence in seconds.

  • Command: CLI command

  • Topic: Topic associated with the output destination.

    Note

     
    Topic can be any string if using an external gRPC server.

If you selected Device Package, click Add icon button and enter values for the following parameters in the Add Device Package Sensor dialog box:

Figure 11. Add Device Package Sensor Dialog Box
Add Device Package Sensor Dialog Box

  • Collection cadence: Push or poll cadence in seconds.

  • Device Package Name: Custom XDE device package ID used while creating device package.

  • Function name: Function name within custom XDE device package.

  • Topic: Topic associated with the output destination.

Enter Key and String value for the paramters.

Click Save.

Step 6

(Applicable only for SNMP collection) Enter the following sensor details:

Figure 12. Sensor Details Window for SNMP Path
Sensor Details Window

  • Select data destination from the Select Data Destination drop-down list.

  • Select sensor type from Sensor Types pane on the left.

If you selected SNMP MIB, Click Add icon button and enter the following parameters in the Add SNMP MIB dialog box:

Figure 13. Add SNMP MIB Dialog Box
Add SNMP MIB Dialog Box

  • Collection Cadence: Push or poll cadence in seconds.

  • OID

  • Operation: Select the operation from the list.

  • Topic: Topic associated with the output destination.

If you selected Device Package, click Add icon button and enter values for the following parameters in the Add Device Package Sensor dialog box:

Figure 14. Add Device Package Sensor Dialog Box
Add Device Package Sensor Dialog Box

  • Collection Cadence: Push or poll cadence in seconds.

  • Device Package Name: Custom device package ID used while creating device package.

  • Function name: Function name within custom device package.

  • Topic: Topic associated with the output destination.

Enter Key and String value for the parameters.

Click Save.

Step 7

Click Create Collection Job.

Note

 

When a collection job is submitted for an external Kafka destination i.e., unsecure Kafka, the dispatch job to Kafka fails to connect. The error seen in collector logs is org.apache.kafka.common.errors.TimeoutException: Topic cli-job-kafka-unsecure not present in metadata after 60000 ms. In Kafka logs, the error seen is SSL authentication error "[2021-01-08 22:17:03,049] INFO [SocketServer brokerId=0] Failed authentication with /80.80.80.108 (SSL handshake failed) (org.apache.kafka.common.network.Selector).

This happens because port is blocked on external Kafka VM. You can use the following command to check if port is listening on Kafka docker/server port:

netstat -tulpn

Fix the problem on the Kafka server and restart the Kafka server process.

Monitor Collection Jobs

You can monitor the status of the collection jobs currently active on all Embedded Collectors instances that are enrolled with Crosswork Network Controller from the Collection Jobs page.

In the Cisco Crosswork UI, from the left navigation bar, choose Administration > Collection Jobs.

This left pane lists all active collection jobs along with their Status, App ID, Context ID, and Action. The Action drop-down lets you:

  • Delete: Removes a collection job.

  • Refresh: Refreshes the status of the collection job and the tasks that are associated with the job.

The Job Details pane shows the details of all collection tasks that are associated with a particular job in the left pane. The overall status of the Collection job in the Collection Jobs pane is the aggregate status of all the collection tasks in the Jobs Details pane.

When you select a job in the Collection Jobs pane, the following details are displayed in the Job Details pane:

  • Application name and context that is associated with the collection job.

  • Status of the collection job.


    Note

    • The status of a collection task associated with a device after it is attached to an Embedded Collector, is Unknown.

    • A job could have status as Unknown for one of the following reasons:

      • Embedded Collectors have not yet reported its status.

      • Loss of connection between Embedded Collectors and Cisco Crosswork.

      • Embedded Collectors have received the collection job, but the actual collection is still pending. For example, traps are not being sent to Embedded Collectors southbound interface, or the device is not sending telemetry updates.

      • The trap condition in an SNMP trap collection job which we are monitoring has not occurred. For example, if you are looking for Link Up or Link down transitions and the link state has not changed since the collector was established, then the state reports as Unknown. To validate that trap-based collections are working, it is therefore necessary to actually trigger the trap.

    • After the collection job is processed, the status changes to 'Successful' if the processing was successful or else it changes to 'Failed'.

    • If a collection job is in a degraded state, one of the reasons might be that the static routes to the device have been erased from Embedded Collectors.

    • Collections to a destination that is in an Error state do not stop. The destination state is identified in the background. If the destination is in an Error state, the error count is incremented. Drill down on the error message that is displayed in the Distribution status to identify and resolve the issue by looking at respective collector logs.

    • Cisco Crosswork Health Insights - KPI jobs must be enabled only on devices mapped to an extended Embedded Collector instance. Enabling KPI jobs on devices that are mapped to a standard Embedded Collector instance reports the collection job status as Degraded and the collection task status as Failed in the Jobs Details pane.

  • Job configuration of the collection job that you pass in the REST API request. Click the icon next to Config Details to view the job configuration. Crosswork Network Controller lets you view configuration in two modes:

    • View Mode

    • Text Mode

  • Collection type

  • Time and date of last modification of the collection job.

  • Collections (x): x refers to requested input collections that span device by sensor paths. The corresponding (y) Issues is the count of input collections that are in the UNKNOWN or FAILED state.

  • Distributions (x): x refers to requested output collections that span device by sensor paths. The corresponding (y) Issues is the count of output collections that are in the UNKNOWN or FAILED state.

Crosswork Network Controller also displays the following details for collections and distributions:

Field

Description

Collection/Distribution Status

Status of the collection/distribution. It is reported on a on change basis from Embedded Collectors. Click next to the collection/distribution status for details.

Hostname

Device hostname with which the collection job is associated.

Device Id

Unique identifier of the device from which data is being collected.

Sensor Data

Sensor path

Click to see collection/distribution summary. From the sensor data summary pop-up you can copy the sensor data by clicking Copy to Clipboard.

Click to see collection/distribution metrics summary. The metrics are reported on a cadence-basis, i.e., once every 10 minutes by default. It shows the following metrics for a collection:

  • last_collection_time_msec

  • total_collection_message_count

  • last_device_latency_msec

  • last_collection_cadence_msec

It shows the following metrics for a collection:

  • total_output_message_count

  • last_destination_latency_msec

  • last_output_cadence_msec

  • last_output_time_msec

  • total_output_bytes_count

Destination

Data destination for the job.

Last Status Change Reported Time

Time and date on which the last status change was reported for that device sensor pair from Embedded Collectors.

Note

  • Create Failed error means out of N devices, some devices failed to set up. However, the collection would happen on the devices that were successfully set up. You can identify one or more devices causing this error by using the Control Status API.

  • If job creation failed on a particular device because of NSO errors, after fixing NSO errors, you have to manually change the administration state of the device first to "Down" and then "Up". However, doing so resets the collection on the device.


Note

Create/Delete failed errors are shown in a different screen pop-up. Click next to the job status to see details of the error.

  • You may also try recreating the job using the PUT collection job API with the same payload.

Collection Status for Event-Based Collection Jobs

  1. When data collection is successful, the status of the Collection job changes from Unknown to Success in the Collection Jobs pane.

  2. When a device is detached from Embedded Collectors, all corresponding collection jobs are deleted and collection job status is displayed as Success in the Collection Jobs pane. There are no devices or collection tasks that are displayed in the Job Details pane.

  3. When a device is attached to an Embedded Collector, Crosswork receives a new collection job with the status set to Unknown that changes to Success after receiving events from the device.

  4. If the device configuration is updated incorrectly on a device that is already attached to Embedded Collectors and after Embedded Collectors has received the job and events, there is no change in status of the collection task in the Jobs Details pane.

  5. If the device inventory is updated with an incorrect device IP, the collection task status in the Jobs Details pane is Unknown.

Delete a Collection Job

System jobs (default jobs created by various Crosswork Applications) should not be deleted as it causes collection issues. Jobs created by Health Insights should only be deleted by disabling the KPI profile which will remove the collection jobs it deployed. When you delete a collection job, it deletes the associated collection tasks.

Use this procedure to delete external collection jobs from the Collection Jobs page. Follow the steps to delete a collection job:

Procedure

Step 1

From the main menu, go to Administration > Collection Jobs.

Step 2

Select either the Bulk Jobs tab or Parametrized Jobs tab.

Step 3

In the Collection Jobs pane on the left-hand side, select the collection job that you want to delete.

Step 4

In the corresponding row, click More icon and select Delete. The Delete Collection Job window is displayed.

Step 5

Click Delete when prompted for confirmation.

Monitor Embedded Collectors Application Health

The CW Embedded Collectors and Offload Components tile provides the operations and health summary of Embedded Collectors. You can find information about the health of pods running on the Crosswork container on this page. The Embedded Collectors application's overall health is influenced by the health of each individual pod service.

Figure 15. Crosswork Embedded Collector and Offload Component Pane
Crosswork Embedded Collector and Offload Component Pane

To review the collector health status, navigate to the Crosswork UI:

Procedure

Step 1

From the main menu, choose Administration > Crosswork Manager > CW Embedded Collectors and Offload Components tile.

Step 2

(Optional) From the Showtech Options drop-down, you can perform the following operations:

  • Request All: Collects both logs and metrics. To view the logs, navigate to Crosswork Manager > Application Management > Showtech Requests to view the logs.

  • Request Metrics: Collects the metrics information.

  • Request Logs: Collects the log information.

  • View Showtech Jobs: Displays the progress of the Showtech jobs. Alternatively, you can also see the job's status and other details from Crosswork Manager > Application Management > Showtech Requests.

  • Change Log Level: Enables you to change the log level of various components within the embedded collector, such as collectors (cli-collector) and infrastructure services (oam-manager). Log level changes only impact the collector where the change is being made.

Step 3

(Optional) From the Application Actions drop-down, you can perform the following operations:

  • Install: Installs new collector with the same information (profile, hostname, management interface) as the previous collectors.

  • Upgrade: Upgrades the collector version.

  • Activate: Activate the collectors.

  • Uninstall: Removes the collector app. The execution of this operation has consequences for the collection jobs currently in progress.

Monitor the Collector's Pod Health

In the Embedded Collector and Offload Component pane, you can get a detailed overview of the health status of the pods that host the collectors or microservices. We suggest regularly monitoring the health of the collector pods in your network to avoid overloading and taking proactive corrective measures, such as adding more resources or reducing the load on the collector in a timely manner.

Figure 16. Microservices Tab
Embedded Collector and Offload Component > Microservices Tab

To view the pod health status, navigate to the Crosswork UI:

Procedure

Step 1

From the main menu, choose Administration > Crosswork Manager > CW Embedded Collectors and Offload Components.

Step 2

Expand the CW Embedded Collectors and Offload Components pane and the click the Microservices tab.

Step 3

(Optional) In the Microservices tab, type the collector name in the Name field to locate the collector pod.

Step 4

(Optional) From this page, click under the Actions column to perform the following actions:

  • Restart– Restarts the collector pod. Restarting a pod causes disruption in the ongoing collection process. Whenever you need to restart, start, or stop a process, we strongly advise to consult the Cisco TAC team.

  • Showtech Requests– Displays the showtech jobs executed for the corresponding collector pod. You have to navigate to Crosswork Manager > Application Management > Showtech Requests to view the logs.

  • Request All– Collects both logs and metrics of the pod. To view the logs, navigate to Crosswork Manager > Application Management > Showtech Requests to view the logs.

  • Request Metrics– Collects the metrics of the pod.

  • Request Logs– Collects the logs of the pod.

Viewing Collector Alarms and Events

Embedded Collectors generate an alarm when it detects an anomaly that prevents data collections. Monitoring the alarms enables you to detect the potential collector issues affecting data collection, and take the remediation action, if necessary.

To view the alarms, from the Crosswork Network Controller UI, choose Administration > Crosswork Manager > CW Embedded Collectors and Offload Components.

The Alarms tab provides a consolidated list of all collector alerts and events. You can toggle between the Alarms and Events subtabs to view the respective details.

Figure 17. Events Tab
Events Tab

Filter the list of alarms or events by filtering columns, changing the Active Alarms Only slider, or adding or removing columns using the Settings icon icon.

Troubleshoot Embedded Collectors

The troubleshooting section contains information about the possible issues and corrective actions that you may observe with Embedded Collectors.

Admin State Automatically Changes from DOWN to UP

In a single VM deployment, the devices are automatically attached to Embedded Collectors, and their Admin State is changed from DOWN to UP.

Workaround: If you want to change the state to DOWN, manually change it through the Edit Device page. For more information about editing a device, see Edit Devices section in the Cisco Crosswork Network Controller 7.0 Device Lifecycle Management.