Monitor cBR-8 Logs in CSPC

Monitor cBR-8 Logs in CSPC

Table 1. Feature History

Feature Name

Release Information

Feature Description

Monitor cBR-8 Logs in CSPC

Cisco IOS XE 17.15.1z

This feature is designed to improve the compatibility of cBR-8 error messages with external monitoring tools, specifically the CX Automated Fault Management (AFM) tool and the CX Common Services Platform Collector (CSPC).

The CX Automated Fault Management (AFM) tool reads syslog messages from a syslog server. The CX Common Services Platform Collector (CSPC) tool is responsible for collecting syslog messages from the cBR8 and sending them to the syslog server. Forwarding rules can be configured for CSPC to define which messages are forwarded from the syslogs to a syslog server. This feature is Part 1 of converting the syslog format to be compatible with AFM.

Compatibility is achieved by replacing the critical buginf with errmsg in the cBR-8 source code.

This feature is designed to improve the compatibility of cBR8 error messages with external monitoring tools, specifically the CX Automated Fault Management (AFM) tool and the CX Common Services Platform Collector (CSPC). The CX Automated Fault Management (AFM) tool relies on syslog messages collected by the CX Common Services Platform Collector (CSPC) from the cBR-8. To ensure that critical error messages generated by the cBR8 are successfully collected and processed by these tools, they need to adhere to a specific format, particularly including the '%' character which the default CSPC forwarding rules look for. Historically, some critical error messages on the cBR8 were printed using `buginf` and did not include the necessary '%' character, making them undiscoverable by AFM/CSPC. This feature converts these specific critical `buginf` messages related to Inter Process Communication (IPC) into `errmsg` logs that conform to the required syslog format, making them visible to AFM and CSPC.

Information about Monitoring cBR-8 Logs in CSPC

This feature focuses on converting critical IPC-related `buginf` messages within specific directories of the cBR8 source code (`src-ubr`) into `errmsg` messages. This conversion ensures that these important error conditions are reported in a standard syslog format (`%FACILITY-SEVERITY-NAME: <text>`) which includes the '%' character, allowing them to be collected and monitored by external systems like AFM and CSPC. The conversion involves identifying the critical `buginf` messages in designated source code directories and replacing their implementation with calls to the `errmsg` function, using defined macros (`facdef`, `msgdef`, `msgsym`) to format the output correctly for syslog.

The critical IPC related buginfs are converted to errmsg in the following directories in vob/ios/sys/src-ubr:

  • apps

  • dataplane

  • docsis/ipc_tdl

  • kman

  • platform

  • ubr-infra/bfd/ipc

For IPC related syslogs, this feature introduces a new class of message definition named CABLE_IPC_ERROR.

msgdef_limit(CABLE_IPC_ERROR, UBR7200, LOG_ERR, 0, "IPC failure: %s", MSGDEF_LIMIT_SLOW);

To avoid the flooding of messages, this message definition rate limits the logging as 1 message per 60 seconds.

Errmsg MACRO: 

#define CMTS_LCHA_IPC_ERRMSG(format, ...) \ 
    if (cmts_debug_tdl_bipc_tx || cmts_debug_tdl_bipc_rx) { \ 
        buginf ("\n" format, ##__VA_ARGS__); \ 
    } else { \ 
        snprintf(cable_ipc_errmsg, MAX_ERR_MSG_SIZE, format , ##__VA_ARGS__); \ 
        errmsg(&msgsym(CABLE_IPC_ERROR, UBR7200), cable_ipc_errmsg); \ 
    }

Sample Log Conversion (Releases belore 17.15.1z)

CMTS_LCHA_IPC_TX_BUGINF("\n%s_TX:%s:LCPR nonblocked send failed, "
                        "dst_slot %d type %d err %d",
                        "UBRIPC_LCHA_DEBUG", __FUNCTION__,
                        msg_ctx_p->d_slot,
                        msg_ctx_p->msg_type, msg_ctx_p->err);

Converted errmsg (Release 17.15.1z and Later)

CMTS_LCHA_IPC_ERRMSG( 
     "Failed to send LCPR nonblocked message, " 
     "dst_slot %d type %d err %d", 
     msg_ctx_p->d_slot, msg_ctx_p->msg_type, msg_ctx_p->err);

Syslog Message with '%': 

CLC 1/0: *Mar 31 02:00:29.491: %CBR-3-CABLE_IPC_ERROR: IPC failure: Failed to send LCPR nonblocked message, dst_slot 1 type 1137 err 14

Benefits of Monitoring cBR-8 Logs in CSPC

The primary benefit of this feature for customers is improved fault management and monitoring capabilities for critical IPC errors on the cBR8. By converting these messages to a standard syslog format compatible with tools like AFM and CSPC, customers can:

  • Ensure critical IPC errors are detected and reported by their existing monitoring infrastructure.

  • Gain better visibility into the health and status of the cBR8 system.

  • Streamline automated fault detection and response processes.

Configuration Overview

The core functionality of converting internal `buginf` messages to `errmsg` messages is an internal software change and does not require any specific configuration to enable the conversion for normal operation. However, the feature includes a test CLI command that allows you to trigger the generation of these converted syslog messages for verification purposes. This test command is the only user "configuration" or interaction related to this feature.

Supported Scenarios

The primary supported scenario is the collection and monitoring of critical IPC error messages from the cBR8 by the CX Automated Fault Management (AFM) tool via the CX Common Services Platform Collector (CSPC). The conversion ensures that these messages are in a format (`%FACILITY-SEVERITY-NAME: <text>`) that CSPC's default forwarding rules can recognize and forward to a syslog server for AFM processing.

Prerequisites for Monitoring cBR-8 Logs in CSPC

The prerequisites relate to the environment where the logs are consumed:

  • The CX Automated Fault Management (AFM) tool must be in use.

  • The CX Common Services Platform Collector (CSPC) tool must be responsible for collecting syslog messages from the cBR-8

  • CSPC forwarding rules should be configured (default rules look for the '%' character).

Monitor cBR-8 Logs in CSPC Update

Table 2. Feature History

Feature Name

Release Information

Feature Description

Monitor cBR-8 Logs in CSPC Update

Cisco IOS XE 17.18.1w

This feature is an update to the existing feature - converting the syslog format to be compatible with AFM, which is introduced in the Cisco IOS XE 17.15.1z release.

The CX Automated Fault Management (AFM) tool reads syslog messages from a syslog server. The CX Common Services Platform Collector (CSPC) tool is responsible for collecting syslog messages from the cBR8 and sending them to the syslog server. Forwarding rules can be configured for CSPC to define which messages are forwarded from the syslogs to a syslog server.

Information about Monitoring cBR-8 Logs in CSPC

Starting with Cisco IOS XE 17.18.1w, the critical IPC related buginfs are converted to errmsg in the following directories in vob/ios/sys/src-ubr:

  • mibs

  • docsis/ipc_tdl

  • ubr-infra/cable-monitor

  • ubr-infra/dcm

  • ubr-infra/md

  • ubr-infra/specmgmt

  • ubr-infra/specsvl

  • ubr-infra/stats

Configuration Overview

While the core conversion is an internal software change, customers manage the visibility and behavior of these logs through standard Cisco IOS XE commands. To ensure the new errmsg format is active and visible, and to manage general syslog output, follow these guidelines:

  1. Enable Console Logging (Optional):

    To view syslog messages directly on the console, use the following commands:

    config terminal
logging console
end
write
  2. Manage Debug Flags for IPC Modules:

    The logging behavior (whether buginf or errmsg is generated) for certain messages is controlled by debug flags. To ensure the converted errmsg format is used for monitoring (which is the default when debug is off), ensure that the relevant debug flags are disabled. If debug flags are enabled, some messages might revert to the buginf format.

    To disable all relevant debug flags:

    no debug cable multicast ipc
no debug cable monitor
no debug cable dcm ipc-debug
no debug cable dcm ipc-2cdman
no debug cable specmgmt ipc-tx
no debug cable specmgmt ipc-rx
no debug cable specsvl ipc-tx
no debug cable specsvl ipc-rx
no debug cable tdl-bipc-tx
no debug cable tdl-bipc-rx

    Note

    Enabling these debug flags (for example, debug cable specmgmt ipc-tx) will cause some messages to be generated using the buginf format, which may not be collected by CSPC.

  3. Clear Syslogs (Optional):

    To clear existing syslog messages, use:

    clear logging
clear logging all

Verify Convert cBR8 logs to be monitored in CSPC

You can verify that the converted IPC error messages are being generated in the correct syslog format using the following show commands:

  1. Check for Specific IPC Error Messages:

    Use the show logging command with a filter to look for the CABLE_IPC_ERROR messages. These messages indicate that the conversion to errmsg format has occurred and they are now properly formatted for CSPC. 

    show logging | i CABLE_IPC_ERROR
    Example Output:
    Jun 16 19:34:46.659 EDT: %CBR-3-TEST_CABLE_IPC_ERROR: IPC failure: TEST: 
DCM_CDMAN_TDL_TX: Failed to set mem for chan config of non mixed mod 1/1 err 2
Jun 16 19:34:46.659 EDT: %CBR-3-TEST_CABLE_IPC_ERROR: IPC failure: TEST: DCM_CDMAN_TDL_TX: 
Failed to set mem for config dpd ccc chan 1/1 err 2
  2. Check Syslog Count:

    Verify the count of CABLE_IPC_ERROR messages to ensure that they are being logged. 

    show logging count
    Example Output:
    CBR TEST_CABLE_IPC_ERROR 3 196 *Jun 16 19:34:46.720
CBR TOTAL 196

Monitor RIB Metric Metric Translation

After the feature is active and generating logs, you can monitor the collected syslog messages through your existing CX Automated Fault Management (AFM) and CX Common Services Platform Collector (CSPC) tools.

  1. Monitor via CSPC

    Ensure CSPC is actively collecting logs from the cBR8 and forwarding them to your syslog server. The converted `errmsg` messages, identifiable by the '%' character and their standard format, will be included in this collection.

  2. Monitor via AFM:

    Use the CX AFM tool to analyze the syslog messages received from the syslog server. The newly formatted critical IPC error messages from the cBR8 will now be visible and actionable within AFM, allowing for centralized fault management and analysis.