Cisco Catalyst SD-WAN Control Components and Device Management Guide, Releases 26.x and Later

PDF

System logging

Want to summarize with AI?

Log in

Describes fundamentals of system logging, including purpose, key features, and its role in monitoring and troubleshooting network devices.


System logging is a process that

  • records a text log of system events using a mechanism similar to the UNIX syslog command,

  • allows devices to send log messages with configurable priority levels to UNIX-style syslog services, and

  • supports secure transmission over the Transport Layer Security (TLS) protocol.

Priority levels

Log messages have levels that indicate their priority. These are the same as for standard UNIX commands. You can configure the priority of syslog messages.

Security

Cisco IOS XE Catalyst SD-WAN devices send syslog messages to syslog servers on configured external hosts using TCP and UDP. When the devices send the syslog messages, the messages might transit several hops to reach the output destination. The intermediate networks during the hops might not be trustworthy, be in a different domain, or have a different security level. Therefore, Cisco IOS XE Catalyst SD-WAN devices support sending secure syslog messages over TLS as described in RFC 5425. To secure the syslog message content from potential tampering, the TLS protocol is used for certificate exchange, mutual authentication, and ciphers negotiation.

Cisco IOS XE Catalyst SD-WAN devices support both mutual and server authentication for sending syslog messages over TLS.

Benefits of using TLS

  • Message confidentiality

    Confidentiality of message content where each TLS session begins with a handshake between the Cisco IOS XE Catalyst SD-WAN device and the syslog server. The Cisco IOS XE Catalyst SD-WAN device and syslog server agree on the specific security key and the encryption algorithms to be used for that session. The TLS session opposes any disclosure of the contents of the syslog message.

  • Message integrity

    Integrity-checking of the content of each message to disable modifications to a message during transit on a hop-by-hop basis.

  • Authentication

    Mutual authentication between the Cisco IOS XE Catalyst SD-WAN device and syslog server ensures that the syslog server accepts log messages only from authorized clients through certificate exchange.


System log files

System log (syslog) messages that are at or above the default or configured priority value are recorded in a number of files in the /var/log directory on the local device of the syslog server. The log files contain these items.

Table 1. Log files

File

Contents

auth.log

Login, logout, and superuser access events, and usage of authorization systems.

kern.log

Kernel messages.

messages.log

Consolidated log file that contains syslog messages from all sources.

vconfd.log

All configuration-related syslog messages.

vdebug.log

All debug messages for modules whose debugging is turned on.

All syslog messages that are above the default priority value. The debug log messages support various levels of logging based on the module. The different modules implement the logging levels differently. For example, the system manager (sysmgr) has two logging levels (on and off), while the chassis manager (chmgr) has four different logging levels (off, low, normal, and high). You cannot send debug messages to a remote host. To enable debugging, use the debug operational command.

vsyslog.log

All syslog messages from Cisco Catalyst SD-WAN processes (daemons) that are above the configured priority value. The default priority value is "informational", so by default, all "notice", "warning", "error", "critical", "alert", and "emergency" syslog messages are saved.

vmanage-syslog.log

Cisco SD-WAN Manager audit log messages

Unused log files

Cisco Catalyst SD-WAN does not use these standard Linux files, which are available in the /var/log directory:

  • cron.log

  • debug.log

  • lpr.log

  • mail.log

  • syslog


System log formats

Syslog messages begin with a percent sign (%) and come in these formats:

  • Sequence and timestamp

    sequence-number:timestamp: %facility-severity-MENEMONIC:description (hostname-n)

  • Format based on RFC5424

    <pri>ver timestamp hostname appname procid msgid structured-data description/msg

    The optional fields such as hostname, appname, procId, msgId, and structured-data are specified with a -.

Table 2. Field descriptions

Field

Description

facility

Sets the logging facility to a value other than 20, which UNIX systems expect.

severity

Importance or severity of the message, 0 to 7. A lower number indicates a greater severity of the system condition.

msg or description

Text string that describes the condition of the syslog server. This portion of the syslog message sometimes includes IP addresses, interface names, port numbers, or usernames.

In syslog message formats based on RFC5424, the description is:

%facility-severity-MENEMONIC:description

Examples

This system logging message includes a priority value, sequence number, and timestamp:

<45>10: polaris-user1: *Jun 21 10:76:84.100: %LINK-5-CHANGED: Interface GigabitEthernet0/0, changed state to administratively down

This RFC5424-format message has a priority value, version of syslog protocol specification, and timestamp:

<45>1 2003-10-11T22:14:15.003Z 10.64.48.125 polaris-user1 - - - %LINK-5-CHANGED: Interface GigabitEthernet0/0, changed state to administratively down
Note

The time stamp formats are not the same in both the syslog message formats. In the message format based on RFC5424, T, and Z are mandatory where T represents a separator and Z represents zero timezone.


System log message levels

Each system log (syslog) message has a severity, or priority, level. A lower severity number means a higher severity. The default priority value is 6 (informational). By default, all syslog messages are recorded.

Table 3. System log message severity levels

Severity level

Name

Description

0

Emergency

System is unusable.

1

Alert

System in a state that requires immediate action.

2

Critical

Serious condition.

3

Error

Error condition that does not fully impair system usability.

4

Warning

Minor error condition.

5

Notice

Normal operation, but with a significant condition requiring notice.

6

Informational

Routine condition (default).

7

Debug

Debug messages.


Sending system log messages to a server, using TLS

Transport layer security (TLS) is a networking protocol that provides secure communication through a network.

Benefits of using TLS for sending syslog messages

The benefits of using TLS for sending syslog messages are:

  • Confidentiality

    Confidentiality of message content where each TLS session begins with a handshake between the Cisco IOS XE Catalyst SD-WAN device and the syslog server. The Cisco IOS XE Catalyst SD-WAN device and syslog server agree on the specific security key and the encryption algorithms to be used for that session. The TLS session opposes any disclosure of the contents of the syslog message.

  • Integrity-checking

    Integrity-checking of the content of each message to disable modifications to a message during transit on a hop-by-hop basis.

  • Mutual authentication

    Mutual authentication between the Cisco IOS XE Catalyst SD-WAN device and syslog server ensures that the syslog server accepts log messages only from authorized clients through certificate exchange.

Authentication type

  • Server

    With server authentication, edge devices verify the identity of the syslog server. If the syslog server and the certificate are legitimate entities, the device establishes a TLS connection with the server.

    As part of server authentication, the syslog server shares its public certificate with the devices.

    See the prerequisite in the "Before you begin" section of this procedure.

    With this option, all information about TLS profiles, except the trustpoint information, is saved.

  • Mutual

    With mutual authentication, edge devices and the syslog server authenticate each other at the same time.

    Devices require root or identity certificates for mutual authentication of the TLS session.

    With this option, a trustpoint, such as SYSLOG-SIGNING-CA certificate, is saved on the device. This enables SD-WAN Manager to install the certificate from the edge device.


Restrictions for system logging

Disabling system logging to disk

Disabling system logging to disk (no system logging disk enable) does not disable vsyslog.

Storage restrictions

The messages sent to syslog files are not rate-limited and consequently:

  • A storage limit of 10 log files with a capacity of up to 16 MB size is set for each syslog file.

    • When the storage capacity exceeds the 16 MB size limit, the log file is saved as a .GZ file along with the date appended to it.

    • When the storage limit exceeds 10 log files, the oldest log file is dropped.

  • If many syslog messages are generated in a short span of time, the overflowing messages are buffered and queued to be stored in the syslog file.

Repeating or identical messages

For repeating syslog messages or identical messages that occur multiple times in succession, only one copy of the message is placed in the syslog file. The message is annotated to indicate the number of times the message occurred.

Maximum length

The maximum length of a log message is 1024 bytes. The longer messages are truncated.

The maximum length of a log message for Cisco SD-WAN Manager audit logs is 1024 bytes. The longer messages are truncated into smaller fragments and each of these fragments are indicated by an identifier. The identifiers are

  • fragment 1/2

  • fragment 2/2

and so on.

For example, a long audit log message when truncated into smaller fragments appears as:

local6.info: 18-Oct-2020 17:42:07 vm10 maintenance-fragment-1/2: {“logid”: “d9ed576a-...”, “entry_time”: 
1576605512190, “statcycletime” 34542398334245, “logmodule”:”maintenance”, “logfeature”: “upgrade”, “loguser”: “admin”, “logusersrcip”: 
“10.0.1.1”, “logmessage”: “Device validation Upgrade to version :  Validation success”, “logdeviceid”:”Validation”, “auditdetails” :
[“[18-Oct-2020 17:42:08 UTC] Published messages to vmanage(s)”, “auditdetails”:[“[18-Oct-2020 17:42:07 UTC] Software image: vmanage-99.99.999-
x86_64.tar.gz”, “Software image download may take up to 60}
local6.info: 18-Oct-2020 17:42:07 vm10 maintenance-fragment-2/2: { minutes”, “logprocessid”: “software_install-7de0ec44-...”, “tenant”:, “default”}

AAA authentication and Netconf CLI access

Syslog messages related to AAA authentication and Netconf CLI access and usage are placed in the auth.log and messages.log files. Each time a Cisco SD-WAN Manager logs into a router to retrieve statistics and status information and to push files to the router, the router generates AAA and Netconf log messages. Over time, these messages can fill the log files. To prevent these messages from filling the log files, you can disable the logging of AAA and Netconf syslog messages.