RFC7519JsonWebToken generates excessive malformed JWT warnings in Splunk Enterprise 10.2.1 during expected token validation

Applicability

Description

After upgrading to Splunk Enterprise 10.2.1 a flood of messages may be written to the splunkd logs by the RFC7519JsonWebToken logger.

The observed user facing behaviour includes :

• Approximately 60,000 to 70,000 warning messages per hour.
• No associated authentication or application failure.
• No clear relationship between the warnings and surrounding events, TCP channel activity, or HTTP 401 responses.
• The warnings are generated by TcpChannelThread threads

Example warning :

WARN RFC7519JsonWebToken [1496259 TcpChannelThread] - Unable to deserialize token header. Error=Malformed JWT string: got 1, expected 3

Investigation

Reviewed events immediately before and after the RFC7519JsonWebToken warnings. No related process or authentication failure was identified.

Reviewed the associated TcpChannelThread activity and searched for HTTP 401 responses. No correlation was found between the warnings and failed authentication.

For deployments using Search Head Cluster members where Agent Management or Deployment Server functionality is not used, the following telemetry setting was tested in $SPLUNK_HOME/etc/system/local/agent_management.conf and through the deployer :

[telemetry]
enabled = false

Splunk was restarted after applying the setting. Disabling Agent Management telemetry did not suppress the warnings in the affected environment.

Checked the current logger level :

$ splunk show log-level RFC7519JsonWebToken
Component: RFC7519JsonWebToken
Level: INFO
Buffering: 0

Resolution

The warning is generated when the token validation flow attempts to interpret a session token as a JSON Web Token. A JWT contains three dot separated sections. When the supplied token contains only one section, the JWT parser reports :

Malformed JWT string: got 1, expected 3

Failure to interpret the session token as a JWT is an expected part of the validation flow. In Splunk Enterprise 10.2.1, this condition is logged at WARN level, resulting in excessive benign messages.

As a workaround, set the RFC7519JsonWebToken logger level to CRIT. Add the following configuration to $SPLUNK_HOME/etc/log-local.cfg. Create the file if it does not exist :

[splunkd]
category.RFC7519JsonWebToken=CRIT

Restart Splunk to apply the persistent configuration.

Alternatively, apply the logger change immediately without restarting Splunk :

$ splunk set log-level RFC7519JsonWebToken -level CRIT

Verify the updated logger level :

$ splunk show log-level RFC7519JsonWebToken

Expected output :

Component: RFC7519JsonWebToken
Level: CRIT
Buffering: 0

Setting the logger to CRIT suppresses all messages from this logger below the CRIT level.