Azure JWT Tokens - user doesn't exist

Applicability

Description

JWT tokens previously working, however, any new attempt to create a new JWT token returns the following response in the UI:

User "$USER" does not exit

When trying to edit or delete any existing token the error is present as well.

Rest API calls to SH using exiting tokens return a response similar to the following:

```
<?xml version="1.0" encoding="UTF-8"?>
<response>
<messages>
<msg type="WARN">call not properly authenticated</msg>
</messages>
</response>
```

Only SAML users are affected by this behavior.
Already created tokens can be seen from the UI.

Investigation

1. Track down `User "$USER" does not exist` response.

index=_internal component=JsonWebTokenHandler ERROR "$USER"ERROR JsonWebTokenHandler [60939 TcpChannelThread] - User "$USER" does not exist.

2. Search for events similar to  `User "$USER" does not exist.` response on splunkd.log.  It is helpful to search for "Events Before or After" option on the UI and select a couple of milliseconds from the original event. e.g:

ERROR JsonWebTokenHandler [60939 TcpChannelThread] - User "$USER" does not exist.
ERROR UserManagerPro [60939 TcpChannelThread] - Did not find any info for user=$USER
ERROR AuthenticationProviderSAML [60939 TcpChannelThread] - Error message from function=getUserInfo() : It appears your clientSecret is incorrect. Search the Microsoft documentation to retrieve that value: "Create an Azure AD app & service principal in the portal - Microsoft identity platform" / "Option 2: Create a new application secret"
ERROR AuthenticationProviderSAML [60939 TcpChannelThread] - Authentication extension function=getUserInfo() returned status=failed for user=$USER
WARN  ScriptedAuthHelper [60939 TcpChannelThread] - Function 'getUserInfo' failed. Could not find '--status=success' in output
WARN  ScriptedAuthHelper [60939 TcpChannelThread] - Function=getUserInfo took ~587.7 milliseconds to execute. elapsed_msec=588

3. In this specific scenario following message provides relevant information:

ERROR AuthenticationProviderSAML [60939 TcpChannelThread] - Error message from function=getUserInfo() : It appears your clientSecret is incorrect. Search the Microsoft documentation to retrieve that value: "Create an Azure AD app & service principal in the portal - Microsoft identity platform" / "Option 2: Create a new application secret"

Taking in consideration that JWT tokens were previously working, this likely means that the `clientSecret` used to configure 
Authentication Extensions has expired (or was modified).

Resolution

Create and configure a new application secret for Splunk Authentication Extensions under Splunk SAML configuration. 
Use the following Microsft Azure documentation:
MS new client secret.
Auth Extensions Azure

Related Links

MS new client secret.
Auth Extensions Azure