Troubleshooting Splunk REST API Authentication Failures and Bearer Token Generation Issues

Applicability

Description

In distributed Splunk environments using LDAP or local authentication, REST API logins might fail even though credentials work in Splunk Web. Typical failure symptoms:

• API responses show:
  {"messages":[{"type":"WARN","text":"call not properly authenticated"}]}

• splunkd_access.log shows HTTP 401

• /services/authorization/tokens/v1/authenticate returns 404 Not Found

• /services/auth/login reports “Login failed”, even with known working accounts

• Wrong header formats are used, e.g. Authorization: Bearer instead of Authorization: Splunk for session keys

Example masked access logs:

[06/Nov/2025:09:13:19 +0000] "POST /servicesNS/<username>/search/search/jobs/export HTTP/1.1" 401 - "-" "curl/8.7.1"

Example masked successful LDAP login entry:

INFO AuditLogger - Audit: [timestamp=..., user=<LDAP_USER_ID>, action=login attempt, info=succeeded ... method=LDAP useragent="curl/8.7.1" ...]

Investigation

1. Check Authentication Method

• Confirm configured authentication type:

  $SPLUNK_HOME/bin/splunk btool authentication list --debug

  Example:

  [authentication]
  authType = LDAP
  authTypePreferredForUserCollision = Splunk

2. Validate Host and User Scope

• Ensure REST calls are sent to the same host used for successful Splunk Web login.

• Confirm local user exists on the node if using local auth ($SPLUNK_HOME/etc/passwd).

• LDAP or local admin names must not include site-specific identifiers in the article.

3. Reproduce and Capture Logs

• In splunkd_access.log → look for HTTP 401 / 403.

• In splunkd.log → look for messages containing:
  AuthenticationManager, Login failed, token, basic auth disabled.

4. Test Session Login (Correct Method)

• /services/auth/login does not accept -u user:pass. It requires POST fields:

  curl -sk https://<HOST>:8089/services/auth/login \
    -d username=<USERNAME> \
    --data-urlencode password='<PASSWORD>' \
    -d output_mode=json

• A valid response returns "sessionKey":"...".

5. Test REST Using Session Key

curl -sk -H "Authorization: Splunk <SESSION_KEY>" \
  'https://<HOST>:8089/services/server/info?output_mode=json'

6. Generate JWT Bearer Token

If token-based auth is enabled:

curl -sk -H "Authorization: Splunk <SESSION_KEY>" \
  https://<HOST>:8089/services/authorization/tokens?output_mode=json \
  -d name='api-token' \
  -d audience='Users' \
  -d type='permanent' \
  -d user='<USERNAME>'

Note: /services/authorization/tokens/v1/authenticate is not a valid endpoint in Splunk Enterprise.

Resolution

To resolve REST API authentication failures and generate a valid token for use with Authorization: Bearer, run the following command (replace placeholders accordingly):

curl -k -u 'admin:<PASSWORD>' \
  https://<HOSTNAME>:8089/services/authorization/tokens?output_mode=json \
  --data name=<USERNAME> \
  --data audience=Users \
  --data-urlencode expires_on=+30d

This will return a JSON response containing a field called "token".
Use that token in API requests like this:

curl -k -X POST \
  -H "Authorization: Bearer <TOKEN_FROM_RESPONSE>" \
  -d 'search=| rest /services/apps/local | table title,label' \
  https://<HOSTNAME>:8089/servicesNS/nobody/search/search/jobs/export?output_mode=json&count=0

Related Links

Use REST to create authentication tokens