Applicability

Description

Splunk's scripttype=persist REST handler mechanism loads a Python script by its absolute filesystem path using Python's standard importlib.util.spec_from_file_location recipe. As a side effect of this path-based loading, the script's own containing directory (bin/) is never added to sys.path. Any attempt to import a sibling module located in the same bin/ directory therefore fails with ModuleNotFoundError, even though the file sits right next to the handler.

The failure mode is compounded in UCC-based add-ons because UCC generates a file called import_declare_test.py, which rebuilds sys.path wholesale when imported and does not add plain bin/ to the result. import_declare_test.py itself lives in bin/, so it cannot even be imported via a normal import statement until something first puts bin/ on sys.path — a chicken-and-egg problem.

When the import fails inside the persistent-connection handler process, Splunk's own error path in appserver.py dumps the entire sys.modules dictionary to stdout. Because stdout is also the framing channel for the persistent-connection wire protocol, this text dump corrupts the frame parser in splunkd and surfaces as a generic HTTP 500 with the message bad character (N) in reply size — giving no indication that the real cause is a missing sys.path entry. This combination of undocumented loader behaviour, a boilerplate helper that silently undoes manual mitigations, and an error path that obscures the real exception is what makes this class of time-consuming to diagnose.

Investigation

How persistconn_handler_file.py Is Loaded and What Is Available in sys.path

1.1 The loader: persistconn/appserver.py

When splunkd handles a request to a scripttype=persist endpoint it does not run the handler script as a normal Python program. Instead, splunkd has already spawned a long-lived Python subprocess that runs Splunk's own $SPLUNK_HOME/lib/python3.x/site-packages/splunk/persistconn/appserver.py as its main program. That subprocess sits in a blocking event loop, waiting for requests from splunkd over stdin/stdout.

appserver.py is part of Splunk Enterprise — it ships with the product and is not written by the add-on developer.

1.2 What happens when the first request arrives

On the first request, the event loop in appserver.py calls _load_file() with the absolute path to the handler script as configured in restmap.conf:

[script:<unique_name>]
match      = /<endpoint_path>
script     = persistconn_handler_file.py
scripttype = persist
handler    = persistconn_handler_file.HandlerClassName
python.version = python3

The relevant code (from the actual Splunk source at $SPLUNK_HOME/lib/python3.x/site-packages/splunk/persistconn/appserver.py) is:

# appserver.py lines 50-64 (actual Splunk source)
@staticmethod
def _load_file(filename):
    (dirname, fileonly) = os.path.split(filename)
    modname = os.path.splitext(fileonly)[0]
    munged_name = PersistentServerConnectionApplicationServer._munge_module_name(
        modname, dirname
    )
    try:
        # Import the module directly from the file.
        # See https://docs.python.org/3/library/importlib.html#importing-a-source-file-directly
        spec = importlib.util.spec_from_file_location(munged_name, filename)
        module = importlib.util.module_from_spec(spec)
        sys.modules[munged_name] = module
        spec.loader.exec_module(module)   # <-- persistconn_handler_file.py starts executing here
    except ImportError as e:
        for name, module in sys.modules.items():
            sys_modules_repr[name] = repr(module)
        sys_modules_str = json.dumps(sys_modules_repr)
        print("ImportError occurred, dumping sys.modules=%s" % (sys_modules_str))
        raise e

spec_from_file_location is a standard Python library function (available since Python 3.1, documented at importlib). Splunk uses it because it allows loading a module from a known absolute path without needing sys.path to locate the file first. It is not a custom Splunk mechanism.

1.3 What spec_from_file_location does and does not do

This is the fundamental difference from running a script normally:

1.4 sys.path at the moment persistconn_handler_file.py begins executing

The persistent Python subprocess inherits a sys.path built from Splunk's own Python environment only:

sys.path = [
    $SPLUNK_HOME/lib/python3.x/site-packages,   # Splunk's own packages
    $SPLUNK_HOME/lib/python3.x/lib-dynload,      # C extension stdlib
    ...standard Python stdlib paths...
]

Notably:

• Only Python's standard library and packages bundled by Splunk itself are available.
• An add-on's bin/ directory is not on sys.path.
• An add-on's lib/ directory (where vendored third-party packages live) is not on sys.path.
• If an add-on vendors a newer version of a package that Splunk also bundles, the older Splunk-bundled version would be imported instead unless sys.path is fixed. For example, one real-world case involved boto3 1.37.14 shipped with Splunk being resolved instead of boto3 1.42.97 that the add-on had vendored in its own lib/, and solnlib being completely unavailable because it is not bundled with Splunk at all.
• <app_name>/bin/ is not on sys.path.
• <app_name>/lib/ is not on sys.path.

At this moment __file__ is correctly set to the absolute path of persistconn_handler_file.py (because spec_from_file_location sets spec.origin and module_from_spec propagates it). __file__ and sys.path are independent of each other; one works, the other is missing what the add-on needs.

1.5 The error path that makes this hard to diagnose

If any import of a non-stdlib, non-Splunk-bundled module is attempted before sys.path is fixed, an ImportError is raised inside exec_module(). This bubbles up to the except ImportError block in _load_file(), which:

1. Serialises the entire sys.modules dictionary to JSON.
2. Calls print(...) — writing the dump to stdout.
3. stdout is the persistent-connection wire protocol channel. splunkd expects length-prefixed binary frames on this channel. A plain text dump corrupts the frame parser.
4. splunkd reports bad character (I) in reply size → HTTP 500.
5. No Python traceback or mention of ModuleNotFoundError reaches the caller.

2. The Problem with Loading import_declare_test.py

2.1 What import_declare_test.py is and what it does

import_declare_test.py is a UCC-generated boilerplate file that lives flat inside bin/. Its purpose is to prevent cross-TA sys.path pollution in a shared persistconn Python process where multiple add-ons may have registered handlers. When executed, it rebuilds sys.path from scratch:

# import_declare_test.py (UCC-generated source, <app_name> substituted at generation time)
ta_name = '<app_name>'
pattern = re.compile(r'[\\/]etc[\\/]apps[\\/][^\\/]+[\\/]bin[\\/]?$')# Keep a path only if it does NOT look like any app's bin/ — OR if it is our own
new_paths = [path for path in sys.path if not pattern.search(path) or ta_name in path]# Add the add-on's lib/ directory (vendored third-party packages)
new_paths.insert(0, os.path.join(dirname(dirname(__file__)), "lib"))# Add bin/<app_name>/ (a UCC namespace subfolder, NOT plain bin/)
new_paths.insert(0, os.path.sep.join([os.path.dirname(__file__), ta_name]))sys.path = new_paths  # wholesale replacement

What it inserts:

• <app_name>/lib/ — the vendored third-party packages (e.g. solnlib, boto3, etc.)
• <app_name>/bin/<app_name>/ — a UCC namespace subfolder (may or may not exist)

What it does not insert:

• Plain <app_name>/bin/ — flat sibling files sitting directly in bin/ are not covered by this.

2.2 The chicken-and-egg problem

import_declare_test.py lives in bin/. A normal import import_declare_test statement requires Python to find it by searching sys.path. At the moment persistconn_handler_file.py starts executing, bin/ is not on sys.path (see Section 2.4). Therefore:

import import_declare_test   →   ModuleNotFoundError: No module named 'import_declare_test'

import_declare_test is meant to fix sys.path, but sys.path must already be partly fixed before import_declare_test can be imported at all.

2.3 Why import_declare_test does not solve the bin/ problem even if imported

Even if import_declare_test is loaded successfully, plain bin/ is only conditionally retained in the rebuilt sys.path. The regex filter keeps a bin/ entry only if the string <app_name> happens to appear in that path — i.e. only if it was already there from a prior insertion. It never actively adds bin/. Flat sibling files in bin/ therefore remain unreachable via a normal import statement unless something explicitly adds bin/ to sys.path in addition.

Also note: the UCC namespace subfolder bin/<app_name>/ that import_declare_test does insert is often not actually created by the developer, making that insertion harmless but also useless for resolving any real modules.

Resolution

Resolution

The key insight is that spec_from_file_location — the exact same standard Python mechanism splunkd uses to load persistconn_handler_file.py — can be used inside persistconn_handler_file.py to load import_declare_test.py by its absolute path, bypassing sys.path entirely for that one load. Since no sys.path search is involved, the chicken-and-egg problem is avoided entirely. Once import_declare_test has run, it adds lib/ to sys.path, making the add-on's vendored third-party packages available. One additional sys.path.insert covers plain bin/ for flat siblings.

__file__ is the anchor for the whole approach: it is reliably set by spec_from_file_location when splunkd loaded persistconn_handler_file.py, so the absolute path to bin/ can always be derived from it without relying on sys.path.

1. The solution code

import importlib.util   # standard library — always available
import os               # standard library — always available
import sys              # standard library — always available# ── Step 1 ─────────────────────────────────────────────────────────────────
_bin = os.path.dirname(os.path.abspath(__file__))
# __file__ was set by spec_from_file_location when splunkd loaded this script.
# os.path.abspath resolves any symlinks or relative components.
# os.path.dirname strips "persistconn_handler_file.py", leaving the absolute
# path to bin/.
# Result example: $SPLUNK_HOME/etc/apps/<app_name>/bin# ── Step 2 ─────────────────────────────────────────────────────────────────
_idt_spec = importlib.util.spec_from_file_location(
    "import_declare_test",                          # module name (arbitrary string)
    os.path.join(_bin, "import_declare_test.py"),   # absolute path to the file
)
# spec_from_file_location creates a loader specification from a FILE PATH.
# It does not search sys.path — it goes directly to the file.
# This is the same function appserver.py uses to load persistconn_handler_file.py itself.# ── Step 3 ─────────────────────────────────────────────────────────────────
_idt_mod = importlib.util.module_from_spec(_idt_spec)
# Creates an empty module object from the spec.
# Sets _idt_mod.__file__ to the absolute path of import_declare_test.py.
# This matters: import_declare_test.py uses __file__ internally to compute
# the paths it inserts into sys.path (lib/ and bin/<app_name>/).# ── Step 4 ─────────────────────────────────────────────────────────────────
_idt_spec.loader.exec_module(_idt_mod)
# Runs the body of import_declare_test.py inside the module object.
# Side effects after this line:
#   - sys.path is rebuilt wholesale (other apps' bin/ entries stripped)
#   - <app_name>/lib/ is inserted at the front of sys.path
#   - <app_name>/bin/<app_name>/ is inserted (whether or not it exists)
# The add-on's vendored packages (e.g. solnlib, boto3 1.42.97) are now importable.# ── Step 5 ─────────────────────────────────────────────────────────────────
if _bin not in sys.path:
    sys.path.insert(0, _bin)
# import_declare_test only conditionally retains bin/ and never actively adds it.
# This one insert guarantees that flat sibling files in bin/ (e.g. sibling_module.py)
# are reachable via a normal import statement.# ── From here, all imports work normally ───────────────────────────────────
import boto3                                            # from lib/ — vendored version (e.g. 1.42.97)
from botocore.config import Config as BotoConfig        # from lib/
from botocore.exceptions import ClientError             # from lib/
from solnlib import conf_manager                        # from lib/ — not in Splunk's Python
from solnlib import log                                 # from lib/
from splunk.persistconn.application import (            # from Splunk's site-packages
    PersistentServerConnectionApplication,
)
import sibling_module                                   # from bin/ — flat siblin

2. Why this works where a plain import would not

At the start of persistconn_handler_file.py execution:  __file__  =  "$SPLUNK_HOME/etc/apps/<app_name>/bin/persistconn_handler_file.py"  ✓  VALID
  sys.path  =  [Splunk site-packages, stdlib, ...]                                  ✗  bin/ missing
                                                                                        lib/ missingStep 1 — compute _bin from __file__:
  Uses the one thing that IS reliably available (__file__) to derive bin/.
  No dependency on sys.path whatsoever.Steps 2–4 — load import_declare_test by path:
  spec_from_file_location goes directly to the file using the absolute path.
  Does not consult sys.path at all — chicken-and-egg is bypassed.
  import_declare_test.py executes and adds lib/ to sys.path.Step 5 — insert bin/ once:
  Covers flat siblings. One line.
  Runs after import_declare_test has already finished, so it cannot be undone.Result:
  sys.path  =  [bin/, bin/<app_name>/, lib/, Splunk site-packages, ...]
  All dependencies reachable.

3 Verification

The fix can be verified without restarting Splunk by simulating exactly what appserver.py does — loading the handler script via spec_from_file_location using Splunk's own Python binary:

$SPLUNK_HOME/bin/python3 -c "
import importlib.utilscript = '$SPLUNK_HOME/etc/apps/<app_name>/bin/persistconn_handler_file.py'
spec = importlib.util.spec_from_file_location('handler', script)
mod  = importlib.util.module_from_spec(spec)
spec.loader.exec_module(mod)print('Module loaded OK')
"

A successful load — no ImportError, no ModuleNotFoundError — confirms that the sys.path bootstrapping is working correctly. With the fix applied, the output will confirm that vendored package versions from lib/ are resolved (not the older Splunk-bundled ones) and flat sibling modules are reachable.

After restarting Splunk, confirm in splunkd.log that the following are absent:

• bad character (N) in reply size
• ImportError occurred, dumping sys.modules
• ModuleNotFoundError