Soar - spawn.log stopped working

Applicability

Description

When telemetry logging was introduced for spawn in 6.0.0, it was added incorrectly as a dependency for spawn log. Without the spawn_telemetry_pipe present in the log directory, spawn log could stop working. 

Investigation

It could affect in the following scenarios:

1. When before upgrade Soar version has no telemetry logging for spawn, the system has no spawn_telemetry_pipe file in the phantom log directory. If telemetry is set off after the upgrade, spawn_telemetry_pipe file is not get created by phantom_watchdogd. The spawn.log would not have any new data written to it.

2. Fresh install 6.0.0 and beyond, if the telemetry is off after installation and phantom_watchdogd doesn’t get to create the file spawn_telemetry_pipe, spawn log would stop working. 

Resolution

There is a workaround to resolve the issue on impacted versions:

1.  Turn on the telemetry in the product setting in web page.

2. Restart phantom_watchdogd from UI or from command line:

/<PHANTOM_HOME>/bin/phsvc restart phantom_watchdogd

3. Check the file was created. You can also check if spawn.log is working:

/<PHANTOM_HOME>/var/log/phantom/spawn_telemetry_pipe 

4. Then it is safe to turn off the telemetry if preferred, spawn_telemetry_pipe should stay and that’s what is needed for spawn to create logger successfully. 

5. It was observed that sometimes the above workaround may not work and the spawn_telemetry_pipe file is not being created. In web page, refreshing the “Data Sharing in Splunk SOAR” page, the telemetry settings turn off. 

Use the following steps:

  - Turn on telemetry again, refresh the page several times, ensure the telemetry setting is on.
  - Restart watchdogd.
  - Observe that “spawn_telemetry_pipe” file is created. After it, the spawn.log will populate new data again.

6. Alternatively, if the above steps fail to work, or if the telemetry page does not load, run the following steps:

  - Move the files below to another directory, such as your home directory:

mv $PHANTOM_HOME/var/log/phantom/spawn_*pipe ~/

  - Restart SOAR services:

$PHANTOM_HOME/bin/stop_phantom.sh
$PHANTOM_HOME/bin/start_phantom.sh