SOAR - Best Practices when auditing playbooks

Applicability

Description

• Using decided instead of actiond. Calls made from rest calls from a playbook or custom function. This is not recommended because it does not utilize the concurrent action limits in actiond. All rest calls should be made from a SOAR app. This is also the preferred method because apps can encrypt API credentials. 

• Concurrent action limits need to be increased. When increasing the action limit, consider the following factors:

  • Do I have the server resources available to increase concurrency?

  • Does the destination server have the ability to accept concurrent requests?

• Python runners need to be increased. The same python runner is reserved for all child playbooks that are executed by a parent playbook. If a SOAR environment creates 5 containers a minute and it takes 1 minute to run the playbook automation, then 5 python runners would be required to keep up with the demand. 

• Querying the SOAR PostgreSQL database instead of running Splunk queries against the phantom_* indexes. Splunk is designed to process large volumes of data. Offload queries to Splunk when possible. The following data can be found in phantom indexes:

  • action run

  • app

  • app run

  • artifact

  • asset

  • container

  • container note

  • custom function

  • decided list

  • note

  • playbook

  • workflow notes.

• Not having error handling in place. After every action block, there should be a decision block to check the HTTP code of the API request. If the status is not 200, tag the container with “error” and write a comment to the container on why it failed. The phantom indexes can be monitored in Splunk to see what actions failed on which containers and why. 

• Creating a large playbook with multiple branches instead of leveraging children playbooks and input playbooks.

• Not saving results to the container. Always save the results from an action to a container as an artifact if you will need it for downstream playbooks or reporting in Splunk.

• Not having a plan in place when restarting the SOAR environment:

  • Have ingest sources been paused?

  • How will I forward and run events that were not imported during restart?

  • Have all active actions completed before restarting?

• Using the classic VPE instead of the new VPE in latest SOAR releases. New VPE provides metrics on action blocks execution time and can identify long running actions.

• Monitor event creation in phantom_* indexes using SPL and alert on any delays or ingest drops using frequently triggered alerts in Splunk. SPL can use stats or tstats commands.

Resolution

Playbook Usage

• For production instances, only have playbooks in non-community repos that are actually being run, i.e. no development or duplicate playbooks. This will help avoid confusion on what playbooks are operational and in-use.
• Minimize active playbooks to only necessary parent playbooks

Playbook Design

• Use I2A2 or another design framework for playbook development. Include design notes in the playbook description for future maintenance and administration.
• Implement small, modular playbooks that perform a single task, and reuse the playbooks across multiple response plans.
• Use a playbook naming scheme to better organize playbooks. For example, [Phase] Playbook Task, e.g. [Ingestion] Filter Artifacts with Allow List, or [Investigation] Investigate IPs
• Classify playbooks according to the procedure phase that the playbook is used for, i.e. ingestion, investigation, containment, recovery, documentation/reporting, utility. Include a playbook phase in the name for better organization and administration.

Playbook Development

• Convert any playbooks & custom functions written in Python 2.7 to the latest supported SOAR python version, i.e. 3.9.
• Include error handling procedures in playbooks, including checking for bad or missing input and checking for errors in action results.
• Never hardcode credentials into code, it is recommended to make rest calls via app code so that credentials are properly protected and take advantage of actiond.
• If playbook development is outsourced, ensure that there is a procedure to transition playbooks to production, perform knowledge transfer and that playbooks are being actively maintained by local staff.
• Have a dedicated team to maintain playbooks that has been trained on playbook implementation and the playbook flows.
• Be aware of critical apps and actions being used in operational playbooks and stay up-to-date on related content.
• Minimize the use of custom code (SOAR apps, custom functions and code blocks) to ensure that code is formally supported and tested by Splunk. If custom code is necessary, ensure that coding best practices are being followed and that the appropriate test procedures are in place and being enforced.
• Limit the amount of custom code that is not in a custom app or custom function. Custom code in code block is run by python runner and not actiond and therefore can block a runner or cause other performance issues. Additionally, there could be issues with custom code being upgraded when SOAR is upgraded.

Playbook CICD

• Follow the Splunk SOAR CICD Best Practices documentation. Most importantly:

  • use separate development and production environments

  • use separate git branches for each environment

  • only develop in the development environment

  • promote playbooks between environments using git

  • only the development environment has read-write access to playbooks

  • all other environments have read-only access to playbooks

  • maintain repo name between environments to minimize playbook reference changes when moving between environments

  • have a process for mapping assets in different environments

• Use separate development and production environments, only develop in the development environment, production environment should have read-only access to playbooks.
• Promote playbooks using git in the back end and moving tested and accepted playbooks to the production branch. Do NOT move any playbooks that have not been tested and accepted from the development branch to the production branch.
• Use a documented testing procedure to test playbooks. Ensure that all playbook paths or Artifacts/Outcomes (I2A2) of the playbook are tested (i.e. no data, bad data, any additional outcomes),
• Use an external git repo that is not on the SOAR local file system.

Playbook Performance

• Create a baseline of playbook run times and monitor for any significant changes.
• Cancel any playbooks that are hanging and investigate playbook runs that are taking significantly longer than expected.

App and Assets

• Only install apps that are needed to avoid any potential unnecessary vulnerabilities and reduce administration costs.
• Maintain the most up-to-date version of the app.
• Minimize the number of custom apps in the environment
• Regularly test connectivity to assets, alert for any issues and make changes/fixes as necessary.
• Baseline action run times to identify anomalies.
• Global action concurrency limit should be tuned according to system resource utilization. Monitoring should be established on system usage to determine if this value can be increased to utilize more system resources.
• Using a third party password vault and using the same key in different environments so no changes needed for asset configuration.
• Hashicorp, CyberArk and Thycotic are supported in SOAR.
• If using a vault that is supported by SOAR, it is recommended to configure it in the Password Vault settings of SOAR.
• Adding any new feature request to ideas.splunk.com
• An automation broker is required to connect to resources that are on an unreachable network from the SOAR instance.
• It is recommended to only enable granular asset control if explicit access for each asset action has already been granted. Otherwise, the policy is default deny, and could render an asset inoperable for automated on manual action runs.
• If granular asset access control is enabled, each asset should have explicit access defined for at least one of the actions. If not, the asset is essentially inoperable as the policy is default deny, no automated on manual run of the app actions would be permitted.

Helpful Splunk Queries

• Average duration by action

index=phantom_action_run
| rename playbook as playbook_id
| join type=left left=main right=playbook where main.playbook_id = playbook.id
    [search index=phantom_playbook earliest=1 latest=now ]
| table _time main.playbook_run main.container main.action playbook.name main.status main.playbook_id main.name main.id
| rename playbook.name as playbook_name
| rename main.* as * playbook.* as *
| transaction playbook_run id action 
| fields playbook_id playbook_name playbook_run, id, container action name status duration eventcount 
| search duration > 0
| sort - duration
| rename id as action_run_id playbook_run as playbook_run_id

• Average duration by playbook

index=phantom_action_run status IN (success,failed)
| rename playbook as playbook_id
| stats latest(close_time) AS close_time, latest(create_time) AS create_time BY playbook_id, playbook_run, status
| eval close_time = strptime(close_time, "%Y-%m-%dT%H:%M:%S")
| eval create_time = strptime(create_time, "%Y-%m-%dT%H:%M:%S")
| eval pb_running_for = floor(close_time - create_time)
| join [search index=phantom_playbook earliest=1 latest=now
| rename id as playbook_id | table playbook_id name]
| stats avg(pb_running_for) as avg_playbook_run_duration_secs by name status

• Check for actions that are stuck in run state

index=phantom_action_run earliest=-2d latest=-1h
| rename playbook as playbook_id
| rename id as action_run_id
| rename container as container_id
| stats list(status) as status by action_run_id playbook_id container_id
| join [search index=phantom_playbook earliest=1 latest=now
| rename id as playbook_id | table playbook_id name]
| table action_run_id playbook_id name status container_id
| search (action_status!=success AND action_status!=failed)

• Container creation by label over time

index=phantom_container earliest=-3d latest=-1h
| eval container_update_time="" 
| dedup id 
| timechart count(id) by container_label span=1d

• Check for apps that are waiting for actiond

index=phantom_app_run "waiting for an ongoing action"
| stats count by app_name

• Alert if ingest is current in phantom_ indexes 

| tstats count WHERE index=phantom_container

• Monitor delay between indextime and container create_time (mapped to _time)

index=phantom_container 
| eval indextime=strftime(_indextime,"%Y-%m-%d %H:%M:%S") 
| eval gap = (_time - _indextime) 
| table id _time indextime gap 
| sort _time