Splunk Observability integration with Jira Comment Not Added When Alert Returns to Normal

Applicability

Description

In Splunk Observability Cloud, integrating with Jira allows for automatic creation of Jira issues when detector alerts are triggered. The expected behavior, as described in documentation, is that when an alert clears (“Back to Normal”), a comment should be added to the corresponding Jira issue to indicate resolution. However, it has been observed that only the initial alert triggers the Jira issue creation, and no comment is added when the alert returns to normal. This results in additional manual monitoring by support teams, as the status of incidents is not updated automatically in Jira.

Example scenario:

• Alert triggered in Splunk Observability → Jira issue created as expected.
• Alert resolves (“Back to Normal”) → No comment appears in the Jira issue.

No error messages are observed, and the issue is not limited to a particular detector.

Investigation

Follow these steps to determine if your environment is experiencing the issue:

1. Verify Integration Configuration

   • Navigate to Settings > Notification Services in Splunk Observability Cloud.
   • Confirm that the Jira integration is set up using the standard, supported method (Jira Cloud, with API token authentication).

2. Test Alert Workflow

   • Trigger a test alert from any detector.
   • Confirm that a Jira issue is created.

3. Test Alert Resolution

   • Resolve the alert (bring the monitored signal back to normal).
   • Check the linked Jira issue for a new comment indicating the alert has cleared.

4. Review Notification Settings

   • In the detector, ensure the Jira integration is added as an alert recipient.
   • Ensure other notification channels (e.g., email, Teams) receive both alert and resolution messages.

5. Check Documentation

   • Review the official documentation for the integration: Add a Jira integration as a detector alert recipient in Splunk Observability Cloud 

     The documentation states:
     “Splunk Observability Cloud sends an alert notification that creates a Jira issue whenever the detector rule condition is met. It also adds a comment to that issue when the alert condition clears.”

6. Verify in Jira

   • Open the created Jira issue.
   • In the issue’s Activity section, set the filter to All (not just Comments) to ensure comments are visible.
   • If no comment is present after the alert clears, this issue is present in the environment.

Resolution

This is a known-issue and it is documented as an internal ticket 

1. Monitor Email or Teams Notifications

   • Other notification channels (email, Teams) will indicate when an alert returns to normal. Use these as interim resolution notifications.

2. Manual Updates

   • Until a product fix is available, consider manually adding a comment to the Jira issue when the alert resolves, to inform stakeholders of the current incident status.

3. Tracking the Issue

   • To track progress on the product fix, open a support case referencing ALEN-2022, or contact your Splunk account team for updates.

4. Configuration Verification

   • No configuration changes or detector customizations will resolve this issue at this time. Ensure that your integration is configured as per documentation, and that the expected behavior is not observed before concluding this is the same product limitation.

Related Links

• Splunk Observability Cloud: Send Alerts to Jira (Official Documentation)
• Splunk Support Contact