Streamlining Splunk On-Call Alert Response via WhatsApp

Applicability

Description

In this scenario, an initial Splunk On‑Call alert notification is received via WhatsApp. The paging policy is configured to send push notifications every minute following the initial WhatsApp notification.

Although the WhatsApp message initially provides actionable options such as Acknowledge or Resolve, continued paging through additional contact methods over time can cause the WhatsApp acknowledgment action for the same alert to become invalid after approximately two hours.

Investigation

The Splunk On‑Call alert was initially delivered via WhatsApp as the first contact method defined in the paging policy. The WhatsApp notification contained actionable options, including Acknowledge and Resolve, which were functional when the alert was first received.

After the initial WhatsApp notification, the paging policy continued to trigger push notifications for the same incident at one‑minute intervals, in accordance with the configured escalation and paging settings. These repeated notifications were delivered consistently over an extended duration.

After approximately two hours of continuous paging through push notifications, the acknowledgment and resolve actions available in the original WhatsApp message were observed to be invalid. Attempts to acknowledge or resolve the incident using the WhatsApp notification were unsuccessful, indicating that the WhatsApp action links were no longer active for that incident.

This behavior confirms that prolonged paging through additional contact methods can invalidate WhatsApp acknowledgment and resolution actions for the same Splunk On‑Call alert.

Resolution

The behavior was confirmed after configuring the paging policy to send an initial Splunk On‑Call alert via WhatsApp, followed by push notifications at one‑minute intervals. After approximately two hours of continuous paging through push notifications, the acknowledgment and resolve actions available in the original WhatsApp notification became invalid.

Once the WhatsApp acknowledgment was no longer valid, the incident could not be acknowledged or resolved through WhatsApp. The incident remained active and actionable through other supported methods, such as push notifications, the Splunk On‑Call web interface, or the mobile application.

To avoid this limitation, paging policies can be reviewed and adjusted to reduce prolonged or continuous paging through additional contact methods when WhatsApp acknowledgment is required. Alternatively, incidents can be acknowledged or resolved using other available contact methods once the WhatsApp action becomes invalid.