Understanding the Delay Throttle Parameter in Message Trace Input

Applicability

Description

When configuring the message trace input to query data from an API, a critical aspect to consider is how the "delay throttle" parameter works. This parameter helps ensure that the system does not miss any late-arriving events by delaying the subsequent data query.

Investigation

The message trace input collects data from the API for a specific time range based on timestamps. Here's a breakdown of how the process works:

1. Data Request by Time Range:
   Each time the input runs, it requests data from the API for a particular time range. This range is defined by two points: the start and end timestamps.

2. Start Date Determination:
   The start date of the next query range is set based on the checkpoint from the previous data collection run. Essentially, it continues where it last stopped.

3. End Date Determination (Query Window):
   The end date is calculated by adding the query window size to the start date. For example, if the query window size is 60 minutes, the range would be the next 60 minutes of data.

   Example:

   • Start Date: 1:00 PM
   • Query Window: 60 minutes
   • Delay Throttle: 60 minutes

   Query Process:

   • At 2:00 PM, the system will query the data for the period from 1:00 PM to 2:00 PM. This is the first input run.
   • Since a delay throttle of 60 minutes is applied, the system will wait an additional hour before moving to the next query.
   • At 3:00 PM, it will then query the data for the period from 2:00 PM to 3:00 PM.

4. Purpose of the Delay Throttle:
   The delay throttle allows the system to handle late-arriving events by waiting before querying the next time range. Without it, the system might miss out on delayed data, which could arrive as late as 24 hours.

5. Default Setting:
   By default, the delay throttle is set to 24 hours to account for the maximum possible delay. However, this parameter can be configured according to the requirements. In this example, the current configuration is set to 60 minutes.

6. API Behavior:
   Events in the MSFT API can arrive out of order, and some data may take as long as 24 hours to appear. The throttle ensures that the system accounts for these delays.

Resolution

To ensure no data is missed due to delayed events, the delay throttle parameter should be configured based on data flow and requirements.

The default value of the delay throttle is set to 1440 minutes(24 hours) because, according to Microsoft documentation, events may be delayed by up to 24 hours before they appear in a report.

Related Links

MessageTrace report