Skip to content

Event Processing

Events represent moments of interest that occur in the components of your data estate. Observability groups these events into five types—Run Status, Message Log, Test Outcomes, Metric Log, and Dataset Operation events—each with its own particular set of event keys and metadata to provide context.

You can see events in the UI from the:

  • Events page, which shows a list of all the events received for a given project.
  • Instance details show you the events associated with a particular instance.
  • Run details list the events associated with a specific run.

Order and timing of events

When an event is sent from an external tool, the event metadata can include an event_timestamp value. If no value is sent with the event data, the system applies its current time (designated by the timestamp value) when it receives the event.

The UI sorts and displays the events by event_timestamp and the system processes events one by one in the order that they are received.

When the system processes events for a run or task, the event_timestamp value is considered and the status and relevant timings are updated to the latest values.

Example

The system might receive events from external tools in an order that is not logical. For example, if the following three events are received in this order:

Event timestamp value event_timestamp value
Run Status event "status": "completed" 2023-01-01T10:21:00 2023-01-01T10:20:00
Run Status event "status": "running" 2023-01-01T10:22:00 20230-01-01T10:15:00
Run Status event "status": "failed" 2023-01-01T10:23:00 2023-01-01T10:18:00

The UI sorts and displays the events using the event_timestamp value. The system processes the events in the order of their timestamp value, and updates the status to maintain consistency with timestamp =, >, or < event_timestamp.

Warning

At this time, the system does not reevaluate run or instance statuses related to component expectations.

Investigate event order

Use the event metadata to better understand the order of events and the status of your instances or runs.

  1. Navigate to the event metadata for an instance or run.

    • For an instance: Select Instances from the menu. Click an instance ID.
    • For a run: Select Events from the menu, then select the Batch Runs tab. Click a Run Key, then select the Events tab.

  2. Click { } View event data for a specific event.

  3. Compare the values for timestamp and event_timestamp (found by expanding the raw_data).

Note

If you need more help, contact your DataKitchen representative.