Skip to content

Events

An event is any moment of interest that occurs in your data estate. For example, an event could be a batch pipeline run starting, a streaming pipeline receiving events with a certain status, a test failing, or a file arriving.

Observability ingests these events, associates them with the correct project and component, uses them to track progress, compares them against expectations, displays their results, and generates indicators and notifications as needed.

Warning

Observability integration: To receive events, your tools and assets must be integrated with Observability.

Event details

Events are described by an event type, a set of keys conditional to the event type, and other optional metadata. From this data, events are associated with:

  • A project, identified by the service account key that is sent with the event.
  • A component, identified by a key that is unique within the project.
  • A run, and optionally a task, if the event is for a batch pipeline component.

When an event is received by the Event Ingestion API, the system attempts to associate the new event with an existing component, run, and/or task. If the component, run, or task for the event does not already exist in the project, the system automatically creates a new one.

Event types

Event types describe common categories of events that can happen in data operations. Observability categorizes events into five types, each with a unique set of properties that provide context.

See Event Types for more information.

View events

To view a list of events received by Observability, select Events from the menu.

See View Events and Runs for more details.

Understand event identifiers

Every event includes required and optional properties for context and metadata.

Key

Keys are defined by users, sent by customer tools, and used to identify and associate events.

Every Event Ingestion API endpoint requires a service account key and <component>_key. Other keys may be required to describe a specific event. Together, these properties allow the system to associate events with a project and a component.

Example

Batch pipeline example: Observability creates a new batch pipeline component when it receives any event with a unique pipeline_key. The system then creates a new run for this pipeline each time it receives an event with a combination of that pipeline_key and a new run_key. Future events are associated with this run if they use the same pipeline_key and run_key. For any given batch pipeline, a run's key is always unique, allowing the system to associate events as they happen over time.

Name

While keys are a unique identifier used by the system, the name property provides a human-readable identifier.

The UI displays the <component>_name, run_name, and/or task_name when the property is sent with an event. If no name is specified for a component or task, the value for the key becomes the name by default; the run name remains null if no name is specified.

The system attributes a name the first time the value is sent. If a new name is sent in future events, the system updates the name.