Tasks

A task represents a single step in a batch pipeline. Depending on your data environment, a task could be a job running on a tool, part of a job, a script, or the arrival of files.

Tasks defined for a batch pipeline are expected to happen during a run. You can monitor the status of tasks from the Events page and run details. With rules, Observability can notify you when tasks take on a certain status.

Create tasks

Tasks are defined during the creation of a batch pipeline.

The system automatically generates all batch pipeline tasks shown in the UI based on relative timings during runs.

Tasks as events

An event is anything of interest that's happened in your data environment. Events that occur during a run can be associated with a specific task.

Tip

While all tasks are ingested as events, not all events are tasks. For example, a batch pipeline starting is an event, but may not be considered a task for your data operations.

task_key

The Event Ingestion API uses the properties task_key and task_name to identify tasks that happen during a run and their statuses.

For more information about events, see Events.

Task status

A task can have the status running, completed, failed, or completed with warnings. You can monitor the status of individual tasks from the run details Graph, Timeline, and Events tabs.

Any change in status received by the system updates the task status regardless of the previous status. Timings are updated as needed.

Tip

The status of instances, runs, and tasks are color-coordinated for better visualization. Colors are repeated in tables, Gantt charts, and graphs.