Skip to content

Define Instances

An instance is the runtime execution of a journey, which is a collection of components in your data estate, as specified by you. An instance shows what events occur in those components during a window that you define with instance conditions.

The following document provides an overview of instance behavior and logic.

Example

For a high-level overview of instance behaviors based on an example journey, see Complex Instance Example.

Instance behaviors

Only one instance can be active for a journey at any time.

Start a new instance

You can select from three types of start conditions.

  • Batch: a new instance starts each time a run begins for a certain batch pipeline in a journey.
  • Schedule: a new instance starts each time a schedule is met.
  • Dataset: a new instance starts each time a certain dataset receives a Dataset read event.

  • Default: if there is no instance currently active for a journey, Observability starts a new instance when it receives the first event for any component included in that journey.

    • If the component is included in several journeys, an instance starts for each journey that does not have an active instance.
    • You can specifically select the default start condition. That is, you can choose not to have a batch or schedule start condition.
    • The default start condition always applies in addition to a batch or schedule condition.

    Example

    Use case: you define instances to start at 8 AM on Mondays for a given journey. Events are also received for the journey at 2 PM on Friday. Observability records both instances—so you don't miss the realities of events in your data estate.

Once an instance is active, the system associates subsequent events received for the components in the journey with that instance.

End an active instance

There are two types of end conditions that apply by default:

  • Observability ends an active instance when each batch pipeline component in the journey has received a Run Status event with a run status of "completed," "completed with warnings," or "failed."
    • This default behavior only applies if no start or end batch or schedule condition is configured.
  • Observability always ends an active instance when a new instance starts for the same journey, per any start condition.

Default end conditions do not need to be configured to apply.

You can then choose to set one or more of the following end conditions:

  • When a run ends (i.e. Run Status event with a run status of "completed," "completed with warnings," or "failed") for a certain batch pipeline in a journey.
  • When a certain dataset receives a Dataset write event.
  • Each time a schedule is met.

Restart an instance

Once ended, instances cannot be restarted.

Note

While runs for a batch pipeline component can change status from Completed back to Running, this does not restart an instance that has ended.

Component progress

  • All events for a streaming pipeline, dataset, and server are always associated with the instance that is active when the events are received.
  • All events during a batch pipeline run are associated with the instance that was active when the run started, even if some of the events occur after that instance ends.

  • The system evaluates the start and end events of batch pipeline runs against the journey's sequence. During an instance, if

    • A run does not occur for each batch pipeline component in the journey

    Or,

    • Batch pipeline components run in a different order than set in the journey's sequence

    Or,

    • A batch pipeline is still running while a subsequent batch pipeline in the journey starts a run

    The system generates an alert.