Skip to content

Payload Instances Overview

An instance represents a collection of the events that occur in the components included in a journey during a certain window, as defined by you. A payload instance is a unique type of instance that tracks the presence of data in the components of a journey. Since payload instances only concern events related to that payload, the instance logic differs from other instances.

Observability highlights payload instances on the Instances page under the Start type / Payload column and in the instance details header.

The following document provides an overview of payload instance behavior and logic.

Example

For a high-level overview of instance behaviors based on an example journey, see Complex Instance Example.

Payload key-value overview

  • A payload identifier is sent from your components as a key-value pair in an event, represented by payload_keys.
  • All components can send an event with a payload and for any event type.
  • Payloads are assumed to be unique pieces of data that are only present in a journey once; only one payload instance can exist for a journey per payload key. For example, a journey can only have one payload instance for "molecule:1234".

Example event

The following shows a Message Log event sent from a database component that helps you monitor the progress of work.

{
  "log_level": "INFO",
  "message": "Details for today's payload instance",
  "dataset_key": "db_01A",
  "component_tool": "postgresql",
  "dataset_name": "Database_01A",
  "event_timestamp": "<dateTime>",
  "payload_keys": [
    "order:1234",
    "order:5678"
    ]
}

Payload instance behaviors

While only one instance can be active at a time, payload instances are tracked as separate instances.

  • There is no limit to the number of payload instances that can be active for a journey at one time.
  • The system creates a separate payload instance per payload value.
  • A payload instance can be active regardless if an instance is active (or not) for the journey.

Start a payload instance

A payload instance starts when payload instances are tracked in the journey Settings and a new payload value is included in any of the following events.

  • A Metric Log, Message Log, or Test Outcomes event received for a server, streaming pipeline, or dataset component.
  • A Dataset Operation event received for a dataset component.
  • The first event (Run Status, Metric Log, Message Log, or Test Outcomes) received for a batch pipeline run.

See Run progress behavior below for more information on batch pipeline runs.

Start conditions apply by default and do not need to be configured.

End a payload instance

An end condition must be configured for payload instances. Payload instances can end:

  • On a schedule.
  • When a specified batch pipeline component receives a run end event (a Run Status event with status of "completed," "completed with warnings," or "failed") for a run associated with the payload.

Run progress behavior

Batch pipeline runs are associated with a payload instance as of the first event received for a run.

  • The system evaluates this first run event for the presence of payload keys.
  • Subsequent events received during a run are associated with every payload instance created for the first event; regardless if every payload key is included in the subsequent events.
  • If a new payload value is sent during a run, no new payload instance is created.