Complex Instance Example

This document provides an example of how Observability handles a journey with multiple components and settings.

Use case: a journey is built with several component types. Instance conditions are set and payload instances are being tracked.

Example instance

The following journey has been built with the following components and settings:

A (batch pipeline) > B (dataset)

C (batch pipeline) > D (batch pipeline) > E (server)

• An Instance Condition is set to start a new instance when runs start for A.
• An Instance Condition is set to end an active instance daily at 8 PM EST.
• Payload instances are being tracked and have an end condition When runs end for D.

Warning

Since multiple settings can be created for a single journey, overlapping conditions and expectations may result in short instances and frequent alerts.

Visual example in Observability

Expected behaviors

Starting instances

A new instance will start for this journey:

• If Observability receives any event for any of the components, and no instance is currently active.
• When Observability receives a Run Status event with status set to "running" for A.

A new payload instance will start for this journey:

• If "payload_keys": [ "payload-Z" ] and/or "payload_keys": [ "payload-Y" ] is received as part of any event for the dataset component B or server component E.
• If a run starts for the batch pipeline components A, C, or D, and the first run event includes either payload key.

Once a start condition is met, the instances are active. Observability then associates subsequent events with the appropriate instances.

Ending instances

An instance will end:

• At 8 pm EST daily.
• If a new instance starts because of the instance condition set for A.

The payload instances will end:

• When D receives a Run Status event with status of "completed," "completed with warnings," or "failed," and the run was associated with a specific payload instance from the run's first event.

Example events and instances

The following events occur:

1. 11 AM a Dataset Operation read event for B with "payload_keys": [ "payload-S" ].
2. 12 PM a Metric Log event for E.
3. 1 PM a Run Status event with status is "running" for A.
4. 2 PM a Dataset Operation write event for B.
5. 3 PM a Message Log event for C that includes a run_key and pipeline_key and "payload_keys": [ "payload-Y" ].
6. 4 PM a Run Status event with status is "completed" for C.
7. 5 PM a Run Status event with status is "running" for D with "payload_keys": [ "payload-S" ].
8. 6 PM a Run Status event with status is "failed" for D.
9. 7 PM a Metric Log event for E.

The system would create four instances from these events, assign the relevant events to each instance, and generate alerts based on the expectations and sequence.