Event Endpoint Details¶

Event types describe common categories of events that can happen in data operations. For technical details on each of the event types, go to the Event Ingestion API documentation.

The system currently supports the following event types:

Run Status event (applies to batch pipeline components only).
Dataset Operation event (applies to dataset components only).
Metric Log event (valid for all component types).
Message Log event (valid for all component types).
Test Outcomes event (valid for all component types).

Component considerations¶

As noted above, three of the event types apply to all components, while two are component-specific.

When you send event data from your data estate to Observability, the first step is to decide which component publishes the event. This relationship is represented in API calls by the <component>_key value. Only one component key and value can be provided per event.

Note, the system creates a new component each time it receives a new <component>_key and associates future events with that component by this key. Users are expected to create and manage keys.

API key¶

Obtain an API key prior to sending events. An API key is required for API authentication, so the system can associate incoming events with the correct project, and for the agent heartbeat signal. You can create a new API key from the API Keys page or use an existing key.

For API calls, enter the API key as the value for the ServiceAccountAuthenticationKey header.

Dataset Operation event¶

Reports that a read or write event has occurred for the specified dataset.

Key	Required?	Details
operation	Required	The read or write operation performed.
dataset_key	Required	A key to identify the target dataset component. Only one component key can be provided per event. Keys are created and managed by users.
dataset_name	Optional	A human-readable display name for the target dataset.
component_tool	Optional	The name of the external tool represented by the dataset component.
path	Optional	Path within the dataset where the operation took place.
external_url	Optional	A link to source information.
metadata	Optional	Additional key-value information for the event. Provided by the user as needed.
event_timestamp	Optional	An ISO8601 timestamp that describes when the event occurred. If no timezone is specified, UTC is assumed. If unset, the Event Ingestion API applies its current time as the value.
payload_keys	Only required when tracking a payload.	An array of payload values present in a payload instance. Payloads must be tracked in the journey Settings.
pipeline_key	Not required for a Dataset Operation event.	Batch pipelines are not compatible with Dataset Operation events.
pipeline_name	Not required for a Dataset Operation event.	Batch pipelines are not compatible with Dataset Operation events.
run_key	Not required for a Dataset Operation event.	Batch pipelines are not compatible with Dataset Operation events.
run_name	Not required for a Dataset Operation event.	Batch pipelines are not compatible with Dataset Operation events.
task_key	Not required for a Dataset Operation event.	Batch pipelines are not compatible with Dataset Operation events.
task_name	Not required for a Dataset Operation event.	Batch pipelines are not compatible with Dataset Operation events.
stream_key	Not required for a Dataset Operation event.	Streaming pipelines are not compatible with Dataset Operations events.
stream_name	Not required for a Dataset Operation event.	Streaming pipelines are not compatible with Dataset Operation events.
server_key	Not required for a Dataset Operation event.	Servers are not compatible with Dataset Operation events.
server_name	Not required for a Dataset Operation event.	Servers are not compatible with Dataset Operation events.

Message Log event¶

Logs a string message related to a component or related to a specific batch pipeline task. Post a MessageLog event to capture error, warning, or debugging messages from external tools and scripts.

Key	Required?	Details
log_level	Required	The severity level of the message.
message	Required	The body of the message to log.
dataset_key	Optional. Only required if the message occurs in a dataset.	A key to identify the dataset. Only one component key can be provided per event. Keys are created and managed by users. When included, can send a `dataset_name` value.
dataset_name	Optional	A human-readable display name for the target dataset.
stream_key	Optional. Only required if the message occurs in a streaming pipeline.	A key to identify the streaming pipeline. Only one component key can be provided per event. Keys are created and managed by users. When included, can send a `stream_name` value.
stream_name	Optional	A human-readable display name for the target streaming pipeline.
server_key	Optional. Only required if the message occurs in a server.	A key to identify the server. Only one component key can be provided per event. Keys are created and managed by users. When included, can send a `server_name` value.
server_name	Optional	A human-readable display name for the target server.
pipeline_key	Optional. Only required if the message occurs in a batch pipeline.	A key to identify the batch pipeline. Only one component key can be provided per event. Keys are created and managed by users. When included, a `run_key` is required and a `task_key` optional. Can send a `pipeline_name` value.
run_key	Optional. Only required if the target component is a batch pipeline.	A key to identify the batch pipeline's run. When included, a `pipeline_key` value is required and a `task_key` optional. Can send a `run_name` value.
task_key	Optional. Only required if the target component is a batch pipeline and the event applies to a task during a run.	A key to identify the task. When included, `run_key` and `pipeline_key` values are required. Can send a `task_name` value.
pipeline_name	Optional	A human-readable display name for the target batch pipeline.
run_name	Optional	A human-readable display name for the target run.
task_name	Optional	A human-readable display name for the target task.
component_tool	Optional	The name of the external tool represented by the component.
external_url	Optional	A link to source information.
metadata	Optional	Additional key-value information for the event. Provided by the user as needed.
event_timestamp	Optional	An ISO8601 timestamp that describes when the event occurred. If no timezone is specified, UTC is assumed. If unset, the Event Ingestion API applies its current time as the value.
payload_keys	Only required when tracking a payload.	An array of payload values present in a payload instance. Payloads must be tracked in the journey Settings.

Metric Log event¶

Logs the value of a user-defined datum of interest, such as a row count. Post a MetricLog event to track the value of a metric through a run or to compare the value of the metric across multiple instances.

Key	Required?	Details
metric_value	Required	The data value to be logged. Decimal numerals only; NaN/INF values not supported.
metric_key	Required	A string to identify the metric_value data source. Used to group like metrics. This value is created and managed by the user.
dataset_key	Optional. Only required if the metric occurs in a dataset.	A key to identify the dataset. Only one component key can be provided per event. Keys are created and managed by users. When included, can send a `dataset_name` value.
dataset_name	Optional	A human-readable display name for the target dataset.
stream_key	Optional. Only required if the metric occurs in a streaming pipeline.	A key to identify the streaming pipeline. Only one component key can be provided per event. Keys are created and managed by users. When included, can send a `stream_name` value.
stream_name	Optional	A human-readable display name for the target streaming pipeline.
server_key	Optional. Only required if the metric occurs in a server.	A key to identify the server. Only one component key can be provided per event. Keys are created and managed by users. When included, can send a `server_name` value.
server_name	Optional	A human-readable display name for the target server.
pipeline_key	Optional. Only required if the metric occurs in a batch pipeline.	A key to identify the batch pipeline. Only one component key can be provided per event. Keys are created and managed by users. When included, a `run_key` is required and a `task_key` optional. Can send a `pipeline_name` value.
run_key	Optional. Only required if the target component is a batch pipeline.	A key to identify the batch pipeline's run. When included, a `pipeline_key` value is required and a `task_key` optional. Can send a `run_name` value.
task_key	Optional. Only required if the target component is a batch pipeline and the event applies to a task during a run.	A key to identify the task. When included, `run_key` and `pipeline_key` values are required. Can send a `task_name` value.
pipeline_name	Optional	A human-readable display name for the target batch pipeline.
run_name	Optional	A human-readable display name for the target run.
task_name	Optional	A human-readable display name for the target task.
component_tool	Optional	The name of the external tool represented by the component.
external_url	Optional	A link to source information.
metadata	Optional	Additional key-value information for the event. Provided by the user as needed.
event_timestamp	Optional	An ISO8601 timestamp that describes when the event occurred. If no timezone is specified, UTC is assumed. If unset, the Event Ingestion API applies its current time as the value.
payload_keys	Only required when tracking a payload.	An array of payload values present in a payload instance. Payloads must be tracked in the journey Settings.

Run Status event¶

Changes the status of a specified batch pipeline run or of a specified task in the run. Post a RunStatus event to alert the system that a run has started and ended with a certain status or that a task has started and ended with a certain status.

Key	Required?	Details
status	Required	The status to be applied. Can set the status for both runs and tasks depending if a task_key is present in the event. A run starts when no `task_key` is present and `status` is “RUNNING” and ends when no `task_key` is present and `status` is not running.
pipeline_key	Required	A key to identify the target batch pipeline component. Only one component key can be provided per event. Keys are created and managed by users.
run_key	Required	A key to identify the target run. Only one run key can be provided per event. Keys are created and managed by users.
task_key	Optional. Only required if the status applies to a batch pipeline task that occurs during a run.	A key to identify the target task. Only one task key can be provided per event. Keys are created and managed by users.
pipeline_name	Optional	A human-readable display name for the target batch pipeline.
run_name	Optional	A human-readable display name for the target run.
task_name	Optional	A human-readable display name for the target task.
component_tool	Optional	The name of the external tool represented by the batch pipeline component.
external_url	Optional	A link to source information.
metadata	Optional	Additional key-value information for the event. Provided by the user as needed.
event_timestamp	Optional	An ISO8601 timestamp that describes when the event occurred. If no timezone is specified, UTC is assumed. If unset, the Event Ingestion API applies its current time as the value.
payload_keys	Only required when tracking a payload.	An array of payload values present in a payload instance. Payloads must be tracked in the journey Settings.
dataset_key	Not required for a Run Status event.	Datasets are not compatible with Run Status events.
dataset_name	Not required for a Run Status event.	Datasets are not compatible with Run Status events.
stream_key	Not required for a Run Status event.	Streaming pipelines are not compatible with Run Status events.
stream_name	Not required for a Run Status event.	Streaming pipelines are not compatible with Run Status events.
server_key	Not required for a Run Status event.	Servers are not compatible with Run Status events.
server_name	Not required for a Run Status event.	Servers are not compatible with Run Status events.

Test Outcomes event¶

Reports the outcomes of a test or a set of tests. Receives a list of test outcomes by test name and status. Post a TestOutcomes event to send test results from any external tool to Observability for tracking, monitoring, or reporting purposes.

Key	Required?	Details
test_outcomes	Required	A list of objects, each representing the outcome of a test.
name	Required	The name of the test.
status	Required	The test status to be applied.
min_threshold	Optional	The minimum acceptable value for the test metric_value.
start_time	Optional	An ISO timestamp of when the test execution started.
integrations	Optional	An object of test data specific to DataKitchen Inc. software integrations. See the API documentation for more parameters.
result	Optional	A string representing the test results
description	Optional	A description of the test outcomes.
end_time	Optional	An ISO timestamp of when the test execution ended.
metric_description	Optional	A description of the unit under measure.
metadata	Optional	Additional key-value information for the event. Provided by the user as needed.
type	Optional	Represents the type or archetype of a test.
max_threshold	Optional	The maximum acceptable value for the test metric_value.
dimensions	Optional	Represents a list of data quality aspects the test is meant to address.
key	Optional	A correlation key. Tests with the same key are assumed to be related.
metric_value	Optional	A numerical test outcome.
metric_name	Optional	The name of the metric, or, its unit of measure.
dataset_key	Optional. Only required if the tests occur in a dataset.	A key to identify the dataset. Only one component key can be provided per event. Keys are created and managed by users. When included, can send a `dataset_name` value.
dataset_name	Optional	A human-readable display name for the target dataset.
stream_key	Optional. Only required if the tests occur in a streaming pipeline.	A key to identify the streaming pipeline. Only one component key can be provided per event. Keys are created and managed by users. When included, can send a `stream_name` value.
stream_name	Optional	A human-readable display name for the target streaming pipeline.
server_key	Optional. Only required if the tests occur in a server.	A key to identify the server. Only one component key can be provided per event. Keys are created and managed by users. When included, can send a `server_name` value.
server_name	Optional	A human-readable display name for the target server.
pipeline_key	Optional. Only required if the tests occur in a batch pipeline.	A key to identify the batch pipeline. Only one component key can be provided per event. Keys are created and managed by users. When included, a `run_key` is required and a `task_key` optional. Can send a `pipeline_name` value.
run_key	Optional. Only required if the target component is a batch pipeline.	A key to identify the batch pipeline's run. When included, a `pipeline_key` value is required and a `task_key` optional. Can send a `run_name` value.
task_key	Optional. Only required if the target component is a batch pipeline and the event applies to a task during a run.	A key to identify the task. When included, `run_key` and `pipeline_key` values are required. Can send a `task_name` value.
pipeline_name	Optional	A human-readable display name for the target batch pipeline.
run_name	Optional	A human-readable display name for the target run.
task_name	Optional	A human-readable display name for the target task.
component_tool	Optional	The name of the external tool represented by the component.
external_url	Optional	A link to source information.
metadata	Optional	Additional key-value information for the event. Provided by the user as needed.
event_timestamp	Optional	An ISO8601 timestamp that describes when the event occurred. If no timezone is specified, UTC is assumed. If unset, the Event Ingestion API applies its current time as the value.
payload_keys	Only required when tracking a payload.	An array of payload values present in a payload instance. Payloads must be tracked in the journey Settings.
component_integrations	Optional	An object of test data specific to DataKitchen Inc. software integrations. Consult the API documentation for specific parameters and details, if needed.