Skip to content

Event Ingestion API Identifiers

Every event sent to Observability through the Event Ingestion API includes required and optional properties to provide context and metadata.

Key

Keys are defined by users, sent by customer tools, and used to identify and associate events.

Every Event Ingestion API endpoint requires an API key and <component>_key. Other keys may be required to describe a specific event. Together, these properties allow the system to associate events with a project and a component.

Example

Batch pipeline example: Observability creates a new batch pipeline component when it receives any event with a unique pipeline_key. The system then creates a new run for this pipeline each time it receives an event with a combination of that pipeline_key and a new run_key. Future events are associated with this run if they use the same pipeline_key and run_key. For any given batch pipeline, a run's key is always unique, allowing the system to associate events as they happen over time.

Name

While keys are a unique identifier used by the system, the name property provides a human-readable identifier.

The UI displays the <component>_name, run_name, and/or task_name when the property is sent with an event. If no name is specified for a component or task, the value for the key becomes the name by default; the run name remains null if no name is specified.

The system attributes a name the first time the value is sent. If a new name is sent in future events, the system updates the name.

Tool

Components represent the tools, software, and assets that exist in your data estate. To provide better context for the infrastructure that sends events to Observability, the component_tool field sets the tool type and icon that display across pages in the UI.

List of available tools

  • Airflow (airflow)
  • Apache Impala (apache_impala)
  • Amazon Redshift (redshift)
  • Amazon S3 (aws_s3)
  • Amazon Sagemaker (aws_sagemaker)
  • Amazon SQS (sqs)
  • AWS Glue (aws_glue)
  • AWS Lambda (aws_lambda)
  • AutoSys (autosys)
  • Azure Blob Storage (blob_storage)
  • Azure Data Factory (data_factory)
  • Azure Functions (azure_functions)
  • Azure Machine Learning (azure_ml)
  • Azure Synapse Analytics (azure_synapse_pipelines)

  • Databricks Workflows on Azure Databricks and AWS (databricks)

  • DataKitchen's DataOps Automation (dataops_automation)

  • Fivetran Log Connector (fivetran)
  • GoAnywhere (GoAnywhere)
  • Google Cloud Composer (gcc)
  • Informatica (informatica)
  • Microsoft Power BI (power_bi)
  • Microsoft SQL Server (mssql)
  • Microsoft SQL Server Integration Services (ssis)
  • Neo4j (neo4j)
  • Oracle Database (oracle_database)
  • PostgreSQL (postgresql)
  • Python (python)
  • Snowflake (snowflake)
  • Tableau (tableau)
  • Talend (talend)

Note

  • This field does not affect the integration between an external tool and Observability.

  • To set this information from the UI, see Manage Components.

  • The component_tool field and UI also accept any custom external tool. For example, "component_tool": "heroku".