Skip to content

Connect to Observability

Use TestGen to generate and execute data quality checks that identify errors in your data. Then, export the results to DataKitchen's DataOps Observability for visibility and impact across your entire data estate.

Tip

Don't have DataOps Observability? Try it out now.

Prerequisites

  • An Observability API key with the ability to Send events to a project is required. Details can be found in the Observability help documentation, API Keys.
  • The API key must be added to the TestGen project configuration.

How to connect TestGen with Observability

To add an Observability API key to a TestGen project:

  1. Select Settings from the menu.

  2. Enter the base API URL for your Observability instance in the Observability Api Url field.

    • Enterprise Observability: https://api.datakitchen.io
    • Minikube application: http://<minikube_ip>:8082/api
    • Minikube application exposed to localhost: http://host.docker.internal:8082/api

  3. Enter the API key in the Observability Api Key field.

  4. Optional. Click Test Observability Connection to test the connection between your Observability and TestGen projects.
  5. Click Save Changes.

Once the connection is established, you can export test results to your Observability project.

How to export to Observability

  1. Follow the steps in Run Tests to execute the tests in a test suite.
  2. On the Test Suites page, click Export results to Observability on the test suite.
  3. Click Start in the dialog to send the test results to Observability.

Considerations

  • The Export to Observability dialog sends the results produced since the last time the process was run for a given test suite.
  • To comply with the Observability Event Ingestion API payload limit of 100 KB and the Test Outcomes event 500 item limit, the events sent from TestGen adhere to a payload limit of up to 100 KB and 500 items per event.

Best practice is to send data as part of your TestGen workflow. Export results per test run or set up a cron schedule to send results automatically.

Mapping

Test results sent from TestGen appear in Observability as a Test Outcomes event for a dataset component. When the dataset component is included in a journey, you can view test results as part of the instance details. The DataOps Observability documentation includes more information on datasets and journeys.

By default, the TestGen table group ID becomes the dataset key, while the table group name becomes the dataset name.

Custom mapping

You can choose to override the default mapping at the test suite level by specifying a component key and name for the test suite.

You can establish this mapping when you create or edit a test suite. Note that the dataset component remains the most relevant Observability component for mapping the component type.

Example

Use case example: one schema, as represented in a specific table group profiling run, can include a large number of tables. These tables can come from different data sources and may even be refreshed on different schedules. To organize results with more granularity, you can export results from the test suite level, rather than at the table group level.

Next steps

After you export test results, log into Observability and dig deeper into the data.

For more information, continue with the Observability documentation: TestGen Dataset Components and Instance Details TestGen Results.