Skip to content

Connect to Observability

Use TestGen to generate and execute data quality checks that identify errors in your data. Then, export the results to DataKitchen's DataOps Observability for visibility and impact across your entire data estate.

Tip

If you do not have DataOps Observability yet, try it out.

Prerequisites

  • An Observability API key with the ability to Send events to a project is required. Details can be found in the Observability help documentation, API Keys.
  • The API key must be added to the TestGen project configuration.

Connect TestGen to Observability

To add an Observability API key to a TestGen project:

  1. Select Project Settings from the menu.
  2. Enter the base API URL for your Observability instance.

    • Enterprise Observability: https://api.datakitchen.io
    • Minikube application: http://<minikube_ip>:8082/api
    • Minikube application exposed to localhost: http://host.docker.internal:8082/api
  3. Enter the API Key.

  4. Optional. Click Test Connection to test the connection between your Observability and TestGen projects.
  5. Click Save.

Once the connection is established, you can export test results to your Observability project.

Export test results to Observability

  1. Follow the steps in Run Tests to run a test suite.
  2. On the Test Suites page, click the (export icon) on the test suite.
  3. Click Start in the dialog to send the test results to Observability.

Considerations

  • The Export to Observability dialog sends the results produced since the last time the process was run for a given test suite.
  • TestGen sends results in events of up to 100 KB and 500 items each.

Best practice is to send data as part of your TestGen workflow. Export results per test run or set up a cron schedule to send results automatically.

Mapping

Test results sent from TestGen appear in Observability as a Test Outcomes event for a dataset component. When the dataset component is included in a journey, you can view test results as part of the instance details. The DataOps Observability documentation includes more information on datasets and journeys.

By default, the TestGen table group ID becomes the dataset key, while the table group name becomes the dataset name.

Custom mapping

You can choose to override the default mapping at the test suite level by specifying a component key and name for the test suite.

You can establish this mapping when you create or edit a test suite. Note that the dataset component remains the most relevant Observability component for mapping the component type.

Example

Use case example: one schema, as represented in a specific table group profiling run, can include a large number of tables. These tables can come from different data sources and may even be refreshed on different schedules. To organize results with more granularity, you can export results from the test suite level, rather than at the table group level.

Next steps

After you export test results, log in to Observability to review them.