Skip to content

Onboard a Dataset

Set up a new dataset for data quality testing from a pipeline: create its table group and test suite, turn on monitoring, profile it, generate tests, and run them.

This suits a pipeline that loads each new dataset into its own schema or its own set of files — one per client, for example — and needs TestGen to profile and test it as soon as the data arrives. Profiling, test generation, and test runs each start a job: submit it, wait for it to finish, then move to the next step.

Prerequisites

  • An access token for a user with edit access to the project.
  • A connection that reaches the new data: a database connection for a new schema, or a file connection for new files. Find its ID with GET /api/v1/projects/{project_code}/connections.

Create the table group

A table group is a set of tables that TestGen profiles and tests together. Create one on the connection.

POST /api/v1/connections/{connection_id}/table-groups

The request names the tables differently depending on the type of connection.

Name the schema that holds the tables:

{
  "table_group_name": "client_42",
  "schema_name": "client_42"
}

To profile only some of the schema's tables, list them in table_set, or filter them with SQL LIKE patterns in include_masks and exclude_masks.

Declare each table in file_declarations, with the file or pattern it reads, relative to the location the connection reads from:

{
  "table_group_name": "client_42",
  "file_declarations": [
    {
      "name": "orders",
      "uri": "client_42/orders_*.csv",
      "format": "csv",
      "delimiter": "|",
      "header": true
    },
    {
      "name": "members",
      "uri": "client_42/members.parquet",
      "format": "parquet"
    }
  ]
}

A pattern is one table whose rows span every file it matches. For CSV files, TestGen detects the delimiter, header, and quote and escape characters; to set them yourself, use delimiter, header, quote, and escape.

Only the name and the tables are required; the API Reference lists the other settings and their defaults. Save the table_group_id from the response — the next steps use it.

Table group names are unique within a project, so a second create with the same name fails.

Create the test suite

A test suite is a collection of data quality tests for a table group. Create one in the table group.

POST /api/v1/table-groups/{table_group_id}/test-suites

{
  "test_suite_name": "client_42"
}

Only the name is required; the API Reference lists the other settings and their defaults. Save the test_suite_id from the response.

Test suite names are not unique, so repeating this call creates a second suite with the same name.

To have the table group's first profiling run generate tests into this suite, add "set_as_default": true. You can then skip Generate tests.

Enable monitoring

Monitors provide automated anomaly detection at the table level: whether tables are updated on schedule, whether row counts change unexpectedly, and whether the schema shifts. This step is optional. Turn monitoring on for the table group with a schedule for the monitors to run on.

PUT /api/v1/table-groups/{table_group_id}/monitoring

{
  "cron_expression": "0 */12 * * *",
  "timezone": "America/New_York"
}

Only the cron expression is required; the API Reference lists the other settings and their defaults. Repeating this call replaces the settings rather than failing, so a retried pipeline run succeeds.

Enable monitoring before profiling: the table group's first profiling run then adds its Freshness monitors. Monitors need roughly 30 runs before they flag anomalies — see Configure Monitors for the settings and how often to run them.

Profile the table group

Profiling discovers the structure and characteristics of the tables in the table group. Its results are the foundation for test generation.

POST /api/v1/table-groups/{table_group_id}/profiling-runs

Wait for the job to complete before the next step. See Run Profiling and Tests for how to poll a job and read its result.

Generate tests

Test generation populates the suite based on the latest profiling results, selecting test types and setting their thresholds.

POST /api/v1/test-suites/{test_suite_id}/test-generation

Omit the body to use the suite's generation sets — Standard, unless you chose others when you created the suite — or pass {"generation_sets": [...]} to override them for this run. Wait for the job to complete before the next step. See Run Profiling and Tests for how to poll a job.

Note

Skip this step if you created the suite with "set_as_default": true — calling it as well generates the tests twice.

Run the tests

Each test run evaluates your data against the thresholds and criteria defined in the suite's test definitions.

POST /api/v1/test-suites/{test_suite_id}/test-runs

Wait for the job to complete, then fetch its results. See Run Profiling and Tests for how to poll a job and read its result.

Test the dataset again

When new data arrives, repeat Run the tests to test it. Profile again when tables or columns change, then generate again to pick up tests for new ones.

Test a dataset whose schema name changes

On a database connection, when your pipeline loads each new version of the data into a schema with a different name, change the table group's schema instead of creating a new table group. The tests keep their history, so trends and predicted thresholds carry over.

PATCH /api/v1/table-groups/{table_group_id}

{
  "schema_name": "client_42_2026_09_16"
}

Profiling, test generation, monitors, and test runs all read the new schema, so run the tests as usual. Referential tests keep reading their second table from the schema they name. To point them at another schema for a run, see Test run options.

If the new schema does not have one of the table group's tables, the response lists it under schema_change.missing_tables. That table's tests report an error on each test run until the table is back. If the test suite was created with "deactivate_invalid_tests": true, the first test run deactivates them instead. The next monitor run deletes the table's monitors, along with the thresholds they learned.