Skip to content

Manage Table Groups

A table group is a set of tables that TestGen profiles and tests together. Each table group belongs to a connection and can have its own test suites, monitors, and profiling schedule.

Tip

You can also create a table group as part of the Add Connection wizard, which walks through connection setup and table group creation in one flow.

Prerequisites

  • A connection is required. Use an existing connection or create a new one from the Connections page.
  • Decide which tables you want to profile. On a database connection, take note of the schema name and any specific tables to include or exclude. On a file connection, take note of the paths to the files, relative to the connection's base path.

Tip

For very large or heterogeneous tables — hundreds of millions to billions of rows, or a mix of different record types — define the table group around homogeneous subsets, such as views that isolate a product category or a recent time window, rather than the full table. This improves both profiling performance and the quality of the generated tests. See Work with large datasets.

Create a table group

  1. Select Table Groups from the menu.
  2. Click Add Table Group.

Fill in the table group details

  1. Give the table group a name, unique within the project.

  2. Choose the tables to profile. This works differently depending on the type of connection.

    In Schema, enter the name of the schema that includes the tables you want to profile. On a Salesforce Data 360 connection, enter the name of the data space in Data Space.

    Use the Tables to Include Mask, Tables to Exclude Mask, and Explicit Table List fields to specify which tables within the schema should be profiled.

    • Mask fields accept wildcard characters.
    • All three fields accept table names.
    • Use commas to separate table names in the Explicit Table List field.
    • The default include and exclude masks are set to: include everything but do not include temporary tables.

    Use the Profiling ID Column Mask and Profiling Surrogate Key Column Mask fields to specify any ID and surrogate key data present in your tables.

    In the Tables section, name each table you want to profile and the files it reads. Click Add Table, then fill in the row.

    • Name — the name for this table in TestGen. Use letters, numbers, and underscores only, and give each table a different name.
    • File or Pattern — the path, relative to the location set on the connection. Name a single file (sales/orders.csv), or a pattern to read every file that matches it: sales/*.csv for the CSV files in a folder, or sales/**/*.csv to include its subfolders. A pattern is one table whose rows span every file it matches, so to profile each file separately, declare one table per file.
    • Format — CSV, Parquet, or JSON. See File formats and column types.

    For a CSV table, click (CSV settings icon) to set the Delimiter, whether the file Has header, and the Quote and Escape characters. TestGen detects each of these automatically when you leave it blank.

    Repeat for each table you want to profile.

  3. Optional. Review the Settings:

    • Detect critical data elements (CDE) during profiling — Automatically flags columns as Critical Data Elements based on profiling patterns. Enabled by default.
    • Detect PII during profiling — Automatically flags columns that match patterns for personally identifiable information (such as SSNs, credit card numbers, and email addresses). Enabled by default.
    • Exclude XDE columns from profiling — Skips columns marked as Excluded Data Elements during profiling, reducing noise and processing time. Enabled by default.
    • Profile for table comparison — Collects hashes of each column's values during profiling, which the hash tests of the Table Comparison generation set need. Adds profiling time. Not shown for SAP HANA and Salesforce Data 360 connections. Disabled by default.
    • Include table group in Project Dashboard — Controls visibility in the Project Dashboard. Enabled by default.
    • Add scorecard for table group — Controls visibility in the Quality Dashboard. Enabled by default.

  4. Optional. Assign a profiling delay for test generation.

    • By default, the delay is 0. That is, any tests generated for a table group use the most recent profiling results.
    • Note that tests are expected to pass when generated against the most recent profiling results.
    • You can choose to generate tests against profiling results that are <X> days old.

  5. Optional. Expand Sampling Parameters to toggle on Use profile sampling, then assign a sample percent and minimum count.

    • Sampling applies to pattern and character analysis, such as letter case, embedded spaces, standard formats, and frequent patterns, and to the list of frequent values. TestGen estimates their counts for the whole table from a sample.
    • Record counts, distinct values, minimums, maximums, average lengths and values, and percentiles always cover the full table.
    • The sample percent sets the share of each table's records to analyze, and the minimum count sets the smallest sample size.
    • For exact pattern, character, and frequent value counts, run profiling without sampling.

  6. Optional. Expand Table Group Tags to add metadata such as a description, data source, business domain, or stakeholder group.

  7. Click Next.

Verify table access

The wizard previews the tables it found for the table group. Optionally, click Verify Access to confirm that TestGen can query the tables.

On a file connection, you can expand each declared table to see the columns TestGen inferred and their types.

Click Next to continue.

Run profiling

  • To profile the table group immediately after creation, leave Run profiling toggled on.
  • To skip profiling and run it at a later time, toggle off Run profiling. You can run profiling later from the Table Groups page.

Click Next to continue.

Set up testing

You can optionally create a test suite and schedule automated test runs as part of the setup.

  • To generate tests and schedule test runs, leave Generate and schedule tests toggled on. Enter a Test Suite Name, set a Test Run Schedule, and select a Timezone.
  • To skip this step, toggle off Generate and schedule tests. You can create a test suite later from the Test Suites page.

Note

Tests are auto-generated after profiling completes and then run on the schedule you configure. See Recommended frequency for guidance.

Click Next to continue.

Set up monitors

You can optionally configure monitors for the table group as part of the setup.

  • To configure monitors, leave Configure monitors toggled on. Set the Monitor Schedule, Timezone, and optionally adjust the Sensitivity and other prediction model settings.
  • To skip this step, toggle off Configure monitors. You can configure monitors later from the Monitors page.

Note

Volume and Schema monitors are created immediately. Freshness monitors are created after the first profiling run completes. See Configure Monitors for details on monitor settings.

Click Finish Setup to complete the setup.

Next steps

After the wizard completes, profiling runs in the background (if enabled). Once profiling finishes, tests are auto-generated and monitors begin tracking your tables. Explore results in the Data Catalog.

Edit a table group

Select Table Groups from the menu, and click the (edit icon) on the table group you want to modify. Make changes as needed in the dialog, and verify table access before saving.

Change the schema of a table group

When your pipeline loads each new version of the data into a schema with a different name, change the table group's schema instead of creating a new table group. The tests keep their history, so trends and predicted thresholds carry over.

  1. Select Table Groups from the menu, and click the (edit icon) on the table group.
  2. In Schema, enter the new schema.
  3. Click Next, and confirm that the list includes every table the table group tests.
  4. Click Save.

Profiling, test generation, monitors, and test runs all read the new schema. Referential tests keep reading their second table from the schema they name. TestGen refuses the change when it cannot read the schema, when no table in it matches the table group's table filters, or while a job for the table group, such as profiling or a test run, is in progress.

Warning

If the new schema does not have a table, that table's tests report an error on each test run until the table is back. When the test suite has Automatically deactivate tests that fail validation selected, the first test run deactivates them instead. See Tests that fail validation. The next monitor run deletes the table's monitors, along with the thresholds they learned.

Delete a table group

Warning

Deleting a table group permanently removes all associated profiling runs, test suites, test results, and monitors. This action cannot be undone.

Select Table Groups from the menu, and click the (delete icon) on the table group you want to remove. Confirm the deletion.

Reference

File formats and column types

On a file connection, TestGen reads each declared table as the format you select, so the format has to match what the files hold. A JSON file read as CSV produces a table of meaningless columns rather than an error. The preview at Verify table access is where you catch that.

A CSV or JSON file carries values rather than declared types, so TestGen infers each column's type from the values it finds. A text column whose values are all digits is read as a number, and a column with no values at all is read as text. Parquet carries its own types, so use Parquet where the exact type matters.

Where one table spans several files, TestGen reads the union of their columns, so a column that appears in only some of the files is still included.

Tables that appear or disappear at the source

TestGen profiles the tables that a table group resolves to at the time of the run. Where a table group names its tables explicitly — through Explicit Table List on a database connection, or through declared tables on a file connection — a table added at the source is not profiled until you add it to the table group, and a table that is no longer there is recorded as dropped.

Manage Connections

Manage Test Suites

Data Catalog