Skip to content

Manage Connections Enterprise

A connection defines how TestGen accesses your data for profiling, testing, and monitoring. TestGen requires only read-only access — see Access Requirements for what each supported source needs. Each project can have multiple connections, and each connection can have multiple table groups.

The open-source version is limited to a single connection. Select Connection from the menu to edit its configuration, then click Setup Table Groups to set up a table group.

Note

Creating, editing, and deleting connections requires the Admin role on the project.

Create a connection

The Data Configuration Setup wizard walks you through creating a connection, setting up a table group, and optionally configuring profiling, testing, and monitors.

  1. Select Connections from the menu.
  2. Click Add Connection.
  3. Select the data source, and click Next.

Fill in the connection details

  1. Give the connection a name, unique within the project.

  2. Enter the connection details for the type you selected.

    Typically a Host, Port, Database, and credentials. For a Docker installation, if the database is running on localhost on the same machine as TestGen, set the Host value to host.docker.internal.

    Typically the location of your files and credentials. Test Connection reports how many files it found there.

  3. Optional. Expand Advanced Tuning to adjust Max Threads and Max Expression Length. The defaults are set for optimal performance and do not need to be changed in most cases.

  4. Optional. Click Test Connection to verify the connection.
  5. Click Next.

Create a table group

  1. Give the table group a name, unique within the project.

  2. Choose the tables to profile. This works differently depending on the type of connection.

    In Schema, enter the name of the schema that includes the tables you want to profile. On a Salesforce Data 360 connection, enter the name of the data space in Data Space.

    Use the Tables to Include Mask, Tables to Exclude Mask, and Explicit Table List fields to specify which tables within the schema should be profiled.

    • Mask fields accept wildcard characters.
    • All three fields accept table names.
    • Use commas to separate table names in the Explicit Table List field.
    • The default include and exclude masks are set to: include everything but do not include temporary tables.

    Use the Profiling ID Column Mask and Profiling Surrogate Key Column Mask fields to specify any ID and surrogate key data present in your tables.

    In the Tables section, name each table you want to profile and the files it reads. Click Add Table, then fill in the row.

    • Name — the name for this table in TestGen. Use letters, numbers, and underscores only, and give each table a different name.
    • File or Pattern — the path, relative to the location set on the connection. Name a single file (sales/orders.csv), or a pattern to read every file that matches it: sales/*.csv for the CSV files in a folder, or sales/**/*.csv to include its subfolders. A pattern is one table whose rows span every file it matches, so to profile each file separately, declare one table per file.
    • Format — CSV, Parquet, or JSON. See File formats and column types.

    For a CSV table, click (CSV settings icon) to set the Delimiter, whether the file Has header, and the Quote and Escape characters. TestGen detects each of these automatically when you leave it blank.

    Repeat for each table you want to profile.

  3. Optional. Review the Settings:

    • Detect critical data elements (CDE) during profiling — Automatically flags columns as Critical Data Elements based on profiling patterns. Enabled by default.
    • Detect PII during profiling — Automatically flags columns that match patterns for personally identifiable information (such as SSNs, credit card numbers, and email addresses). Enabled by default.
    • Exclude XDE columns from profiling — Skips columns marked as Excluded Data Elements during profiling, reducing noise and processing time. Enabled by default.
    • Profile for table comparison — Collects hashes of each column's values during profiling, which the hash tests of the Table Comparison generation set need. Adds profiling time. Not shown for SAP HANA and Salesforce Data 360 connections. Disabled by default.
    • Include table group in Project Dashboard — Controls visibility in the Project Dashboard. Enabled by default.
    • Add scorecard for table group — Controls visibility in the Quality Dashboard. Enabled by default.

  4. Optional. Assign a profiling delay for test generation.

    • By default, the delay is 0. That is, any tests generated for a table group use the most recent profiling results.
    • Note that tests are expected to pass when generated against the most recent profiling results.
    • You can choose to generate tests against profiling results that are <X> days old.

  5. Optional. Expand Sampling Parameters to toggle on Use profile sampling, then assign a sample percent and minimum count.

    • Sampling applies to pattern and character analysis, such as letter case, embedded spaces, standard formats, and frequent patterns. TestGen estimates these counts for the whole table from a sample. The list of frequent values also comes from the sample.
    • Record counts, distinct values, minimums, maximums, average lengths and values, and percentiles always cover the full table.
    • The sample percent sets the share of each table's records to analyze, and the minimum count sets the smallest sample size.
    • For exact pattern and character counts, run profiling without sampling.

  6. Optional. Expand Table Group Tags to add metadata such as a description, data source, business domain, or stakeholder group.

  7. Click Next.

Verify table access

The wizard previews the tables it found for the table group. Optionally, click Verify Access to confirm that TestGen can query the tables.

On a file connection, you can expand each declared table to see the columns TestGen inferred and their types.

Click Next to continue.

Run profiling

  • To profile the table group immediately after creation, leave Run profiling toggled on.
  • To skip profiling and run it at a later time, toggle off Run profiling. You can run profiling later from the Table Groups page.

Click Next to continue.

Set up testing

You can optionally create a test suite and schedule automated test runs as part of the setup.

  • To generate tests and schedule test runs, leave Generate and schedule tests toggled on. Enter a Test Suite Name, set a Test Run Schedule, and select a Timezone.
  • To skip this step, toggle off Generate and schedule tests. You can create a test suite later from the Test Suites page.

Note

Tests are auto-generated after profiling completes and then run on the schedule you configure. See Recommended frequency for guidance.

Click Next to continue.

Set up monitors

You can optionally configure monitors for the table group as part of the setup.

  • To configure monitors, leave Configure monitors toggled on. Set the Monitor Schedule, Timezone, and optionally adjust the Sensitivity and other prediction model settings.
  • To skip this step, toggle off Configure monitors. You can configure monitors later from the Monitors page.

Note

Volume and Schema monitors are created immediately. Freshness monitors are created after the first profiling run completes. See Configure Monitors for details on monitor settings.

Click Finish Setup to complete the setup.

Edit a connection

Select Connections from the menu, and click the (edit icon) on the connection you want to modify. Make changes as needed, and click Test Connection to verify before saving.

Delete a connection

Warning

Deleting a connection permanently removes the connection and all associated table groups, profiling runs, test suites, test results, and monitors. This action cannot be undone.

Select Connections from the menu, and click the (delete icon) on the connection you want to remove. Confirm the deletion.

Manage Table Groups

Manage Test Suites