Skip to content

Size TestGen for DuckDB and File Data Sources

Use this page to plan memory and disk for DuckDB connections and file data sources, and to choose how many of their runs happen at once. These sources are read by a query engine that runs inside TestGen, so their profiling, test, and monitor runs use the memory and disk of the machine or container where TestGen runs, not those of a database server.

Plan memory and disk

What a run needs grows with the number of rows it reads. The number of columns has much less effect on memory, though wide tables take longer.

  • Provision TestGen with memory to match the largest tables you intend to profile and test.
  • Provision disk as well. A run that needs more memory than is available writes temporary files to TestGen's disk.
  • Provision more disk for CSV and JSON files on Amazon S3 or Azure Blob Storage. A run needs about half the size of the CSV files it reads, and about a third the size of the JSON files. If the disk fills, the run reads the remaining tables from storage, which takes longer.

Choose how many runs happen at once

By default, TestGen runs one profiling, test, or monitor run at a time across all DuckDB connections and file data sources. Further runs show the Queued status and start in the order they were submitted. Runs on other connection types are not held back.

Each run can use most of the memory available to TestGen. Before you raise the limit, provision memory and disk for that many runs together.

To raise the limit, set the following environment variable.

Variable Default Purpose
TG_MAX_CONCURRENT_IN_PROCESS_RUNS 1 The number of runs on DuckDB connections and file data sources to allow at once.

  1. Navigate to the directory that contains the docker-compose.yml file for TestGen.

    • If the instance was installed using dk-installer.exe, the file can be located under "AppData\Local\DataKitchenApps".
    • If the instance was installed using dk-installer, the file can be located in the folder from which the Python installer was executed.

  2. Set the following variable under the x-common-variables section.

    x-common-variables: &common-variables
 TG_MAX_CONCURRENT_IN_PROCESS_RUNS: <number of runs>

  3. Restart the application.

    docker compose up -d --wait

  1. Open the ~/.testgen/config.env file in a text editor.

  2. Set the following variable (no quotes).

    TG_MAX_CONCURRENT_IN_PROCESS_RUNS=<number of runs>

  3. Restart the application. Stop the running TestGen process with Ctrl+C in the terminal where it is running, then start it again.

    python3 dk-installer.py tg start

  1. Open the values-tg-app.yaml file used for the installation.

  2. Add the following entry to the extraEnv list.

    extraEnv:
  - name: TG_MAX_CONCURRENT_IN_PROCESS_RUNS
    value: "<number of runs>"

  3. Apply the change.

    helm upgrade -n datakitchen dk-tg-app datakitchen-testgen/testgen \
    --values values-tg-app.yaml --wait

Give TestGen more memory

How you raise the memory available to TestGen depends on the install mode.

  • Docker: Move TestGen to a machine with more memory. If the engine service in docker-compose.yml sets a memory limit, raise it as well.
  • Pip: Move TestGen to a machine with more memory.
  • Kubernetes: Raise the memory limit under resources in values-tg-app.yaml, then apply the change with helm upgrade. Choose a limit that fits within the memory of the node that runs the TestGen pod.

Troubleshoot runs on DuckDB and file data sources

Symptom Cause Solution
A run ends with the Error status and the message The run was stopped by the operating system before it could finish, which usually means TestGen ran out of memory. The run needed more memory than TestGen has. Give TestGen more memory. For profiling, you can turn on sampling instead. If you raised the number of runs allowed at once, lower it.
A run stays in the Queued status. Another run on a DuckDB connection or file data source is in progress, and the limit on runs at once is reached. Wait for the other run to finish, or cancel it. To allow more runs at once, see Choose how many runs happen at once.