Skip to content

Use an External PostgreSQL Database

TestGen stores its application metadata — projects, connections, profiling results, test definitions, and run history — in a PostgreSQL database. By default, the installer runs this database as a PostgreSQL container alongside the application.

For production deployments, DataKitchen recommends pointing TestGen at a dedicated PostgreSQL server instead. Both TestGen open source and TestGen Enterprise support an external database. A separate database server lets you manage backups, high availability, and version upgrades independently of the application.

The server can be either:

  • A PostgreSQL instance on the same virtual machine as TestGen.
  • A managed PostgreSQL service, such as Amazon RDS, Azure Database for PostgreSQL, or Google Cloud SQL.

Note

Choose the database location before installing. TestGen does not migrate metadata between databases, so switching from the container database to an external server after install means starting over with an empty database.

Prerequisites

  • For TestGen Enterprise, request access to the private enterprise Docker image from your DataKitchen representative. TestGen open source uses the public datakitchen/dataops-testgen image.
  • To handle enterprise workloads, install TestGen on a Linux virtual machine with at least 4 CPUs, 16 GB memory, and 100 GB disk space, for example, t3.xlarge on Amazon EC2, c4-standard-4 on Google Compute Engine, or B4ms on Azure Virtual Machines. TestGen profiles and tests DuckDB databases and file data sources with its own memory and disk. If you plan to connect them, see Size TestGen for DuckDB and File Data Sources.
  • Install the following software, and verify that the versions are supported.
Software Supported versions Command to check version
Python 3 3.9+ python3 --version
Docker + Docker Compose Docker: 27+

Compose: 5.0+
docker -v

docker compose version
  • PostgreSQL server, version 14 or later, reachable from the machine where TestGen will run.
  • An administrative PostgreSQL account that can create users.

Create the TestGen database user

Connect to the PostgreSQL server with an administrative account and create a user for TestGen.

CREATE USER testgen WITH PASSWORD '<postgres password>' CREATEDB;

The CREATEDB privilege is required: on first start, TestGen creates its datakitchen database and the testgen schema.

Allow connections from the TestGen container

How TestGen reaches the database depends on where the server runs.

No PostgreSQL server configuration is needed. Make sure that:

  • The service's network rules (security group, firewall, or virtual network rules) allow inbound connections from the TestGen machine on the PostgreSQL port (5432 by default).
  • You note the server's endpoint hostname for the install step.

The TestGen container connects to the server from the datakitchen-network Docker network (192.168.60.0/24), which the installer creates. Edit the PostgreSQL configuration files (paths shown are for PostgreSQL 16 on Ubuntu; adjust the version directory for your install).

  1. In /etc/postgresql/16/main/postgresql.conf, set listen_addresses to listen on all interfaces.

    listen_addresses = '*'
    

    Listing only the Docker gateway address does not work: PostgreSQL starts before Docker creates that address, so TestGen cannot connect after the VM restarts.

  2. In /etc/postgresql/16/main/pg_hba.conf, add a line under # IPv4 local connections to allow password-authenticated connections from the TestGen container network.

    host    all             all             192.168.60.0/24         scram-sha-256
    
  3. Restart PostgreSQL to apply the changes.

    systemctl restart postgresql
    
  4. Block inbound connections to the PostgreSQL port (5432 by default) from outside the VM, using the VM's firewall or your cloud provider's network rules.

Install TestGen against the external database

To use an external database, you supply your own docker-compose.yml instead of letting the installer generate one. When a compose file already exists in the install directory, the installer reuses it.

  1. For TestGen Enterprise, log in to Docker with the credentials provided by your DataKitchen representative. For open source, skip this step.

    docker login -u <username> -p <password>
    
  2. Create and navigate to a new install directory, for example, "/testgen".

  3. In the install directory, create a docker-compose.yml file with the following contents.

    name: dataops-testgen
    
    x-common-variables: &common-variables
      TESTGEN_USERNAME: <UI login username>
      TESTGEN_PASSWORD: <UI login password>
      TG_DECRYPT_SALT: <random string>
      TG_DECRYPT_PASSWORD: <random string>
      TG_JWT_HASHING_KEY: <base64-encoded random string>
      TG_METADATA_DB_HOST: <postgres host>
      TG_METADATA_DB_PORT: "5432"
      TG_METADATA_DB_USER: testgen
      TG_METADATA_DB_PASSWORD: <postgres password>
      TG_INSTANCE_ID: <instance name>
      TG_UI_BASE_URL: <UI base URL>
      TG_BASE_URL: <API base URL>
    
    services:
      engine:
        image: datakitchen/dataops-testgen-enterprise:v5  # open source: datakitchen/dataops-testgen:v5
        container_name: testgen
        restart: unless-stopped
        environment: *common-variables
        volumes:
          - testgen_data:/var/lib/testgen
        ports:
          - 8501:8501
          - 8530:8530
        extra_hosts:
          - host.docker.internal:host-gateway
        networks:
          - datakitchen
    
    volumes:
      testgen_data:
    
    networks:
      datakitchen:
        name: datakitchen-network
        external: true
    
  4. Replace the placeholders.

    Placeholder Value
    TESTGEN_USERNAME, TESTGEN_PASSWORD Username and password for the initial system administrator account. After installing, sign in with these credentials to create accounts for your team. This account has no special status — once other system administrators exist, any of them can remove it if it is no longer needed.
    TG_DECRYPT_SALT, TG_DECRYPT_PASSWORD Random strings used to encrypt stored connection credentials.
    TG_JWT_HASHING_KEY A base64-encoded random string used to sign session tokens.
    TG_METADATA_DB_HOST The PostgreSQL host. For a server on the same VM, use host.docker.internal, not localhost. For a managed service, use its endpoint hostname.
    TG_METADATA_DB_PORT The PostgreSQL port. Defaults to 5432.
    TG_METADATA_DB_USER, TG_METADATA_DB_PASSWORD The user and password created in Create the TestGen database user.
    TG_INSTANCE_ID A name that identifies this TestGen instance.
    TG_UI_BASE_URL The URL used to access the UI, for example, http://<host>:8501. Include a path to serve the UI under it.
    TG_BASE_URL The externally reachable URL for the API and MCP server, for example, http://<host>:8530. Include a path to serve the API and MCP server under it.
  5. Download the latest version of dk-installer into the same directory.

    curl -o dk-installer.py 'https://raw.githubusercontent.com/DataKitchen/data-observability-installer/main/dk-installer.py'
    
  6. Run the install command. Because a docker-compose.yml is present, the installer uses it instead of generating one with a database container.

    python3 dk-installer.py tg install
    
  7. When the install completes, verify that the UI is accessible at the TG_UI_BASE_URL you configured and that you can log in with the credentials from the terminal output.

Tip

To reach TestGen from other machines, enable HTTPS, configure email notifications, or set up single sign-on, see Administer TestGen.

To upgrade an instance of TestGen to the latest released version, see Upgrade TestGen.