Install on Amazon ECS Enterprise

Run DataOps Data Quality TestGen Enterprise as an Amazon ECS service on AWS Fargate, with the application database on Amazon RDS for PostgreSQL.

To run it on ECS, you create a task definition for the service, a one-time database setup task, the service, and a load balancer.

Prerequisites

• Request access to the private enterprise Docker image from your DataKitchen representative.
• Install the AWS CLI and Docker with Buildx.
• Choose the domain names for the TestGen UI and for the API and MCP server, or one domain to serve TestGen under a path, and have a certificate in AWS Certificate Manager covering them.

AWS resources

• An ECS cluster that runs Fargate tasks.
• A VPC with private subnets in at least two Availability Zones. The task subnets need outbound access to Amazon ECR, AWS Secrets Manager, and CloudWatch Logs, through a NAT gateway or VPC endpoints, and to your target databases and, for email notifications, your SMTP server.
• An Amazon ECR repository for the TestGen image.

PostgreSQL server

• Amazon RDS for PostgreSQL 14 or later, reachable from the task subnets on port 5432.
• Recommended compute: 4 vCPUs and 128 GB disk, for example, db.m6g.xlarge.

Prepare the application database

Connect to the RDS instance with its master user and create the user that TestGen will use.

CREATE USER testgen WITH PASSWORD '<postgres password>' CREATEDB;

The CREATEDB privilege is required: during setup, TestGen creates its datakitchen database and the testgen schema.

Note

Do not configure TestGen with the RDS master user. RDS can rotate its password, and a running TestGen task keeps the old one until it restarts.

Copy the image to Amazon ECR

The enterprise image supports both the linux/amd64 and linux/arm64 platforms. Copy the image with docker buildx imagetools create, which copies every platform the image supports.

1. Sign in to Docker Hub with the credentials provided by your DataKitchen representative, and to your ECR registry.

   docker login --username <username>
   aws ecr get-login-password --region <region> | \
       docker login --username AWS --password-stdin <account ID>.dkr.ecr.<region>.amazonaws.com
   

2. Copy the version you are installing. Use an exact version rather than a major version such as v5. For a list of recent releases, see TestGen Release Notes.

   docker buildx imagetools create \
       --tag <ECR repository URI>:v<version> \
       datakitchen/dataops-testgen-enterprise:v<version>
   

Store the secrets

Store the sensitive values in one AWS Secrets Manager secret. The task definitions inject each one.

1. Create the secret, replacing the placeholders. Use the password of the testgen user for TG_METADATA_DB_PASSWORD, and generate a random value for each of the others.

   aws secretsmanager create-secret --name testgen/app --secret-string '{
       "TG_METADATA_DB_PASSWORD": "<postgres password>",
       "TG_DECRYPT_SALT": "<random string>",
       "TG_DECRYPT_PASSWORD": "<random string>",
       "TG_JWT_HASHING_KEY": "<base64-encoded random 32 bytes>",
       "TESTGEN_PASSWORD": "<initial admin password>"
   }'
   

   Warning

   Keep a copy of these values, and do not enable automatic rotation for this secret. If TG_DECRYPT_SALT or TG_DECRYPT_PASSWORD is lost or changed, the credentials of every database connection must be re-entered. If TG_JWT_HASHING_KEY changes, every user is signed out.

2. Give the task execution role permission to read the secret, in addition to the AmazonECSTaskExecutionRolePolicy managed policy.

   {
     "Effect": "Allow",
     "Action": "secretsmanager:GetSecretValue",
     "Resource": "<secret ARN>"
   }
   

Register the task definitions

TestGen uses two task definitions from the same image: one for the service, and one that you run once to set up the application database.

1. Create a testgen-task.json file and replace the placeholders. Keep this file: an upgrade registers a new revision of it.

   {
     "family": "testgen",
     "requiresCompatibilities": ["FARGATE"],
     "networkMode": "awsvpc",
     "cpu": "4096",
     "memory": "16384",
     "runtimePlatform": { "operatingSystemFamily": "LINUX", "cpuArchitecture": "X86_64" },
     "executionRoleArn": "<task execution role ARN>",
     "containerDefinitions": [
       {
         "name": "migrate",
         "image": "<ECR repository URI>:v<version>",
         "essential": false,
         "command": ["upgrade-system-version"],
         "environment": [
           { "name": "TG_METADATA_DB_HOST", "value": "<postgres host>" },
           { "name": "TG_METADATA_DB_USER", "value": "testgen" }
         ],
         "secrets": [
           { "name": "TG_METADATA_DB_PASSWORD", "valueFrom": "<secret ARN>:TG_METADATA_DB_PASSWORD::" },
           { "name": "TG_DECRYPT_SALT", "valueFrom": "<secret ARN>:TG_DECRYPT_SALT::" },
           { "name": "TG_DECRYPT_PASSWORD", "valueFrom": "<secret ARN>:TG_DECRYPT_PASSWORD::" },
           { "name": "TG_JWT_HASHING_KEY", "valueFrom": "<secret ARN>:TG_JWT_HASHING_KEY::" }
         ],
         "logConfiguration": {
           "logDriver": "awslogs",
           "options": {
             "awslogs-group": "/ecs/testgen",
             "awslogs-region": "<region>",
             "awslogs-stream-prefix": "migrate"
           }
         }
       },
       {
         "name": "testgen",
         "image": "<ECR repository URI>:v<version>",
         "essential": true,
         "dependsOn": [{ "containerName": "migrate", "condition": "SUCCESS" }],
         "stopTimeout": 90,
         "portMappings": [
           { "containerPort": 8501, "protocol": "tcp" },
           { "containerPort": 8530, "protocol": "tcp" }
         ],
         "environment": [
           { "name": "TG_METADATA_DB_HOST", "value": "<postgres host>" },
           { "name": "TG_METADATA_DB_USER", "value": "testgen" },
           { "name": "TG_INSTANCE_ID", "value": "<instance name>" },
           { "name": "TG_UI_BASE_URL", "value": "https://<UI domain>" },
           { "name": "TG_BASE_URL", "value": "https://<API domain>" }
         ],
         "secrets": [
           { "name": "TG_METADATA_DB_PASSWORD", "valueFrom": "<secret ARN>:TG_METADATA_DB_PASSWORD::" },
           { "name": "TG_DECRYPT_SALT", "valueFrom": "<secret ARN>:TG_DECRYPT_SALT::" },
           { "name": "TG_DECRYPT_PASSWORD", "valueFrom": "<secret ARN>:TG_DECRYPT_PASSWORD::" },
           { "name": "TG_JWT_HASHING_KEY", "valueFrom": "<secret ARN>:TG_JWT_HASHING_KEY::" }
         ],
         "logConfiguration": {
           "logDriver": "awslogs",
           "options": {
             "awslogs-group": "/ecs/testgen",
             "awslogs-region": "<region>",
             "awslogs-stream-prefix": "testgen"
           }
         }
       }
     ]
   }
   

   The migrate container upgrades the application database before the testgen container starts. Keep stopTimeout at 90 or higher, so that running profiling jobs can pause when the task stops. For what each variable does, see the Environment variables reference.

   TestGen profiles and tests DuckDB databases and file data sources with the task's own memory and disk. If you plan to connect them, see Size TestGen for DuckDB and File Data Sources. For more memory, raise cpu and memory to a combination that Fargate supports. For more disk, add "ephemeralStorage": { "sizeInGiB": <size> }.

2. Create a testgen-setup-task.json file and replace the placeholders. It runs once, so it has a single container.

   {
     "family": "testgen-setup",
     "requiresCompatibilities": ["FARGATE"],
     "networkMode": "awsvpc",
     "cpu": "4096",
     "memory": "16384",
     "runtimePlatform": { "operatingSystemFamily": "LINUX", "cpuArchitecture": "X86_64" },
     "executionRoleArn": "<task execution role ARN>",
     "containerDefinitions": [
       {
         "name": "setup",
         "image": "<ECR repository URI>:v<version>",
         "essential": true,
         "command": ["setup-system-db", "--yes"],
         "environment": [
           { "name": "TG_METADATA_DB_HOST", "value": "<postgres host>" },
           { "name": "TG_METADATA_DB_USER", "value": "testgen" },
           { "name": "TESTGEN_USERNAME", "value": "admin" }
         ],
         "secrets": [
           { "name": "TG_METADATA_DB_PASSWORD", "valueFrom": "<secret ARN>:TG_METADATA_DB_PASSWORD::" },
           { "name": "TESTGEN_PASSWORD", "valueFrom": "<secret ARN>:TESTGEN_PASSWORD::" },
           { "name": "TG_DECRYPT_SALT", "valueFrom": "<secret ARN>:TG_DECRYPT_SALT::" },
           { "name": "TG_DECRYPT_PASSWORD", "valueFrom": "<secret ARN>:TG_DECRYPT_PASSWORD::" },
           { "name": "TG_JWT_HASHING_KEY", "valueFrom": "<secret ARN>:TG_JWT_HASHING_KEY::" }
         ],
         "logConfiguration": {
           "logDriver": "awslogs",
           "options": {
             "awslogs-group": "/ecs/testgen",
             "awslogs-region": "<region>",
             "awslogs-stream-prefix": "setup"
           }
         }
       }
     ]
   }
   

3. Create the log group, and register both task definitions.

   aws logs create-log-group --log-group-name /ecs/testgen
   aws ecs register-task-definition --cli-input-json file://testgen-task.json
   aws ecs register-task-definition --cli-input-json file://testgen-setup-task.json
   

Set up the database

Run the setup task once, before the service's first start. It creates the datakitchen database and the initial system administrator account, and takes about two minutes.

aws ecs run-task --cluster <cluster> --launch-type FARGATE \
    --task-definition testgen-setup \
    --network-configuration "awsvpcConfiguration={subnets=[<private subnet IDs>],securityGroups=[<task security group ID>]}"

The setup container exits with code 0 and logs setup-system-db has successfully finished. when it succeeds.

Warning

Run the setup task only once. Against a database that is already set up, it stops with relation ... already exists. If the setup fails partway, drop the database before running it again.

DROP DATABASE IF EXISTS datakitchen;

After a successful install, deregister the setup task definition so that it cannot be run again by mistake.

aws ecs deregister-task-definition --task-definition testgen-setup:<revision>

Create the load balancer

TestGen serves the UI on port 8501 and the API and MCP server on port 8530. Create an Application Load Balancer that forwards each domain to its port.

To serve TestGen under a path on a domain shared with other applications, use the health check paths and path-based rules in Serve TestGen Through a Reverse Proxy instead of the health check paths in step 1 and the domain rules in step 2.

1. Create two target groups with target type ip, one for each port, with these health checks.

   Port	Health check path	Success codes
   8501	/_stcore/health	200
   8530	/api/v1/health	200

2. Create the load balancer in your VPC, with an HTTPS listener using your certificate. Add a rule that forwards <UI domain> to the 8501 target group and <API domain> to the 8530 target group.

3. Set the load balancer's idle timeout to 90 seconds.
4. Allow inbound traffic on ports 8501 and 8530 to the task security group from the load balancer's security group.
5. Create DNS records pointing the UI and API domains at the load balancer.

Create the service

Create the service with one task. TestGen does not support more than one task.

aws ecs create-service --cluster <cluster> --service-name testgen \
    --task-definition testgen --launch-type FARGATE --desired-count 1 \
    --deployment-configuration "minimumHealthyPercent=0,maximumPercent=100,deploymentCircuitBreaker={enable=true,rollback=false}" \
    --health-check-grace-period-seconds 180 \
    --network-configuration "awsvpcConfiguration={subnets=[<private subnet IDs>],securityGroups=[<task security group ID>]}" \
    --load-balancers \
        "targetGroupArn=<8501 target group ARN>,containerName=testgen,containerPort=8501" \
        "targetGroupArn=<8530 target group ARN>,containerName=testgen,containerPort=8530"

• minimumHealthyPercent=0,maximumPercent=100 stops the running task before starting its replacement, so two tasks never run at once. TestGen is unavailable for a few minutes during each deployment.
• rollback=false keeps ECS from restarting the previous version, which may not run on an upgraded application database.

When the targets are healthy, open https://<UI domain> in a browser and sign in as admin with the TESTGEN_PASSWORD value from the secret.

Tip

To configure email notifications and other settings, see Administer TestGen, and apply the settings as in Change settings. To set up single sign-on, see Enable single sign-on.

Upgrade TestGen

1. In the TestGen UI, check the Profiling Runs and Test Runs pages to ensure there are no actively running processes.
2. Copy the new version to ECR, as in Copy the image to Amazon ECR. For a list of recent releases, see TestGen Release Notes.

3. In testgen-task.json, set both image values to the new version, and register the new revision.

   aws ecs register-task-definition --cli-input-json file://testgen-task.json
   

4. Update the service to the new revision.

   aws ecs update-service --cluster <cluster> --service testgen --task-definition testgen
   

The new task upgrades the application database before TestGen starts.

Change settings

To set an environment variable, such as the email server settings in Configure Email Server:

1. Add the variable to the environment list of the testgen container in testgen-task.json. For a password or another sensitive value, add it to the secret instead, and reference it in the container's secrets list.
2. Register the new revision, and update the service to it, as in Upgrade TestGen.

Stop, start, and restart

# Stop
aws ecs update-service --cluster <cluster> --service testgen --desired-count 0

# Start
aws ecs update-service --cluster <cluster> --service testgen --desired-count 1

# Restart
aws ecs update-service --cluster <cluster> --service testgen --force-new-deployment

Read the logs

The task definition sends the output of each container to the /ecs/testgen log group in CloudWatch Logs. To save the last day of logs to a file, for example to send to support:

aws logs tail /ecs/testgen --since 24h > testgen-logs.txt

The application and audit log files in /var/lib/testgen/log are on the task's own storage, and are lost whenever the task is replaced. To keep them, for example to collect audit records, mount an Amazon EFS volume at /var/lib/testgen/log through an access point whose POSIX user is the container's testgen user (UID 100, GID 101), and whose root directory is created with that owner. See Collect Audit Records.

Provide files to the container

Some settings point to a file in the container, such as the RDS certificate bundle or the sso_auth.toml file for single sign-on. Provide these files on an Amazon EFS volume, mounted in each container that reads them.

Encrypt the connection to the application database

To require a verified TLS connection to RDS, add these variables to the migrate and testgen containers, and to the setup container if you have not installed yet.

{ "name": "TG_METADATA_DB_SSLMODE", "value": "verify-full" },
{ "name": "TG_METADATA_DB_SSLROOTCERT", "value": "/dk/certs/rds-ca.pem" }

The image's trust store does not include the RDS certificate authority. Provide the certificate bundle for your region at that path, in each of those containers.

Enable single sign-on

Create the sso_auth.toml file and set the variables as in Configure Single Sign-On. In the testgen container, provide the file at /dk/.streamlit/secrets.toml, and add the variables to the container's environment list.

Environment variables reference

Variable	Description
TG_METADATA_DB_HOST	Endpoint of the RDS instance.
TG_METADATA_DB_USER, TG_METADATA_DB_PASSWORD	The user created in Prepare the application database.
TG_DECRYPT_SALT, TG_DECRYPT_PASSWORD	Encrypt the credentials of the database connections you configure. Set once; never change.
TG_JWT_HASHING_KEY	Signs user sessions and access tokens. Base64-encoded random 32 bytes.
TESTGEN_USERNAME, TESTGEN_PASSWORD	The initial system administrator account, read only by the setup task. After installing, sign in with these credentials to create accounts for your team.
TG_INSTANCE_ID	A name that identifies this TestGen instance.
TG_UI_BASE_URL	The externally reachable URL of the UI. Include a path to serve the UI under it.
TG_BASE_URL	The externally reachable URL of the API and MCP server. Include a path to serve the API and MCP server under it.

For other settings, see Set Environment Variables, and apply them as in Change settings.

Related topics

• Install on Kubernetes