Access Requirements¶
Before connecting TestGen to your data, verify that it has the access it needs. This page lists the access required for each supported data source, whether that source is a database or a set of files.
How TestGen accesses your data¶
TestGen reads your data where it lives, and stores only computed results, such as aggregates, statistics, and test outcomes, in its own application database.
Database sources. TestGen pushes read-only SQL queries directly into your database to profile data, run tests, and monitor tables. It requires two types of read access:
- Schema metadata — TestGen queries system catalog views (such as
information_schema.columns) to discover tables, columns, and data types in your schema. - Table data — TestGen runs SELECT queries against your tables to compute profiling statistics, execute data quality tests, and collect monitor metrics.
TestGen never creates, modifies, or deletes tables or data in your database. No INSERT, UPDATE, DELETE, CREATE, ALTER, or DROP statements are executed against your database.
DuckDB has no database server or database user. TestGen opens the database file in read-only mode and runs the queries inside TestGen itself, so the file needs only operating system read access.
File data sources. TestGen reads your files using a query engine that runs inside TestGen itself rather than on a server of its own. What it needs to reach your files depends on where they are held. The files are never modified.
Runs on DuckDB connections and file data sources use TestGen's own memory and disk, and can keep a temporary copy of your data on TestGen's disk until the run ends. To plan for them, see Size TestGen for DuckDB and File Data Sources.
Databases¶
Grant the TestGen database user the following permissions. In all cases, only read access is required.
PostgreSQL and Amazon Aurora PostgreSQL¶
Authentication: Username and password.
| Permission | Purpose |
|---|---|
USAGE on target schema |
Access the schema containing your tables |
SELECT on tables in target schema |
Profile data, run tests, and collect monitor metrics |
SELECT on information_schema.columns, information_schema.tables |
Discover tables, columns, and data types |
SELECT on pg_catalog.pg_namespace, pg_catalog.pg_class |
Retrieve approximate row counts for profiling optimization |
Note
Access to pg_catalog is available to all database users by default. information_schema views are accessible to all users but only return metadata for objects the user has privileges on — the SELECT grant on your target tables is sufficient. No additional grants are typically needed for these.
Amazon Redshift¶
Authentication: Username and password.
| Permission | Purpose |
|---|---|
USAGE on target schema |
Access the schema containing your tables |
SELECT on tables in target schema |
Profile data, run tests, and collect monitor metrics |
SELECT on information_schema.columns |
Discover tables, columns, and data types |
SELECT on pg_catalog.pg_namespace, pg_catalog.pg_class |
Retrieve approximate row counts for profiling optimization |
Note
Access to pg_catalog is available to all database users by default. information_schema views are accessible to all users but only return metadata for objects the user has privileges on — the SELECT grant on your target tables is sufficient.
Amazon Redshift Spectrum¶
Authentication: Username and password.
| Permission | Purpose |
|---|---|
USAGE on external schema |
Access the external schema containing your tables |
SELECT on tables in external schema |
Profile data, run tests, and collect monitor metrics |
Note
Redshift Spectrum uses svv_external_columns for schema discovery instead of information_schema.columns. This view is visible to all users and returns metadata for external schemas the user has access to. Approximate row counts are not available for external tables.
Azure SQL Database¶
Authentication: Username and password, Azure Managed Identity, or Entra ID Service Principal.
| Permission | Purpose |
|---|---|
SELECT on tables in target schema |
Profile data, run tests, and collect monitor metrics |
SELECT on information_schema.columns |
Discover tables, columns, and data types |
SELECT on sys.objects, sys.partitions |
Retrieve approximate row counts for profiling optimization |
Note
No additional permissions are needed beyond SELECT on the target tables. information_schema views and sys.partitions are accessible to all database users by default. Metadata in sys.objects is automatically visible for tables the user has SELECT access to.
Azure Synapse Analytics¶
Authentication: Username and password, Azure Managed Identity, or Entra ID Service Principal.
| Permission | Purpose |
|---|---|
SELECT on tables in target schema |
Profile data, run tests, and collect monitor metrics |
SELECT on information_schema.columns |
Discover tables, columns, and data types |
SELECT on sys.objects, sys.partitions |
Retrieve approximate row counts for profiling optimization |
Databricks SQL¶
Authentication: Personal Access Token, or Service Principal with client ID and client secret.
| Permission | Purpose |
|---|---|
USE CATALOG on target catalog |
Access any object within the catalog |
USE SCHEMA on target schema |
Access the schema containing your tables |
SELECT on tables in target schema |
Profile data, run tests, and collect monitor metrics |
CAN USE on the SQL warehouse |
Execute queries using the compute resource specified in the connection |
Note
These permissions apply to workspaces using Unity Catalog. Access to information_schema.columns (used for table and column discovery) is automatic when the user has USE CATALOG and table-level privileges — no separate grant is needed.
DuckDB¶
Authentication: None. DuckDB is an embedded database — TestGen reads a .duckdb file directly instead of connecting to a server, so the connection has no host, port, or credentials. The only setting is the Database File path.
| Requirement | Purpose |
|---|---|
Read access to the .duckdb file for the operating system user TestGen runs as |
Discover tables and columns, profile data, run tests, and collect monitor metrics |
| No other process holding the file open for writing | DuckDB allows a single writer at a time and refuses to open a file while a writer holds it |
| Files read by views in the database kept in the database file's directory or below it | Query views over CSV, Parquet, or JSON files. TestGen reads no files outside that directory |
On a Docker or Kubernetes installation, the database file path is resolved inside the TestGen container. Mount the directory holding your database file into the container, and enter the path as it appears there.
Note
TestGen attaches your database file in read-only mode, so it cannot write to the file even if the operating system permissions would allow it.
Google BigQuery¶
Authentication: Service account key (JSON file).
| Permission | Purpose |
|---|---|
roles/bigquery.dataViewer on the target dataset |
Read table data, metadata, and dataset-level INFORMATION_SCHEMA views |
roles/bigquery.jobUser on the project |
Run query jobs |
Note
TestGen queries INFORMATION_SCHEMA.COLUMNS and the __TABLES__ metadata view to discover tables and estimate row counts. The dataViewer role grants access to both.
Microsoft SQL Server¶
Authentication: Username and password.
| Permission | Purpose |
|---|---|
SELECT on tables in target schema |
Profile data, run tests, and collect monitor metrics |
SELECT on information_schema.columns |
Discover tables, columns, and data types |
SELECT on sys.objects, sys.partitions |
Retrieve approximate row counts for profiling optimization |
Note
No additional permissions are needed beyond SELECT on the target tables. information_schema views and sys.partitions are accessible to all database users by default. Metadata in sys.objects is automatically visible for tables the user has SELECT access to.
Microsoft OneLake (Microsoft Fabric)¶
Authentication: Entra ID Service Principal or Managed Identity. SQL logins are not supported by the Fabric SQL analytics endpoint.
| Permission | Purpose |
|---|---|
| Contributor (or higher) on the Fabric workspace | Access the Lakehouse SQL analytics endpoint and read metadata + table data |
| Tenant setting Service principals can use Fabric APIs — Enabled | Required for Service Principal authentication (Fabric admin portal) |
Note
The Fabric SQL analytics endpoint is read-only by design and derives its schema and data from the underlying Lakehouse or Warehouse. Grant workspace-level access — item-level permissions are not required for a workspace member with the Contributor role.
Oracle Database¶
Authentication: Username and password.
| Permission | Purpose |
|---|---|
SELECT on tables in target schema |
Profile data, run tests, and collect monitor metrics |
SELECT on ALL_TAB_COLUMNS |
Discover tables, columns, and data types |
SELECT on ALL_TABLES |
Retrieve table metadata and approximate row counts |
Note
ALL_TAB_COLUMNS and ALL_TABLES are accessible to any user with the CREATE SESSION privilege and automatically return metadata only for objects the user can access. No additional grants are needed for these views — the SELECT grant on your target tables is sufficient.
Salesforce Data 360¶
Authentication: JWT Bearer flow or Client Credentials flow against a Salesforce external client app.
| OAuth scope | Purpose |
|---|---|
Access Data Cloud Query API (cdp_query_api) |
Profile data, run tests, and collect monitor metrics |
Manage user data via APIs (api) |
Issue the core Salesforce access token that gets exchanged for a Data 360 token |
Perform requests at any time (refresh_token, offline_access) |
Renew the Data 360 access token between long-running operations |
Note
Grant the runtime Salesforce user (the JWT "subject" or the Client Credentials "Run As" user) read access to the target tables.
SAP HANA¶
Authentication: Username and password.
| Permission | Purpose |
|---|---|
SELECT on tables in target schema |
Profile data, run tests, and collect monitor metrics |
SELECT on SYS.TABLE_COLUMNS |
Discover tables, columns, and data types |
SELECT on SYS.M_TABLES |
Retrieve table metadata and row counts |
Note
Access to SYS.TABLE_COLUMNS requires explicit SELECT grants. SYS.M_TABLES provides real-time row counts for in-memory tables. Grant the TestGen user SELECT on both views in addition to the target schema tables.
Snowflake¶
Authentication: Key-pair authentication (recommended), or username and password.
| Permission | Purpose |
|---|---|
USAGE on warehouse, database, and schema |
Access the compute and storage resources |
SELECT on tables in target schema |
Profile data, run tests, and collect monitor metrics |
Warning
Snowflake is deprecating single-factor password authentication — see Snowflake's MFA enforcement rollout plan. New connections should use key-pair authentication. Existing password-based connections continue to work for now but should be migrated before Snowflake disables password sign-in for your account.
Note
information_schema views (used to discover tables, columns, data types, and approximate row counts) are accessible to any role with USAGE on the database — no separate grants are needed. These views automatically return metadata only for objects the role can access.
File data sources¶
Give TestGen access to the location that holds your files. In all cases, only read access is required.
Local Files¶
Authentication: None. A Local Files connection reads CSV, Parquet, and JSON files from a directory, so it has no host, port, or credentials. The only setting is the Base Path of that directory, given as an absolute path.
| Requirement | Purpose |
|---|---|
| Read access to the base path and the files under it, for the operating system user TestGen runs as | Infer columns, profile data, run tests, and collect monitor metrics |
On a Docker or Kubernetes installation, the base path is resolved inside the TestGen container. Mount the directory holding your files into the container, and enter the path as it appears there.
Warning
Mount the directory read-only. A Custom Test runs SQL that you supply, and TestGen cannot restrict what that SQL writes inside the base path. In Docker, append :ro to the volume. In Kubernetes, set readOnly: true on the volume mount.
Amazon S3¶
Authentication: Access key ID and secret access key, or IAM role.
| Permission | Purpose |
|---|---|
s3:ListBucket on the bucket, for the base URI prefix |
Find the files a pattern matches |
s3:GetObject on the objects under the base URI |
Infer columns, profile data, run tests, and collect monitor metrics |
Note
On a Docker or Kubernetes installation that uses an EC2 instance role, set the instance metadata hop limit to 2. The default of 1 does not reach the container.
Warning
Grant read-only access. A Custom Test runs SQL that you supply, and TestGen cannot restrict what that SQL writes if the credentials allow writing.
Azure Blob Storage¶
Authentication: Storage account key, Azure Managed Identity, or Entra ID Service Principal.
| Permission | Purpose |
|---|---|
| Storage Blob Data Reader on the container | Find the files a pattern matches, infer columns, profile data, run tests, and collect monitor metrics |
Warning
Grant read-only access. A Custom Test runs SQL that you supply, and TestGen cannot restrict what that SQL writes if the credentials allow writing. An account key always allows writing, so prefer a managed identity or service principal.
Security considerations¶
- For a database source, best practice is to create a dedicated read-only database user for TestGen, rather than reusing an application or admin account.
- TestGen only needs access to the data you want to profile and test. You can restrict it to specific schemas, or point a file connection at a directory, prefix, or container that holds only the files you intend to test, rather than granting broad read access.