ODCS Mapping Reference¶

Use this page when you write a contract file for TestGen to import, or read a document that TestGen exported. It lists how a contract's tests appear in an exported ODCS document, which rules in an imported document become tests, and how TestGen writes server entries and physical types. See TestGen Test Types for what each test checks.

How tests appear in an exported document¶

Each active test in the exported version becomes a quality rule or a column declaration, except Schema Conformance tests, which do not appear. Every rule TestGen exports carries its test type in a customProperties entry under the testgen vendor, and imports back as the same test.

Library rules¶

These test types export as library rules with unit: rows, except Percent Missing, which has unit: percent.

Test type	Metric	Operator	Applies to
Constant Match	`invalidValues` with `validValues`	`mustBeLessOrEqualTo`	Column
Duplicate Rows	`duplicateValues`	`mustBeLessOrEqualTo`	Column, or the table with its grouping columns in `properties`
Email Format	`invalidValues` with `pattern`	`mustBeLessOrEqualTo`	Column
Match Null Count	`nullValues`	`mustBe`	Column
Match Row Count	`rowCount`	`mustBe`	Table
Pattern Match	`invalidValues` with `pattern`	`mustBeLessOrEqualTo`	Column
Percent Missing	`nullValues`	`mustBeBetween`	Column
Required Entry	`nullValues`	`mustBeLessOrEqualTo`	Column
Required, No Placeholder	`missingValues` with `missingValues`	`mustBeLessOrEqualTo`	Column
Row Count	`rowCount`	`mustBeGreaterOrEqualTo`	Table
Row Range	`rowCount`	`mustBeBetween`	Table
US State	`invalidValues` with `validValues`	`mustBeLessOrEqualTo`	Column
Valid US Zip, Valid US Zip-3, Valid US/Canada Zip	`invalidValues` with `pattern`	`mustBeLessOrEqualTo`	Column
Value Match	`invalidValues` with `validValues`	`mustBeLessOrEqualTo`	Column

A test exports as a custom rule instead when the library rule cannot state exactly what it checks. For example, Constant Match exports as a library rule only when its value is a single literal of the column's logical type.

SQL Metric exports as a sql rule with the test's query and its bounds.

Column declarations¶

A test that allows no exceptions, with a threshold of 0 or none, and has no subset condition exports as a declaration on its column instead of a rule, when the column declares one of the logical types listed.

Test type	Declares	Logical types
Maximum Date	`logicalTypeOptions.maximum`	`date`, `timestamp`
Maximum Value	`logicalTypeOptions.maximum`	`integer`, `number`
Minimum Date	`logicalTypeOptions.minimum`	`date`, `timestamp`
Minimum Value	`logicalTypeOptions.minimum`	`integer`, `number`
Required Entry	`required`	Any
Text Length Range	`logicalTypeOptions.minLength`, `logicalTypeOptions.maxLength`	`string`
Unique Values	`unique` and `required`	Any

Custom rules¶

Every other test type exports as a custom rule with engine: testgen, which only TestGen runs. The rule's implementation holds the test's parameters.

Rules that import as tests¶

TestGen finds a rule's table or column by its physicalName, or by its name when it has none. A rule that TestGen exported, with its testgen test type entry, imports as the same test. Any other rule imports as a test only when it matches one of the shapes below exactly. A rule without type is a library rule, and a rule without unit is read as unit: rows.

Rule	Imports as
`nullValues` on a column, with `mustBeLessOrEqualTo`, `mustBeLessThan`, or `mustBe: 0`	Required Entry
`nullValues` on a column, with `mustBe` and a count above 0	Match Null Count
`rowCount` on a table, with `mustBeGreaterOrEqualTo` or `mustBeGreaterThan`	Row Count
`rowCount` on a table, with `mustBe`	Match Row Count
`invalidValues` on a column, with a `validValues` argument and an upper bound	Value Match
`invalidValues` on a column, with a `pattern` argument and an upper bound	Pattern Match
`duplicateValues` on a column, or on a table that lists its columns in `properties`, with an upper bound	Duplicate Rows
A `sql` rule whose query returns one value, with `mustBe`, `mustBeBetween`, `mustBeGreaterOrEqualTo`, or `mustBeLessOrEqualTo`	SQL Metric

In a sql rule, {object} and ${object} become the table, and {property} and ${property} become the column. TestGen runs the query as written on the contract's connection.

These column declarations also import as tests. The test takes over the declaration, so muting or deleting the test removes the declaration from the exported document.

Declaration	Logical types	Imports as
`logicalTypeOptions.minimum`, `logicalTypeOptions.maximum`	`integer`, `number`	Minimum Value, Maximum Value
`logicalTypeOptions.minimum`, `logicalTypeOptions.maximum`	`date`, `timestamp`	Minimum Date, Maximum Date
`logicalTypeOptions.minLength`, `logicalTypeOptions.maxLength`	`string`	Text Length Range
`unique` with `required`	Any	Unique Values
`required` alone	Any	Required Entry

Rules TestGen does not check¶

A quality rule that matches none of the shapes above is kept in the contract document and marked unverifiable. It is exported with the contract, but it never runs. These rules include:

Rules with a unit other than rows, such as percent.
missingValues rules without the testgen test type entry.
text rules, and custom rules without engine: testgen or the testgen test type entry.
sql rules with a strict or negated operator, such as mustBeGreaterThan or mustNotBe.
invalidValues rules with both validValues and pattern, or with neither.
Rules on nested properties.
relationships on a table or column.

A column declaration that no test type checks, such as unique without required, an exclusive bound, a pattern, or a format, stays in the document and is not checked.

Server entries by connection type¶

A contract's servers entry describes its connection. TestGen writes a typed entry when the connection type has a matching ODCS server type, and a custom entry otherwise.

Connection type	Server type
Amazon Redshift	`redshift`
Azure SQL Database	`sqlserver`
Azure Synapse Analytics	`synapse`
Databricks	`databricks`
DuckDB	`duckdb`
Google BigQuery	`bigquery`
Microsoft SQL Server	`sqlserver`
Oracle	`oracle`
PostgreSQL	`postgres`
SAP HANA	`hana`
Snowflake	`snowflake`
Amazon Redshift Spectrum, Amazon S3, Azure Blob Storage, Local Files, Microsoft OneLake, Salesforce Data 360	`custom`

A connection whose type has a server type also gets a custom entry when it does not supply a field that the server type requires. For example, a Google BigQuery connection without a project gets a custom entry. TestGen reports the reason when it creates the contract.

Physical types¶

On import, TestGen compares each declared physicalType with the type the data catalog records for the column, in the connection's SQL dialect.

A type that names the same type in another spelling, such as int4 for integer, is rewritten in the catalog's spelling. The original text is kept in the column's authoredPhysicalType custom property.
A type that TestGen cannot compare with the catalog's is removed, and kept in authoredPhysicalType. Schema Conformance then compares the column's logicalType instead.
A type that differs from the catalog's is left as written, and Schema Conformance reports the difference.