Data Quality Testing¶
Use data quality testing to verify that your data meets expected standards before it reaches downstream consumers. TestGen runs tests against your data and produces results you can review, confirm, or dismiss β giving you an ongoing record of data quality over time.
What testing checks¶
TestGen includes 83 test types, plus 12 Commercial Pharma tests in TestGen Enterprise, spanning six quality dimensions: accuracy, completeness, consistency, timeliness, uniqueness, and validity. Tests range from simple checks (are required values present? do zip codes match a valid format?) to statistical comparisons (has the average shifted significantly from baseline? has the distribution changed?).
Each test targets a specific column or table and evaluates it against the thresholds and expected values set on the test. For the full catalog of test types, see TestGen Test Types. For business rules that no built-in test type covers, see Custom Tests.
How testing works¶
Testing is organized around test suites, test definitions, and test runs:
- Test suite β A named collection of tests associated with a table group. You generate tests into a suite, and each execution of the suite produces a test run. A contract is a type of test suite that also keeps its tests in versions and holds a contract document that you can import and export as an ODCS file. See Test suite types.
- Test definition β An individual test within a suite, specifying the test type, target table and column, thresholds, and parameters. Definitions can be auto-generated from profiling results or created manually.
- Test run β A single execution of the active tests in a suite, or in one version of a contract. Each run produces a set of results with one of five statuses:
- Passed β The data meets the test criteria.
- Warning β The data does not meet the test criteria, but the test severity is set to warn rather than fail.
- Failed β The data does not meet the test criteria.
- Error β The test could not execute (for example, a missing table or permission issue).
- Log β The test's severity is Log. The result is recorded without passing or failing, and it does not affect the Testing Score.
How testing supports your data quality workflow¶
Testing connects to several other features in TestGen:
- Profiling β Profiling discovers the characteristics of your data, and test generation uses those characteristics to create tests with appropriate thresholds.
- Quality scores β Test results contribute to the Testing Score on the Quality Dashboard, giving you a high-level view of data quality across your project.
- Notifications β Set up alerts to be notified when test runs complete or when failures are detected. See Notifications for configuration details.
- Observability β Export test results to DataKitchen's DataOps Observability for centralized monitoring. See Connect to Observability.
When to run tests¶
Best practice is to run tests on a regular schedule β for example, weekly or after each data pipeline refresh β so that issues are detected close to when they occur. You can schedule test runs to execute automatically during off-hours, with results ready for you to review during working hours.
Regenerate tests every few weeks or when your data has evolved, so that test criteria stay aligned with current data characteristics.
Getting started¶
To start testing, you need a test suite with generated or manually defined tests. See Run Tests for execution steps, and Investigate Test Results to review your results and take action on findings.
Related topics¶
- Generate Tests β populate a test suite from profiling results
- TestGen Test Types β the catalog of test types
- Quality Scores β how test results roll up into scores