Accuracy Tests¶
Accuracy tests evaluate whether data values faithfully represent the real-world quantities or states they are intended to model. These tests compare column values, aggregate measures, and statistical properties against known references, baselines, or expected distributions to detect drift, imbalance, and anomalies.
New Shift¶
Tests for a statistically significant shift in the mean of new values compared to the average calculated at baseline.
Auto-generated for numeric measurement columns with more than 10 distinct values.
| Scope | Column |
| Measures | Z-score of mean shift (2.0 = moderate, 3.0 = strong) |
| Threshold | Maximum acceptable Z-score (default: 2) |
| On failure | Significant shift in average of new values compared to baseline average. |
| Default severity | Warning |
How it works: Calculates the average of values added since the baseline snapshot and compares it to the baseline average using the baseline standard deviation. The result is expressed as a Z-score — the number of standard deviations the new mean falls from the baseline mean. The test fails when this Z-score meets or exceeds the threshold.
When to use: This test is more sensitive than a simple average comparison because it isolates only the values added since the baseline. It is appropriate for cumulative datasets where earlier records remain present. A failure may indicate a legitimate shift in incoming data that warrants business review, or a data quality issue. Consider pairing it with Variability Increase — if variability also rises, process or measurement flaws may be involved; if variability holds steady, the issue is more likely in the source data itself.
Outliers Above¶
Tests that the percentage of values more than two standard deviations above the baseline mean does not exceed a threshold.
Auto-generated for continuous measurement columns with more than 30 distinct values and meaningful variance.
| Scope | Column |
| Measures | Percentage of records over the upper limit |
| Threshold | Maximum acceptable percentage of outliers (default: 0.05) |
| On failure | Percent of outliers exceeding two standard deviations above the mean is greater than expected threshold. |
| Default severity | Warning |
How it works: Counts records where the column value exceeds the baseline mean plus two standard deviations, then divides by the total non-null count. The test fails when this percentage exceeds the threshold.
When to use: Under a normal distribution, roughly 2.3% of values are expected to fall more than two standard deviations above the mean. The default threshold of 5% allows headroom for non-normal distributions. If you observe a stable outlier rate over time, adjust the threshold to match. This test uses the baseline mean rather than the current mean, so it captures systemic upward shifts as well as individual outliers.
Outliers Below¶
Tests that the percentage of values more than two standard deviations below the baseline mean does not exceed a threshold.
Auto-generated for continuous measurement columns with more than 30 distinct values and meaningful variance.
| Scope | Column |
| Measures | Percentage of records under the lower limit |
| Threshold | Maximum acceptable percentage of outliers (default: 0.05) |
| On failure | Percent of outliers exceeding two standard deviations below the mean is greater than expected threshold. |
| Default severity | Warning |
How it works: Counts records where the column value falls below the baseline mean minus two standard deviations, then divides by the total non-null count. The test fails when this percentage exceeds the threshold.
When to use: This test mirrors Outliers Above but watches for an excess of unusually low values. It is useful for detecting measurement floors, truncation errors, or downward drift. As with Outliers Above, the baseline mean anchors the comparison so that gradual shifts are caught alongside individual anomalies.
Variability Increase¶
Tests that the standard deviation of column values has not increased significantly compared to baseline.
Auto-generated for numeric measurement columns with more than 10 distinct values and non-zero standard deviation.
| Scope | Column |
| Measures | Percentage of baseline standard deviation (100% = unchanged) |
| Threshold | Maximum acceptable percentage of baseline standard deviation (default: 120) |
| On failure | The standard deviation of the measure has increased beyond the defined threshold. |
| Default severity | Warning |
How it works: Computes the current standard deviation, divides it by the baseline standard deviation, and expresses the result as a percentage. The test fails when this percentage exceeds the threshold.
When to use: A rising standard deviation signals that values are becoming more spread out — possibly due to new outliers, a more heterogeneous population, or measurement problems. Consider this test alongside New Shift: if both the average and variability shift, the underlying data population may have changed fundamentally. If only variability rises while the average holds, the issue is more likely a data quality or collection problem.
Variability Decrease¶
Tests that the standard deviation of column values has not decreased significantly compared to baseline.
Auto-generated for numeric measurement columns with more than 10 distinct values and non-zero standard deviation.
| Scope | Column |
| Measures | Percentage of baseline standard deviation (100% = unchanged) |
| Threshold | Minimum acceptable percentage of baseline standard deviation (default: 80) |
| On failure | The standard deviation of the measure has decreased below the defined threshold. |
| Default severity | Warning |
How it works: Computes the current standard deviation, divides it by the baseline standard deviation, and expresses the result as a percentage. The test fails when this percentage falls below the threshold.
When to use: A falling standard deviation could indicate improved precision, a more homogeneous cohort, or the removal of legitimate outliers. However, it can also signal data truncation or a narrowed collection scope. This test helps you notice when a measure becomes suspiciously uniform compared to its historical behavior.
Custom Test¶
A flexible business-rule test covering any rule that can be checked by a SQL query against one or more tables in the database.
This test is not auto-generated. To add it, see Add Custom SQL Tests.
| Scope | Custom |
| Measures | Count of failing records returned by the query |
| Threshold | Maximum acceptable count of failing records (default: 0) |
| On failure | Records failed the test definition. |
| Default severity | Fail |
How it works: Runs a user-defined SQL query and counts the rows returned. Each row represents one failing record. The test passes when the query returns no more rows than the threshold. The query results are displayed when you review source data for a failed test.
When to use: Use this test for business rules that cannot be expressed by the built-in test types. Your query can reference any tables in the database and can test row-level conditions (such as a join mismatch) or aggregate conditions (such as a balance that should sum to zero). Include enough detail columns in your query results to support follow-up investigation.
SQL Metric¶
Tests that the single value returned by a query falls within set bounds.
This test is not auto-generated. To add it, see Add Custom SQL Tests.
| Scope | Table |
| Measures | Value returned by the query |
| Threshold | Minimum value, maximum value, or both |
| On failure | Query result outside expected bounds. |
| Default severity | Fail |
How it works: Runs a user-defined SQL query that returns one row with one numeric column, and compares that value to the minimum and maximum. Leave either bound blank to check one side only. The test fails when the value is below the minimum or above the maximum. A query that returns more than one row or column, a value that is not numeric, or no value at all ends the test with the Error status rather than Failed.
When to use: Use this test for any metric a query can compute when the acceptable range is known in advance, such as the number of orders awaiting shipment or the hours since the last load. The query can refer to any tables in the database. Hard-code the schema, or use {DATA_SCHEMA} for the schema defined for the table group, for example SELECT COUNT(*) AS late_orders FROM {DATA_SCHEMA}.orders WHERE shipped_date IS NULL. To count rows that break a rule rather than check a single value, use Custom Test.
Aggregate Minimum¶
Tests that aggregate values per grouping in the source table are at least as large as those in the reference table.
This test is not auto-generated. You can create it manually from the Test Definitions page.
| Scope | Referential |
| Measures | Count of groups below reference value |
| Threshold | Expected count of group totals below reference aggregate (default: 0) |
| On failure | Aggregate measure per set of column values is not at least the same as reference dataset. |
| Default severity | Fail |
How it works: Rolls up an aggregate expression by grouping columns in both tables, then checks that each group's value in the source table meets or exceeds the reference value. Groups where the source value is lower count as failures.
When to use: Use this test to confirm that aggregate values have not dropped between an older and newer version of a cumulative dataset — values may rise but should never fall. Both tables must be accessible at the same time.