Accuracy Tests¶
Accuracy tests evaluate whether data values faithfully represent the real-world quantities or states they are intended to model. These tests compare column values, aggregate measures, and statistical properties against known references, baselines, or expected distributions to detect drift, imbalance, and anomalies.
New Shift¶
Tests for a statistically significant shift in the mean of new values compared to the average calculated at baseline.
Auto-generated for numeric measurement columns with more than 10 distinct values.
| Scope | Column |
| Measures | Z-score of mean shift (2.0 = moderate, 3.0 = strong) |
| Threshold | Maximum acceptable Z-score (default: 2) |
| On failure | Significant shift in average of new values compared to baseline average. |
| Default severity | Warning |
How it works: Calculates the average of values added since the baseline snapshot and compares it to the baseline average using the baseline standard deviation. The result is expressed as a Z-score — the number of standard deviations the new mean falls from the baseline mean. The test fails when this Z-score meets or exceeds the threshold.
When to use: This test is more sensitive than a simple average comparison because it isolates only the values added since the baseline. It is appropriate for cumulative datasets where earlier records remain present. A failure may indicate a legitimate shift in incoming data that warrants business review, or a data quality issue. Consider pairing it with Variability Increase — if variability also rises, process or measurement flaws may be involved; if variability holds steady, the issue is more likely in the source data itself.
Outliers Above¶
Tests that the percentage of values more than two standard deviations above the baseline mean does not exceed a threshold.
Auto-generated for continuous measurement columns with more than 30 distinct values and meaningful variance.
| Scope | Column |
| Measures | Percentage of records over the upper limit |
| Threshold | Maximum acceptable percentage of outliers (default: 0.05) |
| On failure | Percent of outliers exceeding two standard deviations above the mean is greater than expected threshold. |
| Default severity | Warning |
How it works: Counts records where the column value exceeds the baseline mean plus two standard deviations, then divides by the total non-null count. The test fails when this percentage exceeds the threshold.
When to use: Under a normal distribution, roughly 2.3% of values are expected to fall more than two standard deviations above the mean. The default threshold of 5% allows headroom for non-normal distributions. If you observe a stable outlier rate over time, adjust the threshold to match. This test uses the baseline mean rather than the current mean, so it captures systemic upward shifts as well as individual outliers.
Outliers Below¶
Tests that the percentage of values more than two standard deviations below the baseline mean does not exceed a threshold.
Auto-generated for continuous measurement columns with more than 30 distinct values and meaningful variance.
| Scope | Column |
| Measures | Percentage of records under the lower limit |
| Threshold | Maximum acceptable percentage of outliers (default: 0.05) |
| On failure | Percent of outliers exceeding two standard deviations below the mean is greater than expected threshold. |
| Default severity | Warning |
How it works: Counts records where the column value falls below the baseline mean minus two standard deviations, then divides by the total non-null count. The test fails when this percentage exceeds the threshold.
When to use: This test mirrors Outliers Above but watches for an excess of unusually low values. It is useful for detecting measurement floors, truncation errors, or downward drift. As with Outliers Above, the baseline mean anchors the comparison so that gradual shifts are caught alongside individual anomalies.
Variability Increase¶
Tests that the standard deviation of column values has not increased significantly compared to baseline.
Auto-generated for numeric measurement columns with more than 10 distinct values and non-zero standard deviation.
| Scope | Column |
| Measures | Percentage of baseline standard deviation (100% = unchanged) |
| Threshold | Maximum acceptable percentage of baseline standard deviation (default: 120) |
| On failure | The standard deviation of the measure has increased beyond the defined threshold. |
| Default severity | Warning |
How it works: Computes the current standard deviation, divides it by the baseline standard deviation, and expresses the result as a percentage. The test fails when this percentage exceeds the threshold.
When to use: A rising standard deviation signals that values are becoming more spread out — possibly due to new outliers, a more heterogeneous population, or measurement problems. Consider this test alongside New Shift: if both the average and variability shift, the underlying data population may have changed fundamentally. If only variability rises while the average holds, the issue is more likely a data quality or collection problem.
Variability Decrease¶
Tests that the standard deviation of column values has not decreased significantly compared to baseline.
Auto-generated for numeric measurement columns with more than 10 distinct values and non-zero standard deviation.
| Scope | Column |
| Measures | Percentage of baseline standard deviation (100% = unchanged) |
| Threshold | Minimum acceptable percentage of baseline standard deviation (default: 80) |
| On failure | The standard deviation of the measure has decreased below the defined threshold. |
| Default severity | Warning |
How it works: Computes the current standard deviation, divides it by the baseline standard deviation, and expresses the result as a percentage. The test fails when this percentage falls below the threshold.
When to use: A falling standard deviation could indicate improved precision, a more homogeneous cohort, or the removal of legitimate outliers. However, it can also signal data truncation or a narrowed collection scope. This test helps you notice when a measure becomes suspiciously uniform compared to its historical behavior.
Custom Test¶
A flexible business-rule test covering any error condition that can be expressed as a SQL query against one or more tables in the database.
This test is not auto-generated. You can create it manually from the Test Definitions page.
| Scope | Custom |
| Measures | Count of error rows returned by the query |
| Threshold | Maximum acceptable error count (default: 0) |
| On failure | Errors were detected according to test definition. |
| Default severity | Fail |
How it works: Runs a user-defined SQL query and counts the rows returned. Each row represents one error condition. The test passes when the query returns no rows (or fewer rows than the threshold). The query results are displayed when you review source data for a failed test.
When to use: Use this test for business rules that cannot be expressed by the built-in test types. Your query can reference any tables in the database and can test row-level conditions (such as a join mismatch) or aggregate conditions (such as a balance that should sum to zero). Include enough detail columns in your query results to support follow-up investigation.