Skip to content

1 December 2023 Release Notes

Version 1.1.2

Feature highlights

Note

DataOps TestGen now checks for 25 different types of profiling anomalies and offers 40 unique data quality tests.

An improved Quick Start

We've expanded the DataOps TestGen Quick Start to provide more realistic examples of how TestGen can help you better understand the realities of your data.

  • With --simulate-fast-forward, you can now perform an iteration of the Quick Start that reflects how TestGen tests data changes over time.
  • We've included new demo data that demonstrates how database flaws, issues, and inconsistencies can be captured by the profiling and testing capabilities of DataOps TestGen.

Tip

Want to get started now? Go to DataOps TestGen Quick Start to learn more.

New profiling anomalies

Two new profiling anomaly tests have been introduced. TestGen can now alert you when your data includes:

  • Invalid USA Zip-3 formats.
  • Delimited data embedded in a column.

See Profiling Anomaly Suggestions for details, and you can address these anomalies.

New test types

Two new test types have been added to TestGen: Variability decrease and variability increase. For more information on these tests, as well as other tests with a similar data quality dimension, see Accuracy Tests.

Introducing source drill-downs

You will now find the option Review Source Data on both the Profiling Anomalies and Test Results pages. When you select an anomaly or test results, TestGen provides an aggregate summary of the data that triggered the anomaly or test.

For more information on what you can learn from anomalies and test results, see View Profiling Anomalies and View Test Results.

Refined Configuration options

To reduce noise, the Configuration page now provides a space where you can:

  • Create or update projects.
  • View and manage test definitions.

All settings related to table group and test suite maintenance have been related to the new-and-improved Connections flow.

New-and-improved Connections flow

To better reflect the TestGen touchpoints and TestGen entity hierarchy, we've updated the user experience to create and manage connections, table groups, and test suites.

  • A new Connections page shows, at-a-glance the different database connections in a project.
    • The new Add Connection dialog simplifies database connection creation and management.
  • The Table Group button on the Connections page helps you manage the table groups related to a connection.
    • In the Add Table Group dialog, the Preview tab allows you to view a table group as you configure it, while the new Test button provides immediate feedback on the viability of the table group.
  • The Test Suites button on the Table Groups page lets you create and maintain test suites specific to your various table groups.

Plus, in a single click, copy pre-configured CLI commands to run profiling, generate tests, or run tests. For complete details on how to make new database connections and other TestGen entities, start with Create a Connection.

Test the connection between Observability and TestGen

From the Configuration > Project Info page, click Test Observability Export to verify the connection between your TestGen and Observability projects. Once established, include the export-observability command as part of your test runs to send test results to DataOps Observability.

Other fixes and enhancements

  • Confirm and Dismiss test actions cannot be performed on Passed tests. [TG-369]
  • It is confirmed that all system timezones adhere to UTC. [TG-411]
  • When run, the export-observability command now states how many tests have been sent to Observability. [TG-469]
  • We've updated the order of menu items to better represent the TestGen touchpoints and to improve access to your daily tasks. [TG-540]
  • We applied general fixes to reduce errors and improve test efficiencies.
  • We made various improvements to our user experience. Our help documentation has been updated to reflect these changes.