Skip to content

Profiling Characteristics

This reference lists the 60 column-level characteristics that TestGen captures during a profiling run. Use it to understand what a specific statistic measures and how it can help identify data quality concerns.

Column characteristic Description Use case example
25th Percentile For numeric values. The boundary of the 25th percentile. Confirm consistent distribution and outlier detection.
75th Percentile For numeric values. The boundary of the 75th percentile. Confirm consistent distribution and outlier detection.
Average Embedded Spaces The average number of embedded spaces in string values. Identify format changes and potential column parse errors in string values.
Average Length The average length of data in the alpha column. Change in data representation.
Average Value The average numeric value in the column. Shift in the expected average.
Before 1 Year The count of records with date values earlier than one year before the current date. Identify transactional versus historical dates (for example, ship dates versus birthdays).
Before 20 Years The count of records with date values earlier than twenty years before the current date. Similar to the 1yr but with a wider timeframe.
Before 5 Years The count of records with date values earlier than five years before the current date. Similar to the 1yr but with a wider timeframe.
Column Name The name of the column being profiled.
Column Type The data type as defined in the table.
Data Type Suggestion Automatically derived data type based on the data present. Refine data types for better ingestion constraints.
Date Values Count of records with valid date values in the alpha column. Confirm date type and expected ratio.
Days Present Count of distinct days (YYYY-MM-DD) present in date values in a column. Identify changes in date coverage by day.
Distinct Patterns Count of distinct patterns in string data, with alphas shown as A and numeric characters as N. Identify columns with predefined formats.
Distinct Standard Values Count of distinct values in string data transformed to upper case with spaces dropped. Identify potential duplicates formatted differently.
Distinct Value Hash The hash of distinct values present in a column. Identify changes in distinct values present.
Distinct Values Investigate distinct values present in the column. Identify columns with potential domain or coded values, standard categories.
Dummy Values Count of records with miscellaneous potential representations of blank value in the alpha column, exclusive of other potential blank counts. Consistent blank value representation.
Embedded Spaces Count of records with embedded spaces in string values. Identify format changes and potential column parse errors in string values.
Fractional Sum The sum of fractional values in a column Assess decimal truncation.
Frequent Patterns Most common patterns in an alpha column and their counts. Consistent formatting (for example, phone numbers).
Frequent Values Most common values in an alpha column and their counts. Change in most frequent values.
Functional Data Type A more detailed data type category. Reporting and helps in identifying quality control testing.
Functional Table Type A more detailed table type that goes beyond SQL types. Reporting and helps in identifying quality control testing.
Future Dates Count of records with date values beyond the current date. Identify the ratio of scheduled future date values.
General Type Broad data type: (A)lpha, (N)umeric, (D)ate, (B)oolean. A broad category for creating test types.
Includes Digits Count of records with values that contain at least one digit (0-9) in an alpha column. Classify columns with mixed alphanumeric content, such as IDs and codes.
Leading Spaces Count of records with leading space in alpha column. Potential data parsing errors.
Lower Case Count of records with values in all lower case in an alpha column. Consistent letter case.
Maximum Date The maximum date value in a column. Expected range of values.
Maximum Length The maximum length of data in an alpha column. Change in data reporting.
Maximum Text The maximum text value in a column. Expected range of values.
Maximum Value The maximum numeric value in a column. Expected range of values.
Median Value For numeric values. The boundary of the 50th percentile. Confirm consistent distribution and outlier detection.
Minimum Date The minimum date value in a column. Expected range of values.
Minimum Length The minimum length of data in an alpha column. Change in data representation.
Minimum Text The minimum text value in a column. Expected range of values.
Minimum Value The minimum numeric value in a column Expected range of values.
Minimum Value > 0 The minimum numeric value is greater than zero in a column. Expected range of values.
Mixed Case Count of records with values that mix upper and lower case in an alpha column. Consistent letter case.
Months Present Count of distinct months (YYYY-MM) present in date values in a column. Identify changes in date coverage by month.
Non-Alpha Count of records with values that contain no letters in an alpha column. Identify codes, numbers, or symbols in a text column.
Non-Printing Characters Count of records with values that contain non-printing characters, such as non-breaking or zero-width spaces, in an alpha column. Potential matching and parsing errors from invisible characters.
Null Values Count of null values in a column, exclusive of other potential blank counts. Consistent blank value representation and the ratio of null values.
Numeric Values Count of records with valid numeric values in the alpha column. Confirm data type and count.
PII Flag A classification flag identifying Personally Identifiable Information by pattern, column name, and data characteristics. Categorized as high risk (passwords, SSNs, tax IDs), moderate risk (names, addresses, phone numbers, emails), or low risk (user IDs, routing numbers). Identify sensitive data that may require security controls and privacy compliance.
Position The ordinal position of a column in a table. Data consistency.
Quoted Values Count of records with values enclosed in quotes (single or double) in an alpha column. Potential ingestion or processing errors where values were improperly escaped.
Row Count Total count of records in a table. Record keeping and denominator for different ratios.
Standard Deviation The standard deviation of numeric values in a column. For calculating outlying values per standard deviation from the average.
Standard Pattern Match Tag column contents based on pre-defined REGEX pattern matches for 90+% of values. Detailed info about column contents.
Table Name Name of the table being profiled. Data consistency.
True Count Count of records defined as boolean that equates to the True value. The expected ratio of boolean True.
Upper Case Count of records with values in all upper case in an alpha column. Consistent letter case.
Value Count Count of records with values present in a column. Shift in the ratio of missing value.
Weeks Present Count of distinct weeks (YYYY-WW) present in date values in a column. Identify changes in date coverage by week.
Within 1 Month Count of records with date values within one month before the current date. Expected recency of date values.
Within 1 Year Count of records with date values within one year of the current date. Expected recency of date values.
Zero Length Count of records with zero length (blank) in an alpha column, exclusive of other potential blank counts. Consistent blank value representation.
Zero Values Count of records with zero values in alpha or numeric column, exclusive of other potential blank counts. Consistent blank value representation and the ratio of zero values.