# DataKitchen Documentation
> Documentation for DataKitchen's DataOps TestGen, Observability, and Automation products.

## DataOps TestGen
- [Access Tokens](/testgen/access-tokens/): An access token authenticates you to TestGen's programmatic interfaces — the REST API and the MCP server. Each request carries the token in its Authorization header, and TestGen applies the...
- [OAuth Clients](/testgen/access-tokens/oauth-clients/): TestGen supports OAuth 2.1 for programmatic access. You register a client once and sign in through your browser to obtain a long-lived refresh token. Your automation then exchanges that refresh token...
- [Personal Access Tokens 🛡️](/testgen/access-tokens/personal-access-tokens/): A personal access token is a long-lived token created in the TestGen UI and used to authenticate to the REST API and the MCP server. A token carries the identity of the user who created it: requests...
- [Configure Email Server](/testgen/administer/configure-email-server/): TestGen can be configured to use an email server that implements the Simple Mail Transfer Protocol (SMTP), like Amazon Simple Email Service (SES), Azure Communication Services, or Google Workspace...
- [Configure HTTPS](/testgen/administer/configure-https/): By default, TestGen serves its web UI and API over HTTP. To serve them over HTTPS, provide an SSL certificate and private key. When both are configured, TestGen serves the UI and the API over TLS.
- [Configure Single Sign-On 🛡️](/testgen/administer/configure-single-sign-on/): TestGen supports Single Sign-On (SSO) through any authentication provider that implements the OpenID Connect (OIDC) protocol, such as Okta, Microsoft Entra ID, or Auth0, or Ping Identity.
- [Administer TestGen](/testgen/administer/): After installing TestGen, use the topics in this section to manage, configure, and maintain your deployment. These tasks are typically performed by the person who installed TestGen or manages the...
- [Set Custom Logo 🛡️](/testgen/administer/set-custom-logo/): TestGen Enterprise supports replacing the DataKitchen logo and icon on the navigation menu with your custom company logo. Both images must be hosted at a publicly accessible URL.
- [Set Environment Variables](/testgen/administer/set-environment-variables/): Most deployment-level settings in TestGen — SMTP credentials, single sign-on, custom branding, and MCP options — are applied through environment variables. The steps to set a variable depend on the...
- [Uninstall TestGen](/testgen/administer/uninstall-testgen/): This document provides the steps to uninstall an instance of DataOps Data Quality TestGen. The uninstall process deletes all TestGen resources and data from your machine.
- [Upgrade TestGen](/testgen/administer/upgrade-testgen/): This document provides the steps to upgrade an instance of DataOps Data Quality TestGen to the latest released version.
- [Anonymous Analytics](/testgen/anonymous-analytics/): DataKitchen gathers anonymous analytics using Mixpanel. You will be notified the first time you run the update or install program.
- [Authentication](/testgen/api/authentication/): The TestGen REST API authenticates with a bearer token. Include the token in the Authorization header of every request; the API acts with the permissions of the user the token belongs to.
- [Import and Export Tests](/testgen/api/import-and-export-tests/): Manage test definitions in bulk through the API — calibrate thresholds across an entire test suite, promote test definitions between environments, or seed a new test suite from a prepared file.
- [API](/testgen/api/): Automate data profiling and data quality testing by integrating TestGen into your CI/CD pipelines, orchestration tools, or custom workflows.
- [Run Profiling and Tests](/testgen/api/run-profiling-and-tests/): Trigger a profiling or test run through the API and retrieve its results — for CI/CD pipelines, orchestration tools, or any system that needs to drive TestGen in response to upstream events.
- [Connect to Observability](/testgen/connect-to-observability/): Use TestGen to generate and execute data quality checks that identify errors in your data. Then, export the results to DataKitchen's DataOps Observability for visibility and impact across your entire...
- [Database Access Requirements](/testgen/connect-your-database/database-access-requirements/): Before connecting TestGen to your database, verify that the database user you plan to use has the required permissions. This page lists the access TestGen needs for each supported database.
- [Connect your Database](/testgen/connect-your-database/): Before you can profile and test your data, you need to connect TestGen to your database and tell it which tables to analyze.
- [Manage Connections 🛡️](/testgen/connect-your-database/manage-connections/): A connection defines how TestGen accesses your database for profiling, testing, and monitoring. TestGen requires only read-only access — see Database Access Requirements for the specific permissions...
- [Manage Table Groups](/testgen/connect-your-database/manage-table-groups/): A table group is a selection of database tables within a schema that TestGen profiles and tests. Each table group belongs to a connection and can have its own test suites, monitors, and profiling...
- [Manage Test Suites](/testgen/connect-your-database/manage-test-suites/): A test suite organizes data quality tests for a table group. Once created, you can generate tests to populate the suite and then run the tests on a recurring basis.
- [Data Catalog](/testgen/data-catalog/): Use the Data Catalog to investigate data quality issues, explore profiling results across tables and columns, and organize your data assets with metadata tags. The catalog brings together profiling...
- [Profiling Statistics](/testgen/data-catalog/profiling-statistics/): This page describes the profiling statistics that TestGen collects and displays for each column in the Data Catalog. The statistics shown depend on the column's general data type: Alpha (text),...
- [Data Type and Format](/testgen/data-profiling/hygiene-issues/data-type-and-format/): Column values don't match the expected type, format, or structural standard — including detection of sensitive data patterns.
- [Data Hygiene Issues](/testgen/data-profiling/hygiene-issues/): This reference covers the 32 data hygiene issues that TestGen detects during a profiling run. Hygiene issues are structural inconsistencies and content anomalies — such as data type mismatches,...
- [Missing and Incomplete Data](/testgen/data-profiling/hygiene-issues/missing-and-incomplete-data/): Expected values are absent, blank, non-standard, or potentially duplicated.
- [Unexpected Content and Dates](/testgen/data-profiling/hygiene-issues/unexpected-content-and-dates/): Column content doesn't match what the column name suggests, or date values fall outside expected ranges.
- [Value and Pattern Consistency](/testgen/data-profiling/hygiene-issues/value-and-pattern-consistency/): Values are inconsistent within a column, across tables, or with expected value sets.
- [Data Profiling](/testgen/data-profiling/): Use data profiling to discover the structure, content, and quality characteristics of tables in your database. Profiling results populate the Data Catalog, surface hygiene issues, and serve as the...
- [Investigate Profiling Results](/testgen/data-profiling/investigate-profiling-results/): After a profiling run completes, use the Profiling Runs page to review column statistics and hygiene issues, and to take action on issues that need attention. For an overview of what profiling...
- [Profiling Characteristics](/testgen/data-profiling/profiling-characteristics/): This reference lists the 55 column-level characteristics that TestGen captures during a profiling run. Use it to understand what a specific statistic measures and how it can help identify data...
- [Run Profiling](/testgen/data-profiling/run-profiling/): Run profiling to discover the structure and characteristics of tables in a table group. Profiling results populate the Data Catalog, surface hygiene issues, and serve as the foundation for test...
- [Data Quality Testing](/testgen/data-quality-testing/): Use data quality testing to verify that your data meets expected standards before it reaches downstream consumers. TestGen executes tests against your database and produces results you can review,...
- [Investigate Test Results](/testgen/data-quality-testing/investigate-test-results/): After a test run completes, use the Test Runs page to review results, investigate failures, and take action on findings. For an overview of what testing captures and why, see Data Quality Testing.
- [Run Tests](/testgen/data-quality-testing/run-tests/): Run the tests in a test suite to detect data quality issues. Each test run evaluates your data against the thresholds and criteria defined in the suite's test definitions and produces results you can...
- [Generate Tests](/testgen/generate-tests/): Generate tests to automatically populate a test suite based on the latest profiling results for a table group. The system selects relevant test types, sets thresholds, and configures parameters based...
- [Test Definitions](/testgen/generate-tests/test-definitions/): View and manage the individual tests in a test suite — review auto-generated tests, tune parameters, add manual tests, and control which tests run during execution.
- [Accuracy Tests](/testgen/generate-tests/test-types/accuracy-tests/): Accuracy tests evaluate whether data values faithfully represent the real-world quantities or states they are intended to model. These tests compare column values, aggregate measures, and statistical...
- [Completeness Tests](/testgen/generate-tests/test-types/completeness-tests/): Completeness tests verify that your data has no unexpected gaps — whether at the record level (missing values within rows), the table level (fewer rows than expected), or the temporal level (missing...
- [Consistency Tests](/testgen/generate-tests/test-types/consistency-tests/): Consistency tests verify that data values remain stable and aligned with established baselines or business rules. They detect drift in statistical measures, unexpected changes to constant or...
- [TestGen Test Types](/testgen/generate-tests/test-types/): This reference covers the 49 test types available in TestGen. Each test type targets a specific data quality concern — from missing values and format violations to statistical shifts and cross-table...
- [Timeliness Tests](/testgen/generate-tests/test-types/timeliness-tests/): Timeliness tests verify that data arrives on schedule and reflects expected time ranges. They detect stale tables, late deliveries, missing date coverage, and unexpected future dates — problems that...
- [Uniqueness Tests](/testgen/generate-tests/test-types/uniqueness-tests/): Uniqueness tests verify that data values are not inappropriately duplicated — whether at the column level (every value is distinct) or at the table level (no repeated rows). Shifts in uniqueness...
- [Validity Tests](/testgen/generate-tests/test-types/validity-tests/): Validity tests verify that data values conform to expected formats, ranges, and domain rules. A value can be present and non-null but still invalid — for example, a zip code with too few digits, an...
- [Get Started with TestGen](/testgen/get-started/): DataKitchen's DataOps Data Quality TestGen provides database profiling, hygiene issue detection, data quality test generation, test execution, anomaly monitoring, and data quality scoring—so you can...
- [Install Enterprise 🛡️](/testgen/get-started/install-enterprise/): Enterprise
- [Install on Mac/Linux](/testgen/get-started/install-on-mac-linux/): The dk-installer program installs DataOps Data Quality TestGen in one of two modes:
- [Install on Windows](/testgen/get-started/install-on-windows/): The dk-installer.exe program installs DataOps Data Quality TestGen in one of two modes:
- [Use an External PostgreSQL Database 🛡️](/testgen/get-started/use-external-database/): TestGen stores its application metadata — projects, connections, profiling results, test definitions, and run history — in a PostgreSQL database. By default, the installer runs this database as a...
- [Manage Projects 🛡️](/testgen/manage-projects/): Projects organize your data quality work into separate spaces — for example, by department, team, or stakeholder group. All connections, table groups, test suites, and results belong to a specific...
- [MCP Server](/testgen/mcp/): Use the TestGen Model Context Protocol (MCP) server to give AI assistants access to your data quality results, profiling output, and test runs.
- [Set up the MCP Server](/testgen/mcp/setup/): Connect an AI client to the TestGen MCP server. The server authenticates with a bearer token, obtained in one of two ways depending on what your client supports — automatic OAuth registration for...
- [Supported Tools](/testgen/mcp/supported-tools/): The TestGen MCP server exposes the prompts, tools, and resources listed below.
- [Configure Monitors](/testgen/monitor-tables/configure-monitors/): Set up automated anomaly detection for your tables. Monitors track freshness, volume, schema changes, and custom metrics over time, flagging deviations from expected patterns. For an overview of what...
- [Monitor Tables](/testgen/monitor-tables/): TestGen Monitors provide proactive, automated anomaly detection for your data tables. While tests verify specific data content rules (such as column formats, value ranges, and referential integrity),...
- [View Monitor Results](/testgen/monitor-tables/view-monitor-results/): Use the Monitors dashboard to track anomalies detected across your monitored tables. The dashboard shows anomaly counts by type (Freshness, Volume, Schema, Metric) and lets you drill into trend...
- [Notifications](/testgen/notifications/): Set up email notifications to stay informed about data quality events without checking the UI. TestGen can alert you after profiling runs, test runs, monitor anomalies, and quality score changes.
- [Project Dashboard](/testgen/project-dashboard/): The Project Dashboard is the landing page you see after selecting a project. Use it to quickly assess the health of your data across all table groups and navigate to areas that need attention.
- [Explore and Create Scorecards](/testgen/quality-scores/explore-and-create-scorecards/): Use the Score Explorer to explore quality score breakdowns across your data in real time. Add filters, switch categories, and preview scores from different perspectives — then save any combination as...
- [Quality Scores](/testgen/quality-scores/): TestGen quality scores quantify the health of your data by measuring the ratio of clean data to total data. Scores are computed automatically from profiling results and test outcomes, giving you a...
- [View Score Details](/testgen/quality-scores/view-score-details/): Use the Score Details page to investigate a scorecard's scores, track trends over time, and drill down into the individual issues behind a score. To open it, click View details on any scorecard tile...
- [1 October](/testgen/release-notes/enterprise/2024/1-october/): Enterprise Version 2.14.10
- [10 May](/testgen/release-notes/enterprise/2024/10-may/): Enterprise Version 1.6.4
- [16 August](/testgen/release-notes/enterprise/2024/16-august/): Enterprise Version 2.7.2
- [16 October](/testgen/release-notes/enterprise/2024/16-october/): Enterprise Version 2.15.3
- [20 November](/testgen/release-notes/enterprise/2024/20-november/): Enterprise Version 2.24.4
- [25 January](/testgen/release-notes/enterprise/2024/25-january/): Version 1.3.2
- [25 July](/testgen/release-notes/enterprise/2024/25-july/): Enterprise Version 2.7.0
- [25 November](/testgen/release-notes/enterprise/2024/25-november/): Enterprise Version 2.24.7
- [26 September](/testgen/release-notes/enterprise/2024/26-september/): Enterprise Version 2.14.6
- [3 October](/testgen/release-notes/enterprise/2024/3-october/): Enterprise Version 2.15.1
- [6 February](/testgen/release-notes/enterprise/2024/6-february/): Version 1.4.1
- [1 May](/testgen/release-notes/enterprise/2025/1-may/): Enterprise Version 3.7.8
- [10 October](/testgen/release-notes/enterprise/2025/10-october/): Enterprise Version 4.32.5
- [12 August](/testgen/release-notes/enterprise/2025/12-august/): Enterprise Version 4.20.4
- [15 December](/testgen/release-notes/enterprise/2025/15-december/): Enterprise Version 4.38.6
- [16 December](/testgen/release-notes/enterprise/2025/16-december/): Enterprise Version 4.38.7
- [16 July](/testgen/release-notes/enterprise/2025/16-july/): Enterprise Version 4.12.6
- [16 September](/testgen/release-notes/enterprise/2025/16-september/): Enterprise Version 4.26.1
- [17 June](/testgen/release-notes/enterprise/2025/17-june/): Enterprise Version 4.1.2
- [19 February](/testgen/release-notes/enterprise/2025/19-february/): Enterprise Version 3.0.0
- [2 April](/testgen/release-notes/enterprise/2025/2-april/): Enterprise Version 3.7.6
- [2 December](/testgen/release-notes/enterprise/2025/2-december/): Enterprise Version 4.38.3
- [21 May](/testgen/release-notes/enterprise/2025/21-may/): Enterprise Version 4.0.9
- [22 May](/testgen/release-notes/enterprise/2025/22-may/): Enterprise Version 4.0.12
- [24 June](/testgen/release-notes/enterprise/2025/24-june/): Enterprise Version 4.1.3
- [24 March](/testgen/release-notes/enterprise/2025/24-march/): Enterprise Version 3.7.1
- [25 August](/testgen/release-notes/enterprise/2025/25-august/): Enterprise Version 4.22.2
- [28 February](/testgen/release-notes/enterprise/2025/28-february/): Enterprise Version 3.1.0
- [28 July](/testgen/release-notes/enterprise/2025/28-july/): Enterprise Version 4.16.3
- [6 March](/testgen/release-notes/enterprise/2025/6-march/): Enterprise Version 3.1.2
- [2 June](/testgen/release-notes/enterprise/2026/2-june/): Enterprise Version 5.48.0
- [24 February](/testgen/release-notes/enterprise/2026/24-february/): Enterprise Version 5.0.2
- [27 March](/testgen/release-notes/enterprise/2026/27-march/): Enterprise Version 5.9.4
- [30 June](/testgen/release-notes/enterprise/2026/30-june/): Enterprise Version 5.70.2
- [31 March](/testgen/release-notes/enterprise/2026/31-march/): Enterprise Version 5.9.5
- [4 May](/testgen/release-notes/enterprise/2026/4-may/): Enterprise Version 5.32.2
- [6 March](/testgen/release-notes/enterprise/2026/6-march/): Enterprise Version 5.0.3
- [8 January](/testgen/release-notes/enterprise/2026/8-january/): Enterprise Version 4.39.2
- [TestGen Release Notes](/testgen/release-notes/): Find links here to release notes for user-facing features, fixes, and enhancements in DataKitchen's DataOps Data Quality TestGen.
- [1 December](/testgen/release-notes/open-source/2023/1-december/): Version 1.1.2
- [10 October](/testgen/release-notes/open-source/2023/10-october/): Beta version 0.10.2
- [13 July](/testgen/release-notes/open-source/2023/13-july/): Version 0.2.0
- [21 December](/testgen/release-notes/open-source/2023/21-december/): Version 1.2.2
- [23 October](/testgen/release-notes/open-source/2023/23-october/): Beta version 0.12.1
- [25 August](/testgen/release-notes/open-source/2023/25-august/): Beta version 0.6.0
- [25 September](/testgen/release-notes/open-source/2023/25-september/): Beta version 0.9.2
- [3 November](/testgen/release-notes/open-source/2023/3-november/): Beta version 0.15.0
- [30 August](/testgen/release-notes/open-source/2023/30-august/): Beta version 0.7.0
- [30 October](/testgen/release-notes/open-source/2023/30-october/): Beta version 0.14.0
- [31 July](/testgen/release-notes/open-source/2023/31-july/): Beta version 0.4.0
- [5 September](/testgen/release-notes/open-source/2023/5-september/): Beta version 0.8.0
- [Alpha Release](/testgen/release-notes/open-source/2023/alpha-release/): The initial Alpha release of DataKitchen's DataOps TestGen is available to a limited number of customer accounts.
- [13 June](/testgen/release-notes/open-source/2024/13-june/): Open Source Version 2.1.9
- [14 October](/testgen/release-notes/open-source/2024/14-october/): Open Source Version 2.15.2
- [16 August](/testgen/release-notes/open-source/2024/16-august/): Open Source Version 2.7.2
- [16 July](/testgen/release-notes/open-source/2024/16-july/): Open Source Version 2.2.0
- [16 May](/testgen/release-notes/open-source/2024/16-may/): Open Source Version 2.1.1
- [16 October](/testgen/release-notes/open-source/2024/16-october/): Open Source Version 2.15.3
- [20 May](/testgen/release-notes/open-source/2024/20-may/): Open Source Version 2.1.4
- [22 April](/testgen/release-notes/open-source/2024/22-april/): Open Source Version 2.0
- [27 November](/testgen/release-notes/open-source/2024/27-november/): Open Source Version 2.24.7
- [3 December](/testgen/release-notes/open-source/2024/3-december/): Open Source Version 2.24.8
- [30 July](/testgen/release-notes/open-source/2024/30-july/): Open Source Version 2.7.0
- [1 May](/testgen/release-notes/open-source/2025/1-may/): Open Source Version 3.7.8
- [13 August](/testgen/release-notes/open-source/2025/13-august/): Open Source Version 4.20.4
- [13 October](/testgen/release-notes/open-source/2025/13-october/): Open Source Version 4.32.5
- [14 May](/testgen/release-notes/open-source/2025/14-may/): Open Source Version 3.7.9
- [15 December](/testgen/release-notes/open-source/2025/15-december/): Open Source Version 4.38.6
- [16 December](/testgen/release-notes/open-source/2025/16-december/): Open Source Version 4.38.7
- [16 July](/testgen/release-notes/open-source/2025/16-july/): Open Source Version 4.12.6
- [17 June](/testgen/release-notes/open-source/2025/17-june/): Open Source Version 4.1.2
- [17 September](/testgen/release-notes/open-source/2025/17-september/): Open Source Version 4.26.1
- [19 February](/testgen/release-notes/open-source/2025/19-february/): Open Source Version 3.0.0
- [2 April](/testgen/release-notes/open-source/2025/2-april/): Open Source Version 3.7.6
- [2 December](/testgen/release-notes/open-source/2025/2-december/): Open Source Version 4.38.3
- [21 May](/testgen/release-notes/open-source/2025/21-may/): Open Source Version 4.0.9
- [22 May](/testgen/release-notes/open-source/2025/22-may/): Open Source Version 4.0.12
- [24 June](/testgen/release-notes/open-source/2025/24-june/): Open Source Version 4.1.3
- [25 August](/testgen/release-notes/open-source/2025/25-august/): Open Source Version 4.22.2
- [25 March](/testgen/release-notes/open-source/2025/25-march/): Open Source Version 3.7.1
- [28 February](/testgen/release-notes/open-source/2025/28-february/): Open Source Version 3.1.0
- [28 July](/testgen/release-notes/open-source/2025/28-july/): Open Source Version 4.16.3
- [6 March](/testgen/release-notes/open-source/2025/6-march/): Open Source Version 3.1.2
- [15 May](/testgen/release-notes/open-source/2026/15-may/): Open Source Version 5.33.3
- [2 June](/testgen/release-notes/open-source/2026/2-june/): Open Source Version 5.48.0
- [24 February](/testgen/release-notes/open-source/2026/24-february/): Open Source Version 5.0.2
- [27 March](/testgen/release-notes/open-source/2026/27-march/): Open Source Version 5.9.4
- [30 June](/testgen/release-notes/open-source/2026/30-june/): Open Source Version 5.70.2
- [31 March](/testgen/release-notes/open-source/2026/31-march/): Open Source Version 5.9.5
- [4 May](/testgen/release-notes/open-source/2026/4-may/): Open Source Version 5.32.2
- [8 January](/testgen/release-notes/open-source/2026/8-january/): Open Source Version 4.39.2
- [Scheduling](/testgen/scheduling/): Automate your data quality checks by scheduling profiling runs, test runs, and monitor runs to recur on a regular basis.
- [System Architecture and Security](/testgen/system-architecture-and-security/): TestGen is designed to run entirely within your infrastructure, with read-only access to your data. This page describes TestGen's architecture, deployment model, and security posture.
- [Troubleshooting](/testgen/troubleshooting/): When TestGen does not behave as expected, the application logs are the fastest way for DataKitchen support to diagnose the problem. This page explains how to collect your logs and send them to...
- [User Access 🛡️](/testgen/user-access/): TestGen uses role-based access control to manage what each user can do. Roles are assigned per project, so a user can have different levels of access in different projects. Users only see the...
- [What is TestGen?](/testgen/what-is-testgen/): DataKitchen's DataOps Data Quality TestGen is a data quality tool that profiles your databases, generates tests, and monitors your tables, enabling you to confidently assess the health of your data...

## DataOps Observability
- [Configure Email Server](/observability/administer/configure-email-server/): Observability can be configured to use an email server that implements the Simple Mail Transfer Protocol (SMTP), like Amazon Simple Email Service (SES), Azure Communication Services, or Google...
- [Administer Observability](/observability/administer/): After installing Observability, use the topics in this section to manage, configure, and maintain your deployment. These tasks are typically performed by the person who installed Observability or...
- [Uninstall Observability](/observability/administer/uninstall-observability/): This document provides the steps to uninstall an instance of DataOps Observability. The process uninstalls the Docker Compose application and removes all data, containers, and images related to...
- [Upgrade Observability](/observability/administer/upgrade-observability/): This document provides the steps to upgrade an instance of DataOps Observability to the latest released version.
- [Anonymous Analytics](/observability/anonymous-analytics/): DataKitchen gathers anonymous analytics using Mixpanel. You will be notified the first time you run the update or install program.
- [Delete an API Key](/observability/api-keys/delete-an-api-key/): When you delete a key, components using the key in their integration setup lose their connection with Observability. Events sent from those components will be rejected with a 401 code.
- [API Keys](/observability/api-keys/): The integration agents or external scripts that send events from your data estate to Observability are required to provide an API key to authenticate with the Event Ingestion API. API keys are...
- [Observability API](/observability/api-keys/observability-api/): DataOps Observability consists of two REST APIs: the Event Ingestion API, responsible for receiving events sent from your data estate, and the Observability API, responsible for system and management...
- [Batch Pipeline](/observability/components/batch-pipeline/): A batch pipeline is a type of Observability component that represents a series of finite processes running in a data estate.
- [Run Status](/observability/components/batch-pipeline/run-status/): A run status provides context about the progress or results of a batch pipeline execution.
- [Runs](/observability/components/batch-pipeline/runs/): A run is a single execution of a batch pipeline. While a batch pipeline and its configuration reflect what is expected to happen, a run is what actually happens as events are received by the system.
- [Tasks](/observability/components/batch-pipeline/tasks/): A task represents a single step in a batch pipeline. Depending on your data environment, a task could be a job running on a tool, part of a job, a script, or the arrival of files.
- [View Run Details](/observability/components/batch-pipeline/view-run-details/): Observability displays the details for a run across several tabs.
- [Component Bulk Actions](/observability/components/bulk-component-actions/): Make changes to multiple components at one time with bulk actions.
- [Arrival Window](/observability/components/configure-component-expectations/arrival-window/): You can set a frequency and window of time that you expect write Dataset Operation events to occur for dataset components.
- [Cron Expressions](/observability/components/configure-component-expectations/cron-expressions/): DataOps Observability lets you use a cron scheduler to set custom component expectations and instance conditions.
- [Expected Schedule](/observability/components/configure-component-expectations/expected-schedule/): You can set a schedule for when you expect runs to start and/or end for batch pipeline components.
- [Configure Component Expectations](/observability/components/configure-component-expectations/): You can set expectations for when events are known to occur in the components of your data estate.
- [Create Components](/observability/components/create-components/): Components represent the tools and assets that you use in your data estate. Observability automatically creates a new component any time an event is received by the Event Ingestion API with a key...
- [Dataset](/observability/components/dataset/): A dataset is a type of Observability component that represents a collection of data in a data estate.
- [TestGen Dataset Components](/observability/components/dataset/testgen-dataset-components/): DataKitchen's DataOps Data Quality TestGen is a data quality tool that profiles your databases, generates tests, and verifies the health of your data over time. When you run TestGen tests, send the...
- [Delete Components](/observability/components/delete-components/): Delete components to maintain organized projects that represent your data estate.
- [Components](/observability/components/): Components represent the resources, engines, and tools that you use in your day-to-day work to deliver data analytic assets. Components can include batch pipeline runs executed by orchestrators,...
- [Manage Components](/observability/components/manage-components/): Update component information and set expectations in the component details view.
- [Server](/observability/components/server/): A server is a type of Observability component that represents a piece of hardware or software that provides services for other data estate components.
- [Streaming Pipeline](/observability/components/streaming-pipeline/): A streaming pipeline is a type of Observability component that represents a series of ongoing processes running in a data estate.
- [Data Estate](/observability/core-concepts/data-estate/): There are two core concepts in DataKitchen's DataOps Observability: the data estate and the data journey.
- [Data Journeys](/observability/core-concepts/data-journeys/): There are two core concepts in DataKitchen's DataOps Observability: the data estate and the data journey.
- [Expectations and Rules](/observability/core-concepts/expectations-and-rules/): In DataOps Observability, the events received, components generated, journeys built, and instances captured let you see your data estate as it is. You can configure expectations, conditions, and...
- [Core Concepts in Observability](/observability/core-concepts/): At DataKitchen, we believe observability is an important step in achieving DataOps success. DataOps Observability provides end-to-end visibility across your data operations, so you can see how...
- [Integrate your Tools](/observability/core-concepts/integrate-your-tools/): DataOps Observability supports integrating with a wide array of data tools so you can monitor your data operations with end-to-end visibility.
- [Data Retention](/observability/data-retention/): For scalability and data retention considerations, historical records are only kept for a fixed period of time.
- [Event Endpoint Details](/observability/event-ingestion-api/event-endpoints/): Event types describe common categories of events that can happen in data operations. For technical details on each of the event types, go to the Event Ingestion API documentation.
- [Event Ingestion API Identifiers](/observability/event-ingestion-api/event-identifiers/): Every event sent to Observability through the Event Ingestion API includes required and optional properties to provide context and metadata.
- [Event Ingestion API](/observability/event-ingestion-api/): Observability receives, aggregates, and displays event information sent by your tools and data assets (i.e. components). The system does so by exposing a REST API—the Event Ingestion API—where the...
- [Publish Events to the API](/observability/event-ingestion-api/publish-events-to-the-api/): The following integration methods publish events to the Event Ingestion API endpoints directly. Events that are sent to the API are received by the system and displayed in the UI.
- [Publish Events Directly](/observability/event-ingestion-api/publish-events-to-the-api/publish-events-directly-walkthrough/): The following walks you through how you use the Event Ingestion API endpoints directly.
- [Publish Events via Client SDK](/observability/event-ingestion-api/publish-events-to-the-api/publish-events-via-client-sdk/): This integration method uses a client software development kit (SDK) to send events to Observability.
- [Event Processing](/observability/events/event-processing/): Events represent moments of interest that occur in the components of your data estate. Observability groups these events into five types—Run Status, Message Log, Test Outcomes, Metric Log, and...
- [Event Types](/observability/events/event-types/): Event types describe common categories of events that can happen in data operations. For technical details on each of the event types, go to the Event Ingestion API documentation.
- [Events](/observability/events/): An event is any moment of interest that occurs in your data estate. For example, an event could be a batch pipeline run starting, a streaming pipeline receiving events with a certain status, a test...
- [All Events Tab](/observability/events/view-events-and-runs/all-events-tab/): The All Events tab lists all of the events Observability has received for a project.
- [Batch Runs Tab](/observability/events/view-events-and-runs/batch-runs-tab/): The Batch Runs tab shows all of the current and historical runs for the batch pipeline components in a project.
- [View Events and Runs](/observability/events/view-events-and-runs/): Every event that the system receives is aggregated and listed under Events in the UI.
- [Get Started with Observability](/observability/get-started/): DataKitchen's DataOps Observability provides end-to-end visibility across your data operations, so you can see how everything is going in real-time.
- [Install Enterprise 🛡️](/observability/get-started/install-enterprise/): Enterprise
- [Install on Mac/Linux](/observability/get-started/install-on-mac-linux/): The dk-installer program installs DataOps Observability as a Docker Compose application. This is the recommended installation mode, as Docker encapsulates and isolates the application from other...
- [Install on Windows](/observability/get-started/install-on-windows/): The dk-installer.exe program installs DataOps Observability as a Docker Compose application. This is the recommended installation mode, as Docker encapsulates and isolates the application from other...
- [Complex Instance Example](/observability/instances/complex-instance-example/): This document provides an example of how Observability handles a journey with multiple components and settings.
- [Define Instances](/observability/instances/define-instances/): An instance is the runtime execution of a journey, which is a collection of components in your data estate, as specified by you. An instance shows what events occur in those components during a...
- [Instances](/observability/instances/): An instance represents the runtime execution of a journey. Since a journey consists of a collection of components and the relationships that link them, an instance shows the events that occur across...
- [Instance Status](/observability/instances/instance-status/): Observability provides the progress and status of each instance. The system shows an instance as:
- [Payload Instances](/observability/instances/payload-instances/): An instance represents a collection of the events that occur in the components included in a journey during a certain window, as defined by you. A payload instance is a unique type of instance that...
- [Upcoming Instances](/observability/instances/upcoming-instances/): Observability can show you what journey instances are upcoming—so you can better understand what events will happen in your data estate, make appropriate plans, and take action.
- [Batch Runs Tab](/observability/instances/view-instance-details/batch-runs-tab/): Select the Batch Runs tab to see a list of all the runs that have occurred during this instance.
- [Events Tab](/observability/instances/view-instance-details/events-tab/): Select the Events tab for a list of all the events received for the journey's components during this instance.
- [View Instance Details](/observability/instances/view-instance-details/): Observability displays the details and data collected for an instance across several tabs.
- [Status Tab](/observability/instances/view-instance-details/status-tab/): Select the Status tab to open a graphical representation of the instance.
- [TestGen Results](/observability/instances/view-instance-details/testgen-results/): Export data from DataKitchen's DataOps Data Quality TestGen to Observability so you can view test results as part of your data estate journeys.
- [Tests Tab](/observability/instances/view-instance-details/tests-tab/): Select the Tests tab for an aggregated view of all the test results received during this instance.
- [Timeline Tab](/observability/instances/view-instance-details/timeline-tab/): Select the Timeline tab for a Gantt chart of the instance's timings, separated by batch pipeline component and charted in local browser time. The data in this tab refreshes every minute.
- [Add Azure Diagnostic Settings](/observability/integration-agents/add-azure-diagnostic-settings/): DataKitchen provides agents that allow you to monitor Azure services—Data Factory, Blob Storage, and Functions—through Azure Event Hubs.
- [Create an Azure Service Principal](/observability/integration-agents/create-an-azure-service-principal/): An Azure service principal is an identity you can create for automated tools and systems like scripts, apps, and CI/CD platforms. You can configure a service principal to authenticate certain...
- [Airflow](/observability/integration-agents/deploy-an-agent/airflow/): DataKitchen provides an agent that lets you monitor Apache Airflow and Azure Data Factory Managed Airflow .
- [Amazon S3](/observability/integration-agents/deploy-an-agent/amazon-s3/): DataKitchen provides an agent that lets you monitor Amazon S3 through Amazon SQS.
- [AutoSys 🛡️](/observability/integration-agents/deploy-an-agent/autosys/): DataKitchen provides an agent that lets you monitor AutoSys .
- [Azure Blob Storage](/observability/integration-agents/deploy-an-agent/azure-blob-storage/): DataKitchen provides an agent that lets you monitor Azure Blob Storage through Azure Event Hubs. This agent listens to a single Azure Event Hubs instance. To monitor multiple hubs, configure and...
- [Azure Data Factory](/observability/integration-agents/deploy-an-agent/azure-data-factory/): DataKitchen provides an agent that lets you monitor Azure Data Factory through Azure Event Hubs. This agent listens to a single Azure Event Hubs instance. To monitor multiple hubs, configure and...
- [Azure Functions](/observability/integration-agents/deploy-an-agent/azure-functions/): DataKitchen provides an agent that lets you monitor Azure Functions through Azure Event Hubs. This agent listens to a single Azure Event Hubs instance. To monitor multiple hubs, configure and deploy...
- [Azure Synapse](/observability/integration-agents/deploy-an-agent/azure-synapse/): DataKitchen provides an agent that lets you monitor the ETL component of Azure Synapse Analytics .
- [Databricks](/observability/integration-agents/deploy-an-agent/databricks/): DataKitchen provides an agent that lets you monitor Databricks workflows for Azure Databricks, Databricks on AWS, and Databricks on Google Cloud.
- [dbt Core](/observability/integration-agents/deploy-an-agent/dbt-core/): DataKitchen provides an agent to monitor dbt Core . The dbt Core agent parses two artifacts from a dbt run or test (run_results.json and manifest.json) and uses these artifacts to create events in...
- [Fivetran Log Connector](/observability/integration-agents/deploy-an-agent/fivetran-log-connector/): DataKitchen provides an agent that lets you monitor Fivetran Log Connector with a Databricks destination .
- [Deploy an Observability Agent](/observability/integration-agents/deploy-an-agent/): To send events from your components to DataOps Observability, the Observability agents must be deployed in your data estate.
- [Microsoft Power BI](/observability/integration-agents/deploy-an-agent/microsoft-power-bi/): DataKitchen provides an agent that lets you monitor Microsoft Power BI dataset refreshes.
- [Microsoft SSIS](/observability/integration-agents/deploy-an-agent/microsoft-ssis/): DataKitchen provides an agent that lets you monitor the on-prem version of Microsoft SQL Server Integration Services .
- [Qlik](/observability/integration-agents/deploy-an-agent/qlik/): DataKitchen provides an agent that lets you monitor Qlik Cloud Analytics hub .
- [Talend](/observability/integration-agents/deploy-an-agent/talend/): DataKitchen provides an agent that lets you monitor the Talend Management Console .
- [Observability Integration Agents](/observability/integration-agents/): For complex data estates with multiple components and assets, an efficient way to integrate with DataOps Observability is to use agents. Agents can scrape events from event streams, log files, or...
- [Update or Remove an Agent](/observability/integration-agents/update-or-remove-an-agent/): Observability agents consist of a Docker image and a set of configuration variables. This allows each agent to exist as a distinct container that includes all the scripts, configuration values, and...
- [Available Agents](/observability/integrations/available-agents/): The Integrations page provides a list of available Observability agents, complete with steps on how to configure and deploy.
- [Heartbeat Signal](/observability/integrations/heartbeat-signal/): DataOps Observability includes several integration agents that connect your local tools and assets (i.e. components) to your projects.
- [Integrations](/observability/integrations/): Observability receives, aggregates, and displays event information sent by your tools and data assets (i.e. components). The system does so by exposing a REST API—the Event Ingestion API—where the...
- [Component Autodiscovery](/observability/journeys/component-patterns/): Component autodiscovery automatically includes or excludes components in a journey based on their key. When a new component is created that matches a journey's patterns, it is automatically added to...
- [Batch Instance Conditions](/observability/journeys/configure-instance-conditions/batch-instance-conditions/): You can define the start and end of a journey's instances by the behavior of batch pipeline runs.
- [Default Instance Conditions](/observability/journeys/configure-instance-conditions/default-instance-conditions/): You can define the start and end of a journey's instances by the default behavior.
- [Configure Instance Conditions](/observability/journeys/configure-instance-conditions/): Observability starts and ends instances based on the conditions you set for a journey.
- [Payload Instance Conditions](/observability/journeys/configure-instance-conditions/payload-instance-conditions/): You can define the end of a journey's payload instances by duration or batch pipeline run behavior.
- [Schedule Instance Conditions](/observability/journeys/configure-instance-conditions/schedule-instance-conditions/): You can define the start and end of a journey's instances by specifying schedules.
- [Create Journeys](/observability/journeys/create-journeys/): Journeys represent the collection of components and dependencies that deliver a data analytic asset. Create journeys to monitor your data processes from end to end.
- [Create Relationships](/observability/journeys/create-relationships/): You can build a journey from its constituent components—and the dependencies between them—from the Relationships tab of the journey details.
- [Delete Journeys](/observability/journeys/delete-journeys/): Delete journeys to maintain organized projects that represent your data estate.
- [Journeys](/observability/journeys/): A journey represents a collection of components in your data estate that are responsible for creating a data analytic deliverable.
- [Manage Journey Settings](/observability/journeys/manage-journeys/): You can update journey details, set instance conditions, and/or configure payload instance conditions from the journey Settings tab.
- [View Journey Details](/observability/journeys/view-journey-details/): A journey represents a collection of components in your data estate that are responsible for creating a data analytic deliverable.
- [Project Settings](/observability/project-settings/): You can view or update project details from the Settings page.
- [Company Overview 🛡️](/observability/projects/company-overview/): In DataOps Observability, projects are the spaces that separate and distinguish work. Projects might be set up by department, team, or goal, to suit your organizational needs.
- [Create a Project 🛡️](/observability/projects/create-a-project/): Projects are the spaces that separate and distinguish work. Depending on your organizational needs, projects might be set up by department, team, or goal.
- [Delete a Project 🛡️](/observability/projects/delete-a-project/): Delete projects when your work is done and maintain an organized Observability tenant.
- [Projects Enterprise](/observability/projects/): Projects are the spaces that separate and distinguish work. Depending on your organizational needs, projects might be set up by department, team, or goal.
- [Project Overview 🛡️](/observability/projects/project-overview/): Observability data is always associated with a specific project. Use the Project drop-down in the menu to navigate between projects.
- [Agent Changelog](/observability/release-notes/agent-changelog/): All production agents use the same Docker image. The following provides a summary of the image updates and releases.
- [10 January](/observability/release-notes/enterprise/2024/10-january/): Application version 1.0; User interface version 1.97.2
- [12 August](/observability/release-notes/enterprise/2024/12-august/): Enterprise Version 2.2.0
- [12 March](/observability/release-notes/enterprise/2024/12-march/): Application version 1.0; User interface version 1.100.7
- [13 February](/observability/release-notes/enterprise/2024/13-february/): Application version 1.0; User interface version 1.99.0
- [13 July](/observability/release-notes/enterprise/2024/13-july/): Enterprise Version 2.1.0
- [19 February](/observability/release-notes/enterprise/2024/19-february/): Application version 1.0; User interface version 1.100.0
- [20 March](/observability/release-notes/enterprise/2024/20-march/): Application version 1.0; User interface version 1.103.0
- [20 November](/observability/release-notes/enterprise/2024/20-november/): Enterprise Version 2.3.0
- [22 November](/observability/release-notes/enterprise/2024/22-november/): Enterprise Version 2.3.1
- [26 August](/observability/release-notes/enterprise/2024/26-august/): Enterprise Version 2.2.1
- [26 March](/observability/release-notes/enterprise/2024/26-march/): Application version 1.0; User interface version 1.105.0
- [3 April](/observability/release-notes/enterprise/2024/3-april/): Application version 1.0; User interface version 1.105.2
- [12 March](/observability/release-notes/enterprise/2025/12-march/): Enterprise Version 2.4.1
- [14 February](/observability/release-notes/enterprise/2025/14-february/): Enterprise Version 2.3.2
- [17 April](/observability/release-notes/enterprise/2025/17-april/): Enterprise Version 2.6.0
- [22 April](/observability/release-notes/enterprise/2025/22-april/): Enterprise Version 2.6.1
- [25 August](/observability/release-notes/enterprise/2025/25-august/): Enterprise Version 2.7.5
- [25 March](/observability/release-notes/enterprise/2025/25-march/): Enterprise Version 2.5.0
- [28 July](/observability/release-notes/enterprise/2025/28-july/): Enterprise Version 2.7.4
- [30 April](/observability/release-notes/enterprise/2025/30-april/): Enterprise Version 2.7.0
- [4 March](/observability/release-notes/enterprise/2025/4-march/): Enterprise Version 2.4.0
- [5 November](/observability/release-notes/enterprise/2025/5-november/): Enterprise Version 2.8.0
- [10 March](/observability/release-notes/enterprise/2026/10-march/): Enterprise Version 2.9.0
- [21 March](/observability/release-notes/enterprise/2026/21-march/): Enterprise Version 2.11.3
- [3 April](/observability/release-notes/enterprise/2026/3-april/): Enterprise Version 2.12.4
- [Observability Release Notes](/observability/release-notes/): Find links here to release notes for user-facing features, fixes, and enhancements in DataKitchen's DataOps Observability.
- [15 December](/observability/release-notes/open-source/2022/15-december/): Alpha release; User interface version 0.63.0
- [16 November](/observability/release-notes/open-source/2022/16-november/): Alpha release; User interface version 0.57.7
- [19 October](/observability/release-notes/open-source/2022/19-october/): Alpha release; User interface version 0.55.4
- [Alpha Release](/observability/release-notes/open-source/2022/alpha-release/): This initial, limited availability release includes the following basic monitoring and alerting features of the Observability module.
- [Alpha Release Breaking Changes](/observability/release-notes/open-source/2022/alpha-release-breaking-changes/): The expected release date for these changes is the end of January 2023.
- [Deprecation Notes](/observability/release-notes/open-source/2022/deprecation-notes/): To iterate on features, improve consistency, and develop for scalability, the following features will be removed in future releases.
- [1 February](/observability/release-notes/open-source/2023/1-february/): Alpha release; User interface version 0.70.0
- [1 March](/observability/release-notes/open-source/2023/1-march/): Application version 1.0; User interface version 1.0
- [1 May](/observability/release-notes/open-source/2023/1-may/): Application version 1.0; User interface version 1.21
- [10 October](/observability/release-notes/open-source/2023/10-october/): Application version 1.0; User interface version 1.77.0
- [12 June](/observability/release-notes/open-source/2023/12-june/): Application version 1.0; User interface version 1.37.0
- [14 November](/observability/release-notes/open-source/2023/14-november/): Application version 1.0; User interface version 1.85.0
- [17 July](/observability/release-notes/open-source/2023/17-july/): Application version 1.0; User interface version 1.48.1
- [18 April](/observability/release-notes/open-source/2023/18-april/): Application version 1.0; User interface version 1.20.4
- [18 January](/observability/release-notes/open-source/2023/18-january/): Alpha release; User interface version 0.68.1
- [18 May](/observability/release-notes/open-source/2023/18-may/): Application version 1.0; User interface version 1.28.1
- [18 September](/observability/release-notes/open-source/2023/18-september/): Application version 1.0; User interface version 1.71.0
- [21 August](/observability/release-notes/open-source/2023/21-august/): Application version 1.0; User interface version 1.61.2
- [23 February](/observability/release-notes/open-source/2023/23-february/): Alpha; User interface version 0.79
- [23 May](/observability/release-notes/open-source/2023/23-may/): Application version 1.0; User interface version 1.30.0
- [25 May](/observability/release-notes/open-source/2023/25-may/): Application version 1.0; User interface version 1.31
- [25 October](/observability/release-notes/open-source/2023/25-october/): Application version 1.0; User interface version 1.83.0
- [26 December](/observability/release-notes/open-source/2023/26-december/): Application version 1.0; User interface version 1.93.0
- [26 September](/observability/release-notes/open-source/2023/26-september/): Application version 1.0; User interface version 1.75.2
- [27 June](/observability/release-notes/open-source/2023/27-june/): Application version 1.0; User interface version 1.43.4
- [27 March](/observability/release-notes/open-source/2023/27-march/): Application version 1.0; User interface version 1.4.2
- [27 November](/observability/release-notes/open-source/2023/27-november/): Application version 1.0; User interface version 1.88.2
- [28 August](/observability/release-notes/open-source/2023/28-august/): Application version 1.0; User interface version 1.61.2
- [3 April](/observability/release-notes/open-source/2023/3-april/): Application version 1.0; User interface version 1.8
- [30 August](/observability/release-notes/open-source/2023/30-august/): Application version 1.0; User interface version 1.61.2
- [31 July](/observability/release-notes/open-source/2023/31-july/): Application version 1.0; User interface version 1.51.0
- [4 May](/observability/release-notes/open-source/2023/4-may/): Application version 1.0 User interface version 1.23.8
- [5 June](/observability/release-notes/open-source/2023/5-june/): Application version 1.0; User interface version 1.36.2
- [6 December](/observability/release-notes/open-source/2023/6-december/): Application version 1.0; User interface version 1.90.0
- [7 March](/observability/release-notes/open-source/2023/7-march/): Application version 1.0; User interface version 1.0
- [7 September](/observability/release-notes/open-source/2023/7-september/): Application version 1.0; User interface version 1.66.2
- [8 May](/observability/release-notes/open-source/2023/8-may/): Application version 1.0; User interface version 1.23.11
- [8 November](/observability/release-notes/open-source/2023/8-november/): Application version 1.0; User interface version 1.83.3
- [9 October](/observability/release-notes/open-source/2023/9-october/): Application version 1.0; User interface version 1.77.0
- [13 May](/observability/release-notes/open-source/2024/13-may/): Open Source Version 2.0.2
- [20 August](/observability/release-notes/open-source/2024/20-august/): Open Source Version 2.2.0
- [20 November](/observability/release-notes/open-source/2024/20-november/): Open Source Version 2.3.0
- [21 August](/observability/release-notes/open-source/2024/21-august/): Open Source Version 2.2.1
- [21 May](/observability/release-notes/open-source/2024/21-may/): Open Source Version 2.0.3
- [22 April](/observability/release-notes/open-source/2024/22-april/): Open Source Version 2.0
- [11 April](/observability/release-notes/open-source/2025/11-april/): Open Source Version 2.5.0
- [25 August](/observability/release-notes/open-source/2025/25-august/): Open Source Version 2.7.5
- [27 February](/observability/release-notes/open-source/2025/27-february/): Open Source Version 2.3.2
- [28 July](/observability/release-notes/open-source/2025/28-july/): Open Source Version 2.7.4
- [5 November](/observability/release-notes/open-source/2025/5-november/): Open Source Version 2.8.0
- [6 March](/observability/release-notes/open-source/2025/6-march/): Open Source Version 2.4.0
- [11 March](/observability/release-notes/open-source/2026/11-march/): Open Source Version 2.9.0
- [21 March](/observability/release-notes/open-source/2026/21-march/): Open Source Version 2.11.3
- [3 April](/observability/release-notes/open-source/2026/3-april/): Open Source Version 2.12.4
- [Define Rules](/observability/rules/define-rules/): With Observability, you can create rules that notify you of important situations. When the rule condition is met, Observability will send an email to specific recipients or a web request to your...
- [Rules](/observability/rules/): Use rules to define and capture meaningful incidents and results, so you can stay up-to-date and react to situations as they occur.
- [Rule Patterns and Examples](/observability/rules/rule-patterns-and-examples/): Observability provides patterns to help with the construction of meaningful rules based on a standard trigger-condition-action format. The following tables give some examples of the conditions and...
- [Rule Triggers](/observability/rules/rule-triggers/): Observability provides patterns to help with the construction of meaningful rules based on a standard trigger-condition-action format.
- [Web Request](/observability/rules/web-request/): Configure an outgoing web request to send messages or to trigger workflows in your external apps. Start by defining rules for the components in a journey, then, when a rule condition is true, a web...
- [JSON Data Points](/observability/rules/web-request/json-data-points/): The following table shows the data points that are available to use when you create a web request action for a defined rule. These data points can be included in the JSON payload using the format:
- [Web Request Walkthroughs](/observability/rules/web-request-walkthroughs/): These walkthroughs show how you can use an Observability rule to set up a webhook action in Slack, Jira, and Teams—helping you remain up-to-date on events happening in your data estate.
- [Jira](/observability/rules/web-request-walkthroughs/jira/): Use case: When a task fails for a specific batch pipeline in a journey, Observability opens a detailed ticket for you in Jira.
- [Slack](/observability/rules/web-request-walkthroughs/slack/): Use case: As tasks for the batch pipelines in a journey change status, Observability sends the message "Task <name> is now <completed>. Link: <Personalized link to run details in...
- [Teams](/observability/rules/web-request-walkthroughs/teams/): Use case: When a task fails for any batch pipeline in a journey, Observability sends you a detailed message, directly in Microsoft Teams.
- [Empty Instances](/observability/troubleshooting/empty-instances/): One of the following tooltips appears on the Project Overview page to alert you:
- [Inactive Agent](/observability/troubleshooting/inactive-agent/): Use the following troubleshooting steps if you feel that an agent has become inactive or is not sending events to Observability.
- [Observability Troubleshooting](/observability/troubleshooting/): The following documents are designed to help you troubleshoot issues you may experience in DataKitchen's DataOps Observability.
- [Journey Sequence Not Met](/observability/troubleshooting/journey-sequence-not-met/): The following banner appears in Observability to alert you:
- [Pending or Missing Runs](/observability/troubleshooting/pending-or-missing-runs/): One of the following run statuses appears in Observability to alert you:
- [Request Too Large](/observability/troubleshooting/request-too-large/): The following error message appears in an API call, in the browser devtools, or in the UI to alert you:
- [What is Observability?](/observability/what-is-observability/): DataKitchen's DataOps Observability monitors every data journey from data source to customer value, from any team development environment into production, across every tool, team, environment, and...

## DataOps Automation
- [Agent Resources](/automation/agents/agent-resources/): Each agent runs orders concurrently, limited only by computing and storage resources. Kubernetes automatically scales the number of available agents to handle as many concurrent order runs as needed.
- [Automation Agents](/automation/agents/): An Automation agent is a Kubernetes worker node located within a customer's Kubernetes cluster. Agents:
- [Specify Agent Groups](/automation/agents/specify-agent-groups/): Once an organization chooses to divide its agent pool into named agent groups, users can specify a valid agent group for each kitchen.
- [Configure Alert Thresholds](/automation/alerts/configure-alert-thresholds/): The alerts for test warnings and order runs exceeding a time limit require threshold settings at the variation level, regardless of where the alerts are configured (i.e. at the kitchen or recipe...
- [Configure Kitchen Alerts](/automation/alerts/configure-kitchen-alerts/): You can configure kitchen-level alerts from the:
- [Configure Recipe Alerts](/automation/alerts/configure-recipe-alerts/): Recipe-level alerts are useful for temporary, localized testing. This type of alert can be configured in the UI or directly in the recipe's description.json file.
- [Alerts](/automation/alerts/): Automation sends alerts by email for various states of every order run. You can configure notification recipients at both the kitchen and recipe levels.
- [DKUtils](/automation/api/dkutils/): DataKitchen provides a Python package, DKUtils, to help facilitate work with the Automation API.
- [Automation API](/automation/api/): The Automation API is a set of endpoints used by both the Automation system and DKCloudCommand.
- [Request Order Run IDs via API](/automation/api/request-order-run-ids-via-api/): The following example shows how to request the order run IDs associated with a given order ID.
- [Run Orders via API](/automation/api/run-orders-via-api/): Use the following information to run an order using the Automation API.
- [Serve Order Run Info via API](/automation/api/serve-order-run-info-via-api/): Users can request information about an order run or many order runs using the Automation API.
- [Submit Orders via API](/automation/api/submit-orders-via-api/): You can use the Automation API to start or schedule recipe orders.
- [Core Concepts in Automation](/automation/architecture-and-concepts/core-concepts/): The definitions below can help you understand how Automation terminology maps to standard industry terms.
- [Architecture & Concepts](/automation/architecture-and-concepts/): Learn about the technical architecture, key concepts, and operational details of DataOps Automation.
- [Login](/automation/architecture-and-concepts/login/): Your login credentials for access to Automation will be provided by DataKitchen.
- [Naming Conventions](/automation/architecture-and-concepts/naming-conventions/): All objects in Automation should follow these naming conventions and best practices.
- [System Architecture](/automation/architecture-and-concepts/system-architecture/): DataKitchen DataOps Automation was designed with DataOps elements to support rapid change, early error detection, and efficient development. It expands the traditional operations-oriented data...
- [User Account and Sessions](/automation/architecture-and-concepts/user-sessions/): You can access your account information at any time. Log into Automation, then select your username from the gray menu.
- [Versioning and Deployments](/automation/architecture-and-concepts/versioning-deployments/): The recipes contained within any kitchen are subject to version control. Recipe contents (code, configuration, and resource files) are versioned via Git, which is accomplished by underpinning each...
- [Data Source SQL Fields](/automation/data-sources-and-data-sinks/data-source-sql-fields/): The sql and sql-file fields offer methods for injecting inline SQL statements or loading SQL files during node execution.
- [File Settings for SQL Connectors](/automation/data-sources-and-data-sinks/file-settings-for-sql-connectors/): When a recipe node is configured with a SQL database source or sink—to write to or read from a file—the system needs the file format and instruction on how to map and format the data.
- [Data Sources and Data Sinks](/automation/data-sources-and-data-sinks/): Data sources and data sinks are node input/output components that connect to toolchain infrastructure in order to get data to and push data from recipe nodes.
- [Built-In Runtime Variables](/automation/data-sources-and-data-sinks/properties-and-variables/built-in-runtime-variables/): Data sources and sinks provide a number of optional built-in runtime variables as a result of their execution, which can be used in automated tests for quality assurance.
- [Connection Properties](/automation/data-sources-and-data-sinks/properties-and-variables/connection-properties/): Configuration properties required to connect to data sources and data sinks are specific to the source/sink type. The connection configurations can be expressed in two ways: expanded or condensed.
- [Database Properties](/automation/data-sources-and-data-sinks/properties-and-variables/database-properties/): Unless otherwise noted, these properties can be used when configuring nodes for any database data source or sink.
- [Database Variables](/automation/data-sources-and-data-sinks/properties-and-variables/database-variables/): Unless otherwise noted, these built-in runtime variables can be used for all database data sources and sinks. All of these variables are optional.
- [File-Based Properties](/automation/data-sources-and-data-sinks/properties-and-variables/file-based-properties/): Unless otherwise noted, these properties can be used when configuring nodes for any file-based data source or sink.
- [File-Based Variables](/automation/data-sources-and-data-sinks/properties-and-variables/file-based-variables/): Unless otherwise noted, these built-in runtime variables can be used for all file-based data sources and sinks. All of these variables are optional.
- [File-Based Wildcards](/automation/data-sources-and-data-sinks/properties-and-variables/file-based-wildcards/): File-based data sources and sinks allow users to extract and load an arbitrary number of files by supporting a wildcard feature using Glob syntax (*).
- [Properties and Variables](/automation/data-sources-and-data-sinks/properties-and-variables/): The available configuration fields, required properties, optional properties, and runtime variables are specific to the data source or data sink type (file-based or database). In some cases, the...
- [System Connector Properties](/automation/data-sources-and-data-sinks/properties-and-variables/system-connector-properties/): Unless otherwise noted, these properties may be used when configuring nodes for any data source or sink in the system category.
- [Source and Sink Processing](/automation/data-sources-and-data-sinks/source-sink-processing/): Within a given node, data sources are processed first, one by one, followed by the node's notebook.json, and finally, the node's data sinks, one by one. See File Evaluation Sequence for more...
- [Sources and Sinks in Container Nodes](/automation/data-sources-and-data-sinks/sources-sinks-in-container-nodes/): For container nodes, data sources retrieve data and place it inside the container by input files. Data sinks move data out of the container by output files.
- [Amazon Redshift/PostgreSQL](/automation/data-sources-and-data-sinks/supported-connectors/amazon-redshift-postgresql/): Amazon Redshift/PostgreSQL data sources and sinks are in the database category of I/O connectors.
- [Amazon S3](/automation/data-sources-and-data-sinks/supported-connectors/amazon-s3/): Amazon S3 data sources and sinks are in the file-based category of I/O connectors.
- [Azure Blob Storage](/automation/data-sources-and-data-sinks/supported-connectors/azure-blob-storage/): Azure Blob Storage data sources and sinks are in the file-based category of I/O connectors.
- [Azure Data Lake Storage Gen2](/automation/data-sources-and-data-sinks/supported-connectors/azure-data-lake-storage-gen2/): Azure Data Lake Storage data sources and sinks are in the file-based category of I/O connectors.
- [DB2](/automation/data-sources-and-data-sinks/supported-connectors/db2/): DB2 data sources and sinks are in the database category of I/O connectors.
- [Dictionary Setup Examples](/automation/data-sources-and-data-sinks/supported-connectors/dictionary/dictionary-setup-examples/): The following content offers a guided tour of a dictionary data source and data sink configuration and processing in a single container node.
- [Dictionary](/automation/data-sources-and-data-sinks/supported-connectors/dictionary/): Dictionary data sources and sinks enable recipes to use in-memory data as runtime variables or employ Jinja functions to manage or transform files for downstream inputs and testing. It is essentially...
- [FTP](/automation/data-sources-and-data-sinks/supported-connectors/ftp/): FTP data sources and sinks are in the file-based category of I/O connectors.
- [Supported Connectors](/automation/data-sources-and-data-sinks/supported-connectors/): DataOps Automation supports the following data sources and data sinks.
- [Microsoft SQL Server/Azure Synapse](/automation/data-sources-and-data-sinks/supported-connectors/microsoft-sql-server-azure-synapse/): Microsoft SQL Server (MS SQL)/Azure Synapse data sources and sinks are in the database category of I/O connectors.
- [MySQL](/automation/data-sources-and-data-sinks/supported-connectors/mysql/): MySQL data sources and sinks are in the database category of I/O connectors.
- [Oracle](/automation/data-sources-and-data-sinks/supported-connectors/oracle/): Oracle data sources and sinks are in the database category of I/O connectors.
- [Salesforce](/automation/data-sources-and-data-sinks/supported-connectors/salesforce/): Salesforce data sources are in the database category of I/O connectors.
- [SFTP](/automation/data-sources-and-data-sinks/supported-connectors/sftp/): SFTP data sources and sinks are in the file-based category of I/O connectors.
- [Snowflake](/automation/data-sources-and-data-sinks/supported-connectors/snowflake/): Snowflake data sources and sinks are in the database category of I/O connectors.
- [Teradata](/automation/data-sources-and-data-sinks/supported-connectors/teradata/): Teradata data sources and sinks are in the database category of I/O connectors.
- [Tips on Key Generation](/automation/data-sources-and-data-sinks/tips-on-key-generation/): When you use SFTP and other file-based data sources and sinks, it's important that the generation of encryption keys and PEM files follow guidelines to avoid runtime connection issues.
- [Commands Reference](/automation/dkcloudcommand/command-and-option-references/commands-reference/): See Options Reference for a list and descriptions of command options.
- [Command and Option References](/automation/dkcloudcommand/command-and-option-references/): The DKCloudCommand command line interface (CLI) provides many commands and options to perform DataOps work.
- [Options Reference](/automation/dkcloudcommand/command-and-option-references/options-reference/): See Commands Reference for a complete list of the CLI commands and the options available for each.
- [Configure DKCloudCommand](/automation/dkcloudcommand/configure-dkcloudcommand/): Use the config command to configure DKCloudCommand.
- [active-serving-watcher](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/active-serving-watcher/): Shortcut: asw
- [Common DKCloudCommand Sequences](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/common-dkcloudcommand-sequences/): Common command sequences are shown below. Note that references to specific kitchens, recipes, variations, and file paths are omitted here for clarity of command sequencing.
- [config](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/config/): Shortcut: c
- [config-list](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/config-list/): Shortcut: cl
- [context-delete](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/context-delete/): Shortcut: cd
- [context-list](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/context-list/): Shortcut: xl
- [context-switch](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/context-switch/): Shortcut: cs
- [file-compile](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/file-compile/): Shortcut: fc
- [file-delete](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/file-delete/): Shortcut: fd
- [file-diff](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/file-diff/): Shortcut: fdi
- [file-get](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/file-get/): Shortcut: fg
- [file-history](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/file-history/): Shortcut: fh
- [file-merge](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/file-merge/): Shortcut: fm
- [file-resolve](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/file-resolve/): Shortcut: fr
- [file-update](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/file-update/): Shortcut: fu
- [git-setup](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/git-setup/): Shortcut: gs
- [DKCloudCommand Walkthroughs](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/): The following topics contain information on DKCloudCommand best practices, as well as details for each CLI command with usage syntax, available options, and examples.
- [kitchen-config](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/kitchen-config/): Shortcut: kf
- [kitchen-create](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/kitchen-create/): Shortcut: kc
- [kitchen-delete](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/kitchen-delete/): Shortcut: kd
- [kitchen-get](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/kitchen-get/): Shortcut: kg
- [kitchen-history](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/kitchen-history/): Shortcut: kh
- [kitchen-list](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/kitchen-list/): Shortcut: kl
- [kitchen-merge](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/kitchen-merge/): Shortcut: km
- [kitchen-merge-preview](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/kitchen-merge-preview/): Shortcut: kmp
- [kitchen-revert](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/kitchen-revert/): Shortcut: kr
- [kitchen-settings-get](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/kitchen-settings-get/): Shortcut: ksg
- [kitchen-settings-update](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/kitchen-settings-update/): Shortcut: ksu
- [kitchen-which](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/kitchen-which/): Shortcut: kw
- [order-delete](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/order-delete/): Shortcut: od
- [order-list](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/order-list/): Shortcut: ol
- [order-run](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/order-run/): Shortcut: or
- [order-stop](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/order-stop/): Shortcut: os
- [orderrun-delete](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/orderrun-delete/): Shortcut: ord
- [orderrun-info](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/orderrun-info/): Shortcut: ori
- [orderrun-resume](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/orderrun-resume/): Shortcut: orr
- [orderrun-stop](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/orderrun-stop/): Shortcut: ors
- [recipe-compile](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/recipe-compile/): Shortcut: rc
- [recipe-copy](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/recipe-copy/): Shortcut: ry
- [recipe-create](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/recipe-create/): Shortcut: re
- [recipe-delete](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/recipe-delete/): Shortcut: rd
- [recipe-get](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/recipe-get/): Shortcut: rg
- [recipe-ingredient-list](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/recipe-ingredient-list/): Shortcut: ril
- [recipe-list](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/recipe-list/): Shortcut: rl
- [recipe-status](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/recipe-status/): Shortcut: rs
- [recipe-update](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/recipe-update/): Shortcut: ru
- [recipe-validate](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/recipe-validate/): Shortcut: rv
- [recipe-variation-list](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/recipe-variation-list/): Shortcut: rvl
- [secret-delete](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/secret-delete/): Shortcut: sd
- [secret-exists](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/secret-exists/): Shortcut: se
- [secret-list](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/secret-list/): Shortcut: sl
- [secret-write](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/secret-write/): Shortcut: sw
- [Useful DKCloudCommand Techniques](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/useful-dkcloudcommand-techniques/): The following collection comes from veteran DK command line users. The ideas here have not received thorough platform testing, and are provided as suggestions. Use at your own risk.
- [user-info](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/user-info/): Shortcut: ui
- [vault-config](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/vault-config/): Shortcut: vc
- [vault-delete](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/vault-delete/): Shortcut: vd
- [vault-info](/automation/dkcloudcommand/dkcloudcommand-walkthroughs/vault-info/): Shortcut: vi
- [DKCloudCommand](/automation/dkcloudcommand/): DKCloudCommand is the command line tool made for interacting with the DataKitchen Automation API. If you prefer to operate at the command line, rather than use the Automation UI, can do so with...
- [Install and Upgrade DKCloudCommand](/automation/dkcloudcommand/install-upgrade-dkcloudcommand/): Use pip3 to install the Automation command line interface (CLI).
- [Kitchen Settings](/automation/dkcloudcommand/kitchen-settings/): Certain kitchen configurations are governed by the kitchen-settings.json file, including the Create Kitchen wizard. This file is global to all kitchens within a customer account and is only...
- [Manage User Contexts](/automation/dkcloudcommand/manage-user-contexts/): DKCloudCommand supports the configuration of multiple users, both within the same account and across customer accounts. Each of these configurations is called a context.
- [Set Up Local Workspace](/automation/dkcloudcommand/set-up-local-workspace/): Kitchens, recipes, and recipe variations can optionally exist on a local machine.
- [Set Up Merge and Diff Tool](/automation/dkcloudcommand/set-up-merge-diff-tool/): Before configuring DKCloudCommand, set up a local tool to visualize merge conflicts and differences between recipe files.
- [Call Ingredients from Kitchen Wizard](/automation/ingredients/call-ingredients-from-kitchen-wizard/): With an ingredient variation declared, you can call it:
- [Call Ingredients from Recipe Nodes](/automation/ingredients/call-ingredients-from-recipe-nodes/): With an ingredient variation declared, you can call it:
- [Declare Ingredients](/automation/ingredients/declare-ingredients/): The first step required to use any ingredient is to declare a recipe variation as an ingredient variation.
- [Ingredients](/automation/ingredients/): In addition to doing data work for data pipelines, Automation recipe variations can also be built to automate the provisioning and configuration of infrastructure to run data pipelines; to store...
- [Inflowing Variable Validations](/automation/ingredients/inflowing-variable-validations/): Inflowing variables supply values for the execution of an ingredient variation.
- [Results of Ingredient Tests](/automation/ingredients/results-of-ingredient-tests/): Test results from ingredient variation runs can affect the way the system processes the ingredient node calling that ingredient variation. The ingredient tests do not, however, affect the tests...
- [Configure Kitchens](/automation/kitchens/configure-kitchens/): The Configure Kitchen page offers settings for user access, kitchen override variables, agent group, kitchen-level alerts, and a function to delete the kitchen.
- [Create Kitchens](/automation/kitchens/create-kitchens/): Create a child kitchen to do work and make modifications, without impacting the work of others. Child kitchens inherit the kitchen user list, overrides, and recipe copies from the parent kitchen.
- [Delete Kitchens](/automation/kitchens/delete-kitchens/): When you delete a kitchen, you permanently remove the resources. This action cannot be undone.
- [Kitchens](/automation/kitchens/): In DataKitchen's DataOps Automation, users perform all work within virtual workspaces called Kitchens.
- [Kitchen History](/automation/kitchens/kitchen-history/): Each kitchen has a Git branch to store recipes and maintain version control. The kitchen History page displays the branch commits and provides a centralized view of a kitchen's complete update...
- [Kitchen Security](/automation/kitchens/kitchen-security/): In Automation, kitchen security consists of two factors: the connection credentials that provide access to your external data tools and the kitchen-level access for tasks and daily work.
- [Kitchen Roles](/automation/kitchens/kitchen-security/kitchen-roles/): Kitchen roles is a feature that uses custom roles to define kitchen access on a per-user level.
- [Manage Access](/automation/kitchens/kitchen-security/manage-access/): Each kitchen has a designated user list that limits access to managing a kitchen's infrastructure, editing its recipes, and running its orders. Kitchens are locked to users who have not been added to...
- [Set Kitchen Type](/automation/kitchens/kitchen-security/set-kitchen-type/): Minimize the cost for analytic team members to experiment by giving them isolated and disposable workspaces that reflect their production environment. Best practice during kitchen creation is to...
- [Kitchen Users](/automation/kitchens/kitchen-users/): Kitchen users are people with different skill sets, using different tools, working together to deliver high-quality insights. Every kitchen is defined, in part, by the users that have access to do...
- [Kitchens Page](/automation/kitchens/kitchens-page/): In Automation, all work is performed within virtual workspaces called kitchens. When you log in, the first view is of the Kitchen page. This page serves as a dashboard listing all the kitchens in...
- [Merge Kitchens](/automation/kitchens/merge-kitchens/): Most child kitchens are temporary workspaces intended only for development work. When users complete that work, they can merge it into a parent kitchen. Behind the scenes, a kitchen merge is a merge...
- [Revert Kitchens](/automation/kitchens/revert-kitchens/): Kitchen revert returns a kitchen to the immediate previous state, one state at a time, from most recent to oldest.
- [Revert Process](/automation/kitchens/revert-kitchens/revert-process/): While users create temporary kitchens as experimental sandboxes, your Automation tenant likely has kitchens designated as development and production environments. Merges into these environments are...
- [File Evaluation Sequence](/automation/orders/file-evaluation-sequence/): The following sequence describes the underlying actions performed by the system during the execution of recipe variation files.
- [Orders](/automation/orders/): The processing of a recipe variation is known as running an order. Orders are specific to a kitchen-recipe-variation combination.
- [Order Runs](/automation/orders/order-runs/): Each order contains at least one order run. Order runs represent the distinct, one-time processing of a recipe variation in a kitchen.
- [Order Run Details](/automation/orders/order-runs/order-run-details/): Find information and results on each order run in the UI or from the command line.
- [Order Run Logs](/automation/orders/order-runs/order-run-logs/): The Order Run Logs show a list of timestamps, nodes, and messages for the processing of a recipe variation.
- [Order Run Test Results](/automation/orders/order-runs/order-run-test-results/): Automation provides test results for three use cases:
- [Order Run Timings](/automation/orders/order-runs/order-run-timings/): Automation provides time result data for three use cases:
- [Run an Order](/automation/orders/run-an-order/): The processing of a recipe variation is known as running an order. Orders are specific to a kitchen-recipe-variation combination.
- [Order Run Execution](/automation/orders/run-an-order/order-run-execution/): The following provides a summary of what happens during an order run.
- [Order-Time Overrides](/automation/orders/run-an-order/order-time-overrides/): When submitting an order, you have the option to pass runtime overrides.
- [Resume Order Runs](/automation/orders/run-an-order/resume-order-runs/): Both the Resume Order Run function in the UI and the orderrun-resume command in the CLI give you an opportunity to resolve problems in your recipe variation and continue with order processing.
- [Run One Node](/automation/orders/run-an-order/run-one-node/): You can run a single node in a graph. Use this to test iteratively while building a recipe.
- [Runtime Resources](/automation/orders/runtime-resources/): Recipes contain at least one configured resource allocation that assigns memory and disk space on the agent for each order run that's processed. These upper bounds are configured to prevent runaway...
- [Cleanse and Test Raw Data](/automation/recipe-nodes/action-nodes/cleanse-and-test-raw-data/)
- [Configure Action Nodes](/automation/recipe-nodes/action-nodes/configure-action-nodes/): Configure an action node from the Node Editor.
- [Copy and Compute Data Example](/automation/recipe-nodes/action-nodes/copy-and-compute-data-example/)
- [Create an Action Node](/automation/recipe-nodes/action-nodes/create-an-action-node/): There are two steps to add an action node to a recipe graph: create the node, then connect the node to other nodes in the graph.
- [Action Nodes](/automation/recipe-nodes/action-nodes/): With this node type, data sources are intended to perform operations on sources that do not require the retrieval of data. Action nodes do not use data sinks.
- [Compile Node Files](/automation/recipe-nodes/compile-node-files/): As you build recipe nodes, you can use the Compile function (in either the Node Editor or File Editor) to see how the system will read your code. The manual compile process applies values to variable...
- [Check User Status Example](/automation/recipe-nodes/conditional-nodes/check-user-status-example/)
- [Conditional Node Processing](/automation/recipe-nodes/conditional-nodes/conditional-node-processing/): The following examples show how conditional nodes can affect graph processing based on the evaluation of logic statements.
- [Create a Conditional Node](/automation/recipe-nodes/conditional-nodes/create-a-conditional-node/): There are two steps to add a conditional node to a recipe graph: create the node with conditions, then connect the node to other nodes in the graph.
- [Edit a Conditional Node](/automation/recipe-nodes/conditional-nodes/edit-a-conditional-node/): Users can edit all fields of an existing conditional node, except for the node name.
- [Conditional Nodes](/automation/recipe-nodes/conditional-nodes/): Unlike the analytic node types, conditional nodes provide flow control in graph processing. Conditional nodes provide a way to configure a graph to either run or skip some—or all—downstream nodes...
- [Multiple Results Example](/automation/recipe-nodes/conditional-nodes/multiple-results-example/)
- [Process on True Example](/automation/recipe-nodes/conditional-nodes/process-on-true-example/)
- [Processing a Subset Example](/automation/recipe-nodes/conditional-nodes/processing-a-subset-example/)
- [Configure Container Nodes](/automation/recipe-nodes/container-nodes/configure-container-nodes/): Configure a container node from the Node Editor.
- [Container Images](/automation/recipe-nodes/container-nodes/container-images/): To ensure that processes execute efficiently and reliably—and without the need for custom or external infrastructure—Automation uses containers to package code, dependencies, and settings. Automation...
- [Container Node Examples](/automation/recipe-nodes/container-nodes/container-node-examples/): Use this expression to inject files from a data source inside a container. Where:
- [Container Node File Structure](/automation/recipe-nodes/container-nodes/container-node-file-structure/): Automation provides several analytic container images, pre-configured and ready to support any database, computing, programming, provisioning, data visualization, reporting, and business intelligence...
- [Container Node Tests](/automation/recipe-nodes/container-nodes/container-node-tests/): You can define tests in container nodes based on when you want the tests to run, in relation to the node's source and sink operations.
- [Container Notebook Properties](/automation/recipe-nodes/container-nodes/container-notebook-properties/): The properties below are specific to container nodes. The table includes the fields seen in the Node Editor and the corresponding JSON fields.
- [Create a Container Node](/automation/recipe-nodes/container-nodes/create-a-container-node/): Warning
- [Container Nodes](/automation/recipe-nodes/container-nodes/): Automation uses Docker containers to package code, runtime scripts, tools, and other settings so that processes can execute efficiently and reliably without requiring custom infrastructure.
- [Manage Container Icons](/automation/recipe-nodes/container-nodes/manage-container-icons/): When you create or edit a container node in a recipe variation, rather than use the default Docker icon, you can select a custom icon to represent that node in the graph view.
- [Configure DataMapper Nodes](/automation/recipe-nodes/datamapper-nodes/configure-datamapper-nodes/): Configure a DataMapper node from the Node Editor.
- [Create a DataMapper Node](/automation/recipe-nodes/datamapper-nodes/create-a-datamapper-node/): There are two steps to add a DataMapper node to a recipe graph: create the node, then connect the node to other nodes in the graph.
- [DataMapper Node Files](/automation/recipe-nodes/datamapper-nodes/datamapper-node-files/): All mapping, connection, and test configurations of a DataMapper node are reflected in the node files accessible from the File Editor or from DKCloudCommand, the command line interface (CLI).
- [DataMapper Node Properties](/automation/recipe-nodes/datamapper-nodes/datamapper-node-properties/): These fields are found in a DataMapper node's notebook.json file and in the Mappings tab of the Node Editor.
- [DataMapper Nodes](/automation/recipe-nodes/datamapper-nodes/): This node type moves data from a data source to a data sink destination by "mapping" the data between the two locations. DataMapper nodes can map single files between a source and a sink, or they can...
- [Mapping File to Database Example](/automation/recipe-nodes/datamapper-nodes/mapping-file-to-database-example/)
- [Mapping with Variables Example](/automation/recipe-nodes/datamapper-nodes/mapping-with-variables-example/)
- [Wildcard Mapping Example](/automation/recipe-nodes/datamapper-nodes/wildcard-mapping-example/): This example demonstrates the use of wildcard mappings to extract multiple files from a source. In this case, the system retrieves all files that match the wildcard expression input-*.txt from the...
- [GPC Basic Example](/automation/recipe-nodes/general-purpose-container/gpc-basic-example/): This code-based example demonstrates the use of variables, adding tests, and general configuration.
- [GPC Best Practices](/automation/recipe-nodes/general-purpose-container/gpc-best-practices/): This document provides some best practices that can help you get the most out of a General Purpose Container (GPC) container node.
- [GPC description.json](/automation/recipe-nodes/general-purpose-container/gpc-file-structure-and-configuration/gpc-description-json/): The required description.json file defines the node type and node icon (if the icon is not the default Docker icon).
- [GPC docker-share Directory](/automation/recipe-nodes/general-purpose-container/gpc-file-structure-and-configuration/gpc-docker-share-directory/): This required directory stores any scripts or files that need to be executed in the node.
- [GPC notebook.json](/automation/recipe-nodes/general-purpose-container/gpc-file-structure-and-configuration/gpc-notebook-json/): The required notebook.json file defines the container configuration.
- [GPC File Structure and Configuration](/automation/recipe-nodes/general-purpose-container/gpc-file-structure-and-configuration/): Automation provides a pre-configured General Purpose Container (GPC) container image ready to support any database, computing, programming, provisioning, data visualization, reporting, and business...
- [GPC Logging](/automation/recipe-nodes/general-purpose-container/gpc-logging/): The GPC includes LOGGER, a global logging object that wraps Python3 logging. You can use logger to record information, such as error messages, directly in your order run logs.
- [GPC Pre-Installed Packages](/automation/recipe-nodes/general-purpose-container/gpc-pre-installed-packages/): The General Purpose Container (GPC) is built with several tools pre-installed. You can reference any of these apps in your scripts and/or specify other apt-get and Python packages to be installed at...
- [GPC Resource Allocation](/automation/recipe-nodes/general-purpose-container/gpc-resource-allocation/): For resource-intensive order runs—for example, those with large queries, heavy use of variables, or data sources and sinks—you can assign memory and disk space. These allocations can be set for the...
- [General Purpose Container](/automation/recipe-nodes/general-purpose-container/): Automation container nodes work by referencing container images that become Docker containers at runtime, allowing you to build recipes with processes that are containerized and portable.
- [Python Dependency Example](/automation/recipe-nodes/general-purpose-container/python-dependency-example/): This code-based example shows how to add a Python dependency to the container in order to interact with an external tool, such as Tableau.
- [Shared Resources Example](/automation/recipe-nodes/general-purpose-container/shared-resources-example/): This code-based example demonstrates how to use a shared resource from a recipe's resources directory. The files in that directory are available to all nodes within a recipe.
- [Shell Script Exports](/automation/recipe-nodes/general-purpose-container/shell-script-exports/): The configuration for a GPC container node includes an export field that is supported only for Python scripts.
- [Source-Sink Example](/automation/recipe-nodes/general-purpose-container/source-sink-example/): This code-based example shows an active source and sink connected to the container. You can read files from any available data source, load them into a container, transform them using a script, then...
- [Vault Secrets in the GPC](/automation/recipe-nodes/general-purpose-container/vault-secrets-in-the-gpc/): The GPC allows the use of secrets only if they are injected by parameters (for Python scripts) or environment variables (for shell scripts or Python scripts) that are configured in the config.json...
- [Recipe Nodes](/automation/recipe-nodes/): Recipe nodes are isolated steps within a variation graph. Nodes are modular, mini-processes that perform work during an order run. To view all nodes that exist in a recipe, select the Nodes tab, or,...
- [Configure Ingredient Nodes](/automation/recipe-nodes/ingredient-nodes/configure-ingredient-nodes/): Configure an ingredient node from the Node Editor.
- [Create an Ingredient Node](/automation/recipe-nodes/ingredient-nodes/create-an-ingredient-node/): There are two steps to add an ingredient node to a recipe graph: create the node, then connect the node to other nodes in the graph.
- [Email Approval Utility](/automation/recipe-nodes/ingredient-nodes/email-approval-utility/)
- [Get Database Values for Testing](/automation/recipe-nodes/ingredient-nodes/get-database-values-for-testing/)
- [Ingredient Nodes](/automation/recipe-nodes/ingredient-nodes/): You can configure a recipe variation for reuse, to then be embedded within other recipe variation graphs. These reusable variations are called ingredients.
- [Simplify a Variation Graph](/automation/recipe-nodes/ingredient-nodes/simplify-a-variation-graph/)
- [Start/Stop a Cloud Resource](/automation/recipe-nodes/ingredient-nodes/start-stop-a-cloud-resource/): Example Ingredients: In this example there are two variations declared as ingredients that reside in a separate kitchen. One runs a start procedure, the other runs a stop procedure. These ingredients...
- [Node Components](/automation/recipe-nodes/node-components/): The files that make up a node follow a specific structure. Open the File Editor to view the node files for the nodes within a recipe.
- [Rename Nodes](/automation/recipe-nodes/rename-nodes/): Rename nodes from the Recipes page and in the File Editor.
- [Synchronize Nodes](/automation/recipe-nodes/synchronize-nodes/): Synchronize nodes contain no JSON objects (key/value pairs). These nodes typically serve as placeholders during graph creation or can be convergence steps in complex workflows to ensure that all...
- [Declare Runtime Variables](/automation/recipe-variables/declare-runtime-variables/): When a recipe variation runs, any declared runtime variable values are defined and redefined as they are encountered during recipe processing.
- [Define Kitchen Override Variables](/automation/recipe-variables/define-kitchen-override-variables/): You can define a kitchen override variables set from the Configure Kitchen page or from the command line. These overrides do not appear within recipe files; they are maintained outside of version...
- [Define Recipe Base Variables](/automation/recipe-variables/define-recipe-base-variables/): You can declare your recipe's base variables set from the Recipes page or by editing the variables.json file in the File Editor.
- [Define Variation Override Variables](/automation/recipe-variables/define-variation-override-variables/): You can define a variation override variables set from the Recipes page or by editing the variations.json file in the File Editor.
- [Recipe Variables](/automation/recipe-variables/): Automation uses variables of various types, scopes, classes, and override priorities to help you parameterize your processes.
- [Inject Order Override Variables](/automation/recipe-variables/inject-order-override-variables/): When you run an order, you have the option to inject an order override variable from the Run Variation dialog or from the command line with the order-run command.
- [Locate Variable Sets](/automation/recipe-variables/locate-variable-sets/): The Variable Sets tab on the Recipes page shows the base variable set for the recipe, any override variable sets that have been configured, and what variations use each variable set.
- [System Variables Reference](/automation/recipe-variables/system-variables-reference/): Automation's default system variables are read-only. You can reference these variables in any JSON or text file. Note that Python (.py) and shell scripts (.sh) cannot use these variables directly.
- [Variable Use Examples](/automation/recipe-variables/variable-use-examples/): This document provides visual and code excerpt examples of how you can apply variables when configuring nodes in the Node Editor or File Editor and when you configure variation variables.
- [Variable Hierarchy](/automation/recipe-variables/variables-hierarchy/): Recipe variables can be implemented as variable sets at different levels. The system overwrites a variable's value when that same variable is defined in two or more levels.
- [Variable Processing](/automation/recipe-variables/variables-processing/): Automation processes recipe variables as needed during order run execution. At the start of an order run, the system follows a specific hierarchy while compiling default and defined variables.
- [Advanced Jinja Templating](/automation/recipe-variables/variables-syntax/advanced-jinja-templating/): Warning
- [Basic Jinja Variable References](/automation/recipe-variables/variables-syntax/basic-jinja-variable-references/): When referencing variables, it is important to configure valid JSON. The examples below walk through the use of two variables defined as a string and a dictionary. Note how the static JSON files and...
- [Functions for Variable Transformation](/automation/recipe-variables/variables-syntax/functions-for-variable-transformation/): Built-in functions are available for variable transformation and can be called using Jinja syntax.
- [Variables Syntax](/automation/recipe-variables/variables-syntax/): The system supports Jinja variable references with {{ }} syntax in all recipe files—JSON files, and others that might be executed in a node, such as a SQL file.
- [Variables Scope](/automation/recipe-variables/variables-syntax/variables-scope/): By default, all variables are global but can be scoped to a recipe object, configured as private, or limited to a single Jinja template block within a file.
- [Add and Edit Variations](/automation/recipes/add-edit-variations/): You can create new recipe variations, and edit existing variations, from the Recipes page.
- [Add Graphs](/automation/recipes/add-graphs/): You can add new graphs to a recipe:​
- [Add Recipe Files](/automation/recipes/add-recipe-files/): You can upload files directly into a recipe's folder structure.
- [Build and Edit Graphs](/automation/recipes/build-edit-graphs/): You can add new graphs to a recipe:​
- [Graph Processing](/automation/recipes/graph-processing/): Recipe graphs process when users or schedules run orders for a recipe variation.
- [Graphs Best Practices](/automation/recipes/graphs-best-practices/): The following are best practices suggested by DataKitchen for success in DataOps Automation.
- [Recipes](/automation/recipes/): Automation recipes are collections of pipelines or processes that leverage tools and inputs to generate analytic insights.
- [Recipe Graphs](/automation/recipes/recipe-graphs/): Automation recipes are directed acyclic graphs (DAGs) with discrete steps that can either do data work or provision infrastructure.
- [Recipes Page](/automation/recipes/recipes-page/): The Recipes page offers recipe-level configurations and consolidates recipe component management. You can manage variations, nodes, graphs, variable sets (base variables and overrides), alerts, and...
- [Use the Recipe README](/automation/recipes/use-the-recipe-readme/): Every recipe has a README.md file at the root of its file structure. The README displays in the Wiki tab of the Recipes page.
- [Recipe Variations](/automation/recipes/variations/): Automation recipes are collections of pipelines or processes that leverage tools and inputs to generate analytic insights. Each recipe variation is a specific combination of recipe graph, order...
- [1 June](/automation/release-notes/2020/1-june/): Version 1.1.168
- [10 August](/automation/release-notes/2020/10-august/): Version 1.1.178. Note that version 1.1.177 was deployed to DataKitchen testing environments only.
- [11 May](/automation/release-notes/2020/11-may/): Version 1.1.165
- [12 October](/automation/release-notes/2020/12-october/): Version 1.1.187
- [13 July](/automation/release-notes/2020/13-july/): Version 1.1.174
- [14 December](/automation/release-notes/2020/14-december/): Version 1.1.194
- [14 September](/automation/release-notes/2020/14-september/): Version 1.1.183
- [15 June](/automation/release-notes/2020/15-june/): Version 1.1.170
- [16 November](/automation/release-notes/2020/16-november/): Version 1.1.192
- [17 August](/automation/release-notes/2020/17-august/): Version 1.1.179
- [18 May](/automation/release-notes/2020/18-may/): Version 1.1.166
- [19 October](/automation/release-notes/2020/19-october/): Version 1.1.188
- [2 November](/automation/release-notes/2020/2-november/): Version 1.1.190
- [20 April](/automation/release-notes/2020/20-april/): Version 1.1.162
- [20 July](/automation/release-notes/2020/20-july/): Version 1.1.175
- [21 December](/automation/release-notes/2020/21-december/): Version 1.1.195
- [21 September](/automation/release-notes/2020/21-september/): Version 1.1.184
- [22 June](/automation/release-notes/2020/22-june/): Version 1.1.171
- [26 May](/automation/release-notes/2020/26-may/): Version 1.1.167
- [27 April](/automation/release-notes/2020/27-april/): Version 1.1.163
- [27 July](/automation/release-notes/2020/27-july/): Version 1.1.176
- [28 September](/automation/release-notes/2020/28-september/): Version 1.1.185
- [29 June](/automation/release-notes/2020/29-june/): Version 1.1.172
- [4 May](/automation/release-notes/2020/4-may/): Version 1.1.164
- [5 October](/automation/release-notes/2020/5-october/): Version 1.1.186
- [6 April](/automation/release-notes/2020/6-april/): Version 1.0.161
- [6 July](/automation/release-notes/2020/6-july/): Version 1.1.173
- [7 December](/automation/release-notes/2020/7-december/): Versions 1.1.192.2, 1.1.193. NOTE: This release was delayed from its original 23 November release date.
- [8 June](/automation/release-notes/2020/8-june/): Version 1.1.169
- [8 September](/automation/release-notes/2020/8-september/): Version 1.1.182. Note that versions 1.1.180 and 1.1.181 were deployed to DataKitchen testing environments only.
- [9 November](/automation/release-notes/2020/9-november/): Version 1.1.191
- [1 February](/automation/release-notes/2021/1-february/): Version 1.1.201
- [1 March](/automation/release-notes/2021/1-march/): Version 1.1.205. Note that version 1.1.204 was deployed to DataKitchen testing environments only.
- [11 January](/automation/release-notes/2021/11-january/): Version 1.1.198. Note that versions 1.1.196 and 1.1.197 were deployed to DataKitchen testing environments only.
- [12 April](/automation/release-notes/2021/12-april/): Version 1.1.211
- [12 September](/automation/release-notes/2021/12-september/): Version 1.1.228
- [15 February](/automation/release-notes/2021/15-february/): Version 1.1.203
- [15 March](/automation/release-notes/2021/15-march/): Version 1.1.207
- [17 May](/automation/release-notes/2021/17-may/): Version 1.1.216. Note that version 1.1.215 was deployed to DataKitchen testing environments only.
- [18 January](/automation/release-notes/2021/18-january/): Version 1.1.199
- [19 April](/automation/release-notes/2021/19-april/): Version 1.1.212
- [22 March](/automation/release-notes/2021/22-march/): Version 1.1.208
- [22 November](/automation/release-notes/2021/22-november/): Version 1.1.234
- [24 May](/automation/release-notes/2021/24-may/): Version 1.1.217
- [24 October](/automation/release-notes/2021/24-october/): Version 1.1.231
- [25 January](/automation/release-notes/2021/25-january/): Version 1.1.200
- [25 July](/automation/release-notes/2021/25-july/): Version 1.1.223
- [26 April](/automation/release-notes/2021/26-april/): Version 1.1.213
- [26 September](/automation/release-notes/2021/26-september/): Version 1.1.229
- [27 June](/automation/release-notes/2021/27-june/): Version 1.1.221. Note that version 1.1.220 was deployed to DataKitchen testing environments only.
- [29 August](/automation/release-notes/2021/29-august/): Version 1.1.227. Note that versions 1.1.225 and 1.1.226 were deployed to DataKitchen testing environments only.
- [29 March](/automation/release-notes/2021/29-march/): Version 1.1.209
- [3 May](/automation/release-notes/2021/3-may/): Version 1.1.214
- [3 October](/automation/release-notes/2021/3-october/): Version 1.1.230
- [31 October](/automation/release-notes/2021/31-october/): Version 1.1.232
- [5 April](/automation/release-notes/2021/5-april/): Version 1.1.210
- [6 July](/automation/release-notes/2021/6-july/): Version 1.1.222
- [7 June](/automation/release-notes/2021/7-june/): Version 1.1.219. Note that version 1.1.218 was deployed to DataKitchen testing environments only.
- [7 November](/automation/release-notes/2021/7-november/): Version 1.1.233
- [8 February](/automation/release-notes/2021/8-february/): Version 1.1.202
- [8 March](/automation/release-notes/2021/8-march/): Version 1.1.206
- [9 August](/automation/release-notes/2021/9-august/): Version 1.1.224
- [1 December](/automation/release-notes/2022/1-december/): Version 1.1.258
- [1 March](/automation/release-notes/2022/1-march/): Version 1.1.243
- [11 January](/automation/release-notes/2022/11-january/): Version 1.1.235
- [12 April](/automation/release-notes/2022/12-april/): Version 1.1.248
- [15 December Library Upgrades](/automation/release-notes/2022/15-december-library-upgrades/): Version 1.1.258 (no change from the previous release)
- [16 May](/automation/release-notes/2022/16-may/): Version 1.1.249
- [20 June](/automation/release-notes/2022/20-june/): Version 1.1.250
- [22 December](/automation/release-notes/2022/22-december/): Version 1.1.259
- [22 February](/automation/release-notes/2022/22-february/): Version 1.1.242. Note that versions 1.1.237 through 1.1.241 were deployed to DataKitchen testing environments only.
- [23 January](/automation/release-notes/2022/23-january/): Version 1.1.236
- [28 December](/automation/release-notes/2022/28-december/): Version 1.1.260
- [29 March](/automation/release-notes/2022/29-march/): Version 1.1.246
- [8 March](/automation/release-notes/2022/8-march/): Version 1.1.244
- [9 November](/automation/release-notes/2022/9-november/): Version 1.1.257
- [10 July](/automation/release-notes/2023/10-july/): Version 1.1.275
- [13 September](/automation/release-notes/2023/13-september/): Version 1.1.278
- [14 February](/automation/release-notes/2023/14-february/): Version 1.1.264
- [17 April](/automation/release-notes/2023/17-april/): Version 1.1.270
- [18 January](/automation/release-notes/2023/18-january/): Version 1.1.261
- [19 June](/automation/release-notes/2023/19-june/): Version 1.1.274
- [24 January](/automation/release-notes/2023/24-january/): Version 1.1.262
- [27 March](/automation/release-notes/2023/27-march/): Version 1.1.267
- [28 December](/automation/release-notes/2023/28-december/): Version 1.2.2
- [28 September](/automation/release-notes/2023/28-september/): Version 1.1.279
- [31 January](/automation/release-notes/2023/31-january/): Version 1.1.262 (no change from previous release notes)
- [August Changelog](/automation/release-notes/2023/august-changelog/): Version 1.1.277
- [DKCloudCommand Breaking Change](/automation/release-notes/2023/dkcloudcommand-breaking-change/): The expected release date for these changes is prior to 23 February 2023.
- [17 January](/automation/release-notes/2024/17-january/): 17 January 2024 Release Notes
- [22 February](/automation/release-notes/2024/22-february/): 22 February 2024 Release Notes
- [31 January](/automation/release-notes/2024/31-january/): 31 January 2024 Release Notes
- [23 April](/automation/release-notes/2025/23-april/): Version 1.3.13
- [27 September](/automation/release-notes/2025/27-september/): Version 1.4.1
- [30 August](/automation/release-notes/2025/30-august/): Version 1.3.20
- [6 March](/automation/release-notes/2025/6-march/): Version 1.3.9
- [7 February](/automation/release-notes/2025/7-february/): Version 1.3.7
- [9 April](/automation/release-notes/2025/9-april/): Version 1.3.11
- [18 June](/automation/release-notes/2026/18-june/): Version 1.5.16
- [24 June](/automation/release-notes/2026/24-june/): Version 1.5.17
- [7 March](/automation/release-notes/2026/7-march/): Version 1.5.0
- [GPC Image Changelog](/automation/release-notes/gpc-image-changelog/): DataKitchen provides a pre-configured container node called the General Purpose Container (GPC). This node is available as a Docker image:...
- [Automation Release Notes](/automation/release-notes/): Find links here to release notes for user-facing features, fixes, and enhancements in DataKitchen's DataOps Automation.
- [Kubernetes Cluster Changelog](/automation/release-notes/kubernetes-cluster-changelog/): The DataKitchen DataOps Automation software, worker nodes (agents), and scheduler are hosted on Kubernetes clusters. For more information, see Automation Architecture.
- [Reports](/automation/reports/): Automation allows users to embed custom reports from connected tools, such as Tableau, in the Reports page.
- [Configure Schedules](/automation/schedules/configure-schedules/): Set schedules to run orders on a routine basis, assign runtime resources, or add order annotations.
- [Cron Expression Basics](/automation/schedules/cron-expression-basics/): DataKitchen's DataOps Automation uses the Kubernetes scheduler for order scheduling.
- [Schedules](/automation/schedules/): Set schedules to run orders on a routine basis, assign runtime resources, or add order annotations.
- [Pause or Stop Schedules](/automation/schedules/pause-or-stop-schedules/): Stop orders with active schedules, either temporarily or permanently, using controls on the Orders page.
- [Run-Once Order Schedules](/automation/schedules/run-once-order-schedules/): Schedules can be set up for orders that are run once, allowing you to define resource maximums and annotations without assigning a frequency.
- [Schedule Settings](/automation/schedules/schedule-settings/): A recipe can contain several configured schedules, each associated with one or more variations. The following table outlines the scheduling settings used in the UI forms and in the variations.json...
- [Schedules with Heavy Processing](/automation/schedules/schedules-with-heavy-processing/): For resource-intensive order runs—for example, those with large queries, heavy use of variables, or data sources and sinks—you can adjust memory and disk space allocations in the schedule form....
- [Supported Time Zones](/automation/schedules/supported-time-zones/): Reference the following time zones when setting order schedules.
- [Custom Logic Expressions](/automation/tests/custom-logic-expressions/): Complex test conditions are best configured using custom logic expressions that evaluate a runtime variable or the result of a key execution. Expressions are written and evaluated in Jinja.
- [Get Started with Tests](/automation/tests/get-started-with-tests/): One of the primary DataOps practices is to write and automate tests against both code and data to ensure all of the people, tools, and processes of production and development pipelines operate...
- [Tests](/automation/tests/): Automated testing is a foundation of DataOps and serves to provide confidence in the quality of deliverables. It also helps to improve the productivity of a data analytics team—a team cannot increase...
- [Test Coverage](/automation/tests/test-coverage/): Automation allows users to write tests for every step of a variation, then automates the tests during order run execution. Users can view test results as part of the completed details for every order...
- [Declare Tests](/automation/tests/test-declaration/): Tests can be declared:
- [Count Verification with Jinja Example](/automation/tests/test-patterns-and-examples/count-verification-example-using-jinja/): Configure the ingredient variation to export the runtime value as the output_matillion_dict variable with the value recorded in the output_raw_record_count key.
- [Count Verification Test Example](/automation/tests/test-patterns-and-examples/count-verification-test-example/): When configuring the mapping to the source file, declare runtime variables to record the size of the source file and its data.
- [Historical Comparisons](/automation/tests/test-patterns-and-examples/historical-comparisons/): Tests can be declared so the system evaluates them against the test variable values of previous order runs within the same order (i.e. the same kitchen-recipe-variation combination). These historical...
- [Test Patterns and Examples](/automation/tests/test-patterns-and-examples/): Tests can be built into any workflow to ensure that both code and data are error-free.
- [Location Balance Test Example](/automation/tests/test-patterns-and-examples/location-balance-test-example/): When configuring steps in the action node, write a SQL script to query a preconfigured table with invalid ZIP codes and compute the percentage.
- [Recency Test Example](/automation/tests/test-patterns-and-examples/recency-test-example/): When configuring steps in the action node, a SQL script produces a runtime variable for the age of the data set.
- [Statistical Process Control Example](/automation/tests/test-patterns-and-examples/statistical-process-control-example/): The first step in the statistical process control test is to define the inputs for the container.
- [Test Properties](/automation/tests/test-properties/): The table below provides details on the properties you can configure during test creation. When tests are set up in the UI, they are recorded in JSON files.
- [Tests Order of Execution](/automation/tests/tests-order-of-execution/): The system executes tests according to the order that they are declared in any given JSON config file.
- [View Test Results](/automation/tests/view-test-results/): Users can view test results in several locations throughout Automation. Determine the root cause of an order run failure or analyze performance over time, no matter how you work best:
- [504 Errors](/automation/troubleshooting/504-errors/): The following message appears in the Automation UI to alert you:
- [Automation Troubleshooting](/automation/troubleshooting/): The following documents are designed to help you troubleshoot issues you may experience in DataKitchen's DataOps Automation.
- [File Not Found](/automation/troubleshooting/order-problems/file-not-found/): A message similar to one of the following appears in an order run log to alert you:
- [Order Problems](/automation/troubleshooting/order-problems/): The following documents help troubleshoot potential order problems.
- [Insufficient Resources](/automation/troubleshooting/order-problems/insufficient-resources/): A message similar to one of the following appears in an order run log to alert you:
- [Parameter Configuration Problem](/automation/troubleshooting/order-problems/parameter-configuration-problem/): A message similar to one of the following appears in the Run Variation dialog to alert you:
- [Vault Entry Problem](/automation/troubleshooting/order-problems/vault-entry-problem/): A message similar to one of the following appears in the Run Variation dialog to alert you:
- [Pinned Package Dependencies](/automation/troubleshooting/pinned-package-dependencies/): A message similar to one of the following appears in an order run log to alert you:
- [Resolve DKCloudCommand SSL Errors](/automation/troubleshooting/resolve-dkcloudcommand-ssl-errors/): A message similar to one of the following appears in the command line to alert you:
- [Run kitchen-merge-preview Again](/automation/troubleshooting/run-kitchen-merge-preview-again/): The following message appears in the UI while performing a kitchen merge:
- [Cannot Save to Disk](/automation/troubleshooting/troubleshoot-cli-issues/cannot-save-to-disk/): A message similar to the following appears in the command line to alert you:
- [Troubleshoot CLI Issues](/automation/troubleshooting/troubleshoot-cli-issues/): The following documents help troubleshoot potential order problems.
- [Python Package Conflict](/automation/troubleshooting/troubleshoot-cli-issues/python-package-conflict/): A ContextualVersionConflict message similar to the following appears in the command line to alert you:
- [Unknown Error in Command Runs](/automation/troubleshooting/troubleshoot-cli-issues/unknown-error-in-command-runs/): A message similar to the following appears in the command line to alert you:
- [Configure Custom Vault](/automation/vaults-and-secrets/custom-vaults/configure-custom-vault/): A custom vault can be set up as the global vault and kitchen vault.
- [Custom Vaults](/automation/vaults-and-secrets/custom-vaults/): Customers may opt to set up their own, externally-hosted vault services. Custom vaults offer a number of configurations, for example:
- [Configure Global Vault](/automation/vaults-and-secrets/global-vault/configure-global-vault/): By default, the global vault is on and set to the DataKitchen Default vault. Authorized users can deactivate or reactivate the global vault configuration to turn the global vault off or on, or set up...
- [Global Vault](/automation/vaults-and-secrets/global-vault/): The global vault contains secrets that are available for use by recipes in any kitchen. It is turned on by default.
- [Vaults and Secrets](/automation/vaults-and-secrets/): Vaults remotely store sensitive infrastructure and toolchain credentials as secure, encrypted secrets. Teams can use secrets as variables to build kitchens and recipes without needing to view the...
- [Configure Kitchen Vaults](/automation/vaults-and-secrets/kitchen-vault/configure-kitchen-vaults/): Users with access to a kitchen can manage that kitchen's vault connection settings.
- [Kitchen Vault](/automation/vaults-and-secrets/kitchen-vault/): Kitchen vault connection settings are defined at the kitchen level. Each kitchen can only be connected to one kitchen vault at any given time.
- [Set Vault Inheritance](/automation/vaults-and-secrets/kitchen-vault/set-vault-inheritance/): Custom kitchen vault connection settings defined in a parent kitchen can be configured as inheritable by child kitchens. The inheritance of these settings applies across generations of child kitchens.
- [Create and Edit Secrets](/automation/vaults-and-secrets/secrets/create-and-edit-secrets/): Secret values can either be saved as text or as files containing keys.
- [Delete Secrets](/automation/vaults-and-secrets/secrets/delete-secrets/): To delete a kitchen, users must have the appropriate permission or be a member of the kitchen where the secret exists.
- [Secrets](/automation/vaults-and-secrets/secrets/): Secrets are encrypted infrastructure and toolchain credentials that are stored remotely in vaults. Teams can use secrets as variables to build kitchens and recipes without needing to view the actual...
- [Insert Secrets in Recipes](/automation/vaults-and-secrets/secrets/insert-secrets-in-recipes/): There are three ways to insert vault secrets into your recipes.
- [Secrets at Runtime](/automation/vaults-and-secrets/secrets/secrets-at-runtime/): Automation compiles Jinja expressions and resolves vault secrets differently. At runtime, the system first performs Jinja template preprocessing. It then resolves and populates secrets where they are...
- [Secrets Usage](/automation/vaults-and-secrets/secrets/secrets-usage/): Recipes rely on vault secrets to protect and encrypt the credentials for toolchain connections.
- [Security Considerations](/automation/vaults-and-secrets/secrets/security-considerations/): To maintain adequate security, certain behaviors exist when using secrets.
- [What is Automation?](/automation/what-is-automation/): DataKitchen's DataOps Automation helps orchestrate the separate teams, tools, technologies, workflows, operations, tests, and environments that exist across an entire data organization. As a seamless...

## Tutorials
- [AWS Redshift](/tutorials/data-quality/aws-redshift/): Use this guide as an example of how to set up and use open-source DataOps Data Quality TestGen to profile, test, and score the quality of your data in an AWS Redshift database. Automatically define...
- [Azure SQL / Synapse](/tutorials/data-quality/azure-sql-synapse/): Use this guide as an example of how to set up and use open-source DataOps Data Quality TestGen to profile, test, and score the quality of your data in an Azure SQL or Synapse database. Automatically...
- [Databricks](/tutorials/data-quality/databricks/): Use this guide as an example of how to set up and use open-source DataOps Data Quality TestGen to profile, test, and score the quality of your data in a Databricks database. Automatically define and...
- [Data Quality Quickstart Demos](/tutorials/data-quality/): These guides provide an example of how to set up and use Open-Source DataOps Data Quality TestGen to profile, test, and score data quality across various database platforms.
- [Postgres](/tutorials/data-quality/postgres/): Use this guide as an example of how to set up and use open-source DataOps Data Quality TestGen to profile, test, and score the quality of your data in a Postgres database. Automatically define and...
- [Snowflake](/tutorials/data-quality/snowflake/): Use this guide as an example of how to set up and use open-source DataOps Data Quality TestGen to profile, test, and score the quality of your data in a Snowflake database. Automatically define and...
- [Testgen landing](/tutorials/data-quality/testgen-landing/): DataKitchen TestGen
- [Add Automated Data and Logic Tests](/tutorials/dataops-concepts/add-data-logic-tests/): Run tests regularly to catch problems with data, bugs in new code, and regressions in existing code.
- [DataOps Concepts](/tutorials/dataops-concepts/): DataOps is a methodology to build and maintain data analytics and data science systems that reduce cycle time and increase quality over traditional development and operations methods.
- [Monitor for Errors and Trends](/tutorials/dataops-concepts/monitor-for-errors-trends/): Track operational statistics to fine-tune workflow performance, improve cycle times, and ensure quality data and analytic output.
- [Orchestrate Production](/tutorials/dataops-concepts/orchestrate-production/): Coordinate your teams, tools, technologies, workflows, and tests across your data organization into a seamless DataOps factory.
- [Parameterize Your Processing](/tutorials/dataops-concepts/parameterize-your-processing/): Deploy code across multiple environments using variables to eliminate configuration changes.
- [Reuse and Containerize](/tutorials/dataops-concepts/reuse-containerize/): Developing components only once and consolidating complex analytics for use in many pipelines and use cases saves development time.
- [Schedule Pipeline Processing](/tutorials/dataops-concepts/schedule-pipeline-processing/): Deliver analytic insights to your customers on a predictable cadence.
- [Use Multiple Environments](/tutorials/dataops-concepts/use-multiple-environments/): Experiment outside of production environments and production data.
- [Tutorials](/tutorials/): Explore hands-on tutorials and foundational concepts for DataKitchen's data observability and automation products.
- [Anomaly Detection](/tutorials/quickstart-demo/anomaly-detection/): "Polling: Arrival Aberration Alerts"
- [Data Pre-Production](/tutorials/quickstart-demo/data-pre-production/): "Patch Or Pushback Data"
- [Open Source Data Observability Quickstart Demo](/tutorials/quickstart-demo/): Data breaks. Servers break. Your toolchain breaks. Ensure your team is the first to know and the first to solve with visibility across and down your data estate. Save time with simple, fast data...
- [Overview](/tutorials/quickstart-demo/overview/): What does it do?
- [Production Monitoring](/tutorials/quickstart-demo/production-monitoring/): "Production: Check Down and Across"
- [Toolchain Integration](/tutorials/quickstart-demo/toolchain-integration/): "Plug-In: Who Watches the Watchmen?"
- [Data profiling ad q1 2026 direct](/tutorials/topics/data-profiling-ad-q1-2026-direct/)
- [Testgen landing v4](/tutorials/topics/testgen-landing-v4/)

## General
- [Contact Support](/contact-support/): At DataKitchen, we're always here to help. If you have a question about TestGen data quality tests, an Observability journey, an Automation kitchen, or can't find a particular use case in our...