System Architecture¶
DataKitchen DataOps Automation was designed with DataOps elements to support rapid change, early error detection, and efficient development. It expands the traditional operations-oriented data architecture by including support for agile development, robust operations, and statistical process control.
Tip
Customers can implement DataKitchen's DataOps Automation, DataOps Observability, and DataOps Data Quality TestGen separately or together as a comprehensive environment for visibility into—and automation of—their data operations.
How it works¶
The underlying DataOps architecture provides support for dynamic environment creation and management, allowing separate development, test, and production environments, which in turn supports orchestration, monitoring, and test automation. The software automates analytics deployment and impact review so that changes can be vetted and published with CI/CD practices. Agents in each environment operate on behalf of Automation to manage code and configuration, execute tasks, and return test results, logs, and runtime information.
The following diagram and Services summary section below provide a high-level overview of the architecture and information flows across the Automation ecosystem.
Services summary¶
| Service | Purpose and details | Version |
|---|---|---|
| Auth provider | User management. Maintains user identification data. Any OIDC-compliant auth service is required. Auth0, PingFederate, and OKTA are tested and fully supported. | SaaS |
| Container image repo | Stores Automation container images. Interacts with the Kubernetes API endpoints. Typically DockerHub. | SaaS |
| MongoDB | Stores order run data including timings, test results, and other run details. | SaaS or 4.4.x |
| Vault/key storage | Holds credentials to access services and tools. HashiCorp Vault is fully supported. | 1.1 |
| Git-based repo | Stores source code (recipes, tests, and supporting files) and provides version control. | GHE 2.x.x. |
| Kubernetes cluster | Hosts and runs the DataKitchen software, worker nodes (agents), and scheduler. | 1.27 and 1.28 |
| Email service | Used to send order run alerts. An SMTP or AWS SES service is required. | n/a |
Note
DataKitchen uses this configuration for both SaaS and self-hosted implementations. Speak with your DataKitchen representative for more information about the right installation for you.
Presentation: Technical architecture¶
Presentation by DataKitchen CEO, Chris Bergh, March 2019.
See the blog post, DataOps Data Architecture, for even more information.