Skip to content

What is Automation?

DataKitchen's DataOps Automation helps orchestrate the separate teams, tools, technologies, workflows, operations, tests, and environments that exist across an entire data organization. As a seamless DataOps "factory," Automation delivers high-quality data and analytics, and delivers them fast.

Automation facilitates the control of two data pipelines—the production pipeline and the development pipeline—at the same time. This helps to increase innovation and data quality while reducing errors and time to deploy.

(DataOps pipelines image)

The software

Automation is a software as a service (SaaS) that consists of an online user interface (UI), command line interface (CLI) functionality, and a backend system with REST API.

  • UI: includes environment visualizations, a form-based editor, and a file-based editor. The UI is designed to guide you in creating well-structured JSON files as the basis for your pipelines. All work can be performed from the UI.
  • CLI: alternatively, you can work via DKCloudCommand, the command line tool for interacting with the Automation API. The CLI lets you write files locally, then push them to the Automation server to run.
  • API: the majority of the work done in the UI and CLI can also be done directly through the Automation API.

The software is designed to be a flexible and secure integration of SaaS and supporting services, all requiring minimal external communications.

For more technical details on the software, see System Architecture.

DataOps features

Automation includes capabilities for development and production workflows so that cross-functional teams can collaborate, innovate, and deliver error-free, on-demand insight. Governance features are configurable to make sure that Automation suits the needs of your existing infrastructure.

  • Meta-orchestration: Automation helps you orchestrate each data and analytic pipeline, then orchestrates the orchestration to provide a coherent framework across teams, tools, locations, and environments. As your pipelines run, the system orchestrates all the tools involved.
  • Automated testing and monitoring: Automation helps you catch costly or embarrassing data errors early with automated tests at every step in your pipelines and alerts to reduce data downtime.
  • Environment creation and management: Automation delivers infrastructure as code. Developers can create kitchen workspaces with pre-configured tools, datasets, and tests in minutes, then merge work when it's ready into aligned environments. When project work completes, you can tear down those resources.
  • Tool-agnostic DataOps: leverage virtually any analytic tool within Automation using supported connectors or flexible integration methods.
  • DataOps process analytics: with Automation process metrics and reports, you can see how your teams increase collaboration, improve productivity, expand test coverage, reduce errors, shorten deployment cycle times, and meet deadlines.
  • Storage and revision control: version control, using Git and Docker Hub, manages changes in artifacts, essential for governance and iterative development.
  • History and metadata: a robust database centrally manages system and activity logs.
  • Authorization and permissions: identity solutions, such as Auth0, control access to environments.
  • Environment secrets: a secrets management tool, like HashiCorp's Vault, implements access to tools and resources within environments.
  • Collaboration and sharing: team members work in separate but aligned kitchens and integrate their work regularly. And, they can save and share commonly used parts of pipelines as ingredients.
  • Automated deployment: Automation supports continuous deployment for testing and releasing new analytics on demand and safely migrating analytics to production. Solutions such as Jenkins or CircleCI move the code/configuration from one environment (for example, a test environment) to a production environment.

Note

The infrastructure running Automation can adapt to your organization's needs for self-hosted (managed entirely by the customer), SaaS (hosted and managed by DataKitchen), or hybrid implementations. Speak to your DataKitchen representative for more information.

Get started with Automation