Core Concepts in Automation

The definitions below can help you understand how Automation terminology maps to standard industry terms.

Automation object	Comparable industry terms	Description
Kitchen	Environment	A virtual workspace where teams build, manage, and run pipelines that provision infrastructure and perform data work. Team members can make changes and experiment independently, repeatably, and without breaking production. Examples: production kitchen, default Git branch, development kitchen for one project, hotfix branch, sandbox kitchens
Recipe	Collection of related pipelines & components	A collection of related data, analytic, and/or infrastructure provisioning pipelines.
Variation	Pipeline, workflow	A pipeline employing a specific configuration of recipe components for a particular use case and value delivery. A variation automates the tools that drive your data operations and supports testing at every step.
Ingredient	Code reuse	Any variation can be saved as a reusable ingredient so other team members can access, reuse, and incorporate it into their own recipes and kitchens.
Graph	DAG	A representation of the structure of data processes in a pipeline.
Nodes	Vertices	Objects connected by edges (or arcs) in a graph that are steps to a pipeline and are distinct mini-processes executed in sequence.
Tests	Tests	Automated checks on data inputs, evaluations of results against business logic, and comparisons of outputs against expected results for every processing or transformation step of a pipeline.
Order	Job	A submission of a recipe variation, with optional variable overrides, for immediate or scheduled execution.
Order run	Job run	A specific, time-stamped run instance of an order, returning process analytics.

Automation kitchen structure

Kitchens nest the elements listed above, helping you to do scalable, versioned work.

• Kitchen: the overall work area for a project.
  Structure: your active work area in the UI; a parent directory on your local drive.
  Example: your development kitchen for a specific project.
  • Recipe: the general task to be performed.
    Structure: a set of procedures within the UI; a directory under the kitchen on your local drive.
    Example: build a data warehouse database.
    • Variation: one procedural flow within a recipe.
      Structure: a graph defined in JSON that consists of a set of steps (nodes) and order of execution (edges).
      Example: build a particular set of fact or dimension tables within one schema.
      • Node: one major step. Can be shared among variations.
        Structure: an element within a graph in the UI; a directory and JSON files in a local kitchen.
        Example: load a table.
        • Key: a SQL subset — an in-line query or resource file.
          Example: as the final step in a file, SELECT a scalar value (e.g. COUNT(*)) for a test.
        • Test: a QC conditional — validate a key, log result.
          Can use a scalar from a key and/or from other SQL in comparisons.