Container Nodes

Automation uses Docker containers to package code, runtime scripts, tools, and other settings so that processes can execute efficiently and reliably without requiring custom infrastructure.

Automation container nodes work by referencing container images that become Docker containers at runtime. This isolates processes from the general environment and helps to make sure the container node runs the same way every time. Container nodes also provide portability by combining code, runtimes, tools, libraries, and configuration files in a standalone system that can be reused throughout your kitchens.

Tool support

Automation can support any database, computing, programming, provisioning, data visualization, reporting, and business intelligence tools an organization uses. For example, container nodes support the use of scripts (Python, Java, Shell, etc.) and GUI tools like Jupyter and Tableau notebooks.

Tip

DataKitchen provides support for standard I/O connectors. For a list of DataKitchen-supported I/O connectors that do not require the use of container nodes, see Data Sources and Data Sinks.

Ready-to-use containers

DataKitchen provides several analytic container images, pre-configured and ready for use. Any Automation container image available on Docker Hub can be used to support a tool in your toolchain.

See Container Images for more information.

Container node file structure

Container images are built with the DataKitchen Interface Layer and require specific files and file structures to be present in a container node. See Container Node File Structure for complete details.

Note that other, custom images may not have the same requirements.

Container resource cleanup

Over time, as the system executes order runs inside runtime containers, and as container nodes spin up other runtime containers for processing, available disk space can be depleted. To help maintain an efficient environment, Automation includes a cleanup pod to clean all pending, errored, and completed pods (i.e. order runs) every 48 hours.

Note

This service is on and set to 48 hours by default. If you require resource cleanup more or less frequently, speak to your DataKitchen representative.