Node Components

The files that make up a node follow a specific structure. Open the File Editor to view the node files for the nodes within a recipe.

Node file structure

/node_name/
├── description.json
├── notebook.json
│   └── keys and/or tests (optional)
├── /data_sources/
├── /actions/                    ← action nodes only
│   └── keys and/or tests (optional)
├── /data_sinks/
│   └── keys and/or tests (optional)
└── /docker-share/               ← container nodes only
    └── config.json

Node components

Component	Description
description.json	System file that is mandatory for all node types. This file sets the node type and provides a description field used to describe a node's purpose. The file is not used for other purposes.
notebook.json	System file required for action, container, and ingredient nodes. This file defines the node configuration.
docker-share	System folder that is only applicable to container nodes. This directory holds the config.json file and any files used by the container, such as scripts.
config.json	System file that is only applicable to container nodes. Here configuration may be set to import packages, inject parameters into the container, and run scripts within the container.
data sources	Infrastructure connections that get data for use by a recipe node. A node may have multiple data sources or none at all. Data sources within a given node are processed in an indeterminate order. Before the data work performed by a node can occur, data must often be gathered. After the data work by the node is performed, the resulting output must often be placed in some location. These requirements are configured via data sources and data sinks.
data sinks	Infrastructure connections that push data from a recipe node. A node may have multiple data sinks or none at all. Data sinks within a given node are processed in an indeterminate order.
keys	The sub-steps are specific to and executed within a given node. Keys are executed in the order they are defined within a given recipe configuration file. Keys may be defined across multiple files within a node.
resource files	Nodes can call shared files located in the recipe-level /resources directory.
tests	Automated tests against recipe variables may be defined in multiple files within a node. Tests are executed in the order they are defined within a given node file.

Warning

UTF-8 file encoding is required. Files used with data sources and data sinks must be encoded in UTF-8 in order to avoid non-Unicode characters causing problems with sinking data to database tables and errors when running related tests. For CSV and other delimited files, use the "save as" function in your application and select the proper encoding, or consider using a text editor with encoding options.