Data Sources and Data Sinks

Data sources and data sinks are node input/output components that connect to toolchain infrastructure in order to get data to and push data from recipe nodes.

For tools with DataKitchen's DataOps Automation data source and sink support, you have the option of connecting to the tool with a container node or as a data source/sink. A data source/sink provides a tighter coupling to Automation and requires less technical work to configure.

Multiple data sources and/or sinks can be configured for any given node (though their use is optional for certain node types).

Source and sink types

There are two types of data source and data sink connections: file-based or database.

The two types support different sets of connection properties and variables.

Note

The exception to this is the Dictionary data source, which falls under a system category.

Source and sink structure

Keys

Data sources and sinks execute a series of substeps called keys.

Keys can be explicitly expressed within recipe configuration, generated before runtime using Jinja processing and built-in functions or generated at runtime with wildcards. Some built-in runtime variables can be declared in data sources and sinks for use in automated tests.

JSON configuration file

Each data source or sink is configured via a single JSON configuration file.

Data source configurations generally reside within each node's /data_sources directory, while data sink configurations for each node reside in the /data_sinks directory. For action nodes, data source configurations reside within an /actions node directory.

JSON file example

source.json or sink.json

{
    "type": "< DATA SOURCE/SINK TYPE CODE >",
    "name": "my_data_source_or_sink",
    "config-ref": {{config}},
    "keys": {},
    "tests": {}
}

Required configuration properties

When configuring a source or sink, the following properties must have values declared.

JSON field	Source or sink	Field type	Required?	Description
name	source & sink	string	yes	The name of the data source or data sink, as defined by its filename minus the .json extension.
type	source & sink	string	yes	Denotes the type of data source or data sink (file-based, database, or system).

Note

Source and sink naming: Data sources and sinks should be named according to DataKitchen's object-naming guidelines. DataKitchen object names support alpha-numeric and underscore characters and are case-sensitive.