Variables Scope

By default, all variables are global but can be scoped to a recipe object, configured as private, or limited to a single Jinja template block within a file.

Global variables

The lifecycle of a global variable starts as soon as it's created and continues with the order run during execution.

Each assignment on global variables overwrites any previous values in the global scope, although the previous values are retained in lower scopes.

After declaration, variables can be referenced by other files or even other variables using standard Jinja2 referencing syntax {{ }}.

Scoped variables

Since variables are global, any new value overwrites a previous value. When a variable is assigned a new value during the execution of a variation, however, previous values are retained.

It is possible to reference previous values by scoping a variable reference to the position in the graph where that variable held a specific value.

Variables can be scoped to a given node and/or data source or sink with the following syntax:

• Scope to a node: recipe.<node_name>.<variable_name>
• Scope to a data source: recipe.<node_name>.<datasource_name>.<variable_name>
• Scope to a data sink: recipe.<node_name>.<datasink_name>.<variable_name>

Warning

Note that "recipe" in this syntax is the actual word, not the recipe name.

Scoped variable examples

# Reference the latest value set for file_size variable, using variable name only.
{{file_size}}

# Reference the last value set for the file_size variable for the get_file_from_s3 node.
{{recipe.get_file_from_s3.file_size}}

# Reference the last value set for the file_size variable for the source1 data source in the get_file_from_s3 node.
{{recipe.get_file_from_s3.source1.file_size}}

# Reference the last value set for the file_size variable for the sink1 data sink in the get_file_from_s3 node.
{{recipe.get_file_from_s3.sink1.file_size}}

Problematic object names

Node and data source and sink names with unsupported characters, such as hyphens, should use the single-quote syntax as a workaround.

# Reference the last value set for a variable for a specific node.
"{{recipe.['node-1'].my_variable}}"

# Reference the last value set for the variable for a specific data source.
"{{recipe.['node-1'].['datasource-1'].my_variable}}"

Private variables

It's possible to prevent a variable from being set as global and instead isolate it to the scope where it was defined (this includes its children scopes).

You can use the _ prefix for the variable name to set a variable as private.

For example, a private variable set in a node can be accessed by its data sources and sinks, but not by other nodes. A private variable set in a data source can be accessed by resource files being referenced by the data source, but not by the node or its data sinks.

{
   "set-runtime-vars" : {
      "row_count" : "_privateRowCount"
   }
}

Block scoped variables

Block scoped variables are the exception to the default global variable rule (when scoping syntax is not used). These variables are created within Jinja templates. Their lifecycles are limited to the processing of single blocks, after which they are discarded.

{% set items = [1,2,3] %}

{% for item in items %}
....
{% endfor %}