Skip to content

Advanced Jinja Templating

Warning

Advanced Jinja templating limits editing function. Some advanced templating options shown below are not supported in the Node Editor. Use of advanced templating renders recipe files editable only through the File Editor in the UI.

DataKitchen's template engine is based on Jinja2, which functions as a preprocessor for all recipe variation files during recipe execution. To support subsequent file processing, the output of the template engine for any *.json configuration files must be JSON.

Template restrictions

Ineligible files

The following files cannot use complex Jinja templating because they must contain valid JSON at the beginning of recipe processing:

  • variables.json
  • variations.json

Quotations

All files, including variables.json and variations.json, in a recipe do support Jinja variable referencing with {{ }} syntax, but only if they use valid JSON with double quotation marks.

See Basic Jinja Variable References for more information.

  • Supported example: { "data_key": "{{example_string}}" }
  • Not supported example: { "data_key": {{example_string}} }

The unsupported example does not use properly formed JSON and results in an error.

Note

When you compile files in the UI or CLI, you might receive a result that is not in properly formed JSON. This can be because variables set at runtime may not be available yet.

Loops

Jinja templates do not support complex expressions where variable values change during page processing. (This limitation does not apply to runtime variables.)

The following loop, incrementing a variable value, is not supported.

{% set a = 1 %}
{% for i in items %} 
{% set a = a + 1 %} 
{% endor %}

JSON comments

Aside from the exceptions noted above, Jinja comments can be added to *.json configuration files with the {# #} syntax.

Comments are helpful for ongoing operational management and iteration of recipes.

{
    "key1": "value1",

{# single-line jinja comment #}

{#
 multi-line
 jinja
 comment
 #}

    "key2": "value2"
}

Set block scoped variables

Jinja supports setting variables within templates through statement blocks, using the {% %} syntax along with set. These variables are ephemeral (i.e. they are scoped at the template level and not propagated to other recipe files).

Once template variables are set they can be used in the same way runtime variables are used in templates.

{
    {% set tables=['table1','table2','table3'] %}

    {% for table in tables %}
    ....
    {% endfor %}
}

Set runtime variables

In addition to Jinja page-scoped variables, you can export a variable as a runtime variable, making it global for the order run.

The variable is set during the page processing, so it's possible to use it right after its declaration.

{
  {# Example 1 #}
  {% export var1 = 'Hello world'%}

  {# Example 2: export an existing page variable#}
  {% set var1 = 'Hello world %}
  {% export var1 %}

  {# Example 3 #}
  {% export var1, var2 = 'Hello', 'World' #}

  {# Example 4 #}
  {% set var1 = 'Hello world' %}
  {% export var2, var3 = var1.split() %}
}

For loops

Jinja statement blocks ({% %}) may also be used to loop through lists and dictionaries using for and endfor.

# Iteration of a list
{% for user in users %}
...
{% endfor %}

# Iteration of a dictionary
{% for key, value in dictionary.items() %}
...
{% endfor %}

# Iteration of a list of items with index
{% for index, value in enumerate(values) %}
....
{% endfor %}

Example: For loop selects item from array

This example loops through a dictionary array, defined as a kitchen override variable, to select the proper Azure package to install as a container dependency.

{
  "apt-dependencies": [],
  "dependencies": [
    {% for item in ['azure-mgmt-resource'] -%}
      "{{item}}=={{azureAPIVersion[item]}}",
    {% endfor -%}
    "DKUtils=={{DKUtilsVersion}}"
  ],
  "keys": {
    "run_script": {
      "script": "manage_ssas_instance.py",
      "parameters": {
        "AZURE_CLIENT_ID": "{{AZURE_CLIENT_ID}}",
        "AZURE_CLIENT_SECRET": "{{AZURE_CLIENT_SECRET}}",
        "AZURE_RESOURCE_GROUP_NAME": "{{AZURE_RESOURCE_GROUP_NAME}}",
        "AZURE_RESOURCE_URI": "{{AZURE_RESOURCE_URI}}",
        "AZURE_SSAS_SERVER_NAME": "{{AZURE_SSAS_SERVER_NAME}}",
        "AZURE_SUBSCRIPTION_ID": "{{AZURE_SUBSCRIPTION_ID}}",
        "AZURE_TENANT_ID": "{{AZURE_TENANT_ID}}",
        "IGNORE_FAILED_REQUEST": "{{IGNORE_FAILED_REQUEST}}",
        "RESUME_OR_SUSPEND": "resume"
      },
      "exports": [
        "success"
      ]
    }
  }
}

Example: For loop checks status of items

This SQL query in an action node step parses a list of projects and inserts projects that are closed into a database table.

{
  "name": "s3_to_reshift",
  "type": "DKDataSource_PostgreSQL",
  "config": {
    "username": "{{toggl_reshiftConfig.username}}",
    "password": "{{toggl_reshiftConfig.password}}",
    "hostname": "{{toggl_reshiftConfig.hostname}}",
    "port": "{{toggl_reshiftConfig.port}}",
    "database": "{{toggl_reshiftConfig.database}}"
  },
  "keys": {
    "create_and_update_table_closed_projects": {
      "sql": "\nSET SEARCH_PATH TO {{toggl_redshiftConfig.schema}};\n\nDROP table if exists closed_projects;\nCREATE table closed_projects(\nproject nvarchar(100)\n);\n\n{%for PROJECT in CLOSED_PROJECTS.split(',') -%}\nINSERT INTO closed_projects values ('{{PROJECT}}');\n{%endfor - %}\n\nSELECT COUNT(*) from closed_projects;\n",
      "query-type": "execute_scalar",
      "set-runtime-vars": {
        "result": "row_count_closed_projects"
      }
    },
  }
}

Special loop variable

Inside for loops there is a special variable called loop, which contains things like the loop index.

  • index is the loop index starting from 1.
  • index0 is the loop index starting from 0.
# Iteration of a list
{% for user in users %}
    {{ ',' if loop.index0 > 0 else ''}}
    {{user}}
{% endfor %}

Condition blocks

Jinja templates can also include conditional statements that use template or runtime variables. Here, the {{% %}} is used along with if, elif, else, and endif.

Tip

Use conditional Jinja templating to configure keys whose execution is dependent on runtime variables.

{% if a == 0 %}
...
{% endif %}

{% if item.first %}
...
{% elif item.last %}
...
{% else %}
...
{% endif %}

Note

Both JSON and Jinja Boolean literals are supported inside Jinja statements (True, true, False, false).

Example: Parameterized data population in databases

datasource.json

{
    "keys": {
    {% for i,tablename in enumerate(tables) %}
        {{ ',' if i >0 else '' }}

        "create-table-{{tablename}}": {
            "sql" : "{{load_text('table-ddl.sql',params={'table':tablename})}}",
          "query-type" : "execute_non_query"
        },
        "create-data-{{tablename}}": {
          "sql" : "{{load_text('insert-data.sql',params={'table':tablename})}}",
          "query-type" : "execute_non_query"
        }

    {% endfor %}
    }
}
 

/resources/table_ddl.sql

create table {{table}} (id integer primary key, description varchar(100), value float);
 

/resources/insert_data.sql

{% for i in range(100) %}
insert into {{table}} values({{i}},'Item #{{i}}',{{random() * 100}});
{% endfor %}

Example: Use JSON resource files

/resources/zip_codes.json

[
  {
    "state"   : "AL",
    "city"    : "Huntsville",
    "zipfrom" : 35801,
    "zipto"   : 35816
  },
  {
    "state"   : "AK",
    "city"    : "Anchorage",
    "zipfrom" : 99501,
    "zipto"   : 99524
  }
  ....
]
 

/resources/insert_data.sql

{% for entry in load_json('zip-codes.json') %}

insert into ZIPCODES (state,city,zipfrom,zipto)
    values ({{entry.state}},{{entry.city}},{{entry.zipfrom}},{{entry.zipto}});

{% endfor %}