Skip to content

Configure DataMapper Nodes

Configure a DataMapper node from the Node Editor.

You can also review the node files and update the configurations in the File Editor. See DataMapper Node Files for more information.

Runtime variable considerations

Runtime variables can be configured for DataMapper nodes at the mapping-level and/or at the node-level.

  • Mapping-level: These runtime variables appear with the source and sink configurations for most source/sink mapping forms. These variables are associated solely with each mapping that is defined.
    • You can define these runtime variables for both source and sink mappings for use in tests, to ensure that the file is not altered or corrupted during transmission.
    • For details on mapping runtime variables, see File-Based Variables and Database Variables.
  • Node-level: These variables are associated with the source as a whole or with the sink as a whole.
    • These fields can also be used to store data for tests of wildcard mappings; the data about the JSON keys in source or sink mappings are dynamically generated when processing the wildcard.
    • For details on these built-in runtime variables, see Built-In Runtime Variables.

Configure a node

  1. Navigate to the Node Editor.
  2. Select a tab to configure the DataMapper node.

Required fields are marked with an * asterisk.

Note

Source/sink names are auto-generated. The system generates the default names, "source" and "sink," upon creation of a DataMapper node, as well as the default folders, "data_sources" and "data_sinks." Other elements inherit those names: the file generated within the node's data_sources folder is source.json, the file in the node's data_sinks folder is sink.json, and mapping keys are mappingname_source and mappingname_sink. Any name changes must be updated in the node's notebook.json and in the source and sink files.

Mappings tab

  1. Click Add Mapping to move a single file between source and sink, or click Add Wildcard to move multiple files.

  2. For single-file mapping, specify the properties of the data transmission for the source and the sink; use variables or  secrets where available.

    1. Enter a Mapping Name and Description. See DataMapper Node Properties for more information on these fields.
    2. Define the data to be retrieved from the Source.

    3. Set the source's Runtime Variables.

      These are mapping-level runtime variables.

    4. Select Sink.

    5. Specify the properties of the data to be placed in the sink.

    6. Set the sink's Runtime Variables.

      These are mapping-level runtime variables.

  3. For wildcard/multiple-file mapping, use variables or secrets (where available) to specify the properties of the source and the sink files.

  4. Optional. Under the list of mappings at the left, select Runtime Variables.

    1. If needed, define runtime variables for the node's source and/or sink.

      These are node-level runtime variables.

  5. Select Add notes for changelog , describe the changes, then click Update.

Additional resources

More context on the DataMapper node Mappings tab properties can be found in the following documentation:

Connections tab

Tip

Connection properties differ for each source and sink connector. Details on how to fill out these fields can be found in the Supported Connectors documentation.

  1. Select the Connections tab.
  2. Use variables, secrets, or enter credentials to define the data source and data sink connections.
    • Specify Credentials: Enter each credential manually, or select variables or secrets where available.
    • Use Variable: Select this option if a dictionary variable, containing all required credentials, has been defined in the kitchen or recipe.
  3. Select Add notes for changelog , describe the changes, then click Update.

Tests tab

  1. Select the Tests tab.
  2. Click Add Source Test to create data verification tests for the file that the system will receive from the data source.

  3. Click Add Sink Test to create validation tests on the output file.

    These settings are recorded in /data_sinks/sink.json

  4. Select Add notes for changelog , describe the changes, then click Update.

For technical details on test properties, see Test Declaration and Test Properties.

Copy and delete

Click Copy next to a test or mapping to duplicate its configuration within the DataMapper node. Adjust the configuration to make the new test or mapping unique.

Click Delete to remove a mapping or test from the node.