Skip to content

DataMapper Node Files

All mapping, connection, and test configurations of a DataMapper node are reflected in the node files accessible from the File Editor or from DKCloudCommand, the command line interface (CLI).

Mappings Recorded in Node Files

  • Node Type: DataMapper
  • Node Name: sftp_to_s3

Mappings example: web app

  • Mapping Type: single-file mapping

()

Extended image of both source and sink configuration forms in the web app.

Mappings example: File Editor

The same configurations made in the web app appear in JSON files when viewing the node in the File Editor.

()

The notebook.json file records the high-level mappings.

Note that both source and sink JSON files contain an auto-generated use-only-file-keys field. This field is slated for deprecation but works in the following manner.

  • "use-only-file-key": true = instructs the system to use the file path in the file-key field.
  • "use-only-file-key": false = instructs the system to calculate the file path by concatenating the key name and the value in the file-key field.

()

The JSON file in the data_sources directory contains the detailed mapping properties for the source.

()

The JSON file in the data_sinks directory contains the detailed mapping properties for the sink.

Note

Source/sink names are auto-generated. The system generates the default names, "source" and "sink," upon creation of a DataMapper node, as well as the default folders, "data_sources" and "data_sinks." Other elements inherit those names: the file generated within the node's data_sources folder is source.json, the file in the node's data_sinks folder is sink.json, and mapping keys are mappingname_source and mappingname_sink. Any name changes must be updated in the node's notebook.json and in the source and sink files.

Wildcard Mappings examples: web app

  • Mapping Type: multiple-file mapping using wildcard

()

Mappings tab with a wildcard configured to retrieve multiple CSV files from the data source.

  • Mapping Type: multiple-folder/multiple-file mapping using wildcard

()

Wildcard mapping configured to retrieve multiple CSV files from multiple folders.

Wildcard Mappings example: File Editor app

()

A composite image showing how three node files record a wildcard mapping.

Connections Recorded in Node Files

  • Source Connection Type: SFTP credentials defined as variable using vault secrets and PEM file
{
    "sftpConfig": {
        "username": "#{vault://sftp/username}",
        "pem_file": "#{vault://sftp/pem_file}",
        "hostname": "#{vault://sftp/hostname}",
        "port": "#{vault://sftp/port}"
    }
}
  • Sink Connection Type: S3 credentials defined as variable using vault secrets
{
    "s3Config": {
        "secret-key": "#{vault://s3/secret_key}",
        "access-key": "#{vault://s3/access_key}",
        "bucket": "#{vault://s3/bucket}"
    }
}

Connections example: web app

()

Connections tab for source and sink credentials in the web app.

Connections example: File Editor

()

The JSON file in the data_sources directory contains source connection properties.

()

The JSON file in the data_sinks directory contains sink connection properties.

Tests Recorded in Node Files

  • Source Test: Verifies that the file retrieved from the SFTP server contains data and stops the node execution if it does not.

Tests example: web app

()

Tests tab with a source test defined.

Tests example: File Editor

()

The JSON file in the data_sources directory contains test definitions for the source file.