File-Based Properties¶
Unless otherwise noted, these properties can be used when configuring nodes for any file-based data source or sink.
In JSON node files, these fields can be used to configure the source or sink generally or used to configure specific keys (steps) within a source or sink definition.
For the properties specific to each connector, consult the docs for Supported Connectors.
Note
Source/sink names are auto-generated. The system generates the default names, "source" and "sink," upon creation of a DataMapper node, as well as the default folders, "data_sources" and "data_sinks." Other elements inherit those names: the file generated within the node's data_sources folder is source.json, the file in the node's data_sinks folder is sink.json, and mapping keys are mappingname_source and mappingname_sink. Any name changes must be updated in the node's notebook.json and in the source and sink files.
Access file-based properties¶
File-based source and sink mapping/step and wildcard properties can be found in the Mappings tab of the Node Editor and in the JSON file of the File Editor.
File-based source properties¶
The following properties can be used when connecting to a source.
Mapping/step¶
| Mapping/step field in the UI | Mapping/step field in JSON | Description and details |
|---|---|---|
| File Path | file-key | Path and filename to get on the data source. The path is built using key/file-key. Availability: Single-file mappings, not wildcard mappings. Required field Value type: string Node file: Recorded in /data_sources/source.json |
| N/A | use-only-file-key | The system writes this value as true. When set to true, it uses only the value file-key as path/file spec to reference a file, ignoring the key. Availability: Single-file mappings at the key level. Node file: Recorded in /data_sources/source.json in key dictionary for single-file mappings. |
| Cache File (checkbox) | cache | When checked, instructs the system to store the source data in local memory to use again in downstream nodes. Other downstream data sources of the same type may leverage cached data by referencing upstream key names. Using the cache feature prevents repetitive file retrievals, but does not presently support order run resume. Availability: Single-file mappings, at key level. Value type: Boolean Node file: Recorded in /data_sources/source.json in key dictionary for single-file mappings. |
| Decrypt File (checkbox) | decrypt-key, decrypt-passphrase | When checked, displays fields for file access credentials. Private Key: Encryption key for the secured file. Define as a secret to follow best practices. Key Passphrase: Encryption passphrase for the secured file. Define as a secret to follow best practices. Availability: Single-file source mappings, not wildcard mappings. Value type: string Node file: Recorded in /data_sources/source.json |
Wildcard¶
| Wildcard field in the UI | Wildcard field in JSON | Description and details |
|---|---|---|
| Folder Path | wildcard-key-prefix | Path to folder on data source containing multiple files to be retrieved using the given wildcard expression. The folder path may include variables and wildcards to retrieve files from multiple folders. Wildcards are limited to the final segment of a path. For example, /folder/child_folder/prefix_{{yymmdd}}* Availability: Wildcard mappings, at source level. Node file: Recorded in /data_sources/source.json |
| File Glob Pattern | wildcard | Definition of filename pattern using wildcard characters. For example, *.csv Specifies a glob wildcard expression to pick/push a set of arbitrary files that match the declared wildcard expression. Wildcards apply only to a single directory; they are not recursive. Use multiple data sources or data sinks if pulling or pushing files across multiple directories. Required field: Yes, if wildcard mapping used. Availability: Wildcard mappings, at source level. Value type: string Node file: Recorded in /data_sources/source.json |
| Cache Files (checkbox) | cache | When checked, instructs the system to store the source data in local memory for faster retrieval. Other downstream data sources of the same type may leverage cached data by referencing upstream key names. Using the cache feature prevents repetitive file retrievals, but does not presently support order run resume. Availability: Wildcard mappings, at source level. Value type: Boolean Node file: Recorded in /data_sources/source.json at source level for wildcard mappings. |
File-based sink properties¶
The following properties can be used when connecting to a sink.
Mapping/step¶
| Mapping/step field in the UI | Mapping/step field in JSON | Description and details |
|---|---|---|
| Target File Path | file-key | Path and filename for copying the output file to the data sink. The path is built using key/file-key. Node file: Recorded in /data_sinks/sink.json |
| Encrypt File (checkbox) | encrypt-key | When checked, displays a field to configure file encryption. Private Key: Encryption key for securing the output file on the data sink. Define as a secret to follow best practices. Availability: Single-file sink mappings, not wildcard mappings. Value type: string Node file: Recorded in /data_sinks/sink.json |
Wildcard¶
| Wildcard field in the UI | Wildcard field in JSON | Description and details |
|---|---|---|
| Sink Files: Target Folder Path | file-based | Path to folder on data sink where the files will be sent. The folder path may include variables and wildcards to put files in multiple folders. Wildcards are limited to the final segment of a path. For example, /folder/child_folder/prefix_{{yymmdd}}* Availability: Wildcard mappings, at source level. Node file: Recorded in /data_sinks/sink.json |