File-Based Variables¶
Unless otherwise noted, these built-in runtime variables can be used for all file-based data sources and sinks. All of these variables are optional.
In JSON node files, these variables can be used in configuring the keys (steps) within a step or source/sink mapping definition.
See Built-In Runtime Variables for the variables used in configuring the node or source/sink, itself, and for testing wildcard mappings.
Access runtime variables¶
Runtime variables can be found in the Mappings tab of the Node Editor and in the JSON file of the File Editor.
Runtime variables¶
Runtime variables for file-based source and sink mappings/steps.
| Runtime variables field in the UI | Runtime variables field in JSON | Description and details |
|---|---|---|
| File Size (in bytes) | size | For data source: Specifies the variable, specific to the source mapping, that stores the file size in bytes of the source file. For data sink: Specifies the variable, specific to the sink mapping, that stores the file size in bytes of the output file. Tests comparing input and output file size can validate file transmissions. Value type: string Node file: Recorded in /data_sources/source.json and/or /data_sinks/sink.json. |
| Line Count | row_count | For data source: Specifies the variable, specific to the source mapping, that stores the number of lines in any kind of source text file. Tests comparing input and output line counts can validate file transmissions. Value type: string Node file: Recorded in /data_sources/source.json and/or /data_sinks/sink.json. |
| MD5 Hash | md5 | For data source: Specifies the variable, specific to the source mapping, that stores a calculated checksum of the source file contents. For data sink: Specifies the variable, specific to the sink mapping, that stores a calculated checksum of the output file contents. Tests on MD5 can provide assurance that a file has been transmitted without corruption. Value type: string Node file: Recorded in /data_sources/source.json and/or /data_sinks/sink.json. |
| SHA-1 Hash | sha | For data source: Specifies the variable, specific to the source mapping, that stores a hash value of the source file contents. For data sink: Specifies the variable, specific to the sink mapping, that stores a hash value of the output file. Tests on SHA-1 can provide assurance that a file has been transmitted without corruption. Value type: string Node file: Recorded in /data_sources/source.json and/or /data_sinks/sink.json. |
Warning
Row count test results may not match. When row_count variables are used with file-based data sources and sinks, the resulting value includes a header row if one is present. Conversely, when row_count variables are used with database sources and sinks, the resulting value does not count the header row when database table headers are present. A database table header row is only included in the data set if the property key insert-column-names is set to "true."
Warning
UTF-8 file encoding is required. Files used with data sources and data sinks must be encoded in UTF-8 in order to avoid non-Unicode characters causing problems with sinking data to database tables and errors when running related tests. For CSV and other delimited files, use the "save as" function in your application and select the proper encoding, or consider using a text editor with encoding options.