Skip to content

File-Based Wildcards

File-based data sources and sinks allow users to extract and load an arbitrary number of files by supporting a wildcard feature using Glob syntax (*).

Any file-based data source or sink may include both explicit, single-file mappings and a wildcard-generated mapping with the following requirements.

  • Only one wildcard mapping may be used for each data source or sink.
  • Wildcard values for files are set at the source/sink level, using the JSON wildcard and wildcard-key-prefix fields.
  • Wildcard values in folder paths are set for data sources using the Folder Path (or JSON wildcard-key-prefix) field
  • Wildcards in folder paths are set for data sinks with the Target Folder Path (or JSON file-based) field.
  • Single-file mappings, not using wildcards, are set at the key level, with values defined under each mapping name.

Note

Wildcard Limitation for Folder Paths: Wildcard glob syntax (*) in folder paths of wildcard mappings is only supported in the final segment of a path. For example, /folder/child_folder/* is supported, while /folder/*/final_folder is not supported.

Wildcard Mappings examples: web app

  • Mapping Type: multiple-file mapping using wildcard

()

Mappings tab with a wildcard configured to retrieve multiple CSV files from the data source.

  • Mapping Type: multiple-folder/multiple-file mapping using wildcard

()

Wildcard mapping configured to retrieve multiple CSV files from multiple folders.

Wildcard Mappings example: File Editor app

()

A composite image showing how three node files record a wildcard mapping.