Wildcard Mapping Example

Map SFTP Files to S3

This example demonstrates the use of wildcard mappings to extract multiple files from a source. In this case, the system retrieves all files that match the wildcard expression input-*.txt from the stfp/in folder on the SFTP server. Then, the system renames those files following the pattern output-*.txt and places them in the wildcard-test folder in the specified S3 bucket.

Connections to the source and sink are contained in config dictionaries, but could otherwise be defined in Kitchen Overrides and referenced here.

notebook.json

{
    "wildcard-will-automatically-create-mappings": [
        {
            "data-source": "sftp_datasource",
            "data-sink": "s3_datasink"
        }
    ],
    "mappings": {}
}

 

/data_sources/sftp_datasource.json

{
    "type": "DKDataSource_SFTP",
    "name": "sftp_datasource",
    "config": {
      "username": "uid234",
      "hostname": "#{vault://dk-sftp-03/host}",
      "port": 22,
      "pem_file": "#{vault://dk-sftp-03/credentials.pem}"
    },
    "wildcard": "input-*.txt",
    "wildcard-key-prefix": "sftp/in/",
    "keys": {}
}

 

/data_sinks/s3_datasink.json

{
    "type": "DKDataSink_S3",
    "name": "s3_datasink",
    "config": {
      "bucket": "#{vault://s3_schema/bucket}",
      "access-key": "#{vault://s3_schema/s3-access-key}",
      "secret-key": "#{vault://s3_schema/s3-secret-key}"
    },
    "wildcard": "output-*.txt",
    "wildcard-key-prefix": "wildcard-test",
}