SFTP

SFTP data sources and sinks are in the file-based category of I/O connectors.

Tool documentation

• SFTP documentation

Connector type values

The "type": value to use in the source or sink JSON files.

Connector type	Value
Data source	DKDataSource_SFTP
Data sink	DKDataSink_SFTP

Connection properties

The properties to use when connecting to an SFTP instance from Automation.

Field	Scope	Type	Required?	Description
hostname	source/sink	string	yes	STFP hostname.
password	source/sink	string	yes, if pem_file is not set.	SFTP password.
pem_file	source/sink	string	no	A path to the SSH key file, or the contents of the key file if reading from Vault. Required if password not set.
support-legacy-sftp	source/sink	boolean	optional, for use with pem_file	Can be used to support legacy servers that themselves do not support SHA-2 signature algorithms or the server-sig-algs protocol.
port	source/sink	string/number	no	SFTP port. Default is 22.
username	source/sink	string	yes	SFTP user name.
reconnect-interval	source/sink	integer	no	Specifies the frequency in seconds to attempt a connection. Default is 3.
retry-count	source/sink	integer	no	Specifies the number of attempts the system makes to establish a connection. Default is 3.

Tip

Passing password in shell commands: SCP/SFTP always expects passwords passed by keyboard. Use the command expect to send passwords in cases where you do not wish to use a PEM file.

Connections

See Connection Properties for an overview on connection configurations and syntax.

Defined in kitchen-level variables

Using password

sftpconfig in kitchen overrides

{
    "sftpconfig": {
        "username": "#{vault://sftp/username}",
        "password": "#{vault://sftp/password}",
        "hostname": "#{vault://sftp/hostname}",
        "port": "#{vault://sftp/port}"
    }
}

Using PEM file

sftpconfig in kitchen overrides

{
    "sftpconfig": {
        "username": "#{vault://sftp/username}",
        "pem_file": "#{vault://sftp/pem_file}",
        "hostname": "#{vault://sftp/hostname}",
        "port": "#{vault://sftp/port}"
    }
}

The Connection tab in a Node Editor

Expanded connection syntax

Using password: for a data source

sftp_datasource.json

{
    "type": "DKDataSource_SFTP",
    "name": "sftp_datasource",
    "config": {
        "username": "{{sftpconfig.username}}",
        "password": "{{sftpconfig.password}}",
        "hostname": "{{sftpconfig.hostname}}",
        "port": "{{sftpconfig.port}}"
    },
    "keys": {},
    "tests": {}
}

Using password: for a data sink

sftp_datasink.json

{
    "type": "DKDataSink_SFTP",
    "name": "sftp_datasink",
    "config": {
        "username": "{{sftpconfig.username}}",
        "password": "{{sftpconfig.password}}",
        "hostname": "{{sftpconfig.hostname}}",
        "port": "{{sftpconfig.port}}"
    },
    "keys": {},
    "tests": {}
}

Using PEM file: for a data source

sftp_datasource.json

{
    "type": "DKDataSource_SFTP",
    "name": "sftp_datasource",
    "config": {
        "username": "{{sftpconfig.username}}",
        "pem_file": "{{sftpconfig.pem_file}}",
        "hostname": "{{sftpconfig.hostname}}",
        "port": "{{sftpconfig.port}}"
    },
    "keys": {},
    "tests": {}
}

Using PEM file: for a data sink

sftp_datasink.json

{
    "type": "DKDataSink_SFTP",
    "name": "sftp_datasink",
    "config": {
        "username": "{{sftpconfig.username}}",
        "pem_file": "{{sftpconfig.pem_file}}",
        "hostname": "{{sftpconfig.hostname}}",
        "port": "{{sftpconfig.port}}"
    },
    "keys": {},
    "tests": {}
}

Warning

SFTP connector PEM format. If you use a PEM file for storing private keys, you must use an unencrypted key in RSA format. Your SFTP source or sink connection will fail on a PEM file if it requires decryption and/or uses another format. See Tips on Key Generation for more information.

Condensed connection syntax

For a data source

sftp_datasource.json

{
    "type": "DKDataSource_SFTP",
    "name": "sftp_datasource",
    "config-ref": "sftpconfig",
    "keys": {},
    "tests": {}
}

For a data sink

sftp_datasink.json

{
    "type": "DKDataSink_SFTP",
    "name": "sftp_datasink",
    "config-ref": "sftpconfig",
    "keys": {},
    "tests": {}
}

Local connection

SFTP file contents can be viewed locally by configuring connections with file-transfer applications like Transmit.

Other configuration properties

See the following topics for common properties, wildcards, and runtime variables:

• File-Based Properties
• File-Based Wildcards
• File-Based Variables

File encoding requirements

Pre-Existing path requirement: SFTP data sinks can only PUT files to existing output directories. They cannot create subfolders during processing. Other file-based data sinks, such as S3 or Google Cloud Storage, have the underlying functionality to create folders automatically.

Files used with data sources and data sinks must be encoded in UTF-8 in order to avoid non-Unicode characters causing problems with sinking data to database tables and errors when running related tests

For CSV and other delimited files, use Save as in the program and select the proper encoding, or consider using a text editor with encoding options.

Data source examples

Example source 1

Get a TXT file from SFTP via an explicit key and generate runtime variables for tests.

sftp_datasource.json

{
    "type": "DKDataSource_SFTP",
    "name": "sftp_datasource",
    "config": {
        "username": "{{sftpconfig.username}}",
        "hostname": "{{sftpconfig.hostname}}",
        "port": "{{sftpconfig.port}}",
        "pem_file": "{{sftpconfig.pem_file}}"
    },
    "keys": {
        "sftp": {
            "file-key": "in/file1.txt",
            "set-runtime-vars": {
                "md5": "key_md5",
                "sha": "key_sha",
                "size": "key_size",
                "row_count": "key_row_count"
            }
        }
    },
    "tests": {

    }
}

Example source 1

Use a wildcard to get an arbitrary list of files.

sftp_datasource.json

{
    "type": "DKDataSource_SFTP",
    "name": "sftp_datasource",
    "config-ref": "sftpconfig",
    "decrypt-key": "",
    "decrypt-passphrase": "",
    "keys": {
        "example-key": {
            "file-key": "",
            "decrypt-key": "",
            "decrypt-passphrase": "",
        }
    },
    "tests": {}
}

Data sink examples

Example sink 1

sftp_datasink.json

{
    "type": "DKDataSink_SFTP",
    "name": "sftp_datasink",
    "config-ref": "sftpconfig",
    "wildcard": "",
    "keys": {
        "example-key-name": {
            "file-key": "file.csv",
            "encrypt-key": "vault://secret-key"
        },
    },
    "tests": {}
}

Example sink 2

sftp_datasink.json

{
    "type": "DKDataSink_SFTP",
    "name": "sftp_datasink",
    "config-ref": "sftpconfig",
    "wildcard": "",
    "keys": {
        "shared": {
            "file-key": "test-out/test.txt"
        }
    },
    "tests": {}
}