Copy and Compute Data Example¶

Use Case: Data analysts require a data file to be loaded to an Amazon Redshift database in order to produce insight reports in Tableau.
Example Recipe: In this example, the variation uses a DataMapper node to copy a sales data file from an SFTP server to an Amazon S3 data store. The variation then loads the data to a Redshift database where it runs a series of SQL queries to compute profits. Finally, it publishes the output to a Tableau workbook and uses an ingredient to direct the data to a second target.
Action Node Role: With the data file in Amazon S3 and the proper table created in a Redshift database, the third node in this variation copies the data to the new table, runs SQL queries against it to compute profits, and tests for expected results. While a DataMapper node could have been used to transfer data from a file-based source to a database sink, the action node is used here because it is more efficient to take advantage of features available within the AWS environment.

An action node copies data to an Amazon Redshift instance.

Web App Configuration¶

The following images show how the node is configured using the web app forms.

Note

Tests Follow Steps: Tests declared for a data source in an action node are executed only after all keys/steps have been executed successfully.

The data source connection could use either a pre-defined variable dictionary or call the variables for each credential.

The anatomy of the steps configured in the web app for the action node.

Test 1 compares a value from step 1 with a value from another node in the variation.

Test 2 uses the value from step 2 to test for any loss of expected data.

Test 3 uses the value from step 4 to ensure that data entries are standard.

The platform records the connection, step, and test configurations from the web app forms shown above in the actions node file: source.json.