Source and Sink Processing

Within a given node, data sources are processed first, one by one, followed by the node's notebook.json, and finally, the node's data sinks, one by one. See File Evaluation Sequence for more information.

If multiple data sources are configured for a given node, they will process in a series of indeterminate order. The same applies to data sinks.

Tip

Consider processing in graph design. The processing order of data sources and sinks is an important consideration as to whether multiple sources or sinks should be included in a single node or broken out into multiple nodes.

Key execution

Data sources and sinks execute a series of substeps called keys.

Keys are executed in the order with which they are defined within any given data source or sink. For example, if three keys (A, B, C) are defined in that order, key A will process, then key B, and finally, key C. If key B fails, key A will process successfully, but key B will generate a run error, meaning key C will not be executed.

Test execution

Data source and sink tests are evaluated in the order they are defined within any given data source or sink.

Warning

UTF-8 file encoding is required. Files used with data sources and data sinks must be encoded in UTF-8 in order to avoid non-Unicode characters causing problems with sinking data to database tables and errors when running related tests. For CSV and other delimited files, use the "save as" function in your application and select the proper encoding, or consider using a text editor with encoding options.