1 March 2021 Release Notes¶

Version 1.1.205. Note that version 1.1.204 was deployed to DataKitchen testing environments only.

Feature Highlights¶

Order Runs Show New Node Status¶

The Order Run Details page now differentiates nodes with test failure warnings from nodes that process with no test failures.

Node tests may be configured to take different actions on failures—stop execution, generate a warning, or log the incident. When a user configures a test for the Warn failure mode, the node continues to process despite a test failure unless the number of warnings for the entire order run exceeds a pre-determined threshold. Now, users can see at a glance which nodes finished execution but had test result warnings along the way. [DEV-7253]

Nodes display in yellow if they completed but had test failures with a "warning" failure mode.

Ingredient Updates Improve Logging, Performance¶

The ingredients feature, which enables users to reuse common processes in multiple recipe variations, has been updated to address a number of performance and access issues. [DEV-7480]

Changes to ingredients

Some temporary kitchens eliminated: Previously, the system created temporary child kitchens to process all ingredients, increasing runtime and resource usage. With this update, the system processes an ingredient that resides in the same kitchen as the calling variation within that kitchen as a separate order. Ingredients that reside in different kitchens than their calling variations still require the temporary child kitchens for processing.
Polling frequency increased: The default polling interval, used to determine both ingredient processing and log status, has been decreased from 60 seconds to 5 seconds. With polling occurring at a faster rate, the system can identify completed ingredient execution much sooner.
Log polling extended: The log information that the system previously transferred from an ingredient order run to the calling order run was often incomplete because the system did not continue to check for new entries. Now, the system polls for a full minute during ingredient execution and transfers the full log set only from ingredients running in a separate kitchen.
Improved log access: Instead of transferring all log information from an ingredient order run to the calling order run, the system now handles logs in two ways.
- For an ingredient that resides in the same kitchen as the calling variation, the system writes a cross-reference statement in the calling order run log to show the user the specific order run for the ingredient execution.
- For ingredients that reside in separate kitchens, the system pulls the logs from the temporary kitchens where they execute and mark them accordingly in the logs of the calling order run.

Log Excerpt: Ingredient in same kitchen

1 2021-02-23 10:14:40.880850 INFO [MainThread]: DKCommandServer: starting

...

72 2021-02-23 10:15:08.640299 TRACE [NodeExecutorThread:0]: Found servings: ['9ca15f74-7784-11eb-8801-d640e9f75f37']

73 2021-02-23 10:15:08.640495 INFO [NodeExecutorThread:0]: Ingredient order status=30

74 2021-02-23 10:15:08.640609 INFO [NodeExecutorThread:0]: DKNode_Ingredient: Ingredient node logs can be found in orderrun 9ca15f74-7784-11eb-8801-d640e9f75f37

75 2021-02-23 10:15:09.040039 INFO [NodeExecutorThread:0]: NodeConsumer:Run: got status DKNodeNotebookStatus_completed_success for node ingredient

76 2021-02-23 10:15:09.040249 INFO [NodeExecutorThread:0]: Post conditions ok

...

Users can find cross-reference entries with IDs in the calling order runs to locate logs from ingredients that run in the same kitchen.

Log Excerpt: Ingredient in different kitchen

1 2021-02-25 14:37:11.757525 INFO [MainThread]: DKCommandServer: starting

...

62 2021-02-25 14:37:16.727736 TRACE [NodeExecutorThread:0]: Start processing node child_ingred

...

65 2021-02-25 14:37:16.737937 TRACE [NodeExecutorThread:0]: Running ingredient in kitchen test-child_e80d335a16b5b236f27b883af76c4ded

...

78 2021-02-25 14:37:49.272646 TRACE [NodeExecutorThread:0]: DKNode_Ingredient: waiting for all log data (0 logs retrieved so far)

79 2021-02-25 14:37:54.616119 INFO [NodeExecutorThread:0]: DKNode_Ingredient: log data retrieved:

80 2021-02-25 14:37:54.616337 INFO [NodeExecutorThread:0]: Ingredient log: [2021-02-25 14:37:39.695115]: [MainThread] DKCommandServer: starting

...

84 2021-02-25 14:37:54.617825 INFO [NodeExecutorThread:0]: Ingredient log: [2021-02-25 14:37:40.871071]: [MainThread] Order ID:e14bc742-77a0-11eb-b31c-9ebd09c45fee

85 2021-02-25 14:37:54.618166 INFO [NodeExecutorThread:0]: Ingredient log: [2021-02-25 14:37:40.871754]: [MainThread] Order Run ID:ec0d9bc4-77a0-11eb-8022-7a66919433e4

...

107 2021-02-25 14:37:54.623234 INFO [NodeExecutorThread:0]: Ingredient log: [2021-02-25 14:37:42.605465]: [MainThread] Loading recipe with variation ingredient_base

...

147 2021-02-25 14:37:54.633959 TRACE [NodeExecutorThread:0]: Ingredient log: [2021-02-25 14:37:43.183061]: [NodeExecutorThread:0] Start processing node test_node ...

150 2021-02-25 14:37:54.634190 INFO [NodeExecutorThread:0]: Ingredient log: [2021-02-25 14:37:43.187187]: [NodeExecutorThread:0] NodeConsumer:Run: got status DKNodeNotebookStatus_completed_success for node test_node ...

158 2021-02-25 14:37:54.636863 INFO [NodeExecutorThread:0]: Ingredient log: [2021-02-25 14:37:44.481489]: [VariationThread] Set Order Run(ec0d9bc4-77a0-11eb-8022-7a66919433e4) Status (COMPLETED_SERVING)

...

Users can view ingredient logs from separate, temporary kitchens among the entries in the logs of the calling order runs.

Automation Supports Multiple Version Control Systems¶

The original Automation platform was built to use GitHub repositories exclusively for kitchen branching and merging, recipe commits, and version control. Now, the platform has been expanded to support any Git-based solution for managing repositories. DataKitchen is currently beta-testing with select customers. In the future, customers may request account configurations using GitHub, GitLab, Bitbucket, and many others. [DEV-3546]

Automation now supports any Git-based repository.

Other Fixes & Enhancements¶

Jinja2 Upgraded in Automation Platform: The platform supports referencing variables using the Jinja templating language. This release upgrades Jinja to version 2.11.2 to remediate known security issues and support newer features. This latest version no longer supports complex templating where the values of page variables are changed during template processing, for example the use of a for loop that increments a numeric value. Note that this limitation does not affect runtime variables. See Advanced Jinja Templating and its subtopics for more information. [DEV-6187]

Containers Allow Multi-Line Errors: Automation's analytic container images, including the General Purpose Container (GPC), were not handling error log messages with multiple lines. When an error occurred that triggered a user's custom log message, the order run logs identified only the first line as an error and the remaining lines as info-level entries. The logs did not link these lines together in a useful stack trace. This issue has been resolved, and the images are updated to allow error messages written with more than one line. [DEV-7637]

Platform Disregards Case in Object Names: Until this release, the platform treated all names as case-sensitive. This validation system caused some problems because database schema names do recognize case. The new validation treats all new object names created via the web app as case-insensitive, so a user cannot create a kitchen named "Test_Kitchen" if "test_kitchen" already exists within her customer account. See Naming Conventions for more information. [DEV-7288]

Note

Case Validation Notes: This validation will not generate errors for existing object names, only new names entered via the web app user interface. Additionally, the new case-insensitive validation has not yet been implemented in the command line interface or in the DataKitchen API.

Python Exit Codes Handled Properly: A user reported that a container node was failing on any parameter in a Python exit() function. Automation's General Purpose Container (GPC) was interpreting all exit codes as failures. This issue has been resolved; nodes complete with a success status when exit codes equal zero (0) and fail when exit codes return non-zero values. Log statements now indicate the return codes as expected. [DEV-7176]

Web App SQL Fields Can Expand: The fixed-length SQL fields in the web app's node configuration forms have been updated to expandable fields. Users can drag the field from the lower right corner to increase the vertical space. The SQL fields appear for database source or sink settings in action node steps, DataMapper node mappings, and container node source inputs. [DEV-7493]

Users can drag to expand the corner of a SQL field.

Orders Page Navigation Fixed: Users found that the :material-: Next Page button on the Orders page was not working correctly after taking the user to the second page of orders. This issue was resolved immediately, and the button works as expected. [DEV-7828]

Variations, Secrets Lists Alphabetized: The variations in Compile drop-down lists and secrets in various drop-down lists were not displayed in case-insensitive, alphabetical order. This issue has been resolved, and all drop-down lists display in alpha order as expected. [DEV-7516]