Skip to content

Order Run Test Results

Automation provides test results for three use cases:

Note

Resume function creates a new order run. When a stopped or failed order run is resumed, the system puts the initial run into a Resumed state and starts a new order run to process the remaining nodes of the recipe variation. Note that the two order runs record separate timings and test results, and the initial run retains the Resumed status permanently.

View order run test results

The test results for an individual order run are accessible from the Order Run Details page. Click Test Results in the order run toolbar.

The Test Results dialog sorts results by test failure status (failed, warning, etc.), then by node and tests in execution order. Links open the relevant Tests tab in the Node Editor.

Visual example in Automation

(Screenshot of the Test Results dialog)

Order run test results in the command line

Test results for an individual order run can be viewed from the orderrun-info command using the --test option.

View node test results

The Order Run Details page includes test data in the Node Details for each graph node.

Select any node in the graph, then expand the Tests section.

View test results in order run logs

From the Order Run Details page, click Logs. To provide visibility on test outcomes, tests with a Failure Action of Stop appear in order run logs as an Error, and tests set as Warn appear as a Warning.

Logs can provide debugging insight for failed orders, for example, when an order is set to Stop on Error in the test configuration.

View test results over time

Get graphical views of test results over time and across multiple order runs and orders for a given kitchen-recipe-variation combination.

  1. Use one of the following to navigate to the Variation Metrics page:
    • On the Orders page, click a linked recipe/variation name.
    • On the Order Run Details page, click Variation Metrics over Time.

From the Tests tab, hover your cursor over any data point in the graphs to see more details. To dig deeper into test results:

  • Type a part or all of a test name or regex (test logic expression) in the search field to find a specific results chart.
  • Refine the Tests tab using the node name and order run status filters.

  • Reorder test charts using sort options: Order of Execution, Test Name, or Node Name.

    The default sort order is by order of test execution.

  • Click Edit Test to access the related Tests configuration tab in the Node Editor.

  • Click Export to download graphs as PNG, SVG, or JSON files.

Tip

Click to open the Timings tab to view order run durations. Click on a specific order run in the bar chart to view its node durations in bar or Gantt chart formats.

Visual example in Automation

(Screenshot of the Variation Metrics page)