GPC Resource Allocation¶
For resource-intensive order runs—for example, those with large queries, heavy use of variables, or data sources and sinks—you can assign memory and disk space. These allocations can be set for the order run or for container nodes.
- Resources for order runs: these variation-level resource settings affect an overall order run and the temporary storage
used by data sources and data sinks; they are not relevant to processing that occurs inside a container node. Runtime settings for order run resources are configured in the UI schedule forms or in the
variations.jsonfile. - Resources for container nodes: these container settings only affect a container node itself. Here, boosting RAM and disk can help with heavy processing needs by running scripts within that container. Assign resource values from the Configuration tab of the Node Editor.
Potential resource issues¶
General purpose container nodes can be configured to perform heavy data processing. Two potential issues may occur with container node resource allocation:
-
Allocated resources are greater than what's available.
When a user configures resource usage that exceeds the resources available for a single node, the system will not complete the schedule. A warning message appears that says, "Not enough allocatable resources."
-
The resource allocation is insufficient to execute the order run.
An order run fails in an error state because it does not have enough memory or disk space to complete the processing. Log entries and metrics must be used to determine how much memory and disk space to add for the node.
Best practice¶
On your local machine, examine the memory consumption of any scripts you want to insert into container nodes in a recipe. Memory leaks affecting these standalone scripts will also impact any order runs that include these scripts.