Agent Resources

Each agent runs orders concurrently, limited only by computing and storage resources. Kubernetes automatically scales the number of available agents to handle as many concurrent order runs as needed.

To manage performance and cost, and to create a setup that best suits your requirements, you can:

• Configure the minimum and maximum number of available agents.
• Set up agent groups to run specific orders on Kubernetes worker nodes.

Tip

One benefit of using Kubernetes services is that resources can be scaled by increasing or decreasing the number of agents (i.e. Kubernetes worker nodes) that run pods, or sets of running containers.

Allocate resources with schedules

You can allocate resources during schedule configuration to assign memory and disk space for order processing.

The cron expression configured for order runs in a variation becomes layers of Kubernetes jobs that dispatch each other. This means that a Kubernetes cron job creates jobs on a repeating schedule, and those jobs create one or more pods to execute the work.

If multiple orders run at the same time, an order that has insufficient resources stays in a pending state until resources become available in the agent. If the agent runs out of memory or disk space completely, the order will end with errors.

For more information about memory and disk space, see Runtime Resources.

Agent pools and agent groups

Customers can host and maintain a private pool of agents. In this case, DataKitchen implementation engineers will configure all kitchens to constrain to the private pool. Once set up, customers can define agent groups to allocate agents by environment.

DataKitchen also offers a default agent pool to allocate resources. At the time of any order run, the system provides the best available agent for use.

Agent groups

Customers can place constraints on resource usage by specifying agent groups that subdivide their agent pools. Agent groups allow you to reserve or restrict the resources for specific kitchens.

For example, in a pool of three agents, a DataKitchen customer may want a production kitchen to use agent1, a staging or UAT kitchen to use agent2, and all remaining kitchens to share agent3. This method can help make sure that production deployments will always have sufficient resources.

See Specify Agent Groups for more information.

Troubleshooting: Insufficient Resources