Skip to content

Databricks

DataKitchen provides an agent that lets you monitor Databricks workflows for Azure Databricks, Databricks on AWS, and Databricks on Google Cloud.

Deploy

Prerequisites

  • Databricks requirements:
    • Choose the authentication method you'd like to use (personal access token, Azure service principal, or basic authentication) and configure the authentication as needed.
    • Consult the Databricks documentation for steps on how to create a personal access token.
    • Service principal authentication applies to Azure Databricks only and requires Workspace access entitlement on the workspace. Start with Create an Azure Service Principal for technical details.
    • Basic authentication is not recommended by Databricks; however, when used, the password must meet certain complexity requirements .
  • Docker with Docker Compose. If not installed, see Get Docker .
  • An API key for this agent — see step 5 below to create one.
  • Enterprise Access to the agent's private Docker repository. Login with the provided credentials using docker login or through the Docker Compose application.

Steps

  1. Log into Observability and select a project.
  2. Select Integrations from the menu, then click View Available Agents.
  3. Select the tool from the list.
  4. Under Step 1: Prerequisites, verify any requirements for the tool have been completed.
  5. In the New API Key section, create an API key for this agent.
    1. Enter a name, expiration, and description.
    2. Configure the key to send events, manage entities, and transmit heartbeat.
    3. Click Create Key and Continue.

  6. Under Step 2: Configuration, fill in any remaining variables as needed.

    • Required values are noted by an asterisk.
    • Some values are pre-populated with project-specific configuration details.

  7. Click Continue.

  8. Under Step 3: Deploy, enter the agent's Image Tag (format: vx.x.x).
  9. Select Docker as the Deployment Location.
  10. Click Download Script.
  11. Save the file anywhere it can be accessed by Docker.

  12. Open a terminal and run the deployment script:

    chmod +x deploy-agent-databricks.sh
./deploy-agent-databricks.sh

Prerequisites

  • Databricks requirements:
    • Choose the authentication method you'd like to use (personal access token, Azure service principal, or basic authentication) and configure the authentication as needed.
    • Consult the Databricks documentation for steps on how to create a personal access token.
    • Service principal authentication applies to Azure Databricks only. Start with Create an Azure Service Principal for technical details.
    • Basic authentication is not recommended by Databricks; however, when used, the password must meet certain complexity requirements .
  • A Kubernetes cluster with kubectl access.
  • An API key for this agent — see step 5 below to create one.
  • Enterprise Access to the agent's private Docker repository. Login with the provided credentials using docker login.

Steps

  1. Log into Observability and select a project.
  2. Select Integrations from the menu, then click View Available Agents.
  3. Select the tool from the list.
  4. Under Step 1: Prerequisites, verify any requirements for the tool have been completed.
  5. In the New API Key section, create an API key for this agent.
    1. Enter a name, expiration, and description.
    2. Configure the key to send events, manage entities, and transmit heartbeat.
    3. Click Create Key and Continue.

  6. Under Step 2: Configuration, fill in any remaining variables as needed.

    • Required values are noted by an asterisk.
    • Some values are pre-populated with project-specific configuration details.

  7. Click Continue.

  8. Under Step 3: Deploy, enter the agent's Image Tag (format: vx.x.x).
  9. Select Kubernetes as the Deployment Location.
  10. Click Download Script.
  11. Copy the script to a machine with access to the Kubernetes cluster.

  12. Open a terminal and run the deployment script:

    chmod +x deploy-agent-databricks.sh
./deploy-agent-databricks.sh

Follows the steps in Microsoft's Deploy a container instance in Azure , with notes specific to Observability agents.

Prerequisites

  • Databricks requirements:
    • Choose the authentication method you'd like to use (personal access token, Azure service principal, or basic authentication) and configure the authentication as needed.
    • Consult the Databricks documentation for steps on how to create a personal access token.
    • Service principal authentication applies to Azure Databricks only. Start with Create an Azure Service Principal for technical details.
    • Basic authentication is not recommended by Databricks; however, when used, the password must meet certain complexity requirements .
  • An API key for this agent. Create and manage keys from the API Keys page. Configure the key to send events, manage entities, and transmit heartbeat.
  • Enterprise Access to the agent's private Docker repository.

Steps

  1. Sign into your Azure portal and create a container instance.
    1. On the Azure portal homepage, select Create a resource.
    2. Select Containers.
    3. Under Container Instances, click Create.

  2. On the Create container instance page, under the Basics tab, fill in the following:

    • Resource group: Use an existing resource group or create a new one.
    • Container name: Enter a name for the container (e.g., observabilityagent).
    • Image source: Select Other registry.
    • Image: Enter the full name of the agent's Docker image (e.g., datakitchen/dataops-observability-agents:v2.0.0).
    • Image type: Select Private and enter the provided Docker credentials. Enterprise
    • Update the OS type and Size if needed. Leave all other values as default.

  3. Click Next: Networking.

  4. Under the Networking tab, select Public, Private, or None for the networking type, depending on your organization's needs.
  5. Click Next: Advanced.

  6. Under the Advanced tab, fill in the following:

    • Restart Policy: Set to On failure.
    • Environment variables: Enter all agent configuration variables. Set Mark as secure to Yes for any API or token values.
    • Leave all other settings as default.

  7. Click Review + create, then click Create.

After deploying, navigate to Resource groups > {your resource group} > {your container name}. A Running status indicates the agent is sending events to Observability.

Tip

To test the status of your container, copy the FQDN from the container overview and open it in your browser. Then check Containers > Logs for the HTTP GET request, confirming the system is active.

Configuration variables

Variable Required? Description Value
AGENT_KEY Yes Assigns the identifier of the agent. Appears in the UI on the Integrations page. production-project-agent
DK_API_URL Yes The base API URL for the Observability instance. Enterprise: https://api.datakitchen.io
Docker/minikube: http://<minikube_ip>:8082/api
Docker/localhost: http://host.docker.internal:8082/api
K8s/minikube: http://observability-ui.datakitchen.svc.cluster.local:8082/api
DK_API_KEY Yes An API key for the Observability project. A key unique to this agent is recommended. Provides authorization for events and agent heartbeat.

Agent-specific variables

Variable Required? Description Value
AGENT_TYPE Yes Identifier for the agent. Retain the default value. databricks
HOST Yes The URL of the Databricks instance. https://dbc-a1b2345c-d6e7.cloud.databricks.com
TOKEN Yes, if using token-based authentication A personal access token for the Databricks instance.
USERNAME Yes, if using basic authentication A username for the Databricks instance.
PASSWORD Yes, if using basic authentication A password for the Databricks instance.
TENANT_ID Yes, if authenticating with service principal The Directory (tenant) ID of the service principal made for monitoring your Azure Databricks instance. See Create an Azure Service Principal for more details.
CLIENT_ID Yes, if authenticating with service principal The Application (client) ID of the service principal made for monitoring your Azure Databricks instance.
CLIENT_SECRET Yes, if authenticating with service principal The client secret Value that was created during the setup of the service principal.
JOBS Optional The Databricks jobs to observe. Format: a comma-separated list of Databricks job names. If this variable is not defined, the agent will observe all Databricks jobs. Note, the list inside the square brackets requires double quotation marks. '["abc123","abc321"]'
PERIOD Optional Cadence at which the agent polls for information, in seconds. Default: 5.0
TIMEOUT Optional Timeout for requests made by the agent to your tool, in seconds. Default: 120.0
FAILED_WATCH_PERIOD Optional Cadence at which the agent polls for repaired runs, in seconds. Default: 600.0
FAILED_WATCH_MAX_TIME Optional Maximum time the agent watches for repaired runs, in seconds. Default: 604800.0