Skip to content

Azure Synapse Agent

DataKitchen provides an agent that lets you monitor the ETL component of Azure Synapse Analytics .

Deploy

Prerequisites

  • A service principal with a workspace-scoped Synapse Artifact User role assignment is required for authentication. For a walkthrough, see Create an Azure Service Principal.
  • Docker with Docker Compose. If not installed, see Get Docker .
  • An API key for this agent — see step 5 below to create one.
  • Enterprise Access to the agent's private Docker repository. Login with the provided credentials using docker login or through the Docker Compose application.

Steps

  1. Log into Observability and select a project.
  2. Select Integrations from the menu, then click View Available Agents.
  3. Select the tool from the list.
  4. Under Step 1: Prerequisites, verify any requirements for the tool have been completed.
  5. In the New API Key section, create an API key for this agent.
    1. Enter a name, expiration, and description.
    2. Configure the key to send events, manage entities, and transmit heartbeat.
    3. Click Create Key and Continue.

  6. Under Step 2: Configuration, fill in any remaining variables as needed.

    • Required values are noted by an asterisk.
    • Some values are pre-populated with project-specific configuration details.

  7. Click Continue.

  8. Under Step 3: Deploy, enter the agent's Image Tag (format: vx.x.x).
  9. Select Docker as the Deployment Location.
  10. Click Download Script.
  11. Save the file anywhere it can be accessed by Docker.

  12. Open a terminal and run the deployment script:

    chmod +x deploy-agent-azure_synapse_pipelines.sh
./deploy-agent-azure_synapse_pipelines.sh

Prerequisites

  • A service principal with a workspace-scoped role assignment is required for authentication. For a walkthrough, see Create an Azure Service Principal.
  • A Kubernetes cluster with kubectl access.
  • An API key for this agent — see step 5 below to create one.
  • Enterprise Access to the agent's private Docker repository. Login with the provided credentials using docker login.

Steps

  1. Log into Observability and select a project.
  2. Select Integrations from the menu, then click View Available Agents.
  3. Select the tool from the list.
  4. Under Step 1: Prerequisites, verify any requirements for the tool have been completed.
  5. In the New API Key section, create an API key for this agent.
    1. Enter a name, expiration, and description.
    2. Configure the key to send events, manage entities, and transmit heartbeat.
    3. Click Create Key and Continue.

  6. Under Step 2: Configuration, fill in any remaining variables as needed.

    • Required values are noted by an asterisk.
    • Some values are pre-populated with project-specific configuration details.

  7. Click Continue.

  8. Under Step 3: Deploy, enter the agent's Image Tag (format: vx.x.x).
  9. Select Kubernetes as the Deployment Location.
  10. Click Download Script.
  11. Copy the script to a machine with access to the Kubernetes cluster.

  12. Open a terminal and run the deployment script:

    chmod +x deploy-agent-azure_synapse_pipelines.sh
./deploy-agent-azure_synapse_pipelines.sh

Follows the steps in Microsoft's Deploy a container instance in Azure , with notes specific to Observability agents.

Prerequisites

  • A service principal with a workspace-scoped role assignment is required for authentication. For a walkthrough, see Create an Azure Service Principal.
  • An API key for this agent. Create and manage keys from the API Keys page. Configure the key to send events, manage entities, and transmit heartbeat.
  • Enterprise Access to the agent's private Docker repository.

Steps

  1. Sign into your Azure portal and create a container instance.
    1. On the Azure portal homepage, select Create a resource.
    2. Select Containers.
    3. Under Container Instances, click Create.

  2. On the Create container instance page, under the Basics tab, fill in the following:

    • Resource group: Use an existing resource group or create a new one.
    • Container name: Enter a name for the container (e.g., observabilityagent).
    • Image source: Select Other registry.
    • Image: Enter the full name of the agent's Docker image (e.g., datakitchen/dataops-observability-agents:v2.0.0).
    • Image type: Select Private and enter the provided Docker credentials. Enterprise
    • Update the OS type and Size if needed. Leave all other values as default.

  3. Click Next: Networking.

  4. Under the Networking tab, select Public, Private, or None for the networking type, depending on your organization's needs.
  5. Click Next: Advanced.

  6. Under the Advanced tab, fill in the following:

    • Restart Policy: Set to On failure.
    • Environment variables: Enter all agent configuration variables. Set Mark as secure to Yes for any API or token values.
    • Leave all other settings as default.

  7. Click Review + create, then click Create.

After deploying, navigate to Resource groups > {your resource group} > {your container name}. A Running status indicates the agent is sending events to Observability.

Tip

To test the status of your container, copy the FQDN from the container overview and open it in your browser. Then check Containers > Logs for the HTTP GET request, confirming the system is active.

Configuration variables

Variable Required? Description Value
AGENT_KEY Yes Assigns the identifier of the agent. Appears in the UI on the Integrations page. production-project-agent
DK_API_URL Yes The base API URL for the Observability instance. Enterprise: https://api.datakitchen.io
Docker/minikube: http://<minikube_ip>:8082/api
Docker/localhost: http://host.docker.internal:8082/api
K8s/minikube: http://observability-ui.datakitchen.svc.cluster.local:8082/api
DK_API_KEY Yes An API key for the Observability project. A key unique to this agent is recommended. Provides authorization for events and agent heartbeat.

Agent-specific variables

Variable Required? Description Value
AGENT_TYPE Yes Identifier for the agent. Retain the default value. synapse_analytics
TENANT_ID Yes The Directory (tenant) ID of the service principal made for monitoring your Azure Synapse workspace. See Create an Azure Service Principal for more details.
CLIENT_ID Yes The Application (client) ID of the service principal made for monitoring your Azure Synapse workspace.
CLIENT_SECRET Yes The client secret Value that was created during the setup of the service principal.
WORKSPACE_NAME Yes The name of your Azure Synapse workspace.
SUBSCRIPTION_ID Optional The ID of the subscription where the resource group and Azure Synapse workspace exist. This value is set when you create your Synapse workspace and can be found in the workspace Overview page, under Essentials. Use this variable to generate click-back URLs from Observability events to your Azure Synapse workspace.
RESOURCE_GROUP_NAME Optional The name of the resource group where your Azure Synapse workspace exists. This value is set when you create your Synapse workspace and can be found in the workspace Overview page, under Essentials. Use this variable to generate click-back URLs from Observability events to your Azure Synapse workspace.
PIPELINES_FILTER Optional A list of specific pipelines you want to monitor. If this variable is not defined, the agent will observe all Synapse pipelines in the workspace. Format: a comma-separated list of Azure Synapse pipeline names. Names must be surrounded by double quotation marks, inside square brackets within single quotation marks (i.e. a JSON array in a string). '["pipeline 1", "pipeline 2"]'
PERIOD Optional Cadence at which the agent polls for information, in seconds. Default: 5.0