Skip to content

Azure Data Factory

DataKitchen provides an agent that lets you monitor Azure Data Factory through Azure Event Hubs. This agent listens to a single Azure Event Hubs instance. To monitor multiple hubs, configure and deploy an agent per hub.

Deploy

Prerequisites

  • An active Azure Event Hubs instance. If you need to create one, see Microsoft's Quickstart .
  • Diagnostic settings in your Azure service must be set up to stream logs to the Azure Event Hubs instance. For a walkthrough, see Add Azure Diagnostic Settings.
  • Optional: If you're using a Microsoft Azure Checkpoint Store, an active Azure Blob container is required to store Event Hubs Checkpoint information. A checkpoint store determines the position the agent should resume receiving events if a restart occurs and can be used to load balance multiple instances of your application. See Microsoft's Checkpoint Store and checkpointStore documentation for more details.
  • Docker with Docker Compose. If not installed, see Get Docker .
  • An API key for this agent — see step 5 below to create one.
  • Enterprise Access to the agent's private Docker repository. Login with the provided credentials using docker login or through the Docker Compose application.

Steps

  1. Log into Observability and select a project.
  2. Select Integrations from the menu, then click View Available Agents.
  3. Select the tool from the list.
  4. Under Step 1: Prerequisites, verify any requirements for the tool have been completed.
  5. In the New API Key section, create an API key for this agent.
    1. Enter a name, expiration, and description.
    2. Configure the key to send events, manage entities, and transmit heartbeat.
    3. Click Create Key and Continue.

  6. Under Step 2: Configuration, fill in any remaining variables as needed.

    • Required values are noted by an asterisk.
    • Some values are pre-populated with project-specific configuration details.

  7. Click Continue.

  8. Under Step 3: Deploy, enter the agent's Image Tag (format: vx.x.x).
  9. Select Docker as the Deployment Location.
  10. Click Download Script.
  11. Save the file anywhere it can be accessed by Docker.

  12. Open a terminal and run the deployment script:

    chmod +x deploy-agent-data_factory.sh
./deploy-agent-data_factory.sh

Prerequisites

  • An active Azure Event Hubs instance. If you need to create one, see Microsoft's Quickstart .
  • Diagnostic settings in your Azure service must be set up to stream logs to the Azure Event Hubs instance. For a walkthrough, see Add Azure Diagnostic Settings.
  • Optional: If you're using a Microsoft Azure Checkpoint Store, an active Azure Blob container is required to store Event Hubs Checkpoint information. A checkpoint store determines the position the agent should resume receiving events if a restart occurs and can be used to load balance multiple instances of your application. See Microsoft's Checkpoint Store and checkpointStore documentation for more details.
  • A Kubernetes cluster with kubectl access.
  • An API key for this agent — see step 5 below to create one.
  • Enterprise Access to the agent's private Docker repository. Login with the provided credentials using docker login.

Steps

  1. Log into Observability and select a project.
  2. Select Integrations from the menu, then click View Available Agents.
  3. Select the tool from the list.
  4. Under Step 1: Prerequisites, verify any requirements for the tool have been completed.
  5. In the New API Key section, create an API key for this agent.
    1. Enter a name, expiration, and description.
    2. Configure the key to send events, manage entities, and transmit heartbeat.
    3. Click Create Key and Continue.

  6. Under Step 2: Configuration, fill in any remaining variables as needed.

    • Required values are noted by an asterisk.
    • Some values are pre-populated with project-specific configuration details.

  7. Click Continue.

  8. Under Step 3: Deploy, enter the agent's Image Tag (format: vx.x.x).
  9. Select Kubernetes as the Deployment Location.
  10. Click Download Script.
  11. Copy the script to a machine with access to the Kubernetes cluster.

  12. Open a terminal and run the deployment script:

    chmod +x deploy-agent-data_factory.sh
./deploy-agent-data_factory.sh

Follows the steps in Microsoft's Deploy a container instance in Azure , with notes specific to Observability agents.

Prerequisites

  • An active Azure Event Hubs instance. If you need to create one, see Microsoft's Quickstart .
  • Diagnostic settings in your Azure service must be set up to stream logs to the Azure Event Hubs instance. For a walkthrough, see Add Azure Diagnostic Settings.
  • Optional: If you're using a Microsoft Azure Checkpoint Store, an active Azure Blob container is required to store Event Hubs Checkpoint information. A checkpoint store determines the position the agent should resume receiving events if a restart occurs and can be used to load balance multiple instances of your application. See Microsoft's Checkpoint Store and checkpointStore documentation for more details.
  • An API key for this agent. Create and manage keys from the API Keys page. Configure the key to send events, manage entities, and transmit heartbeat.
  • Enterprise Access to the agent's private Docker repository.

Steps

  1. Sign into your Azure portal and create a container instance.
    1. On the Azure portal homepage, select Create a resource.
    2. Select Containers.
    3. Under Container Instances, click Create.

  2. On the Create container instance page, under the Basics tab, fill in the following:

    • Resource group: Use an existing resource group or create a new one.
    • Container name: Enter a name for the container (e.g., observabilityagent).
    • Image source: Select Other registry.
    • Image: Enter the full name of the agent's Docker image (e.g., datakitchen/dataops-observability-agents:v2.0.0).
    • Image type: Select Private and enter the provided Docker credentials. Enterprise
    • Update the OS type and Size if needed. Leave all other values as default.

  3. Click Next: Networking.

  4. Under the Networking tab, select Public, Private, or None for the networking type, depending on your organization's needs.
  5. Click Next: Advanced.

  6. Under the Advanced tab, fill in the following:

    • Restart Policy: Set to On failure.
    • Environment variables: Enter all agent configuration variables. Set Mark as secure to Yes for any API or token values.
    • Leave all other settings as default.

  7. Click Review + create, then click Create.

After deploying, navigate to Resource groups > {your resource group} > {your container name}. A Running status indicates the agent is sending events to Observability.

Tip

To test the status of your container, copy the FQDN from the container overview and open it in your browser. Then check Containers > Logs for the HTTP GET request, confirming the system is active.

Configuration variables

Variable Required? Description Value
AGENT_KEY Yes Assigns the identifier of the agent. Appears in the UI on the Integrations page. production-project-agent
DK_API_URL Yes The base API URL for the Observability instance. Enterprise: https://api.datakitchen.io
Docker/minikube: http://<minikube_ip>:8082/api
Docker/localhost: http://host.docker.internal:8082/api
K8s/minikube: http://observability-ui.datakitchen.svc.cluster.local:8082/api
DK_API_KEY Yes An API key for the Observability project. A key unique to this agent is recommended. Provides authorization for events and agent heartbeat.

Agent-specific variables

Variable Required? Description Value
AGENT_TYPE Yes Identifier for the agent. Retain the default value. eventhubs
MESSAGE_TYPES Yes Identifier for the message types handled by the agent. Retain the default value. ["ADF"]
NAME Yes The name of your Azure Event Hubs instance. Note, this is not the name of the Event Hubs namespace .
CONNECTION_STRING Yes The connection string for your Azure Event Hubs instance. See Microsoft's Get an Event Hubs connection string for steps on how to find the string in Azure.
STARTING_POSITION Optional The event position from which the agent starts receiving at startup on a new install. From the beginning of the stream: -1. Only new events: @latest. Specific position: an integer index or a datetime. Default: -1
BLOB_NAME Optional The name of your Blob storage container to use as a Checkpoint store. If this variable is not defined, the agent will not use checkpointing.