Skip to content

Azure Blob Storage Community

DataKitchen provides an agent that lets you monitor Azure Blob Storage through Azure Event Hubs. This agent listens to a single Azure Event Hubs instance. To monitor multiple hubs, configure and deploy an agent per hub.

Deploy

Prerequisites

  • An active Azure Event Hubs instance. If you need to create one, see Microsoft's Quickstart .
  • Diagnostic settings in your Azure service must be set up to stream logs to the Azure Event Hubs instance. For a walkthrough, see Add Azure Diagnostic Settings.
  • Optional: If using a Microsoft Azure Checkpoint Store, an active Azure Blob container is required to store Event Hubs Checkpoint information. See Microsoft's Checkpoint Store and checkpointStore documentation for more details.
  • Docker with Docker Compose. If not installed, see Get Docker .
  • An API key for this agent — see step 5 below to create one.

Steps

  1. Log into Observability and select a project.
  2. Select Integrations from the menu, then click View Available Agents.
  3. Select the tool from the list.
  4. Under Step 1: Prerequisites, verify any requirements for the tool have been completed.
  5. In the New API Key section, create an API key for this agent.
    1. Enter a name, expiration, and description.
    2. Configure the key to send events, manage entities, and transmit heartbeat.
    3. Click Create Key and Continue.

  6. Under Step 2: Configuration, fill in any remaining variables as needed.

    • Required values are noted by an asterisk.
    • Some values are pre-populated with project-specific configuration details.

  7. Click Continue.

  8. Under Step 3: Deploy, enter the agent's Image Tag (format: vx.x.x).
  9. Select Docker as the Deployment Location.
  10. Click Download Script.
  11. Save the file anywhere it can be accessed by Docker.

  12. Open a terminal and run the deployment script:

    chmod +x deploy-agent-blob_storage.sh
./deploy-agent-blob_storage.sh

Prerequisites

  • An active Azure Event Hubs instance. If you need to create one, see Microsoft's Quickstart .
  • Diagnostic settings in your Azure service must be set up to stream logs to the Azure Event Hubs instance. For a walkthrough, see Add Azure Diagnostic Settings.
  • Optional: If using a Microsoft Azure Checkpoint Store, an active Azure Blob container is required to store Event Hubs Checkpoint information. See Microsoft's Checkpoint Store and checkpointStore documentation for more details.
  • A Kubernetes cluster with kubectl access.
  • An API key for this agent — see step 5 below to create one.

Steps

  1. Log into Observability and select a project.
  2. Select Integrations from the menu, then click View Available Agents.
  3. Select the tool from the list.
  4. Under Step 1: Prerequisites, verify any requirements for the tool have been completed.
  5. In the New API Key section, create an API key for this agent.
    1. Enter a name, expiration, and description.
    2. Configure the key to send events, manage entities, and transmit heartbeat.
    3. Click Create Key and Continue.

  6. Under Step 2: Configuration, fill in any remaining variables as needed.

    • Required values are noted by an asterisk.
    • Some values are pre-populated with project-specific configuration details.

  7. Click Continue.

  8. Under Step 3: Deploy, enter the agent's Image Tag (format: vx.x.x).
  9. Select Kubernetes as the Deployment Location.
  10. Click Download Script.
  11. Copy the script to a machine with access to the Kubernetes cluster.

  12. Open a terminal and run the deployment script:

    chmod +x deploy-agent-blob_storage.sh
./deploy-agent-blob_storage.sh

Follows the steps in Microsoft's Deploy a container instance in Azure , with notes specific to Observability agents.

Prerequisites

  • An active Azure Event Hubs instance. If you need to create one, see Microsoft's Quickstart .
  • Diagnostic settings in your Azure service must be set up to stream logs to the Azure Event Hubs instance. For a walkthrough, see Add Azure Diagnostic Settings.
  • Optional: If using a Microsoft Azure Checkpoint Store, an active Azure Blob container is required to store Event Hubs Checkpoint information. See Microsoft's Checkpoint Store and checkpointStore documentation for more details.
  • An API key for this agent. Create and manage keys from the API Keys page. Configure the key to send events, manage entities, and transmit heartbeat.

Steps

  1. Sign into your Azure portal and create a container instance.
    1. On the Azure portal homepage, select Create a resource.
    2. Select Containers.
    3. Under Container Instances, click Create.

  2. On the Create container instance page, under the Basics tab, fill in the following:

    • Resource group: Use an existing resource group or create a new one.
    • Container name: Enter a name for the container (e.g., observabilityagent).
    • Image source: Select Other registry.
    • Image: Enter the full name of the agent's Docker image (e.g., datakitchen/dataops-observability-agents:v2.0.0).
    • Image type: Select Private and enter the provided Docker credentials. Enterprise
    • Update the OS type and Size if needed. Leave all other values as default.

  3. Click Next: Networking.

  4. Under the Networking tab, select Public, Private, or None for the networking type, depending on your organization's needs.
  5. Click Next: Advanced.

  6. Under the Advanced tab, fill in the following:

    • Restart Policy: Set to On failure.
    • Environment variables: Enter all agent configuration variables. Set Mark as secure to Yes for any API or token values.
    • Leave all other settings as default.

  7. Click Review + create, then click Create.

After deploying, navigate to Resource groups > {your resource group} > {your container name}. A Running status indicates the agent is sending events to Observability.

Tip

To test the status of your container, copy the FQDN from the container overview and open it in your browser. Then check Containers > Logs for the HTTP GET request, confirming the system is active.

Configuration variables

Variable Required? Description Value
EVENTS_API_HOST Yes The base API URL for the Observability instance. Enterprise: https://api.datakitchen.io
Docker/minikube: http://<minikube_ip>:8082/api
Docker/localhost: http://host.docker.internal:8082/api
K8s/minikube: http://observability-ui.datakitchen.svc.cluster.local:8082/api
EVENTS_API_KEY Yes An API key for the Observability project. A key unique to this agent is recommended. Provides authorization for events and agent heartbeat.
EVENTS_PROJECT_ID Yes The current project ID. Found in your Observability URL. https://example-account.datakitchen.io/projects/{the-project-id}/
EXTERNAL_PLUGINS_PATH Yes The location of the plugin. Retain the default value. /plugins
PUBLISH_EVENTS Yes Indicates to the system events are being published by the external tool. Retain the default value. True
LOGGING_MODE Optional Use to set the logging level. Accepts values DEBUG or INFO. INFO

Agent-specific variables

Variable Required? Description Value
ENABLED_PLUGINS Yes The plugin that represents and specifies the agent. Retain the default value. Accepts a comma-separated list of plugins. blob_event_handler
EVENT_HUB_CONN_STR Yes The connection string for your Azure Event Hubs instance. See Microsoft's Get an Event Hubs connection string .
EVENT_HUB_NAME Yes The name of your Azure Event Hubs instance. This is not the name of the Event Hubs namespace .
AZURE_STORAGE_CONN_STR Optional The connection string for your Azure storage. Only required if using a Checkpoint Store. If using a Checkpoint Store, also include the BLOB_CONTAINER_NAME value.
BLOB_CONTAINER_NAME Optional The name of your Blob storage container. Only required when using a Checkpoint Store.