Skip to content

Add Azure Diagnostic Settings

DataKitchen provides agents that allow you to monitor Azure services—Data Factory, Blob Storage, and Functions—through Azure Event Hubs.

The following summarizes steps provided by Microsoft's documentation, with an example for configuring Azure Data Factory. For more technical details or notes specific to other Azure services, see Microsoft's documentation Create diagnostic settings in Azure Monitor .

Warning

This step requires technical knowledge about your tools and higher-level permissions. A DevOps Engineer or IT Manager from your team may be needed. And don't forget, we're always here to help. Contact DataKitchen if you have more questions.

Azure monitoring overview

Azure Event Hubs is a data streaming service that uses Azure Monitor to create its monitoring data. The metrics and logs from your Azure services can be sent to Azure Event Hubs through designated diagnostic settings that you configure in your Azure portal.

The Observability agents can then listen to events that are broadcast from an Event Hubs instance and display the events in the Observability UI.

Configure diagnostic settings: Azure Data Factory example

  1. Sign in to the Azure portal.
  2. Search for and select Data factories.
  3. Select the Data Factory instance you want to monitor.
  4. Select Diagnostic settings from the menu.
  5. Click Add diagnostic setting.
  6. On the Diagnostic setting page, configure the diagnostic settings to publish events to your event hub.
    1. Under Logs, check the Pipeline runs log and Pipeline activity runs log.
    2. Select Stream to an event hub as the Destination details.
    3. Enter your event hub details as needed.
    4. Click Save.

After a few moments, the new setting appears in your list of settings for this service. Logs are streamed to the event hub as new data is generated. It can take up to 15 minutes between when an event is emitted and when it appears in a Log Analytics workspace.

Next steps

Once you add the diagnostic settings, you can then configure any of the Observability agents mentioned above.