Skip to content

Google Cloud Composer Community

DataKitchen provides an agent that lets you monitor Google Cloud Composer , versions 1 and 2.

Deploy

Prerequisites

  • A Google Cloud service account JSON file. If you do not have one, see Create a service account key . Once created, Base64 encode the contents of the file and use the resulting string in the agent configuration.
  • Docker with Docker Compose. If not installed, see Get Docker .
  • An API key for this agent — see step 5 below to create one.

Steps

  1. Log into Observability and select a project.
  2. Select Integrations from the menu, then click View Available Agents.
  3. Select the tool from the list.
  4. Under Step 1: Prerequisites, verify any requirements for the tool have been completed.
  5. In the New API Key section, create an API key for this agent.
    1. Enter a name, expiration, and description.
    2. Configure the key to send events, manage entities, and transmit heartbeat.
    3. Click Create Key and Continue.

  6. Under Step 2: Configuration, fill in any remaining variables as needed.

    • Required values are noted by an asterisk.
    • Some values are pre-populated with project-specific configuration details.

  7. Click Continue.

  8. Under Step 3: Deploy, enter the agent's Image Tag (format: vx.x.x).
  9. Select Docker as the Deployment Location.
  10. Click Download Script.
  11. Save the file anywhere it can be accessed by Docker.

  12. Open a terminal and run the deployment script:

    chmod +x deploy-agent-gcc.sh
./deploy-agent-gcc.sh

Prerequisites

  • A Google Cloud service account JSON file. If you do not have one, see Create a service account key . Once created, Base64 encode the contents of the file and use the resulting string in the agent configuration.
  • A Kubernetes cluster with kubectl access.
  • An API key for this agent — see step 5 below to create one.

Steps

  1. Log into Observability and select a project.
  2. Select Integrations from the menu, then click View Available Agents.
  3. Select the tool from the list.
  4. Under Step 1: Prerequisites, verify any requirements for the tool have been completed.
  5. In the New API Key section, create an API key for this agent.
    1. Enter a name, expiration, and description.
    2. Configure the key to send events, manage entities, and transmit heartbeat.
    3. Click Create Key and Continue.

  6. Under Step 2: Configuration, fill in any remaining variables as needed.

    • Required values are noted by an asterisk.
    • Some values are pre-populated with project-specific configuration details.

  7. Click Continue.

  8. Under Step 3: Deploy, enter the agent's Image Tag (format: vx.x.x).
  9. Select Kubernetes as the Deployment Location.
  10. Click Download Script.
  11. Copy the script to a machine with access to the Kubernetes cluster.

  12. Open a terminal and run the deployment script:

    chmod +x deploy-agent-gcc.sh
./deploy-agent-gcc.sh

Configuration variables

Variable Required? Description Value
EVENTS_API_HOST Yes The base API URL for the Observability instance. Enterprise: https://api.datakitchen.io
Docker/minikube: http://<minikube_ip>:8082/api
Docker/localhost: http://host.docker.internal:8082/api
K8s/minikube: http://observability-ui.datakitchen.svc.cluster.local:8082/api
EVENTS_API_KEY Yes An API key for the Observability project. A key unique to this agent is recommended. Provides authorization for events and agent heartbeat.
EVENTS_PROJECT_ID Yes The current project ID. Found in your Observability URL. https://example-account.datakitchen.io/projects/{the-project-id}/
EXTERNAL_PLUGINS_PATH Yes The location of the plugin. Retain the default value. /plugins
PUBLISH_EVENTS Yes Indicates to the system events are being published by the external tool. Retain the default value. True
POLLING_INTERVAL_SECS Yes The frequency at which the agent captures events. Retain the default value. 10
LOGGING_MODE Optional Use to set the logging level. Accepts values DEBUG or INFO. INFO
MAX_WORKERS Optional Sets the number of workers available. Accepts integer values. Default value: 10. Set to 1 for Databricks agents to avoid rate limit errors. 10

Agent-specific variables

Variable Required? Description Value
ENABLED_PLUGINS Yes The plugin that represents and specifies the agent. Retain the default value. Accepts a comma-separated list of plugins if you have more than one. airflow_composer
BASE64_ENCODED_SERVICE_ACCOUNT_STR Yes The resulting string of your Base64 encoded service account JSON file.
COMPOSER_CLIENT_ID Yes, for version 1. The client ID of the Identity and Access Management proxy that protects the Airflow web server. See Google's documentation on how to retrieve your client ID for more information.
COMPOSER_WEBSERVER_ID Yes, for version 1. The webserver ID. Can be found in the instance as: https://{webserver-id}.appspot.com
COMPOSER_2_AUTH_SCOPE Yes, for version 2. Retain the default value. https://www.googleapis.com/auth/cloud-platform
COMPOSER_2_FLAG Yes, for version 2. Retain the default value. True
COMPOSER_2_WEB_URL Yes, for version 2. The web URL of your Cloud Composer instance.