Skip to content

Amazon S3 Community

DataKitchen provides an agent that lets you monitor Amazon S3 through Amazon SQS.

Deploy

Prerequisites

  • Create an Amazon SQS queue, if you do not already have one. See Amazon's Create a queue .
  • Configure your Amazon S3 instance to send events to the SQS queue. See Amazon's Amazon S3 Event Notifications .
  • Docker with Docker Compose. If not installed, see Get Docker .
  • An API key for this agent — see step 5 below to create one.

Steps

  1. Log into Observability and select a project.
  2. Select Integrations from the menu, then click View Available Agents.
  3. Select the tool from the list.
  4. Under Step 1: Prerequisites, verify any requirements for the tool have been completed.
  5. In the New API Key section, create an API key for this agent.
    1. Enter a name, expiration, and description.
    2. Configure the key to send events, manage entities, and transmit heartbeat.
    3. Click Create Key and Continue.

  6. Under Step 2: Configuration, fill in any remaining variables as needed.

    • Required values are noted by an asterisk.
    • Some values are pre-populated with project-specific configuration details.

  7. Click Continue.

  8. Under Step 3: Deploy, enter the agent's Image Tag (format: vx.x.x).
  9. Select Docker as the Deployment Location.
  10. Click Download Script.
  11. Save the file anywhere it can be accessed by Docker.

  12. Open a terminal and run the deployment script:

    chmod +x deploy-agent-sqs.sh
./deploy-agent-sqs.sh

Prerequisites

  • Create an Amazon SQS queue, if you do not already have one. See Amazon's Create a queue .
  • Configure your Amazon S3 instance to send events to the SQS queue. See Amazon's Amazon S3 Event Notifications .
  • A Kubernetes cluster with kubectl access.
  • An API key for this agent — see step 5 below to create one.

Steps

  1. Log into Observability and select a project.
  2. Select Integrations from the menu, then click View Available Agents.
  3. Select the tool from the list.
  4. Under Step 1: Prerequisites, verify any requirements for the tool have been completed.
  5. In the New API Key section, create an API key for this agent.
    1. Enter a name, expiration, and description.
    2. Configure the key to send events, manage entities, and transmit heartbeat.
    3. Click Create Key and Continue.

  6. Under Step 2: Configuration, fill in any remaining variables as needed.

    • Required values are noted by an asterisk.
    • Some values are pre-populated with project-specific configuration details.

  7. Click Continue.

  8. Under Step 3: Deploy, enter the agent's Image Tag (format: vx.x.x).
  9. Select Kubernetes as the Deployment Location.
  10. Click Download Script.
  11. Copy the script to a machine with access to the Kubernetes cluster.

  12. Open a terminal and run the deployment script:

    chmod +x deploy-agent-sqs.sh
./deploy-agent-sqs.sh

Configuration variables

Variable Required? Description Value
EVENTS_API_HOST Yes The base API URL for the Observability instance. Enterprise: https://api.datakitchen.io
Docker/minikube: http://<minikube_ip>:8082/api
Docker/localhost: http://host.docker.internal:8082/api
K8s/minikube: http://observability-ui.datakitchen.svc.cluster.local:8082/api
EVENTS_API_KEY Yes An API key for the Observability project. A key unique to this agent is recommended. Provides authorization for events and agent heartbeat.
EVENTS_PROJECT_ID Yes The current project ID. Found in your Observability URL. https://example-account.datakitchen.io/projects/{the-project-id}/
EXTERNAL_PLUGINS_PATH Yes The location of the plugin. Retain the default value. /plugins
PUBLISH_EVENTS Yes Indicates to the system events are being published by the external tool. Retain the default value. True
LOGGING_MODE Optional Use to set the logging level. Accepts values DEBUG or INFO. INFO

Agent-specific variables

Variable Required? Description Value
ENABLED_PLUGINS Yes The plugin that represents and specifies the agent. Retain the default value. Accepts a comma-separated list of plugins. aws_s3_event_handler
ACCESS_KEY Yes The access key for your AWS account.
AWS_SECRET_KEY Yes The secret key of your AWS account.
SESSION_TOKEN Optional The session key for your AWS account. Only required if using temporary credentials.
AWS_DEFAULT_REGION Yes The default region of your AWS account.
SQS_QUEUE_NAME Yes The name of the Amazon SQS queue.