dbt Core Community

DataKitchen provides an agent to monitor dbt Core . The dbt Core agent parses two artifacts from a dbt run or test (run_results.json and manifest.json) and uses these artifacts to create events in Observability.

Deploy

LocalGitHub Actions

Prerequisites

• Docker with Docker Compose. If not installed, see Get Docker .
• An API key for this agent is recommended.
  • Create and manage API keys from the API Keys page.
  • Configure the key to send events and manage entities.

Steps

After you run dbt Core and generate results, deploy the dbt Core agent so those results can be sent to Observability.

1. Open a terminal window in your workstation.

2. Use the commands below to run the agent, substituting your configuration variable values into the code:

   docker pull datakitchen/dbt-core-connector:<version>
   
   docker run -v /dbt-jobs/target:/target \
   -e EVENTS_API_KEY="${EVENTS_API_KEY}" \
   -e PIPELINE_NAME=jaffle_shop_test \
   -e PIPELINE_KEY=jaffle_shop_test \
   -e RUN_RESULTS_PATH=/target/run_results.json \
   -e MANIFEST_PATH=/target/manifest.json \
   datakitchen/dbt-core-connector:<version>
   

3. The commands instruct Docker to fetch the image, read the configuration variables, and run the agent.

4. Along with the required configuration variables, you will need to map a volume to the folder where dbt Core has deposited the results.
5. The dbt Core agent is not persistent but is invoked by the same mechanism used for your dbt pipelines. For each dbt Core execution that you want to monitor, rerun the commands starting from docker run to invoke and use the agent.
6. You can find the <version> number in DataKitchen's Docker Hub: https://hub.docker.com/r/datakitchen/dbt-core-connector/tags .
7. If you call dbt Core from a shell script in a cron job, add the commands to the shell script to run automatically.
8. By default, dbt Core puts output files in the same location with each run. If you expect to run multiple commands from the same folder (for example, build, run, test), it's recommended that you use the target-path config to send the results to different folders.

Prerequisites

• A GitHub Actions instance is required. To create one, see GitHub's quickstart .
• An API key for this agent is recommended.
  • Create and manage API keys from the API Keys page.
  • Configure the key to send events and manage entities.

Steps

1. Upload the run_results.json and manifest.json files into your existing GitHub Actions instance. Note, the path to the results will vary depending on your configuration.

   Example

   steps:
     - uses: actions/upload-artifact@v3
       with:
         name: run-results
         path:
           ${{github.workspace}}/target/run_results.json
     - uses: actions/upload-artifact@v3
       with:
         name: manifest
         path:
           ${{github.workspace}}/target/manifest.json
   

2. Define a new job, substituting your configuration variable values into the code.

   Example

   jobs:
     datakitchen:
       needs: [run-dbt]
       runs-on: ubuntu-latest
       steps:
         # This downloads the artifacts into ${{github.workspace}}, where we should have permission
         - uses: actions/download-artifact@v3
           with:
             name: run-results
         - uses: actions/download-artifact@v3
           with:
             name: manifest
         - name: Run my container
           run: |
             docker run \
              -v ${{github.workspace}}/:/results \
             -e EVENTS_API_KEY="${{secrets.EVENTS_API_KEY}}" \
             -e PIPELINE_NAME=jaffle_shop_test_github \
             -e PIPELINE_KEY=jaffle_shop_test_github \
             -e RUN_RESULTS_PATH="/results/run_results.json" \
             -e MANIFEST_PATH="/results/manifest.json" \
             datakitchen/dbt-core-connector:<version>
   

3. The new DataKitchen job depends on the run-dbt job where the artifacts were created. You can find the job in your action file and use this as your upstream dependency.

4. To make the dbt results accessible, the local workspace folder (github_workspace) is mapped into a /results folder inside the container.
5. The default download path for the artifacts is ${{github_workspace}}.
6. You can find the <version> number in DataKitchen's Docker Hub: https://hub.docker.com/r/datakitchen/dbt-core-connector/tags .

Configuration variables

Variable	Required?	Description
EVENTS_API_KEY	Yes	An API key for a project. Provides authorization for events sent by the tool. Allows the agent to manage Observability entities.
EVENTS_API_HOST	Yes	The base URL to the Observability API. Examples: https://api.datakitchen.io, http://127.0.0.1:8082/api
PIPELINE_NAME	Yes	A unique name for this dbt job. This value remains the same between runs. This value will be used as the component name in the Observability UI. It's recommended to make this human-readable.
PIPELINE_KEY	Yes	A unique key for this dbt job. This value remains the same between runs. This value will be used as the component key. It's not necessary to make this value human readable.
RUN_RESULTS_PATH	Yes	The path to the run_results.json file generated by dbt.
MANIFEST_PATH	Yes	The path to the manifest.json file generated by dbt.