Core Concepts in Observability

At DataKitchen, we believe observability is an important step in achieving DataOps success. DataOps Observability provides end-to-end visibility across your data operations, so you can see how everything is going in real time.

Observability collects metadata, metrics, and statuses from your data estate by ingesting events from your tools and data assets. The system processes the events, displays the details and results in the UI, and triggers actions based on the schedules, rules, expectations, and conditions that you set—ensuring you can monitor important situations as they occur.

Monitor your data operations in one place, with all the details you need, and none that you don't. In time, make research-based changes to improve efficiency and effectiveness, then close the loop by observing how these changes have impacted your data operations.

System details

Observability is available as open-source (Apache 2.0 license) and enterprise software. The open-source version is fully functional for a single data engineer, while the enterprise version has additional features for teams and enterprises.

The software consists of a browser-based user interface (UI), REST APIs, and a logic and rules engine. All user tasks—including rule creation, setting schedules, or making conditions—can be performed in the UI and are covered in the Observability Help documentation.

Core concepts

Data estate

A data estate is a collection of all the data within an organization. This includes not only the data values, but also the infrastructure, tools, and assets.

Data journey

A data journey represents a collection of components in your data estate, that together, are responsible for creating a data analytic deliverable.

Tip

A data estate is your data operation "as-is," with separate tools, distinct assets, and different teams. A data journey is the representation of a data estate, viewed holistically from the perspective of doing work, with components and their dependencies. For more information, see Data Estate and Data Journeys.

Project

A project is the space that separates and distinguishes work. In a company account, projects might be set up by department or by team. The Projects page lists all the projects in an organization's Observability instance. The Project Overview page serves as a dashboard for users to view recent activity in the project.

The open-source version is limited to a single project.

Events

An event is anything of interest that has happened in a data estate. For example, an event can be a job starting, a job finishing with a certain status, an error occurring, a file arriving, tests starting, or tests finishing with specific results. Events that are received by the Event Ingestion API:

• Are always associated with a project and component.
• Can be associated with a journey and an instance of that journey.

Components

Each component represents a tool or asset in a customer's data estate. Observability components include batch pipelines, datasets, streaming pipelines, and servers.

Rule

Use rules to define and capture events, so you can monitor and react to situations as needed. Observability provides patterns to help with the construction of meaningful rules based on a standard trigger-condition-action format.

Alert

Observability can generate alerts at the run-level and instance-level to highlight situations that don't align with your expectations.

Get started with Observability

See Get Started for information on how to install and set up Observability.