Alpha Release Breaking Changes¶
The expected release date for these changes is the end of January 2023.
To iterate on features, improve scalability, and continue innovating, the Alpha release of Observability requires some changes that may impact your setup.
Warning
Changes to your setup will be required: The following updates are considered breaking changes and will require you to update scripts or codebase.
The run_key will soon be a required value for all event types—RunStatusEvent, MetricLogEvent, MessageLogEvent, and TestOutcomesEvent.
New events that do not include the run_key will not be accepted by the Event Ingestion API. Existing events that are already logged in Observability will be
unaffected. [PD-882]
Previous breaking releases¶
The changes below were implemented as part of the release on December 15th, 2022.
The expected release date for these changes is 13 December 2022.
To iterate on features, improve scalability, and continue innovating, the Alpha release of Observability requires some changes that are not backward compatible.
Warning
Changes to your setup will be required: The following updates are considered breaking changes and will require you to update scripts or codebase prior to the release date.
Event Ingestion API updates¶
- Certain internal-use API properties and endpoints have been updated, deprecated, and/or removed. The user interface and all documentation have been updated to align with these updates and no changes are needed to your current setup. However, for pipeline runs prior to 13 December 2022, there may no longer be events listed in the run details Events tab. The runs will appear as historical references to the pipeline executions. [PD-726, PD-764]
TaskStatusEventandCloseRunEventhave been removed (4 November 2022) and a singleRunStatusEventis now used. The new event optionally accepts atask_keyproperty to differentiate the status for a run task and the run itself. [PD-618]- The
_namefield that appeared in the propertiespipeline_nameandtask_namehas been replaced with a_keyfield (4 November 2022). While the function of these properties remains the same—that is, to uniquely identify the pipeline or task related to events—they are nowpipeline_keyandtask_key. Additionally,run_taghas been changed torun_key. [PD-613]- As pipeline and task names have been uncoupled from their IDs (i.e. key), you can now change the
pipeline_nameortask_namewhile maintaining thepipeline_keyandtask_keythat uniquely identify the pipeline or task associated with events.
- As pipeline and task names have been uncoupled from their IDs (i.e. key), you can now change the
AddRunStateEventandCustomEventhave been removed (4 November 2022). [PD-754]- The
TestResultsEventhas been deprecated and replaced with aTestOutcomesEvent. [PD-720]
Required changes to your setup¶
In your existing scripts or codebase, replace any instances of:
| Previous API endpoints and properties | New API endpoints and properties |
|---|---|
| POST /events/v1/task-status | POST /events/v1/run-status - In the request body: specify the task_key field |
| POST/events/v1/close-run | POST /events/v1/run-status - In the request body: do not specify the task_key field - The end status for the run should be specified (i.e. "COMPLETED", "FAILED", "COMPLETED_WITH_WARNINGS") |
| POST /events/v1/test-results | POST /events/v1/test-outcomes - In the request body: replace test_results property with test_outcomes |
| "pipeline_name" | "pipeline_key" |
| "task_name" | "task_key" |
| "run_tag" | "run_key" |
| POST /events/v1/add-run-state | no replacement |
| POST /events/v1/custom-event | no replacement |
Complete details can be found in the Event Ingestion API documentation here, https://api.docs.datakitchen.io/production/events.html.
Integration agent updates¶
The Observability integration agent polls or listens to events in your data environment and forwards them to the Event Ingestion API. The integration agent has been updated for compatibility with the improvements to the Event Ingestion API. [PD-723]
Required changes to your setup¶
At this time, the code repository for the integration agent is limited to DataKitchen internal use only. No changes need to be made by you, to your Observability setup. For any additional questions, do not hesitate to contact your DataKitchen representative.
Events Ingestion Client updates¶
DataKitchen provides a Python API client, Events Ingestion Client, to assist with sending events to the Event Ingestion API. With the changes to the Event Ingestion API, the API client has been updated. [PD-770]
Required changes to your setup¶
The client is automatically generated by Swagger Codegen to reflect the Event Ingestion API updates.
If you are using the Events Ingestion Client, the methods called in your scripts need to be updated. Refer to the information outlined in the project README for details.
DKUtils updates¶
The Python package, DKUtils, facilitates interactions with the Automation API. A new version of DKUtils has been released that is compatible with updates made to the Observability system. [PD-771]
Required changes to your setup¶
Customers who use the Order Monitor Node to observe Automation events in Observability will need to update the version of DKUtils in the General Purpose Container node config file.
Additional recommendations¶
- Use the GPC image
dk_general_purpose_container:ubuntu20-latest(equivalent todk_general_purpose_container:ubuntu20-0.0.5). - Update to version 2.11.3Â in any other locations where you may be using DKUtils.