Repository navigation
Use task instance end date as OpenLineage terminal event time - #75328
Merged
Merged
Conversation
kacpermuda
force-pushed
the
ol-end-date-as-event-time
branch
from
October 26, 2026 17:35
5a7763f to
6a63973
Compare
1 task done
kacpermuda
force-pushed
the
ol-end-date-as-event-time
branch
3 times, most recently
from
October 27, 2026 11:17
6eb73a9 to
4defb65
Compare
kacpermuda
marked this pull request as ready for review
October 27, 2026 11:34
kacpermuda
marked this pull request as draft
October 27, 2026 12:18
kacpermuda
marked this pull request as ready for review
October 27, 2026 12:18
mobuchowski
reviewed
Oct 27, 2026
kacpermuda
force-pushed
the
ol-end-date-as-event-time
branch
3 times, most recently
from
October 27, 2026 15:22
5771e62 to
f481092
Compare
The Airflow 3 task runner sets the end date and reports it to the API server before it notifies listeners, but the listener stamped complete, fail and skipped events with the time of its own notification. That is later than the stored end date by however long the Airflow callbacks took. The task instance facet also never carried a duration on Airflow 3: the runtime task instance has none, as the API server computes it after the listener ran. It is now derived from the same end date as the event time. Co-Authored-By: Claude <noreply@anthropic.com>
Airflow 2 stores the duration before the listener is notified, while the end date passed to the facet there is the notification time, taken after the success callback ran. Computing the duration from it made the facet disagree with the duration shown in the UI. Co-Authored-By: Claude <noreply@anthropic.com>
Since apache#75685 the task runner sets the end date on the task instance in the skip and existing Dag run cases too, so the fallback to the notification time there only applies to older Airflow versions. Co-Authored-By: Claude <noreply@anthropic.com>
Consumers had only the event time and a duration to work out when a task instance ran, and the event time of a terminal event isn't always its end date. The dates the task instance holds are the ones the UI shows. Co-Authored-By: Claude <noreply@anthropic.com>
The facet now carries the start and end dates; locking them in on the success, skip, failure and retry paths keeps a regression there from going unnoticed. Co-Authored-By: Claude <noreply@anthropic.com>
A numeric string would pass the previous check, while the duration must be emitted as a JSON number, as in the Airflow 2 expectations. Co-Authored-By: Claude <noreply@anthropic.com>
Address review: on Airflow 2 the event time and the facet's end date came from different values, a retry reported the previous try's duration, and before Airflow 3.2.2 a task resumed from deferral reported its resume time as start date and a duration that left out the time before it deferred. Co-Authored-By: Claude <noreply@anthropic.com>
…em tests A task resumed from deferral reports no start date and duration before Airflow 3.2.2, and Airflow 2 events now carry the dates too, so the expectations need to say which versions get what. That boundary is a patch release, which the expected-event files couldn't express. Co-Authored-By: Claude <noreply@anthropic.com>
…Lineage test Its stored duration is the previous try's, which the facet no longer reports. Co-Authored-By: Claude <noreply@anthropic.com>
kacpermuda
force-pushed
the
ol-end-date-as-event-time
branch
from
October 27, 2026 16:17
f481092 to
202431a
Compare
mobuchowski
approved these changes
Oct 28, 2026
7 of 36 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
OpenLineage complete/fail/skipped events from the worker are now timed with the task instance's
end_dateinstead of the listener's ownnow(), which is taken only after the task's callbacks have run. Airflow 2 and the Airflow 3 task runner both setend_datebefore they notify listeners, and that is the value Airflow stores, so the event time now matches what the UI shows.The listener falls back to
now()when the task instance carries noend_date:AirflowSkipException, or aTriggerDagRunOperatortask skipped or failed because the Dag run it triggers already exists (the task runner setsend_datethere since Let listeners see the end date of a skipped task instance #75685).The task instance facet now also carries
start_dateandend_date. Itsend_dateis the task instance's own when it has one, and otherwise the end date the terminal event is timed with, so the event and the facet always agree. Thedurationis now filled in for the worker's events too: the runtime task instance has none, as the API server computes it after the listener ran.The stored
durationwins over the computed one, so for scheduler-detected failures, manual state changes and Airflow 2 the facet keeps matching the UI. It's used only together with a storedend_date, because Airflow 2 clearsend_datebut keeps the previous try'sdurationwhen the next try starts. Scheduler and API server events are otherwise unchanged.Before Airflow 3.2.2 (#63247), the worker's
start_dateof a task resumed from deferral is the resume time, while the stored one is the original start. The facet leaves outstart_datethere, and with it the duration computed from it, rather than reporting values that disagree with the UI.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Sonnet 5.5, Opus 5.5) following the guidelines
🤖 Generated with Claude Code