Skip to content

Use task instance end date as OpenLineage terminal event time - #75328

Merged
mobuchowski merged 9 commits into
apache:mainfrom
kacpermuda:ol-end-date-as-event-time
Oct 28, 2026
Merged

mobuchowski merged 9 commits into
apache:mainfrom
kacpermuda:ol-end-date-as-event-time

Conversation

@kacpermuda

@kacpermuda kacpermuda commented Oct 21, 2026 •

Copy link
Copy Markdown
Collaborator

OpenLineage complete/fail/skipped events from the worker are now timed with the task instance's end_date instead of the listener's own now(), which is taken only after the task's callbacks have run. Airflow 2 and the Airflow 3 task runner both set end_date before they notify listeners, and that is the value Airflow stores, so the event time now matches what the UI shows.

The listener falls back to now() when the task instance carries no end_date:

  • Airflow 3.0.0 to 3.0.4, whose task runner never sets it,
  • before Airflow 3.4.0, a task that raised AirflowSkipException, or a TriggerDagRunOperator task skipped or failed because the Dag run it triggers already exists (the task runner sets end_date there since Let listeners see the end date of a skipped task instance #75685).

The task instance facet now also carries start_date and end_date. Its end_date is the task instance's own when it has one, and otherwise the end date the terminal event is timed with, so the event and the facet always agree. The duration is now filled in for the worker's events too: the runtime task instance has none, as the API server computes it after the listener ran.

The stored duration wins over the computed one, so for scheduler-detected failures, manual state changes and Airflow 2 the facet keeps matching the UI. It's used only together with a stored end_date, because Airflow 2 clears end_date but keeps the previous try's duration when the next try starts. Scheduler and API server events are otherwise unchanged.

Before Airflow 3.2.2 (#63247), the worker's start_date of a task resumed from deferral is the resume time, while the stored one is the original start. The facet leaves out start_date there, and with it the duration computed from it, rather than reporting values that disagree with the UI.


Was generative AI tooling used to co-author this PR?
  • Yes — Claude Code (Sonnet 5.5, Opus 5.5)

Generated-by: Claude Code (Sonnet 5.5, Opus 5.5) following the guidelines

🤖 Generated with Claude Code

@kacpermuda
kacpermuda force-pushed the ol-end-date-as-event-time branch from 5a7763f to 6a63973 Compare October 26, 2026 17:35
@kacpermuda
kacpermuda force-pushed the ol-end-date-as-event-time branch 3 times, most recently from 6eb73a9 to 4defb65 Compare October 27, 2026 11:17
@kacpermuda
kacpermuda marked this pull request as ready for review October 27, 2026 11:34
@kacpermuda kacpermuda added the full tests needed We need to run full set of tests for this PR to merge label Oct 27, 2026
@kacpermuda
kacpermuda marked this pull request as draft October 27, 2026 12:18
@kacpermuda
kacpermuda marked this pull request as ready for review October 27, 2026 12:18
@kacpermuda
kacpermuda force-pushed the ol-end-date-as-event-time branch 3 times, most recently from 5771e62 to f481092 Compare October 27, 2026 15:22
kacpermuda and others added 9 commits October 27, 2026 17:17
The Airflow 3 task runner sets the end date and reports it to the API
server before it notifies listeners, but the listener stamped complete,
fail and skipped events with the time of its own notification. That is
later than the stored end date by however long the Airflow callbacks took.

The task instance facet also never carried a duration on Airflow 3: the
runtime task instance has none, as the API server computes it after the
listener ran. It is now derived from the same end date as the event time.

Co-Authored-By: Claude <noreply@anthropic.com>
Airflow 2 stores the duration before the listener is notified, while the
end date passed to the facet there is the notification time, taken after
the success callback ran. Computing the duration from it made the facet
disagree with the duration shown in the UI.

Co-Authored-By: Claude <noreply@anthropic.com>
Since apache#75685 the task runner sets the end date on the task
instance in the skip and existing Dag run cases too, so the fallback to the
notification time there only applies to older Airflow versions.

Co-Authored-By: Claude <noreply@anthropic.com>
Consumers had only the event time and a duration to work out when a task
instance ran, and the event time of a terminal event isn't always its end
date. The dates the task instance holds are the ones the UI shows.

Co-Authored-By: Claude <noreply@anthropic.com>
The facet now carries the start and end dates; locking them in on the
success, skip, failure and retry paths keeps a regression there from
going unnoticed.

Co-Authored-By: Claude <noreply@anthropic.com>
A numeric string would pass the previous check, while the duration must be
emitted as a JSON number, as in the Airflow 2 expectations.

Co-Authored-By: Claude <noreply@anthropic.com>
Address review: on Airflow 2 the event time and the facet's end date came
from different values, a retry reported the previous try's duration, and
before Airflow 3.2.2 a task resumed from deferral reported its resume time
as start date and a duration that left out the time before it deferred.

Co-Authored-By: Claude <noreply@anthropic.com>
…em tests

A task resumed from deferral reports no start date and duration before
Airflow 3.2.2, and Airflow 2 events now carry the dates too, so the
expectations need to say which versions get what. That boundary is a
patch release, which the expected-event files couldn't express.

Co-Authored-By: Claude <noreply@anthropic.com>
…Lineage test

Its stored duration is the previous try's, which the facet no longer reports.

Co-Authored-By: Claude <noreply@anthropic.com>
@kacpermuda
kacpermuda force-pushed the ol-end-date-as-event-time branch from f481092 to 202431a Compare October 27, 2026 16:17
@mobuchowski
mobuchowski merged commit ce75cee into apache:main Oct 28, 2026
288 of 290 checks passed
@kacpermuda
kacpermuda deleted the ol-end-date-as-event-time branch October 28, 2026 10:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers full tests needed We need to run full set of tests for this PR to merge provider:openlineage AIP-53

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants