Skip to content

Fix Dag callbacks dropped when bundle versioning is disabled - #72930

Merged
ephraimbuddy merged 3 commits into
apache:mainfrom
astronomer:callback-bundle-init-unversioned
Sep 18, 2026
Merged

ephraimbuddy merged 3 commits into
apache:mainfrom
astronomer:callback-bundle-init-unversioned

Conversation

@ephraimbuddy

Copy link
Copy Markdown
Contributor

A Dag run that is not pinned to a bundle version -- the normal case under
disable_bundle_versioning -- produces a callback request that carries no
bundle version. Callback queueing built a second, uninitialized bundle
instance for it and read a path off it, so for a versioned bundle whose path
is only defined after initialization the callback was queued against an
unresolved location and lost. Nothing surfaced to the user, because a
callback-only run records no import error.

Such a request wants the bundle's current contents, which is what this
processor's own instance already tracks, so serving it from there resolves the
same path the file scan queues and keeps a fetch and checkout out of the
parsing loop. Nothing else is a valid answer for an unversioned request:
callbacks are claimed only for bundles this processor parses, and a versioned
instance that is still uninitialized has just failed initialization in the
refresh that precedes callback fetching, which is also where it is retried.

closes: #64891


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Opus 5, GPT-6 astra following the guidelines

A Dag run that is not pinned to a bundle version -- the normal case under
disable_bundle_versioning -- produces a callback request that carries no
bundle version. Callback queueing built a second, uninitialized bundle
instance for it and read a path off it, so for a versioned bundle whose path
is only defined after initialization the callback was queued against an
unresolved location and lost. Nothing surfaced to the user, because a
callback-only run records no import error.

Such a request wants the bundle's current contents, which is what this
processor's own instance already tracks, so serving it from there resolves the
same path the file scan queues and keeps a fetch and checkout out of the
parsing loop. Nothing else is a valid answer for an unversioned request:
callbacks are claimed only for bundles this processor parses, and a versioned
instance that is still uninitialized has just failed initialization in the
refresh that precedes callback fetching, which is also where it is retried.

closes: apache#64891
An initialization outage must not consume pending callbacks before the processor can execute them. Callback work needs to survive temporary bundle failures and remain diagnosable until recovery.
@ephraimbuddy ephraimbuddy added this to the 3.3.3 milestone Sep 18, 2026
@ephraimbuddy
ephraimbuddy merged commit b6d67b7 into apache:main Sep 18, 2026
78 checks passed
@ephraimbuddy
ephraimbuddy deleted the callback-bundle-init-unversioned branch September 18, 2026 09:59
@github-actions

Copy link
Copy Markdown
Contributor

Backport failed to create: v3-3-test. View the failure log Run details

Note: As of Merging PRs targeted for Airflow 3.X
the committer who merges the PR is responsible for backporting the PRs that are bug fixes (generally speaking) to the maintenance branches.

In matter of doubt please ask in #release-management Slack channel.

Status Branch Result
❌ v3-3-test Commit Link

You can attempt to backport this manually by running:

cherry_picker b6d67b7 v3-3-test

This should apply the commit to the v3-3-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

If you don't have cherry-picker installed, see the installation guide.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Callback queueing skips versioned bundle initialization when DagCallbackRequest.bundle_version is None

2 participants