Repository navigation
Allow publishing asset events to a single consumer - #76980
Open
ataulmujeeb-cyber wants to merge 16 commits into
Open
ataulmujeeb-cyber wants to merge 16 commits into
ataulmujeeb-cyber wants to merge 16 commits into
Conversation
Add an optional consumer_dag_id to event publication so a missing partition input can be supplied without queueing sibling consumers. Require target run permission and retain normal consumer eligibility. Cover both dependency-change recovery sequences through the public API and scheduler, and document clearing stale queued progress.
ataulmujeeb-cyber
requested review from
Lee-W,
bbovenzi,
bugraoz93,
choo121600,
dheerajturaga,
ephraimbuddy,
guan404ming,
henry3260,
jason810496,
pierrejeambrun,
potiuk,
rawwar,
ryanahamilton,
shubhamraj-git,
uranusjr and
vatsrahul1001
as code owners
November 12, 2026 12:59
Contributor
Author
|
Could a maintainer please re-run the failed CI Job please? The Kubernetes/Celery job failed while fetching the production-image artifact during setup, before the tests started. The core database tests and static checks passed. Thanks |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adding an asset to a partitioned consumer can leave the current partition missing an input that was published before the dependency existed. Republishing that input currently queues work for every eligible consumer, including consumers that already processed the partition.
This adds an optional
consumer_dag_idtoPOST /api/v2/assets/events. Operators can publish the missing input for one eligible consumer without queueing another run for its siblings. The caller must have both asset-event creation permission and run creation permission for the selected Dag. Normal consumer resolution, team restrictions, paused/draining state and partition mapping still apply.The request creates a new event. Asset history, listeners and metrics retain their normal behavior; historical events are not attached retroactively. If pending progress belongs to an old dependency or mapper definition, the documented recovery is to clear the affected consumer's queued inputs and publish scoped events for every required input under the new definition. Queue deletion filters source partition keys, so non-identity mappings require care. This does not change automatic historical catchup or introduce a migration.
Closes #76979. Related to #76706: reevaluating an existing pending run cannot supply an input link that was never created.
Validation against main at
29331a8325698266e380a88e6aece1dad7669210:airflow-core/tests/unit/assets/test_manager.pyandairflow-core/tests/unit/api_fastapi/core_api/routes/public/test_assets.py.The focused runtime command is:
The database run used an isolated PostgreSQL database, the workspace sources and the matching PostgreSQL driver. The broader Breeze matrix was not run locally because the CI-image build exhausted Docker's available disk space. The OpenAPI generator ran directly under the supported workspace environment because the installed Breeze launcher was broken. The UI hook's generated-file lint step reports "No files found to lint" because those paths are deliberately ignored by its linter; code generation and the production build were run separately. These are local validation limits, not CI passes.