Repository navigation
Order partition keys before creating pending asset runs - #75524
Open
ataulmujeeb-cyber wants to merge 1 commit into
Open
ataulmujeeb-cyber wants to merge 1 commit into
ataulmujeeb-cyber wants to merge 1 commit into
Conversation
This was referenced Oct 23, 2026
Lee-W
approved these changes
Nov 20, 2026
Lee-W
enabled auto-merge (squash)
November 20, 2026 07:07
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Related: #71074
A partition mapper fans one incoming asset event out into several downstream keys and is free to return them in any order. Pending runs are then created in that order, so two producers can take the same keys in opposing sequences and wait on each other.
Root Cause
Each pending run inserted for a batched consumer holds a
(target_dag_id, pending_partition_key)unique-index entry until its transaction commits. The fan-out loop acquires in mapper order:With two producers feeding the same consumer:
["k2", "k1"]for consumerC; asset B's mapper returns["k1", "k2"].k2; the producer for B insertsk1.k1and B asks fork2. Each now waits on an index entry the other holds.Consumer Dags are already sorted before acquisition; the keys within each consumer are not.
Fix
Sort the keys so every producer acquires them in the same order within a registration:
The complete key set is preserved and the fan-out cap check still runs before acquisition, so an over-cap fan-out is skipped as a whole rather than truncated. No migration, no schema change.
This orders keys within one registration. A transaction that registers several assets makes several registrations and can still take keys in conflicting orders, so this does not remove every deadlock — it removes the single-registration cycle cheaply. The remaining cross-registration case is handled separately in #75525.
Changes
airflow-core/src/airflow/assets/manager.py: sort mapper fan-out keys in_queue_partitioned_dags, after the existing cap check.airflow-core/tests/unit/assets/test_manager.py: two new cases driving real asset registration with a mapper that returns keys in descending order:(dag_id, key)acquisition and event-link sequenceTesting
The same selection passed 10 on PostgreSQL, MySQL and SQLite during implementation on an equivalent tree.
Removing the sort in production makes both new cases fail on each of the three backends:
The assertions compare full
(dag_id, key)sequences rather than set membership, because a membership check still passes with the sort removed. This establishes per-registration acquisition order, not transaction-wide ordering and not the absence of all deadlocks. SQLite results do not establish concurrent-writer behaviour.Affected Components
AssetManager._queue_partitioned_dagsRelated Follow-ups
Each stands alone and can be reviewed independently.