Skip to content

Order partition keys before creating pending asset runs - #75524

Open
ataulmujeeb-cyber wants to merge 1 commit into
apache:mainfrom
ataulmujeeb-cyber:apdr-order-partition-keys
Open

ataulmujeeb-cyber wants to merge 1 commit into
apache:mainfrom
ataulmujeeb-cyber:apdr-order-partition-keys

Conversation

@ataulmujeeb-cyber

@ataulmujeeb-cyber ataulmujeeb-cyber commented Oct 23, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Related: #71074

A partition mapper fans one incoming asset event out into several downstream keys and is free to return them in any order. Pending runs are then created in that order, so two producers can take the same keys in opposing sequences and wait on each other.

Root Cause

Each pending run inserted for a batched consumer holds a (target_dag_id, pending_partition_key) unique-index entry until its transaction commits. The fan-out loop acquires in mapper order:

target_keys: list[str] = list(target_key)

With two producers feeding the same consumer:

  1. Asset A's mapper returns ["k2", "k1"] for consumer C; asset B's mapper returns ["k1", "k2"].
  2. The producer for A inserts the pending run for k2; the producer for B inserts k1.
  3. A asks for k1 and B asks for k2. Each now waits on an index entry the other holds.

Consumer Dags are already sorted before acquisition; the keys within each consumer are not.

Fix

Sort the keys so every producer acquires them in the same order within a registration:

target_keys: list[str] = sorted(target_key)

The complete key set is preserved and the fan-out cap check still runs before acquisition, so an over-cap fan-out is skipped as a whole rather than truncated. No migration, no schema change.

This orders keys within one registration. A transaction that registers several assets makes several registrations and can still take keys in conflicting orders, so this does not remove every deadlock — it removes the single-registration cycle cheaply. The remaining cross-registration case is handled separately in #75525.

Changes

  • airflow-core/src/airflow/assets/manager.py: sort mapper fan-out keys in _queue_partitioned_dags, after the existing cap check.
  • airflow-core/tests/unit/assets/test_manager.py: two new cases driving real asset registration with a mapper that returns keys in descending order:
    • single consumer: asserts the complete (dag_id, key) acquisition and event-link sequence
    • two consumers: asserts the same across both, so consumer-Dag order and key order are both covered

Testing

pytest airflow-core/tests/unit/assets/test_manager.py -k 'fanout or fan_out or dag_id_order'
  • PostgreSQL 16 / psycopg3: 10 passed, 0 skipped
  • MySQL 8.4 / PyMySQL: 10 passed, 0 skipped

The same selection passed 10 on PostgreSQL, MySQL and SQLite during implementation on an equivalent tree.

Removing the sort in production makes both new cases fail on each of the three backends:

pytest airflow-core/tests/unit/assets/test_manager.py::TestAssetManager::test_partition_fanout_creates_apdrs_in_key_order

The assertions compare full (dag_id, key) sequences rather than set membership, because a membership check still passes with the sort removed. This establishes per-registration acquisition order, not transaction-wide ordering and not the absence of all deadlocks. SQLite results do not establish concurrent-writer behaviour.

Affected Components

  • AssetManager._queue_partitioned_dags
  • Partitioned consumers fed by fan-out mappers

Related Follow-ups

Each stands alone and can be reviewed independently.

@Lee-W
Lee-W enabled auto-merge (squash) November 20, 2026 07:07
@eladkal eladkal added this to the Airflow 3.4.1 milestone Nov 20, 2026
@eladkal eladkal added type:bug-fix Changelog: Bug Fixes backport-to-v3-4-test Backport to v3-4-test labels Nov 20, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backport-to-v3-4-test Backport to v3-4-test type:bug-fix Changelog: Bug Fixes

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants