Skip to content

Consume all mapped asset events on the next triggered run - #71991

Closed
Vamsi-klu wants to merge 3 commits into
apache:mainfrom
Vamsi-klu:asset-mapped-outlet-consume-all
Closed

Vamsi-klu wants to merge 3 commits into
apache:mainfrom
Vamsi-klu:asset-mapped-outlet-consume-all

Conversation

@Vamsi-klu

@Vamsi-klu Vamsi-klu commented Aug 23, 2026 •

Copy link
Copy Markdown

Consume all mapped asset events on the next triggered run

closes: #54659

What I did

Tests only. Mapped producer @task(outlets=[asset]).expand(...) succeeds, then a scheduler tick. Assert all N events land on the consumer run and in triggering_asset_events. Also pins empty expand, a failed map index, leftovers on the next tick, AND/OR schedules, alias yields, catchup, and two producers in the same second. No production change. No newsfragment.

Why I did

#54659 is three mapped outlets, one consumer run, triggering_asset_events showing only one event. #70972 already switched ADRQ to (target_dag_id, asset_event_id) and consume-by-id. Existing tests insert ADRQ by hand, so they never covered this emit path. These tests are that coverage, including one dag_maker.run_ti path.

How I did

Emit: register_asset_changes_in_db after mapped SUCCESS, plus one dag_maker.run_ti(..., map_index=N) case.
Consume: SchedulerJobRunner._create_dagruns_for_dags.
Context: get_template_context after loading consumed_asset_events.

AND waits for the second asset, then consumes the leftover queue. One tick batches all visible events. That is the default, not a bug. Leftovers stay for the next tick.

What's the impact

None at runtime. If consume-by-id or the mapped emit path regresses, these tests fail instead of silently dropping events.

What's the testing

airflow-core/tests/unit/jobs/test_scheduler_job.py

  • test_mapped_asset_empty_expand_creates_no_run
  • test_mapped_asset_events_consumed_together
  • test_mapped_asset_run_ti_consumes_all_outlets
  • test_mapped_asset_leftovers_consumed_on_next_tick
  • test_mapped_asset_failed_index_not_queued
  • test_mapped_asset_two_producers_same_second
  • test_mapped_asset_alias_yields_consumed_together
  • test_mapped_asset_and_condition_keeps_adrq
  • test_mapped_asset_or_condition_consumes_fan_out
  • test_mapped_asset_catchup_keeps_queued_events
uv run --project airflow-core pytest airflow-core/tests/unit/jobs/test_scheduler_job.py \
  -k mapped_asset_ -q

13 passed.


Was generative AI tooling used to co-author this PR?
  • Yes (Grok 4.6)

Generated-by: Grok 4.6 following the guidelines

Vamsi-klu and others added 3 commits August 23, 2026 06:22
The scheduler already attaches every queued mapped outlet event, but
the suite never drove that through TaskInstance.register_asset_changes_in_db.
Without that fixture the original mapped-outlet miss can come back unnoticed.

Co-authored-by: Cursor <cursoragent@cursor.com>
The leftover case never checked that events 1 and 2 were already queued
before the second tick, so a fourth-event rescue would still pass.
Failed-index and OR cases had the same hole: they never marked a failed
TI or left a second asset's ADRQ in place.

Co-authored-by: Cursor <cursoragent@cursor.com>
apache#54659 is only closed if the consumer task sees every mapped event in
template context, not just consumed_asset_events on the DagRun.

Co-authored-by: Cursor <cursoragent@cursor.com>
@Vamsi-klu

This comment was marked as spam.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Asset-triggered DAGs miss events from concurrently completing dynamic mapped tasks

1 participant