Repository navigation
Evaluate pending partition Dag runs only when their inputs change - #76706
Draft
hussein-awala wants to merge 3 commits into
Draft
hussein-awala wants to merge 3 commits into
hussein-awala wants to merge 3 commits into
Conversation
Each scheduler loop locked and evaluated the 500 oldest pending partition-driven Dag runs, whether or not anything had changed for them. Runs that wait by design, such as a rollup whose window is not complete, filled those slots, so newer runs whose partitions were complete were never evaluated. An idle deployment also paid for the claim and the evaluations on every loop. The result of an evaluation changes only when one of its inputs does, so evaluating a run only after such a change removes both the starvation and the idle cost. Writers record the change as an inserted request rather than as a flag on the run, because an update would wait on the scheduler's lock of the run inside the producer's transaction. Passes over every pending run, at startup and on an interval, cover the changes that no request reports, such as key logs written by components that still run an older version during an upgrade.
The pending-run lookup that decides whether a new definition asks for an evaluation cost one more statement for every Dag the Dag processor rewrites, which the PostgreSQL statement-budget test of the Dag file processor caught. Writing a Dag already looks up its task instances in the same place, so both lookups now share one round trip.
This was referenced Nov 12, 2026
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
On
main, each scheduler loop locks the 500 oldest pending partition-driven Dag runs (AssetPartitionDagRun) and evaluates them, whether or not anything changed for them:A natural alternative is for writers to flag the runs whose inputs changed. A flag on the run row, though, would be an UPDATE inside the producer's task-success transaction, while the execution API holds the task instance's row lock and while the scheduler may hold the run's row: the producer would wait on the scheduler, and a lock timeout would fail the producer task. This PR has the writer insert a request instead, which never waits.
How
A request queue. The new table
asset_partition_dag_run_wake(migration 0157) holds requests to evaluate pending runs:id,apdr_id(NULL for a request about every pending run of a Dag),target_dag_id,reasonandcreated_at. It has no foreign key and no unique key, so an insert never waits on the scheduler's lock of the run and never conflicts with a request the scheduler already read. Rows are never updated: writers insert them, and the scheduler deletes the ones it read, by primary key.Who writes requests, each in the transaction of the change it reports:
PartitionedAssetKeyLogthat a session flush inserts. This covers the execution API, the REST API, the triggerer and any[core] asset_manager_classthat adds key logs to the session, without changing them.SerializedDagModel.write_dagwrites one request for the whole Dag when the timetable section of the definition changes, for a partitioned Dag or a Dag that still has pending runs. The timetable holds the asset condition and the partition mappers; nothing else in the definition changes an evaluation._activate_referenced_assets) and the Dag processor (activate_assets_if_possible) write one request for each Dag scheduled on an asset they activate, because key logs of an inactive asset do not count.What a scheduler loop evaluates, within the same budget of 500 runs per loop:
[scheduler] asset_partition_resync_interval(new, 3600 s by default). They catch what no request reports: key logs written by components that still run an older version during an upgrade, key logs written with a bulk insert, and a new link from an alias to an asset that is already active.While the fan-outs or a pass have runs left, each keeps a tenth of the loop.
Claim. As in #76463:
FOR UPDATE SKIP LOCKEDandmain's paused, draining and stale filter, with up to 3 refills.Request lifecycle.
@retry_db_transactionretries does it again.[scheduler] parsing_cleanup_intervaldrops the requests of Dags that no longer exist. It also merges those of paused, draining or stale Dags into one request per Dag, in batches committed on their own, so the queue the claim reads stays small.Metrics:
scheduler.asset_partitions.wake_lag: the age of the oldest waiting request of a Dag that can run (not paused, draining or stale).scheduler.asset_partitions.evaluated: the runs evaluated.scheduler.asset_partitions.resync_fired: runs created that only a pass found. It is taggedstartuporperiodic. A periodic count above 0 means that a change reached no request: key logs written without one, an alias newly linked to an asset that was already active, or custom code whose result changes while its inputs do not.scheduler.asset_partitions.resync_pass_duration.Behaviour changes
asset_partition_resync_intervalafter the previous one started, or when that one ends if it took longer.create_dagrunstill fails the loop, as onmain.main.SerializedDagModeldrops its decoded data when the instance is expired, as on a rollback (Airflow's sessions do not expire on commit), so a loop retried after a definition was written in place evaluates the new one.Relation to #76463
_create_dagruns_for_partitioned_asset_dags,metrics_template.yaml,assets.rstand a comment in the UI's assets route, so whichever lands second rebases. If Prevent pending partitions from starving ready Dag runs #76463 lands first, the rebase removes its per-loop rotation. Its metrics (pass_loops,pass_duration,oldest_pending_age) are then either removed or kept for the resync pass. Log and audit when the partitioned Dag run per-tick cap is reached #71072, which logs when the per-loop cap is reached, also changes that function.Open questions for reviewers
Testing
On the final commit:
tests/unit/migrations, 1,623 passed;test_serialized_dag.py, 114 passed.test_scheduler_job.py, 100 passed; the other changed test files excepttest_db_cleanup.py, 278 passed.test_db_cleanup.py, 278 passed.breeze verify: the core DB tests (13,009 tests in 6 types, no failures), the core non-DB tests (5,718 passed), thestandardprovider suites (DB and non-DB), thecommon.compatsuite (non-DB; it has no DB tests), andbreeze build-docs apache-airflow.breeze verifyalso lists every provider suite, becausedevel-commonchanged: before each provider DB test module, the_clear_dbfixture runsclear_db_apdr(), which this PR changes. ThestandardDB suite above ran it; CI runs the other suites, including the runs against older Airflow versions, which have noAssetPartitionDagRunWake.Existing tests that change:
test_partitioned_dag_run_resumes_when_asset_reactivates: anAssetActiverow that something other than the scheduler or the Dag processor inserts now waits for the next pass.test_partition_cap_at_exactly_n_processes_allandtest_partition_cap_at_n_minus_one_leaves_one_pending: the order within a Dag now comes from the requests, so thecreated_atpin of the second test is gone, and both docstrings name the per-loop budget.test_pending_apdr_select_locks_rows_for_skip_locked_claim: the claim no longer runs a locking query when it has no run to evaluate, so the test creates a pending run.TestNextRunAssets::test_pending_apdr_uses_fifo_order(UI), docstring only: the pending run that waited longest is not necessarily the next to fire.test_no_models_missing: the request table is not part ofairflow db clean; the scheduler deletes its rows.test_pending_apdr_select_breaks_tie_on_idis removed. It ran its ownSELECT ... ORDER BY created_at, id, which stood for the claim query that this PR removes, and called no scheduler code.Not from this PR: on
maintoo,test_serialized_dag.pyfails aftertest_scheduler_job.pyin one pytest process unless the database was just initialized, because an existing scheduler test fills the plugin cache whileload_examplesis off. The uv runs above keep the two files in separate processes; the breeze run initializes the database first.Cost of the claim
Measured as the guide for performance-sensitive components describes:
main's claim (one lockingSELECT ... ORDER BY created_at, id LIMIT 500) against this PR's claim, in alternating rounds with a server-side control query. The numbers are medians of 6 rounds of 10 loops, each rolled back; the control drifted at most 2.5%. Each database holds about 1,050,000 partition-driven Dag runs. Of them, 55,000 (PostgreSQL) and 50,101 (MySQL) are pending, in about 2,300 and 2,400 Dags, and 50,000 and 45,075 belong to Dags that are not paused, draining or stale.mainmain's claim: a third more with a request for every pending run of a Dag that can run, because the signal read groups every request in a temporary table. The backlog drains by up to 500 runs per loop, whilemainpays its cost on every loop.main, each loop also evaluates up to 500 runs; here, only the runs that something asked for, plus every pending run once in each resync pass.related: #76463, #64571
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines
🤖 Generated with Claude Code