diff --git a/airflow-core/docs/authoring-and-scheduling/assets.rst b/airflow-core/docs/authoring-and-scheduling/assets.rst index f2a5d147bdf36..942f2119f3b9f 100644 --- a/airflow-core/docs/authoring-and-scheduling/assets.rst +++ b/airflow-core/docs/authoring-and-scheduling/assets.rst @@ -702,6 +702,31 @@ uses ``StartOfDayMapper`` (which expects a timestamp-like key), no downstream partition match can be produced, so the downstream Dag is not triggered for that key. +While partitions wait for their upstream events, Airflow keeps pending +partition-driven Dag runs for them. The scheduler evaluates a pending run, and creates +its Dag run once the condition is met, when something happens that can change the +result: an asset event reaches the run, the Dag's timetable changes, or an asset that +the Dag is scheduled on becomes active. A run that keeps waiting, such as a rollup whose +window is not complete yet, is not evaluated again until then, so it cannot hold the +others back. Each scheduler also evaluates every pending run once when it starts, and +then once every :ref:`[scheduler] asset_partition_resync_interval +`. These passes catch asset events +that were recorded without a request for evaluation, for example by components that +still run an older Airflow version during an upgrade. The +``scheduler.asset_partitions.wake_lag`` metric reports how long the oldest request has +waited, and ``scheduler.asset_partitions.resync_fired`` counts the Dag runs that only +such a pass found. + +This relies on the evaluation of a pending run depending only on the Dag's timetable, +the partition keys that reached the run, which assets are active, and how the Dag's +asset references and aliases resolve. A new link from an alias to an asset that is +already active is picked up only by the next pass. Custom partition mappers, rollup +windows and timetables must keep to this: for example, a mapper that reads the clock, +or fails now and then, has its runs evaluated again only by the next pass. A run whose +evaluation fails because of the Dag itself, for example because its mapper raises, is +not evaluated again on every loop either: only when an asset event reaches it, the Dag's +timetable changes, an asset that the Dag is scheduled on becomes active, or a pass runs. + Inside partitioned Dag runs, access the resolved partition through ``dag_run.partition_key``. When the consumer's partition mapper can resolve the key to a ``datetime``, that value is also available as diff --git a/airflow-core/docs/migrations-ref.rst b/airflow-core/docs/migrations-ref.rst index 5bbf2b43216cd..a4b1daaf1a644 100644 --- a/airflow-core/docs/migrations-ref.rst +++ b/airflow-core/docs/migrations-ref.rst @@ -39,7 +39,9 @@ Here's the list of all the Database Migrations that are executed via when you ru +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | Revision ID | Revises ID | Airflow Version | Description | +=========================+==================+===================+==============================================================+ -| ``d8c6f3a129b0`` (head) | ``687eb253de24`` | ``3.5.0`` | Add durable Dag processor work claims. | +| ``6b0e4c1f2a37`` (head) | ``d8c6f3a129b0`` | ``3.5.0`` | Add asset_partition_dag_run_wake. | ++-------------------------+------------------+-------------------+--------------------------------------------------------------+ +| ``d8c6f3a129b0`` | ``687eb253de24`` | ``3.5.0`` | Add durable Dag processor work claims. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ | ``687eb253de24`` | ``a8b48633fa60`` | ``3.5.0`` | Add Dag processor publication receipts and source revisions. | +-------------------------+------------------+-------------------+--------------------------------------------------------------+ diff --git a/airflow-core/newsfragments/76706.significant.rst b/airflow-core/newsfragments/76706.significant.rst new file mode 100644 index 0000000000000..0ab196bf5ecd1 --- /dev/null +++ b/airflow-core/newsfragments/76706.significant.rst @@ -0,0 +1,28 @@ +Pending partition-driven Dag runs are evaluated only when their inputs change + +The scheduler no longer evaluates the oldest pending partition-driven Dag runs on every loop. It +evaluates a pending run when an asset event reaches it, when its Dag's timetable changes, or when an +asset that its Dag is scheduled on becomes active. Each scheduler also evaluates every pending run once +when it starts, and then once every ``[scheduler] asset_partition_resync_interval`` (3600 seconds by +default). + +The evaluation of a pending run must depend only on the Dag's timetable, the partition keys that reached +the run, which assets are active, and how the Dag's asset references and aliases resolve. A custom +partition mapper, rollup window or timetable that reads the clock, or that fails now and then, has its +runs evaluated again only by the next pass. So do runs whose key logs are written in bulk rather than +through a session, and runs that a new link from an alias to an asset that is already active satisfies. + +Upgrade the schedulers after the other components, or restart them once every component runs this +version: components that run an older version do not ask for an evaluation, so their asset events are +picked up by the next pass. + +* Types of change + + * [ ] Dag changes + * [x] Config changes + * [ ] API changes + * [ ] CLI changes + * [x] Behaviour changes + * [ ] Plugin changes + * [ ] Dependency changes + * [ ] Code interface changes diff --git a/airflow-core/src/airflow/api_fastapi/core_api/routes/ui/assets.py b/airflow-core/src/airflow/api_fastapi/core_api/routes/ui/assets.py index 25c74fae94293..e85bae5741013 100644 --- a/airflow-core/src/airflow/api_fastapi/core_api/routes/ui/assets.py +++ b/airflow-core/src/airflow/api_fastapi/core_api/routes/ui/assets.py @@ -278,9 +278,9 @@ def next_run_assets( return NextRunAssetsResponse.model_validate(model_data) # Partitioned Dags: enrich with per-asset received/required counts and rollup flag. - # FIFO matches the scheduler's pending-APDR processing order - # (``_create_dagruns_for_partitioned_asset_dags``), so the "next run" the UI - # surfaces is the same one the scheduler will fire next. + # This shows the pending APDR that has waited longest. It is not necessarily the + # next run: the scheduler (``_create_dagruns_for_partitioned_asset_dags``) fires + # whichever pending APDR is satisfied, whatever its age. pending_apdr = session.execute( select(AssetPartitionDagRun.id, AssetPartitionDagRun.partition_key) .where( diff --git a/airflow-core/src/airflow/config_templates/config.yml b/airflow-core/src/airflow/config_templates/config.yml index d69dd736e57b6..3affb7080d046 100644 --- a/airflow-core/src/airflow/config_templates/config.yml +++ b/airflow-core/src/airflow/config_templates/config.yml @@ -2984,6 +2984,20 @@ scheduler: type: integer example: ~ default: "1000" + asset_partition_resync_interval: + description: | + How often (in seconds) the scheduler evaluates every pending partition-driven Dag run once, + whether or not anything asked for it. Otherwise the scheduler evaluates a pending run only + when a key log is added to it, when its Dag's timetable changes, or when an asset its Dag is + scheduled on is activated. This pass also evaluates key logs that were written without such a + request: for example, by components that still run an older Airflow version during a rolling + upgrade, or by code that inserts key logs in bulk rather than through a session. Each scheduler + also runs one pass when it starts. A pass starts when the previous one is complete and this + interval has passed since that one started. + version_added: 3.5.0 + type: integer + example: ~ + default: "3600" use_job_schedule: description: | Set to ``False`` to stop the scheduler creating Dag runs on its own, whether a Dag is due diff --git a/airflow-core/src/airflow/dag_processing/collection.py b/airflow-core/src/airflow/dag_processing/collection.py index fbbf3d14a20f7..86e41341cf7b0 100644 --- a/airflow-core/src/airflow/dag_processing/collection.py +++ b/airflow-core/src/airflow/dag_processing/collection.py @@ -32,7 +32,7 @@ from typing import TYPE_CHECKING, Any, NamedTuple, TypeVar import structlog -from sqlalchemy import delete, false, func, insert, or_, select, tuple_, update +from sqlalchemy import delete, false, func, insert, inspect, or_, select, tuple_, update from sqlalchemy.exc import OperationalError from sqlalchemy.orm import Session, joinedload, load_only @@ -44,6 +44,7 @@ AssetActive, AssetAliasModel, AssetModel, + AssetPartitionDagRunWake, DagScheduleAssetAliasReference, DagScheduleAssetNameReference, DagScheduleAssetReference, @@ -70,6 +71,7 @@ from airflow.serialization.enums import Encoding from airflow.serialization.serialized_objects import BaseSerialization, DagSerialization, LazyDeserializedDAG from airflow.triggers.base import BaseEventTrigger +from airflow.utils.helpers import chunks from airflow.utils.retries import MAX_DB_RETRIES, run_with_db_retries from airflow.utils.sqlalchemy import build_upsert_stmt, get_dialect_name, with_row_locks from airflow.utils.types import DagRunType @@ -1207,7 +1209,9 @@ def sync_assets(self, *, session: Session) -> dict[tuple[str, str], AssetModel]: orm_assets: dict[tuple[str, str], AssetModel] = { (am.name, am.uri): am for am in session.scalars( - select(AssetModel).where(tuple_(AssetModel.name, AssetModel.uri).in_(self.assets)) + select(AssetModel) + .where(tuple_(AssetModel.name, AssetModel.uri).in_(self.assets)) + .options(joinedload(AssetModel.active)) ) } for key, model in orm_assets.items(): @@ -1268,8 +1272,24 @@ def activate_assets_if_possible(self, models: Iterable[AssetModel], *, session: from sqlalchemy.dialects.sqlite import insert as sqlite_insert stmt = sqlite_insert(AssetActive).on_conflict_do_nothing() + models = list(models) if values := [{"name": m.name, "uri": m.uri} for m in models]: session.execute(stmt, values) + # An inactive asset may hold key logs of pending partition Dag runs, which activating it can + # satisfy. sync_assets loads whether each existing asset is active; the assets it creates have + # no key logs, and their activation state is not loaded. + if inactive := [m for m in models if inspect(m).attrs.active.loaded_value is None]: + active = { + pair + for chunk in chunks(inactive, 400) + for pair in session.execute( + select(AssetActive.name, AssetActive.uri).where( + tuple_(AssetActive.name, AssetActive.uri).in_([(m.name, m.uri) for m in chunk]) + ) + ).tuples() + } + if activated := [m for m in inactive if (m.name, m.uri) in active]: + AssetPartitionDagRunWake.add_for_activated_assets(activated, session=session) def add_dag_asset_references( self, diff --git a/airflow-core/src/airflow/jobs/scheduler_job_runner.py b/airflow-core/src/airflow/jobs/scheduler_job_runner.py index 8566da6cb832e..aa2d9906d57ef 100644 --- a/airflow-core/src/airflow/jobs/scheduler_job_runner.py +++ b/airflow-core/src/airflow/jobs/scheduler_job_runner.py @@ -22,19 +22,24 @@ import multiprocessing import operator import os +import random import signal import sys import time +from array import array +from bisect import bisect_left from collections import Counter, defaultdict, deque from collections.abc import Callable, Collection, Iterable, Iterator, Sequence from contextlib import ExitStack, suppress +from dataclasses import dataclass, field from datetime import datetime, timedelta from functools import lru_cache, partial from itertools import groupby -from typing import TYPE_CHECKING, Any, cast +from typing import TYPE_CHECKING, Any, NamedTuple, cast from uuid import UUID from sqlalchemy import ( + Integer, Text, and_, case, @@ -42,11 +47,15 @@ delete, exists, func, + insert, inspect, + literal_column, + null, or_, select, text, tuple_, + union_all, update, ) from sqlalchemy.exc import DBAPIError, OperationalError @@ -85,10 +94,12 @@ AssetEvent, AssetModel, AssetPartitionDagRun, + AssetPartitionDagRunWake, AssetWatcherModel, DagScheduleAssetAliasReference, DagScheduleAssetReference, PartitionedAssetKeyLog, + PartitionWakeReason, TaskInletAssetReference, TaskOutletAssetReference, alias_association_table, @@ -189,6 +200,138 @@ :meta private: """ +# Max number of lock queries per loop that refill the partition Dag runs to evaluate, when rows found +# in the wake queue are locked by another scheduler or no longer pending. +MAX_PARTITION_CLAIM_REFILLS_PER_LOOP = 3 +# Max number of ids per IN-list on a secondary index. MySQL estimates the rows of longer lists from +# sampled statistics, and has then chosen a full scan of the table. +_PARTITION_IN_LIST_SIZE = 199 +# Max number of Dags whose pending partition Dag runs one loop starts evaluating for a Dag-level wake row. +_PARTITION_FAN_OUT_STARTS_PER_LOOP = 50 +# Max number of Dag-level wake rows of one Dag that a fan-out reads, and deletes once it is done. +_PARTITION_FAN_OUT_WAKE_ROWS = 500 +# Max number of wake rows one loop deletes after evaluating their runs. Rows left over wake their +# runs once more. +_PARTITION_MAX_CONSUMED_WAKE_ROWS = 20_000 +# Max number of wake rows of fired or deleted runs one loop deletes; the next loops find the rest. +_PARTITION_MAX_ORPHANED_WAKE_ROWS = 20_000 +# Max number of batches of wake rows one firing of the cleanup timer deletes or merges. +_PARTITION_WAKE_CLEANUP_BATCHES = 20 + + +class _PartitionSweep: + """ + Pending partition Dag runs to evaluate once each, held per Dag in id order. + + Ids leave only once a committed loop evaluated their run, or found it fired, deleted or owned by + another Dag. An id whose run a loop could not lock stays, so that a run another transaction holds + is evaluated in a later loop. + """ + + def __init__(self) -> None: + self._ids_by_dag: dict[str, array[int]] = {} + self._dag_order: deque[str] = deque() + + def __len__(self) -> int: + """Return the number of Dags with runs left.""" + return len(self._ids_by_dag) + + def __contains__(self, dag_id: str) -> bool: + return dag_id in self._ids_by_dag + + def add(self, dag_id: str, ids: Iterable[int]) -> None: + if (sorted_ids := array("q", sorted(set(ids)))) and dag_id not in self._ids_by_dag: + self._ids_by_dag[dag_id] = sorted_ids + self._dag_order.append(dag_id) + + def rotate(self, positions: int) -> None: + self._dag_order.rotate(-positions) + + def peek(self, limit: int) -> list[str]: + """Return the Dags the next loop serves first.""" + return list(itertools.islice(self._dag_order, limit)) + + def take( + self, dag_ids: Sequence[str], budget: int, *, taken: set[int] + ) -> tuple[list[tuple[int, str]], list[tuple[int, str]]]: + """ + Take up to *budget* ids, one Dag at a time in turn, skipping the ids in *taken*. + + Return the ids taken, and the ids passed over because another source already took them. + Neither leaves the sweep: :meth:`commit` removes them once their loop is committed. + """ + candidates: list[tuple[int, str]] = [] + covered: list[tuple[int, str]] = [] + positions = dict.fromkeys(dag_ids, 0) + remaining = list(dag_ids) + while remaining and len(candidates) < budget: + still_remaining = [] + for dag_id in remaining: + ids, position = self._ids_by_dag[dag_id], positions[dag_id] + while position < len(ids) and ids[position] in taken: + covered.append((ids[position], dag_id)) + position += 1 + if position < len(ids): + candidates.append((ids[position], dag_id)) + taken.add(ids[position]) + position += 1 + positions[dag_id] = position + if position < len(ids): + still_remaining.append(dag_id) + if len(candidates) == budget: + break + remaining = still_remaining + return candidates, covered + + def is_left_empty(self, dag_id: str, done: Collection[int]) -> bool: + return all(apdr_id in done for apdr_id in self._ids_by_dag.get(dag_id, ())) + + def holds(self, dag_id: str, apdr_id: int) -> bool: + """Whether the sweep still has to evaluate this run of this Dag.""" + ids = self._ids_by_dag.get(dag_id, ()) + position = bisect_left(ids, apdr_id) + return position < len(ids) and ids[position] == apdr_id + + def commit(self, *, done: dict[str, set[int]], dropped: Collection[str], served: int) -> None: + """Remove what a committed loop finished, and serve the next Dags first in the next loop.""" + for dag_id, ids in done.items(): + if (dag_ids := self._ids_by_dag.get(dag_id)) is not None: + self._ids_by_dag[dag_id] = array("q", (i for i in dag_ids if i not in ids)) + for dag_id in dropped: + self._ids_by_dag.pop(dag_id, None) + for dag_id in [d for d, ids in self._ids_by_dag.items() if not ids]: + del self._ids_by_dag[dag_id] + self._dag_order.rotate(-served) + self._dag_order = deque(d for d in self._dag_order if d in self._ids_by_dag) + + +class _PartitionClaim(NamedTuple): + """The pending partition Dag runs a loop evaluates, and the wake rows it writes or deletes.""" + + apdrs: list[AssetPartitionDagRun] + consumed: list[int] + orphaned: list[int] + finished_wakes: list[int] + resync_wakes: list[str] + unsignalled: set[int] + + +@dataclass +class _PartitionLoop: + """What a loop of the partition claim changes in memory, applied once the loop is committed.""" + + resync_kind: str | None + fan_out_starts: dict[str, tuple[list[int], list[int]]] = field(default_factory=dict) + fan_out_done: dict[str, set[int]] = field(default_factory=lambda: defaultdict(set)) + fan_out_dropped: set[str] = field(default_factory=set) + fan_out_served: int = 0 + resync_done: dict[str, set[int]] = field(default_factory=lambda: defaultdict(set)) + resync_dropped: set[str] = field(default_factory=set) + resync_served: int = 0 + evaluated: int = 0 + resync_fired: int = 0 + oldest_signal_at: datetime | None = None + def _eager_load_dag_run_for_validation() -> tuple[LoaderOption, LoaderOption]: """ @@ -400,6 +543,23 @@ def __init__( self._multi_team = conf.getboolean("core", "multi_team") self._dag_tags_in_metrics = conf.getboolean("metrics", "dag_tags_in_metrics", fallback=False) self._max_partition_dag_runs_per_loop = MAX_PARTITION_DAG_RUNS_PER_LOOP + self._partition_resync_interval = conf.getint("scheduler", "asset_partition_resync_interval") + if self._partition_resync_interval <= 0: + raise ValueError("[scheduler] asset_partition_resync_interval must be greater than 0") + # The pending runs of Dags with a Dag-level wake row, and the rows to delete once they are done. + self._partition_fan_out = _PartitionSweep() + self._partition_fan_out_wakes: dict[str, list[int]] = {} + self._partition_resync = _PartitionSweep() + self._partition_resync_kind: str | None = None + self._partition_resync_passes = 0 + self._partition_resync_started_at = 0.0 + self._partition_rng = random.Random() + # Monotonic time from which the next resync pass may start; the first loop starts the + # startup pass. + self._partition_next_resync_at = 0.0 + self._partition_loop: _PartitionLoop | None = None + # A gauge keeps the value a previous process reported, so report 0 once if nothing is waiting. + self._partition_wake_lag_reported = True self._dag_id_to_team_name: dict[str, str | None] = {} self.executors: list[BaseExecutor] = executors if executors else ExecutorLoader.init_executors() @@ -2065,6 +2225,11 @@ def _run_scheduler_loop(self) -> None: self._update_asset_orphanage, non_fatal=True, ) + timers.call_regular_interval( + conf.getfloat("scheduler", "parsing_cleanup_interval"), + self._clean_up_partition_wakes, + non_fatal=True, + ) timers.call_regular_interval( conf.getfloat("scheduler", "parsing_cleanup_interval"), self._remove_unreferenced_triggers, @@ -2458,63 +2623,479 @@ def _create_dagruns_for_partitioned_asset_dags(self, session: Session) -> set[st return ``dag_id`` strings rather than full Dag/DagRun objects because the only downstream use is membership lookup, and a heavier return type would just be discarded. + A pending APDR is evaluated when a row of :class:`AssetPartitionDagRunWake` asks for it: + one written with each key log added to it, or one for its Dag, written when the Dag's + timetable changes or when an asset the Dag is scheduled on is activated. The outcome of the + evaluation depends only on the timetable, the APDR's key logs, which assets are active, and + how the Dag's asset references and aliases resolve. A pass when the scheduler starts, and then + one every ``[scheduler] asset_partition_resync_interval``, also evaluates every pending APDR + once: for key logs written without a wake row, and for a new link from an alias to an asset + that is already active. + Asset deactivation freezes pending APDRs: when an asset becomes inactive (orphan — no Dag declares it any more), its ``PartitionedAssetKeyLog`` rows stop contributing to the rollup. If the consumer Dag still depends on that asset, firing on stale history would conflict with the declared topology, - so the APDR waits. Reactivating the asset resumes evaluation automatically. + so the APDR waits. Reactivating the asset wakes its consumer Dags again. This matches the UI's progress view (``_fetch_active_assets_per_dag``). Pausing and draining freeze pending APDRs, mirroring the ``is_paused`` / ``is_draining`` half of :meth:`~airflow.models.dag.DagModel.dags_needing_dagruns`. ``has_import_errors`` needs no predicate of its own: it is only ever set together with ``is_stale`` (``_update_import_errors``) and cleared together with it - (``DagModelOperation.update_dags``), so the ``is_stale`` filter above already - excludes those Dags. ``exceeds_max_non_backfill`` is the one genuine divergence -- - an APDR for a Dag already at ``max_active_runs`` still creates its run, which then - waits at the QUEUED->RUNNING gate rather than being held back here. + (``DagModelOperation.update_dags``), so the ``is_stale`` filter of + :meth:`_lock_pending_partition_dag_runs` already excludes those Dags. + ``exceeds_max_non_backfill`` is the one genuine divergence -- an APDR for a Dag already at + ``max_active_runs`` still creates its run, which then waits at the QUEUED->RUNNING gate + rather than being held back here. Nothing accrues while a Dag is inactive -- ``AssetManager.register_asset_change`` drops paused and draining Dags before any ``PartitionedAssetKeyLog`` row is written -- so an event produced during the pause is never recorded and a partially satisfied APDR cannot advance past it. On reactivation the APDR resumes from the keys logged before it went inactive, unless the rollup definition changed in the - meantime, in which case the stale-fingerprint cleanup below drops it instead. + meantime, in which case the stale-fingerprint cleanup of + :meth:`_evaluate_partition_dag_runs` drops it instead. + """ + loop = self._partition_loop = _PartitionLoop(resync_kind=self._partition_resync_kind) + claim = self._claim_partition_dag_runs(loop, session=session) + partition_dag_ids = ( + self._evaluate_partition_dag_runs(claim.apdrs, session=session) if claim.apdrs else set() + ) + loop.evaluated = len(claim.apdrs) + if fired := [ + apdr + for apdr in claim.apdrs + if apdr.id in claim.unsignalled and apdr.created_dag_run_id is not None + ]: + # Any Dag-level row of the run's Dag asked for it too, except one that a running fan-out + # read, which asks only for the runs that fan-out holds. + unrequested = set( + self._find_dags_without_dag_level_wakes( + sorted({apdr.target_dag_id for apdr in fired}), session=session + ) + ) + loop.resync_fired = sum(apdr.target_dag_id in unrequested for apdr in fired) + self._write_partition_wakes(claim, session=session) + return partition_dag_ids + + def _claim_partition_dag_runs(self, loop: _PartitionLoop, *, session: Session) -> _PartitionClaim: + """ + Lock the pending partition Dag runs this loop evaluates, and read the wake rows they consume. + + The runs come from three sources, in this order: the wake rows of single runs, oldest first + within a Dag and in turn across Dags; the pending runs of Dags that have a Dag-level wake row; + and the startup or periodic resync. While the last two have runs left, each keeps a share of + the loop, and a source with fewer runs than its share leaves the rest to the others. + + The wake rows are read before the serialized Dags and the key logs. A wake row is committed + with its key log, so the evaluation sees every key log whose wake row this loop deletes, and + a wake row committed later survives and wakes the run again. """ - # Cap per-tick work so the scheduler transaction stays bounded and other - # scheduling work isn't starved. Remaining APDRs drain across subsequent ticks. - # FIFO is intentional: the oldest pending APDR fires first. A persistently - # unsatisfiable APDR at the head (e.g. broken mapper, upstream that will - # never arrive) blocks newer ones until an operator removes it or fixes - # the underlying mapper. We surface the stuck state rather than silently - # rotating past it. - # `with_row_locks(skip_locked=True)` mirrors the sibling ADRQ claim path: - # in HA two schedulers can otherwise both grab the same satisfied APDR - # and race the `created_dag_run_id` UPDATE, orphaning whichever DagRun - # loses. The `id` tiebreaker on `order_by` keeps LIMIT deterministic when - # two APDRs share a `created_at` under bulk asset-event ingestion. - # SQLite is single-writer and silently drops `FOR UPDATE`, which is fine. - pending_apdrs = session.scalars( + budget = self._max_partition_dag_runs_per_loop + share = max(1, budget // 10) + signals = self._read_partition_signals(budget, session=session) + apdr_signals = [(apdr_id, dag_id, at) for dag_id, apdr_id, at in signals if apdr_id is not None] + dag_signals = [(dag_id, at) for dag_id, apdr_id, at in signals if apdr_id is None] + finished_wakes = self._start_partition_fan_outs( + [dag_id for dag_id, _ in dag_signals], loop, session=session + ) + self._start_partition_resync_if_due(session=session) + loop.resync_kind = self._partition_resync_kind + + fan_out_window = self._partition_fan_out.peek(budget) + resync_window = self._partition_resync.peek(budget) + can_run = self._read_dag_eligibility({*fan_out_window, *resync_window}, session=session) + loop.fan_out_dropped = {dag_id for dag_id in fan_out_window if not can_run.get(dag_id)} + loop.resync_dropped = {dag_id for dag_id in resync_window if not can_run.get(dag_id)} + # The resync never evaluates the runs of a Dag that cannot run now, so a Dag-level wake row + # makes the Dag evaluate them once it can. + resync_wakes = [dag_id for dag_id in resync_window if can_run.get(dag_id) is False] + + taken: set[int] = set() + signalled = [] + for apdr_id, dag_id, _ in apdr_signals: + if apdr_id not in taken: + taken.add(apdr_id) + signalled.append((apdr_id, dag_id)) + fan_out, fan_out_covered = self._partition_fan_out.take( + [dag_id for dag_id in fan_out_window if can_run.get(dag_id)], budget, taken=taken + ) + resync, resync_covered = self._partition_resync.take( + [dag_id for dag_id in resync_window if can_run.get(dag_id)], budget, taken=taken + ) + fan_out_share, resync_share = min(share, len(fan_out)), min(share, len(resync)) + signalled_count = min(len(signalled), budget - fan_out_share - resync_share) + fan_out_count = min(len(fan_out), budget - signalled_count - resync_share) + resync_count = min(len(resync), budget - signalled_count - fan_out_count) + queue = signalled[:signalled_count] + fan_out[:fan_out_count] + resync[:resync_count] + spare = signalled[signalled_count:] + fan_out[fan_out_count:] + resync[resync_count:] + + locked: dict[int, AssetPartitionDagRun] = {} + tried: set[int] = set() + for refill in range(MAX_PARTITION_CLAIM_REFILLS_PER_LOOP + 1): + tried.update(apdr_id for apdr_id, _ in queue) + for batch in chunks([apdr_id for apdr_id, _ in queue], _ADRQ_DELETE_CHUNK_SIZE): + locked.update( + (apdr.id, apdr) for apdr in self._lock_pending_partition_dag_runs(batch, session=session) + ) + missing = budget - len(locked) + if missing <= 0 or not spare or refill == MAX_PARTITION_CLAIM_REFILLS_PER_LOOP: + break + queue, spare = spare[:missing], spare[missing:] + + def count_served(window: list[str], candidates: list[tuple[int, str]]) -> int: + """Return how many Dags at the head of *window* this loop served; the next loop starts after them.""" + served = {dag_id for apdr_id, dag_id in candidates if apdr_id in tried} + return max((i + 1 for i, dag_id in enumerate(window) if dag_id in served), default=0) + + loop.fan_out_served = count_served(fan_out_window, fan_out) + loop.resync_served = count_served(resync_window, resync) + + # The run of a source that could not be locked: owned by which Dag, and whether it fired. + found: dict[int, tuple[str, bool]] = {} + for batch in chunks(sorted(tried.difference(locked)), _ADRQ_DELETE_CHUNK_SIZE): + found.update( + (apdr_id, (dag_id, run_id is not None)) + for apdr_id, dag_id, run_id in session.execute( + select( + AssetPartitionDagRun.id, + AssetPartitionDagRun.target_dag_id, + AssetPartitionDagRun.created_dag_run_id, + ).where(AssetPartitionDagRun.id.in_(batch)) + ) + ) + + def is_settled(apdr_id: int, dag_id: str) -> bool: + """Whether nothing is left to evaluate for this run of this Dag.""" + if apdr_id in locked: + return True + if apdr_id in found: + owner, fired = found[apdr_id] + return fired or owner != dag_id + return apdr_id in tried + + # A wake row whose run fired, was deleted, or now belongs to another Dag, which SQLite + # allows by reusing the id of a deleted run. + orphans = { + (apdr_id, dag_id) + for apdr_id, dag_id, _ in apdr_signals + if is_settled(apdr_id, dag_id) + and not (apdr_id in locked and locked[apdr_id].target_dag_id == dag_id) + } + loop.oldest_signal_at = min( + itertools.chain( + (at for apdr_id, dag_id, at in apdr_signals if (apdr_id, dag_id) not in orphans), + (at for _, at in dag_signals), + ), + default=None, + ) + for sweep_done, candidates in ( + (loop.fan_out_done, fan_out + fan_out_covered), + (loop.resync_done, resync + resync_covered), + ): + for apdr_id, dag_id in candidates: + if is_settled(apdr_id, dag_id): + sweep_done[dag_id].add(apdr_id) + for dag_id in fan_out_window: + if can_run.get(dag_id) and self._partition_fan_out.is_left_empty( + dag_id, loop.fan_out_done[dag_id] + ): + finished_wakes.extend(self._partition_fan_out_wakes.get(dag_id, ())) + + consumed: list[int] = [] + orphaned: list[int] = [] + read: set[int] = set() + wake = AssetPartitionDagRunWake + for batch in chunks(sorted({*locked, *(apdr_id for apdr_id, _ in orphans)}), _PARTITION_IN_LIST_SIZE): + for wake_id, apdr_id, dag_id in session.execute( + select(wake.id, wake.apdr_id, wake.target_dag_id).where(wake.apdr_id.in_(batch)) + ): + if apdr_id in locked and locked[apdr_id].target_dag_id == dag_id: + consumed.append(wake_id) + read.add(apdr_id) + elif (apdr_id, dag_id) in orphans: + orphaned.append(wake_id) + # A run that a running fan-out still has to evaluate was asked for by the fan-out's Dag-level row. + resynced = { + apdr_id for apdr_id, dag_id in resync if not self._partition_fan_out.holds(dag_id, apdr_id) + } + return _PartitionClaim( + apdrs=sorted(locked.values(), key=lambda apdr: (apdr.created_at, apdr.id)), + consumed=sorted(consumed)[:_PARTITION_MAX_CONSUMED_WAKE_ROWS], + orphaned=sorted(orphaned)[:_PARTITION_MAX_ORPHANED_WAKE_ROWS], + finished_wakes=sorted(finished_wakes), + resync_wakes=self._find_dags_without_dag_level_wakes(resync_wakes, session=session), + unsignalled={apdr_id for apdr_id in locked if apdr_id in resynced and apdr_id not in read}, + ) + + def _find_dags_without_dag_level_wakes(self, dag_ids: list[str], *, session: Session) -> list[str]: + """ + Return the Dags of *dag_ids* that have no Dag-level wake row, which would make a new one redundant. + + The rows that a running fan-out of this scheduler read do not count: the fan-out deletes them + once it is done with the runs it read when it started. + """ + if not dag_ids: + return [] + wake = AssetPartitionDagRunWake + woken: set[str] = set() + for batch in chunks(dag_ids, _PARTITION_IN_LIST_SIZE): + woken.update( + dag_id + for dag_id, wake_id in session.execute( + select(wake.target_dag_id, wake.id).where( + wake.target_dag_id.in_(batch), wake.apdr_id.is_(None) + ) + ) + if wake_id not in self._partition_fan_out_wakes.get(dag_id, ()) + ) + return [dag_id for dag_id in dag_ids if dag_id not in woken] + + def _read_partition_signals( + self, budget: int, *, session: Session + ) -> list[tuple[str, int | None, datetime]]: + """ + Return the wake rows of Dags that can run, one per run and one per Dag for Dag-level rows. + + Within a Dag the oldest row comes first; across Dags they come in turn, so that many rows of + one Dag do not hold back the rows of the others. Rows of single runs and Dag-level rows are + capped apart: a Dag-level row stays until the evaluation of every pending run of its Dag is + done, and Dags whose evaluation is running must not hide the rows of other runs or Dags. On an + idle scheduler this is the only query of the partition claim. + """ + wake = AssetPartitionDagRunWake + can_run = ( + DagModel.is_paused.is_(False), + DagModel.is_draining.is_(False), + DagModel.is_stale.is_(False), + ) + runs = ( + select( + wake.target_dag_id.label("dag_id"), + wake.apdr_id.label("apdr_id"), + func.min(wake.id).label("first_id"), + func.min(wake.created_at).label("first_at"), + ) + .join(DagModel, DagModel.dag_id == wake.target_dag_id) + .where(wake.apdr_id.is_not(None), *can_run) + .group_by(wake.target_dag_id, wake.apdr_id) + .subquery() + ) + ranked = select( + runs, + func.row_number().over(partition_by=runs.c.dag_id, order_by=runs.c.first_id).label("turn"), + ).subquery() + # Each kind takes its first rows with a top-N sort of its own; ranking both kinds in one + # window would sort every group once more. SQLite rejects ORDER BY and LIMIT on the members + # of a compound select, so each kind is a subquery. + run_level = select(ranked).order_by(ranked.c.turn, ranked.c.first_id).limit(2 * budget).subquery() + dag_level = ( + select( + wake.target_dag_id.label("dag_id"), + sql_cast(null(), Integer).label("apdr_id"), + func.min(wake.id).label("first_id"), + func.min(wake.created_at).label("first_at"), + literal_column("1", Integer).label("turn"), + ) + .join(DagModel, DagModel.dag_id == wake.target_dag_id) + .where(wake.apdr_id.is_(None), *can_run) + .group_by(wake.target_dag_id) + .order_by(func.min(wake.id)) + .limit(len(self._partition_fan_out) + _PARTITION_FAN_OUT_STARTS_PER_LOOP) + .subquery() + ) + signals = union_all( + *( + select(kind.c.dag_id, kind.c.apdr_id, kind.c.first_at, kind.c.turn, kind.c.first_id) + for kind in (run_level, dag_level) + ) + ).subquery() + return [ + (dag_id, apdr_id, first_at) + for dag_id, apdr_id, first_at in session.execute( + select(signals.c.dag_id, signals.c.apdr_id, signals.c.first_at).order_by( + signals.c.turn, signals.c.first_id + ) + ) + ] + + def _start_partition_fan_outs( + self, dag_ids: list[str], loop: _PartitionLoop, *, session: Session + ) -> list[int]: + """ + Read the pending runs of Dags with a Dag-level wake row, to evaluate them in the next loops. + + The Dag-level rows are read before the pending runs and before any of them is evaluated, so + the evaluation sees the definition or asset they report. Return the Dag-level rows of Dags + that have no pending run, which are done already. + """ + if not ( + new := [dag_id for dag_id in dict.fromkeys(dag_ids) if dag_id not in self._partition_fan_out][ + :_PARTITION_FAN_OUT_STARTS_PER_LOOP + ] + ): + return [] + wake = AssetPartitionDagRunWake + wake_ids: dict[str, list[int]] = defaultdict(list) + pending: dict[str, list[int]] = defaultdict(list) + for batch in chunks(new, _PARTITION_IN_LIST_SIZE): + ranked = ( + select( + wake.id, + wake.target_dag_id, + func.row_number().over(partition_by=wake.target_dag_id, order_by=wake.id).label("turn"), + ) + .where(wake.target_dag_id.in_(batch), wake.apdr_id.is_(None)) + .subquery() + ) + for wake_id, dag_id in session.execute( + select(ranked.c.id, ranked.c.target_dag_id).where( + ranked.c.turn <= _PARTITION_FAN_OUT_WAKE_ROWS + ) + ): + wake_ids[dag_id].append(wake_id) + for batch in chunks(new, _PARTITION_IN_LIST_SIZE): + for apdr_id, dag_id in session.execute( + select(AssetPartitionDagRun.id, AssetPartitionDagRun.target_dag_id).where( + AssetPartitionDagRun.target_dag_id.in_(batch), + AssetPartitionDagRun.created_dag_run_id.is_(None), + ) + ): + pending[dag_id].append(apdr_id) + done: list[int] = [] + for dag_id in new: + if pending[dag_id]: + loop.fan_out_starts[dag_id] = (wake_ids[dag_id], pending[dag_id]) + else: + done.extend(wake_ids[dag_id]) + return done + + def _start_partition_resync_if_due(self, *, session: Session) -> None: + """ + Read every pending run, once a pass is due and none is running. + + The pass starts only once the read completes, so that a loop retried after a failed read + reads again. A pass started by a loop that is then retried is kept: the retried attempt + evaluates the same runs, because a run leaves the pass only once a committed loop is done + with it. + """ + if self._partition_resync or (now := time.monotonic()) < self._partition_next_resync_at: + return + ids_by_dag = self._read_pending_partition_dag_runs(session=session) + for dag_id in sorted(ids_by_dag): + self._partition_resync.add(sys.intern(dag_id), ids_by_dag.pop(dag_id)) + if self._partition_resync: + # A random start, so that a scheduler that restarts often does not favour the same Dags. + self._partition_resync.rotate(self._partition_rng.randrange(len(self._partition_resync))) + self._partition_next_resync_at = now + self._partition_resync_interval + self._partition_resync_kind = "periodic" if self._partition_resync_passes else "startup" + self._partition_resync_passes += 1 + self._partition_resync_started_at = now + + @staticmethod + def _read_pending_partition_dag_runs(*, session: Session) -> dict[str, list[int]]: + """Return the ids of every pending partition Dag run of a Dag that exists, per Dag.""" + ids_by_dag: dict[str, list[int]] = defaultdict(list) + for apdr_id, dag_id in session.execute( + # The runs of Dags that no longer exist are left to come back with their Dag, whose first + # definition then wakes them. Streamed: the result holds every pending run. + select(AssetPartitionDagRun.id, AssetPartitionDagRun.target_dag_id) + .join(DagModel, DagModel.dag_id == AssetPartitionDagRun.target_dag_id) + .where(AssetPartitionDagRun.created_dag_run_id.is_(None)) + .execution_options(yield_per=10_000) + ): + ids_by_dag[dag_id].append(apdr_id) + return ids_by_dag + + @staticmethod + def _read_dag_eligibility(dag_ids: Collection[str], *, session: Session) -> dict[str, bool]: + """Return, for each of *dag_ids* that exists, whether it can run: not paused, draining or stale.""" + can_run: dict[str, bool] = {} + for batch in chunks(sorted(dag_ids), _ADRQ_DELETE_CHUNK_SIZE): + can_run.update( + (dag_id, not (paused or draining or stale)) + for dag_id, paused, draining, stale in session.execute( + select( + DagModel.dag_id, DagModel.is_paused, DagModel.is_draining, DagModel.is_stale + ).where(DagModel.dag_id.in_(batch)) + ) + ) + return can_run + + @staticmethod + def _lock_pending_partition_dag_runs( + apdr_ids: list[int], *, session: Session + ) -> Iterable[AssetPartitionDagRun]: + # Rows another scheduler holds are skipped; they keep their wake rows. The Dag filter is checked + # again on the locked row's own Dag, because the Dag can stop running after the claim read it. + return session.scalars( with_row_locks( select(AssetPartitionDagRun) .join(DagModel, DagModel.dag_id == AssetPartitionDagRun.target_dag_id) .where( + AssetPartitionDagRun.id.in_(apdr_ids), AssetPartitionDagRun.created_dag_run_id.is_(None), DagModel.is_paused.is_(False), DagModel.is_draining.is_(False), DagModel.is_stale.is_(False), - ) - .order_by(AssetPartitionDagRun.created_at, AssetPartitionDagRun.id) - .limit(self._max_partition_dag_runs_per_loop), + ), of=AssetPartitionDagRun, skip_locked=True, key_share=False, session=session, ) - ).all() - if not pending_apdrs: - return set() + ) + + @staticmethod + def _write_partition_wakes(claim: _PartitionClaim, *, session: Session) -> None: + """ + Delete the wake rows this loop is done with, and write those the resync asks for. + + Deleting the rows of the runs the loop evaluated is atomic with the Dag runs it creates, so + these writes go in the loop's transaction rather than in batches committed apart. Each is + bounded per loop instead, and the next loops do the rest: + + * the rows of the runs the loop evaluated, at most ``_PARTITION_MAX_CONSUMED_WAKE_ROWS``; + * the rows of runs that fired or were deleted, at most ``_PARTITION_MAX_ORPHANED_WAKE_ROWS``; + * the Dag-level rows of the fan-outs the loop finished, at most + ``_PARTITION_FAN_OUT_WAKE_ROWS`` per fan-out; + * one Dag-level row per Dag that the resync found unable to run, at most one per Dag the loop + serves. + """ + wake = AssetPartitionDagRunWake + for batch in chunks(claim.consumed, _ADRQ_DELETE_CHUNK_SIZE): + session.execute(delete(wake).where(wake.id.in_(batch))) + # Another scheduler can delete the same orphan and Dag-level rows: skip the rows it holds + # rather than waiting for its loop to commit. + still_orphaned = ~exists().where( + AssetPartitionDagRun.id == wake.apdr_id, + AssetPartitionDagRun.target_dag_id == wake.target_dag_id, + AssetPartitionDagRun.created_dag_run_id.is_(None), + ) + for wake_ids, condition in ((claim.orphaned, still_orphaned), (claim.finished_wakes, None)): + for batch in chunks(wake_ids, _ADRQ_DELETE_CHUNK_SIZE): + query = select(wake.id).where(wake.id.in_(batch)) + if condition is not None: + query = query.where(condition) + if held := session.scalars( + with_row_locks(query, of=wake, skip_locked=True, key_share=False, session=session) + ).all(): + session.execute(delete(wake).where(wake.id.in_(held))) + if claim.resync_wakes: + now = timezone.utcnow() + session.execute( + insert(wake.__table__), + [ + {"target_dag_id": dag_id, "reason": PartitionWakeReason.RESYNC, "created_at": now} + for dag_id in claim.resync_wakes + ], + ) + def _evaluate_partition_dag_runs( + self, pending_apdrs: list[AssetPartitionDagRun], *, session: Session + ) -> set[str]: + """Evaluate the locked pending runs, oldest first, and create the Dag runs that are satisfied.""" # Pre-fetch all required serialized Dags in one query. The same map # serves the stale-version cleanup below and the downstream rollup # evaluation, so the table is only hit once per tick. @@ -2538,9 +3119,15 @@ def _create_dagruns_for_partitioned_asset_dags(self, session: Session) -> set[st # The fingerprint per dag is computed once and cached to avoid redundant # serialization when multiple APDRs share the same target dag. latest_fp_by_dag: dict[str, dict] = {} + # A Dag that is no longer partitioned is stale even when its fingerprint matches: a timetable + # scheduled on aliases only has the same empty fingerprint either way. + not_partitioned: set[str] = set() for dag_id, serdag in serdags_by_dag_id.items(): try: - latest_fp_by_dag[dag_id] = compute_rollup_fingerprint(serdag.dag.timetable) + timetable = serdag.dag.timetable + latest_fp_by_dag[dag_id] = compute_rollup_fingerprint(timetable) + if not timetable.partitioned: + not_partitioned.add(dag_id) except Exception: # If deserialization fails, skip rather than treating as stale — # a broken serdag should not silently wipe pending progress. @@ -2552,7 +3139,8 @@ def _create_dagruns_for_partitioned_asset_dags(self, session: Session) -> set[st if serdags_by_dag_id.get(apdr.target_dag_id) is not None and apdr.target_dag_id in latest_fp_by_dag # fingerprint failed to compute → skip and ( - apdr.rollup_fingerprint is None + apdr.target_dag_id in not_partitioned + or apdr.rollup_fingerprint is None or apdr.rollup_fingerprint != latest_fp_by_dag[apdr.target_dag_id] ) ] @@ -2708,8 +3296,122 @@ def _create_dagruns_for_dags(self, guard: CommitProhibitorGuard, session: Sessio # commit the session - Release the write lock on DagModel table. guard.commit() + self._commit_partition_loop() # END: create dagruns + def _commit_partition_loop(self) -> None: + """ + Keep what the committed loop changed in the fan-outs and the resync, and report its metrics. + + A loop that ``@retry_db_transaction`` retries never gets here, so the next attempt starts + from the state before it. + """ + if (loop := self._partition_loop) is None: + return + self._partition_loop = None + for dag_id, (wake_ids, pending) in loop.fan_out_starts.items(): + self._partition_fan_out.add(dag_id, pending) + self._partition_fan_out_wakes[dag_id] = wake_ids + self._partition_fan_out.commit( + done=loop.fan_out_done, dropped=loop.fan_out_dropped, served=loop.fan_out_served + ) + for dag_id in [d for d in self._partition_fan_out_wakes if d not in self._partition_fan_out]: + del self._partition_fan_out_wakes[dag_id] + self._partition_resync.commit( + done=loop.resync_done, dropped=loop.resync_dropped, served=loop.resync_served + ) + if self._partition_resync_kind and not self._partition_resync: + stats.timing( + "scheduler.asset_partitions.resync_pass_duration", + timedelta(seconds=time.monotonic() - self._partition_resync_started_at), + ) + self._partition_resync_kind = None + if loop.evaluated: + stats.incr("scheduler.asset_partitions.evaluated", loop.evaluated) + if loop.resync_fired: + stats.incr( + "scheduler.asset_partitions.resync_fired", + loop.resync_fired, + tags={"pass": loop.resync_kind or "periodic"}, + ) + if loop.oldest_signal_at is not None: + stats.gauge( + "scheduler.asset_partitions.wake_lag", + max((timezone.utcnow() - loop.oldest_signal_at).total_seconds(), 0.0), + ) + self._partition_wake_lag_reported = True + elif self._partition_wake_lag_reported: + stats.gauge("scheduler.asset_partitions.wake_lag", 0) + self._partition_wake_lag_reported = False + + def _clean_up_partition_wakes(self) -> None: + """ + Delete the wake rows of Dags that no longer exist, and merge those of Dags that cannot run. + + The claim reads only the rows of Dags that can run, so the rows of others would build up. + One Dag-level row replaces the rows of a paused, draining or stale Dag: once the Dag can run + again, the scheduler evaluates all its pending runs, which covers what the rows asked for. + Each batch commits on its own, rows another transaction holds are skipped, and one firing runs + at most a fixed number of batches. The Dags whose rows are oldest are merged first, so that a + Dag merged in a firing goes behind the others in the next. + """ + wake = AssetPartitionDagRunWake + batches = 0 + while batches < _PARTITION_WAKE_CLEANUP_BATCHES: + batches += 1 + with create_session() as session: + wake_ids = session.scalars( + with_row_locks( + select(wake.id) + .where(~exists().where(DagModel.dag_id == wake.target_dag_id)) + .order_by(wake.id) + .limit(_ADRQ_DELETE_CHUNK_SIZE), + of=wake, + skip_locked=True, + key_share=False, + session=session, + ) + ).all() + if wake_ids: + session.execute(delete(wake).where(wake.id.in_(wake_ids))) + if len(wake_ids) < _ADRQ_DELETE_CHUNK_SIZE: + break + if batches >= _PARTITION_WAKE_CLEANUP_BATCHES: + return + with create_session() as session: + dag_ids = session.scalars( + select(wake.target_dag_id) + .join(DagModel, DagModel.dag_id == wake.target_dag_id) + .where(or_(DagModel.is_paused, DagModel.is_draining, DagModel.is_stale)) + .group_by(wake.target_dag_id) + .having(func.count() > 1) + .order_by(func.min(wake.id)) + .limit(_PARTITION_WAKE_CLEANUP_BATCHES - batches) + ).all() + for dag_id in dag_ids: + while batches < _PARTITION_WAKE_CLEANUP_BATCHES: + batches += 1 + with create_session() as session: + wake_ids = session.scalars( + with_row_locks( + select(wake.id) + .where(wake.target_dag_id == dag_id) + .order_by(wake.id) + .limit(_ADRQ_DELETE_CHUNK_SIZE), + of=wake, + skip_locked=True, + key_share=False, + session=session, + ) + ).all() + if len(wake_ids) > 1: + # Written in the transaction that deletes the rows it replaces, so no request is + # lost; a later batch of the same Dag merges it again with the rest. + session.add(wake(target_dag_id=dag_id, reason=PartitionWakeReason.COALESCED)) + session.execute(delete(wake).where(wake.id.in_(wake_ids))) + if len(wake_ids) < _ADRQ_DELETE_CHUNK_SIZE: + break + @provide_session def _mark_backfills_complete(self, *, session: Session = NEW_SESSION) -> None: """Mark completed backfills as completed.""" @@ -4343,6 +5045,7 @@ def _activate_referenced_assets(assets_query: Subquery, *, session: Session) -> active_name_to_uri: dict[str, str] = {name: uri for name, uri in active_assets} active_uri_to_name: dict[str, str] = {uri: name for name, uri in active_assets} + activated: list[AssetModel] = [] def _generate_warning_message( offending: AssetModel, attr: str, value: str @@ -4392,6 +5095,7 @@ def _activate_assets_generate_warnings() -> Iterator[tuple[str, str]]: incoming_name_to_uri[asset.name] = asset.uri incoming_uri_to_name[asset.uri] = asset.name session.add(AssetActive.for_asset(asset)) + activated.append(asset) warnings_to_have = { dag_id: DagWarning( @@ -4403,6 +5107,8 @@ def _activate_assets_generate_warnings() -> Iterator[tuple[str, str]]: sorted(_activate_assets_generate_warnings()), key=operator.itemgetter(0) ) } + if activated: + AssetPartitionDagRunWake.add_for_activated_assets(activated, session=session) session.execute( delete(DagWarning).where( diff --git a/airflow-core/src/airflow/migrations/versions/0157_3_5_0_add_asset_partition_dag_run_wake.py b/airflow-core/src/airflow/migrations/versions/0157_3_5_0_add_asset_partition_dag_run_wake.py new file mode 100644 index 0000000000000..4b96dcdd353de --- /dev/null +++ b/airflow-core/src/airflow/migrations/versions/0157_3_5_0_add_asset_partition_dag_run_wake.py @@ -0,0 +1,61 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +""" +Add asset_partition_dag_run_wake. + +Revision ID: 6b0e4c1f2a37 +Revises: d8c6f3a129b0 +Create Date: 2026-11-08 12:00:00.000000 +""" + +from __future__ import annotations + +import sqlalchemy as sa +from alembic import op + +from airflow.models.base import StringID +from airflow.utils.sqlalchemy import UtcDateTime + +revision = "6b0e4c1f2a37" +down_revision = "d8c6f3a129b0" +branch_labels = None +depends_on = None +airflow_version = "3.5.0" + + +def upgrade(): + """Add the table through which writers ask the scheduler to evaluate pending partition Dag runs.""" + op.create_table( + "asset_partition_dag_run_wake", + sa.Column("id", sa.Integer(), autoincrement=True, nullable=False), + sa.Column("apdr_id", sa.Integer(), nullable=True), + sa.Column("target_dag_id", StringID(), nullable=False), + sa.Column("reason", sa.String(length=20), nullable=False), + sa.Column("created_at", UtcDateTime(), nullable=False), + sa.PrimaryKeyConstraint("id", name=op.f("asset_partition_dag_run_wake_pkey")), + sqlite_autoincrement=True, + ) + op.create_index("idx_apdr_wake_apdr_id", "asset_partition_dag_run_wake", ["apdr_id"]) + op.create_index( + "idx_apdr_wake_target_dag_id", "asset_partition_dag_run_wake", ["target_dag_id", "apdr_id"] + ) + + +def downgrade(): + """Drop asset_partition_dag_run_wake.""" + op.drop_table("asset_partition_dag_run_wake") diff --git a/airflow-core/src/airflow/models/asset.py b/airflow-core/src/airflow/models/asset.py index 4ddb46b00cd87..d3c635d26d89a 100644 --- a/airflow-core/src/airflow/models/asset.py +++ b/airflow-core/src/airflow/models/asset.py @@ -18,6 +18,7 @@ from __future__ import annotations from datetime import datetime +from enum import StrEnum from typing import TYPE_CHECKING from urllib.parse import urlsplit from uuid import UUID @@ -35,14 +36,18 @@ Table, UniqueConstraint, delete, + event, + insert, select, + union, ) from sqlalchemy.ext.associationproxy import association_proxy -from sqlalchemy.orm import Mapped, mapped_column, relationship, validates +from sqlalchemy.orm import Mapped, Session, mapped_column, object_session, relationship, validates from airflow._shared.timezones import timezone from airflow.configuration import conf as airflow_conf from airflow.models.base import Base, StringID +from airflow.utils.helpers import chunks from airflow.utils.session import NEW_SESSION, provide_session from airflow.utils.sqlalchemy import UtcDateTime, get_dialect_name @@ -51,7 +56,6 @@ from typing import Any, TypeAlias from sqlalchemy import Insert - from sqlalchemy.orm import Session from sqlalchemy.sql.elements import ColumnElement, SQLColumnExpression from airflow.models.dag import DagModel @@ -1055,3 +1059,131 @@ class PartitionedAssetKeyLog(Base): def __repr__(self): args = (f"{x.name}={getattr(self, x.name)!r}" for x in self.__mapper__.primary_key) return f"{self.__class__.__name__}({', '.join(args)})" + + +class PartitionWakeReason(StrEnum): + """Why an :class:`AssetPartitionDagRunWake` row was written.""" + + KEY_LOG = "key_log" + DAG_CHANGED = "dag_changed" + ASSET_ACTIVATED = "asset_activated" + RESYNC = "resync" + COALESCED = "coalesced" + + +class AssetPartitionDagRunWake(Base): + """ + Request for the scheduler to evaluate pending :class:`AssetPartitionDagRun` rows. + + The scheduler evaluates a pending partition Dag run when a row asks for it: one with its + ``apdr_id``, written with each key log, or one for its Dag with a null ``apdr_id``, written when + the Dag's timetable changes or an asset it is scheduled on is activated. A row is written in the + same transaction as the change it reports, and the scheduler deletes only the rows it read, by + primary key, so a row written while the scheduler evaluates the run survives and wakes it again. + + No foreign key and no unique key, so that writing a row never waits on the scheduler's lock of + the run, and never conflicts with a row the scheduler already read. + """ + + id: Mapped[int] = mapped_column(Integer, primary_key=True, autoincrement=True) + apdr_id: Mapped[int | None] = mapped_column(Integer, nullable=True) + target_dag_id: Mapped[str] = mapped_column(StringID(), nullable=False) + reason: Mapped[str] = mapped_column(String(20), nullable=False) + created_at: Mapped[datetime] = mapped_column(UtcDateTime, default=timezone.utcnow, nullable=False) + + __tablename__ = "asset_partition_dag_run_wake" + __table_args__ = ( + Index("idx_apdr_wake_apdr_id", apdr_id), + Index("idx_apdr_wake_target_dag_id", target_dag_id, apdr_id), + {"sqlite_autoincrement": True}, + ) + + @classmethod + def add_for_activated_assets(cls, assets: Iterable[AssetModel], *, session: Session) -> None: + """ + Ask the scheduler to evaluate the pending runs of every Dag scheduled on *assets* again. + + Key logs of an inactive asset do not count towards its runs, so activating the asset can + satisfy runs that already hold its key logs. + """ + dag_ids: set[str] = set() + # Each asset takes four bind parameters, its id twice, its name and its uri; SQLite before 3.32 + # allows only 999 per statement. + for chunk in chunks(list(assets), 999 // 4): + ids = [asset.id for asset in chunk] + dag_ids.update( + session.scalars( + union( + select(DagScheduleAssetReference.dag_id).where( + DagScheduleAssetReference.asset_id.in_(ids) + ), + select(DagScheduleAssetNameReference.dag_id).where( + DagScheduleAssetNameReference.name.in_({asset.name for asset in chunk}) + ), + select(DagScheduleAssetUriReference.dag_id).where( + DagScheduleAssetUriReference.uri.in_({asset.uri for asset in chunk}) + ), + select(DagScheduleAssetAliasReference.dag_id) + .join( + alias_association_table, + alias_association_table.c.alias_id == DagScheduleAssetAliasReference.alias_id, + ) + .where(alias_association_table.c.asset_id.in_(ids)), + ) + ) + ) + if dag_ids: + now = timezone.utcnow() + session.execute( + insert(cls.__table__), + [ + { + "target_dag_id": dag_id, + "reason": PartitionWakeReason.ASSET_ACTIVATED, + "created_at": now, + } + for dag_id in sorted(dag_ids) + ], + ) + + +_PARTITION_KEY_LOGS_FLUSHED = "airflow_partition_key_logs_flushed" + + +@event.listens_for(PartitionedAssetKeyLog, "after_insert") +def _collect_partition_key_log(mapper, connection, target: PartitionedAssetKeyLog) -> None: + if (session := object_session(target)) is not None: + session.info.setdefault(_PARTITION_KEY_LOGS_FLUSHED, []).append( + (target.asset_partition_dag_run_id, target.target_dag_id) + ) + + +@event.listens_for(Session, "before_flush") +def _reset_partition_key_logs(session: Session, flush_context, instances) -> None: + # A flush that failed after inserting key logs leaves them collected; they were rolled back. + session.info.pop(_PARTITION_KEY_LOGS_FLUSHED, None) + + +@event.listens_for(Session, "after_flush") +def _wake_partition_dag_runs(session: Session, flush_context) -> None: + """ + Write a wake row for every key log the flush inserted, in the same transaction. + + Every key log that a session flush inserts gets its row here, whichever asset manager adds it. Key + logs inserted with a Core or bulk insert get none; the scheduler's periodic resync evaluates their runs. + """ + if not (flushed := session.info.pop(_PARTITION_KEY_LOGS_FLUSHED, None)): + return + now = timezone.utcnow() + session.connection().execute( + insert(AssetPartitionDagRunWake.__table__), + [ + { + "apdr_id": apdr_id, + "target_dag_id": target_dag_id, + "reason": PartitionWakeReason.KEY_LOG, + "created_at": now, + } + for apdr_id, target_dag_id in flushed + ], + ) diff --git a/airflow-core/src/airflow/models/serialized_dag.py b/airflow-core/src/airflow/models/serialized_dag.py index 6cab8defb816b..785804ab0946d 100644 --- a/airflow-core/src/airflow/models/serialized_dag.py +++ b/airflow-core/src/airflow/models/serialized_dag.py @@ -28,7 +28,19 @@ from uuid import UUID import uuid6 -from sqlalchemy import JSON, ForeignKey, Index, LargeBinary, String, Uuid, exists, select, tuple_, update +from sqlalchemy import ( + JSON, + ForeignKey, + Index, + LargeBinary, + String, + Uuid, + event, + exists, + select, + tuple_, + update, +) from sqlalchemy.dialects.postgresql import ARRAY, JSONB from sqlalchemy.orm import Mapped, backref, foreign, mapped_column, relationship from sqlalchemy.sql.expression import func, literal @@ -39,6 +51,9 @@ from airflow.models.asset import ( AssetAliasModel, AssetModel, + AssetPartitionDagRun, + AssetPartitionDagRunWake, + PartitionWakeReason, ) from airflow.models.base import ID_LEN, Base from airflow.models.dag import DagModel @@ -62,6 +77,7 @@ from sqlalchemy.orm.attributes import InstrumentedAttribute from sqlalchemy.sql import Select from sqlalchemy.sql.elements import ColumnElement + from sqlalchemy.sql.selectable import Exists from airflow.sdk.importers import DagSourceCode # noqa: SDK001 from airflow.serialization.definitions.dag import SerializedDAG @@ -728,18 +744,29 @@ def write_dag( return False has_task_instances: bool = False + has_pending_partition_dag_runs: bool | None = None if dag_version: - has_task_instances = bool( - session.scalar( + # Both lookups in one round trip: rewriting a Dag counts against the Dag processor's + # statement budget per file. + has_task_instances, has_pending_partition_dag_runs = ( + bool(found) + for found in session.execute( select( exists().where( # Using dag_id filter to speed up query via the composite index. TaskInstance.dag_id == dag.dag_id, TaskInstance.dag_version_id == dag_version.id, - ) + ), + cls._build_pending_partition_dag_runs_check(dag.dag_id), ) - ) + ).one() ) + wake_partition_dag_runs = serialized_dag_hash != new_dag_hash and cls._changes_partition_evaluation( + dag, + dag_version, + has_pending_partition_dag_runs=has_pending_partition_dag_runs, + session=session, + ) if dag_version and not has_task_instances: # This is for dynamic DAGs that the hashes changes often. We should update @@ -763,6 +790,8 @@ def write_dag( if getattr(result, "rowcount", 0) == 0: # No rows updated - serialized DAG doesn't exist return False + if wake_partition_dag_runs: + cls._wake_partition_dag_runs(dag.dag_id, session=session) if deadline_uuid_mapping: updated_serialized_dag = session.scalar( @@ -810,6 +839,8 @@ def write_dag( new_serialized_dag = cls(dag, _dag_hash=new_dag_hash) new_serialized_dag.dag_version = dagv session.add(new_serialized_dag) + if wake_partition_dag_runs: + cls._wake_partition_dag_runs(dag.dag_id, session=session) cls._create_deadline_alert_records(new_serialized_dag, deadline_uuid_mapping) log.debug("DAG: %s written to the DB", dag.dag_id) @@ -820,6 +851,58 @@ def write_dag( ) return True + @staticmethod + def _build_pending_partition_dag_runs_check(dag_id: str) -> Exists: + return exists().where( + AssetPartitionDagRun.target_dag_id == dag_id, + AssetPartitionDagRun.created_dag_run_id.is_(None), + ) + + @classmethod + def _changes_partition_evaluation( + cls, + dag: LazyDeserializedDAG, + dag_version: DagVersion | None, + *, + has_pending_partition_dag_runs: bool | None, + session: Session, + ) -> bool: + """ + Return whether writing *dag* can change the evaluation of its pending partition Dag runs. + + Evaluation and the stale-run cleanup read only the timetable, which holds the asset condition + and the partition mappers, so a change anywhere else in the Dag cannot change their result. + Whether the Dag has pending runs is looked up here when the caller passes None. + """ + if not dag.timetable.partitioned: + if has_pending_partition_dag_runs is None: + has_pending_partition_dag_runs = bool( + session.scalar(select(cls._build_pending_partition_dag_runs_check(dag.dag_id))) + ) + if not has_pending_partition_dag_runs: + return False + if dag_version is None: + return True + previous = session.execute( + select(cls._data, cls._data_compressed).where(cls.dag_version_id == dag_version.id) + ).one_or_none() + if previous is None: + return True + data, data_compressed = previous + if data_compressed: + previous_data = json.loads(zlib.decompress(data_compressed)) + elif isinstance(data, str): + previous_data = json.loads(data) + else: + previous_data = data or {} + return previous_data.get("dag", {}).get("timetable") != dag.data.get("dag", {}).get("timetable") + + @staticmethod + def _wake_partition_dag_runs(dag_id: str, *, session: Session) -> None: + # Written with the definition, so the scheduler reads this definition, or a later one, + # whenever it reads the row. + session.add(AssetPartitionDagRunWake(target_dag_id=dag_id, reason=PartitionWakeReason.DAG_CHANGED)) + @classmethod def latest_item_select_object(cls, dag_id): from airflow.settings import engine @@ -1021,3 +1104,15 @@ def load_json(deps_data): resolver = _DagDependenciesResolver(dag_id_dependencies=dag_depdendencies, session=session) dag_depdendencies_by_dag = resolver.resolve() return dag_depdendencies_by_dag + + def _drop_decoded_data(self) -> None: + self.__data_cache = None + + +@event.listens_for(SerializedDagModel, "expire") +def _drop_decoded_data_on_expire(target: SerializedDagModel | None, attrs) -> None: + # Expiring reloads the columns but not the data decoded from them. Without this, a session that + # rolls back and reads the row again after an in-place write would keep serving the old data. + # The target is None for an instance that is already garbage collected. + if target is not None: + target._drop_decoded_data() diff --git a/airflow-core/src/airflow/utils/db.py b/airflow-core/src/airflow/utils/db.py index 5e88b05c137b9..264956d8b0f50 100644 --- a/airflow-core/src/airflow/utils/db.py +++ b/airflow-core/src/airflow/utils/db.py @@ -112,7 +112,7 @@ class MappedClassProtocol(Protocol): "3.2.0": "1d6611b6ab7c", "3.3.0": "d2f4e1b3c5a7", "3.4.0": "b4f1a7d2c903", - "3.5.0": "d8c6f3a129b0", + "3.5.0": "6b0e4c1f2a37", } # Prefix used to identify tables holding data moved during migration. diff --git a/airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_assets.py b/airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_assets.py index f964cd7a8ed61..8a52108b18b52 100644 --- a/airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_assets.py +++ b/airflow-core/tests/unit/api_fastapi/core_api/routes/ui/test_assets.py @@ -453,11 +453,11 @@ def test_partitioned_dag_last_update( def test_pending_apdr_uses_fifo_order(self, test_client, dag_maker, session): """ - The next-run view must surface the oldest pending APDR so it matches the - one the scheduler will fire next (``_create_dagruns_for_partitioned_asset_dags`` - is strict FIFO over ``created_at``). When multiple partitions are - backlogged, showing the newest would confuse the operator about which - run is actually queued. + The next-run view must surface the pending APDR that has waited longest. + + When multiple partitions are backlogged, showing the newest would hide the one + that has been queued longest. It is not necessarily the next to fire: the + scheduler fires whichever pending APDR is satisfied, whatever its age. """ asset = Asset(uri="s3://bucket/fifo", name="fifo") with dag_maker( diff --git a/airflow-core/tests/unit/dag_processing/test_collection.py b/airflow-core/tests/unit/dag_processing/test_collection.py index f8834b57095f1..8bb0c55ffff28 100644 --- a/airflow-core/tests/unit/dag_processing/test_collection.py +++ b/airflow-core/tests/unit/dag_processing/test_collection.py @@ -55,6 +55,7 @@ from airflow.models.asset import ( AssetActive, AssetModel, + AssetPartitionDagRunWake, DagScheduleAssetNameReference, DagScheduleAssetUriReference, ) @@ -91,10 +92,12 @@ from airflow.timetables.simple import PartitionedAtRuntime from airflow.timetables.trigger import CronTriggerTimetable from airflow.triggers.base import BaseEventTrigger +from airflow.utils.session import create_session from airflow.utils.types import DagRunType from tests_common.test_utils.config import conf_vars from tests_common.test_utils.db import ( + clear_db_apdr, clear_db_assets, clear_db_dag_bundles, clear_db_dags, @@ -619,6 +622,7 @@ def clean_db(): clear_db_dags() clear_db_assets() clear_db_triggers() + clear_db_apdr() @pytest.fixture(autouse=True) def per_test(self) -> Generator: @@ -688,6 +692,59 @@ def test_add_asset_activate_conflict(self, dag_maker, session, existing_assets): session.flush() assert orm_assets["myasset", "file://myasset/"].active is None, "should not activate due to conflict" + @staticmethod + def _sync_and_activate(dag) -> None: + # The Dag processor syncs the assets of each file in a session of its own. + with create_session(scoped=False) as parse_session: + asset_op = AssetModelOperation.collect({dag.dag_id: LazyDeserializedDAG.from_dag(dag)}) + orm_assets = asset_op.sync_assets(session=parse_session) + parse_session.flush() + asset_op.activate_assets_if_possible(orm_assets.values(), session=parse_session) + + @staticmethod + def _read_dag_level_wakes(session) -> list[tuple[str, str]]: + wake = AssetPartitionDagRunWake + return [ + (dag_id, reason) + for dag_id, reason in session.execute( + select(wake.target_dag_id, wake.reason).where(wake.apdr_id.is_(None)) + ) + ] + + def test_activating_an_inactive_asset_wakes_the_dags_scheduled_on_it(self, dag_maker, session): + with dag_maker(dag_id="consumer", schedule=[Asset(name="myasset", uri="file://myasset/")]) as dag: + EmptyOperator(task_id="mytask") + session.execute(delete(AssetActive)) + session.commit() + + self._sync_and_activate(dag) + + assert self._read_dag_level_wakes(session) == [("consumer", "asset_activated")] + + def test_activating_a_new_or_an_active_asset_wakes_no_dag(self, dag_maker, session): + with dag_maker(dag_id="consumer", schedule=[Asset(name="myasset", uri="file://myasset/")]) as dag: + EmptyOperator(task_id="mytask") + session.commit() + assert self._read_dag_level_wakes(session) == [] + + self._sync_and_activate(dag) + + assert self._read_dag_level_wakes(session) == [] + + def test_asset_a_conflict_keeps_inactive_wakes_no_dag(self, dag_maker, session): + with dag_maker(dag_id="consumer", schedule=[Asset(name="myasset", uri="file://myasset/")]) as dag: + EmptyOperator(task_id="mytask") + session.execute(delete(AssetActive)) + conflicting = SerializedAsset("myasset", "file://different/asset", "", {}, []) + session.add(AssetModel.from_serialized(conflicting)) + session.flush() + session.add(AssetActive.for_asset(conflicting)) + session.commit() + + self._sync_and_activate(dag) + + assert self._read_dag_level_wakes(session) == [] + @pytest.mark.need_serialized_dag @pytest.mark.db_test diff --git a/airflow-core/tests/unit/jobs/test_scheduler_job.py b/airflow-core/tests/unit/jobs/test_scheduler_job.py index b15a82ace30a5..d116d2264abcd 100644 --- a/airflow-core/tests/unit/jobs/test_scheduler_job.py +++ b/airflow-core/tests/unit/jobs/test_scheduler_job.py @@ -20,7 +20,9 @@ import contextlib import datetime import logging +import math import os +import random import re import time from collections import Counter, deque @@ -39,9 +41,22 @@ import psutil import pytest import time_machine -from sqlalchemy import Delete, delete, event as sa_event, func, inspect, select, update +from sqlalchemy import ( + Delete, + create_engine, + delete, + event as sa_event, + func, + insert, + inspect, + select, + text, + update, +) from sqlalchemy.dialects import mysql -from sqlalchemy.orm import joinedload +from sqlalchemy.exc import OperationalError +from sqlalchemy.orm import joinedload, sessionmaker +from sqlalchemy.pool import NullPool from airflow import settings from airflow._shared.module_loading import qualname @@ -73,6 +88,8 @@ MAX_TI_REFILL_QUERIES_PER_LOOP, TASK_STUCK_IN_QUEUED_RESCHEDULE_EVENT, SchedulerJobRunner, + _PartitionLoop, + _PartitionSweep, ) from airflow.listeners.listener import get_listener_manager from airflow.models.asset import ( @@ -82,8 +99,10 @@ AssetEvent, AssetModel, AssetPartitionDagRun, + AssetPartitionDagRunWake, DagScheduleAssetReference, PartitionedAssetKeyLog, + PartitionWakeReason, ) from airflow.models.backfill import Backfill, BackfillDagRun, ReprocessBehavior, _create_backfill from airflow.models.callback import Callback, ExecutorCallback @@ -166,7 +185,7 @@ from airflow.triggers.base import AssetTriggerEvent, TriggerEvent from airflow.utils.event_scheduler import EventScheduler from airflow.utils.session import NEW_SESSION, create_session, provide_session -from airflow.utils.sqlalchemy import CommitProhibitorGuard, with_row_locks +from airflow.utils.sqlalchemy import CommitProhibitorGuard, prohibit_commit, with_row_locks from airflow.utils.state import CallbackState, DagRunState, DagSchedulingState, State, TaskInstanceState from airflow.utils.types import DagRunTriggeredByType, DagRunType @@ -4666,6 +4685,21 @@ def test_dag_processor_maintenance_runs_on_the_parsing_cleanup_interval(self, ca call_regular_interval.assert_any_call(mock.ANY, 123.0, cleanup_processor_metadata, non_fatal=True) + @conf_vars({("scheduler", "parsing_cleanup_interval"): "123"}) + @mock.patch.object(EventScheduler, "call_regular_interval", autospec=True) + def test_partition_wake_cleanup_runs_on_the_parsing_cleanup_interval(self, call_regular_interval): + scheduler_job = Job() + scheduler_job.heartbeat = mock.create_autospec(scheduler_job.heartbeat) + self.job_runner = SchedulerJobRunner( + job=scheduler_job, num_runs=1, executors=[MockExecutor(do_update=False)] + ) + + self.job_runner._run_scheduler_loop() + + call_regular_interval.assert_any_call( + mock.ANY, 123.0, self.job_runner._clean_up_partition_wakes, non_fatal=True + ) + def test_adopt_or_reset_orphaned_tasks(self, dag_maker, session): with dag_maker("test_execute_helper_reset_orphaned_tasks", session=session): op1 = EmptyOperator(task_id="op1") @@ -14327,13 +14361,17 @@ def _patch() -> ExitStack: @pytest.fixture def clear_asset_partition_rows() -> Iterator: + # dag_maker leaves the assets of its Dags behind. A test that deactivates assets, followed by one that + # declares the same assets again, would see wake rows for their activation. clear_db_apdr() clear_db_pakl() + clear_db_assets() yield clear_db_apdr() clear_db_pakl() + clear_db_assets() def _produce_and_register_asset_event( @@ -15817,14 +15855,22 @@ def test_partitioned_dag_run_rollup_survives_scheduler_restart_partial_arrival( assert partition_dags == {"rollup-consumer"} -@pytest.mark.db_test +@pytest.mark.need_serialized_dag @pytest.mark.usefixtures("clear_asset_partition_rows") -def test_pending_apdr_select_locks_rows_for_skip_locked_claim(session: Session): +def test_pending_apdr_select_locks_rows_for_skip_locked_claim(dag_maker: DagMaker, session: Session): """ The pending-APDR select must wrap the query in ``with_row_locks(skip_locked=True)`` so HA scheduler replicas don't both grab the same satisfied APDR and race the ``created_dag_run_id`` UPDATE (which would orphan whichever DagRun loses the race). """ + # The claim runs its locking query only when it has a pending run to evaluate. + _make_n_satisfied_apdrs( + consumer_dag_id="locked-consumer", + asset=Asset(name="locked-asset"), + partition_keys=["k"], + session=session, + dag_maker=dag_maker, + ) runner = SchedulerJobRunner( job=Job(job_type=SchedulerJobRunner.job_type), executors=[MockExecutor(do_update=False)] ) @@ -15843,33 +15889,6 @@ def test_pending_apdr_select_locks_rows_for_skip_locked_claim(session: Session): assert call.kwargs["session"] is session -@pytest.mark.db_test -@pytest.mark.usefixtures("clear_asset_partition_rows") -def test_pending_apdr_select_breaks_tie_on_id(session: Session): - """ - Two APDRs sharing a ``created_at`` (possible under bulk asset-event ingestion) - must be ordered by ``id`` ascending so concurrent HA scheduler ticks pick the - same LIMIT slice — otherwise replicas can disagree on which APDRs the LIMIT - selects even when the row lock would otherwise prevent the duplicate fire. - """ - shared_ts = timezone.utcnow() - first_inserted = AssetPartitionDagRun(target_dag_id="d1", partition_key="b", created_at=shared_ts) - session.add(first_inserted) - session.flush() - second_inserted = AssetPartitionDagRun(target_dag_id="d2", partition_key="a", created_at=shared_ts) - session.add(second_inserted) - session.flush() - - fetched = session.scalars( - select(AssetPartitionDagRun) - .where(AssetPartitionDagRun.created_dag_run_id.is_(None)) - .order_by(AssetPartitionDagRun.created_at, AssetPartitionDagRun.id) - ).all() - - assert first_inserted.id < second_inserted.id - assert [a.id for a in fetched] == [first_inserted.id, second_inserted.id] - - @pytest.mark.need_serialized_dag @pytest.mark.usefixtures("clear_asset_partition_rows") @pytest.mark.parametrize( @@ -16267,6 +16286,7 @@ def _make_n_satisfied_apdrs( partition_keys: list[str], session: Session, dag_maker: DagMaker, + producer_prefix: str = "asset-event-producer", ) -> list[AssetPartitionDagRun]: """Build a consumer Dag plus *N* satisfied APDRs (one per partition_key).""" with dag_maker( @@ -16282,7 +16302,7 @@ def _make_n_satisfied_apdrs( return [ _produce_and_register_asset_event( - dag_id=f"asset-event-producer-{i}", + dag_id=f"{producer_prefix}-{i}", asset=asset, partition_key=key, session=session, @@ -16299,7 +16319,7 @@ def test_partition_cap_at_exactly_n_processes_all(dag_maker: DagMaker, session: cap == APDR count: every pending APDR fires in a single tick. Pairs with :func:`test_partition_cap_at_n_minus_one_leaves_one_pending` to pin - the ``LIMIT`` boundary against off-by-one regressions (``>`` vs ``>=``). + the per-loop budget against off-by-one regressions (``>`` vs ``>=``). """ apdrs = _make_n_satisfied_apdrs( consumer_dag_id="cap-consumer-n", @@ -16328,10 +16348,11 @@ def test_partition_cap_at_n_minus_one_leaves_one_pending(dag_maker: DagMaker, se """ cap == APDR count - 1: oldest cap APDRs fire, the newest remains pending. - FIFO ordering means the *last-created* APDR is the one left for the next - tick. Together with :func:`test_partition_cap_at_exactly_n_processes_all` - this pins the ``LIMIT`` boundary — a ``LIMIT N-1`` regression would still - pass the at-cap test, but the older sibling here would fail to fire. + Within a Dag the oldest request for evaluation comes first, so the *last-created* + APDR is the one left for the next tick. Together with + :func:`test_partition_cap_at_exactly_n_processes_all` this pins the per-loop budget — a + budget of N-1 would still pass the at-cap test, but the older sibling here would fail + to fire. """ apdrs = _make_n_satisfied_apdrs( consumer_dag_id="cap-consumer-n-minus-one", @@ -16340,13 +16361,6 @@ def test_partition_cap_at_n_minus_one_leaves_one_pending(dag_maker: DagMaker, se session=session, dag_maker=dag_maker, ) - # Pin created_at so FIFO ordering is stable across fast in-memory runs — - # otherwise three APDRs created in the same microsecond tie under - # ``ORDER BY created_at`` and "which two fire" becomes nondeterministic. - base = timezone.utcnow() - for i, apdr in enumerate(apdrs): - apdr.created_at = base + timedelta(seconds=i) - session.commit() runner = SchedulerJobRunner( job=Job(job_type=SchedulerJobRunner.job_type), executors=[MockExecutor(do_update=False)] @@ -16408,11 +16422,13 @@ def test_partitioned_dag_run_skips_when_asset_is_inactive(dag_maker: DagMaker, s @pytest.mark.usefixtures("clear_asset_partition_rows") def test_partitioned_dag_run_resumes_when_asset_reactivates(dag_maker: DagMaker, session: Session): """ - A frozen APDR fires automatically once its asset is reactivated. + A frozen APDR fires once its asset is reactivated, at the latest in the next resync pass. Pairs with :func:`test_partitioned_dag_run_skips_when_asset_is_inactive` to pin - the freeze ↔ resume contract: deactivation halts fire, reactivation resumes it - without any other state change. + the freeze ↔ resume contract: deactivation halts fire, reactivation resumes it. The + scheduler and the Dag processor ask for an evaluation when they activate an asset + (:func:`test_activating_an_asset_wakes_the_pending_runs_frozen_on_it`); an + ``AssetActive`` row inserted any other way waits for the resync. """ asset = Asset(name="asset-reactivate") [apdr] = _make_n_satisfied_apdrs( @@ -16424,19 +16440,1270 @@ def test_partitioned_dag_run_resumes_when_asset_reactivates(dag_maker: DagMaker, ) _set_asset_active(name=asset.name, uri=asset.uri, session=session, active=False) + runner = _make_partition_runner(startup_pass=False) + _run_partition_loops(runner, session) + assert not _is_fired(apdr, session) + + _set_asset_active(name=asset.name, uri=asset.uri, session=session, active=True) + _run_partition_loops(runner, session) + assert not _is_fired(apdr, session) + + runner._partition_next_resync_at = 0.0 + _run_partition_loops(runner, session) + assert _is_fired(apdr, session) + + +# --------------------------------------------------------------------------- +# Signal-driven evaluation of pending partition Dag runs. +# +# The scheduler evaluates a pending run when an AssetPartitionDagRunWake row asks +# for it, and every pending run in its startup and periodic resync passes. These +# tests drive whole loops, because the claim keeps its in-memory progress only +# once a loop commits. +# --------------------------------------------------------------------------- + + +def _make_partition_runner(*, startup_pass: bool = True, per_loop: int | None = None) -> SchedulerJobRunner: runner = SchedulerJobRunner( job=Job(job_type=SchedulerJobRunner.job_type), executors=[MockExecutor(do_update=False)] ) - partition_dags = runner._create_dagruns_for_partitioned_asset_dags(session=session) - session.refresh(apdr) - assert apdr.created_dag_run_id is None - assert partition_dags == set() + if not startup_pass: + # No resync pass is due, so only the wake rows decide which pending runs are evaluated. + runner._partition_next_resync_at = math.inf + if per_loop is not None: + runner._max_partition_dag_runs_per_loop = per_loop + return runner - _set_asset_active(name=asset.name, uri=asset.uri, session=session, active=True) - partition_dags = runner._create_dagruns_for_partitioned_asset_dags(session=session) + +def _run_partition_loops(runner: SchedulerJobRunner, session: Session, loops: int = 1) -> None: + """Run the Dag run creation step of *loops* scheduler loops, each of which commits.""" + for _ in range(loops): + with prohibit_commit(session) as guard: + runner._create_dagruns_for_dags(guard, session) + + +def _read_wakes(session: Session) -> list[tuple[int | None, str, str]]: + wake = AssetPartitionDagRunWake + return [ + (apdr_id, dag_id, reason) + for apdr_id, dag_id, reason in session.execute( + select(wake.apdr_id, wake.target_dag_id, wake.reason).order_by(wake.id) + ) + ] + + +def _make_blocked_apdrs( + *, consumer_dag_id: str, partition_keys: list[str], session: Session, dag_maker: DagMaker +) -> list[AssetPartitionDagRun]: + """Build a consumer Dag on ``a & b`` plus one pending APDR per key that only ``a`` has reached.""" + asset_a = Asset(name=f"{consumer_dag_id}-a") + asset_b = Asset(name=f"{consumer_dag_id}-b") + with dag_maker( + dag_id=consumer_dag_id, + schedule=PartitionedAssetTimetable( + assets=asset_a & asset_b, default_partition_mapper=IdentityMapper() + ), + session=session, + ): + EmptyOperator(task_id="hi") + session.commit() + return [ + _produce_and_register_asset_event( + dag_id=f"{consumer_dag_id}-producer-{i}", + asset=asset_a, + partition_key=key, + session=session, + dag_maker=dag_maker, + ) + for i, key in enumerate(partition_keys, start=1) + ] + + +def _widen_to_either_asset(consumer_dag_id: str, *, session: Session) -> None: + """Rewrite a ``_make_blocked_apdrs`` consumer on ``a | b``, through the Dag processor's write.""" + with DAG( + dag_id=consumer_dag_id, + schedule=PartitionedAssetTimetable( + assets=Asset(name=f"{consumer_dag_id}-a") | Asset(name=f"{consumer_dag_id}-b"), + default_partition_mapper=IdentityMapper(), + ), + start_date=DEFAULT_DATE, + ) as dag: + EmptyOperator(task_id="hi") + sync_dag_to_db(dag, session=session) + session.commit() + + +def _is_fired(apdr: AssetPartitionDagRun, session: Session) -> bool: session.refresh(apdr) - assert apdr.created_dag_run_id is not None - assert partition_dags == {"reactivate-asset-consumer"} + return apdr.created_dag_run_id is not None + + +def _delete_wakes(session: Session) -> None: + """Drop every wake row, as for key logs written by a writer that writes none.""" + session.execute(delete(AssetPartitionDagRunWake)) + session.commit() + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +def test_partition_signal_of_a_dag_is_not_held_back_by_older_signals_of_another( + dag_maker: DagMaker, session: Session +): + blocked = _make_blocked_apdrs( + consumer_dag_id="waiting-consumer", + partition_keys=[f"k{i}" for i in range(5)], + session=session, + dag_maker=dag_maker, + ) + [ready] = _make_n_satisfied_apdrs( + consumer_dag_id="ready-consumer", + asset=Asset(name="ready-asset"), + partition_keys=["k"], + session=session, + dag_maker=dag_maker, + ) + runner = _make_partition_runner(startup_pass=False, per_loop=3) + + _run_partition_loops(runner, session) + + assert _is_fired(ready, session) + assert not any(_is_fired(apdr, session) for apdr in blocked) + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +def test_pending_partition_dag_run_without_a_signal_waits_for_the_periodic_resync( + dag_maker: DagMaker, session: Session +): + [apdr] = _make_n_satisfied_apdrs( + consumer_dag_id="unsignalled-consumer", + asset=Asset(name="unsignalled-asset"), + partition_keys=["k"], + session=session, + dag_maker=dag_maker, + ) + _delete_wakes(session) + runner = _make_partition_runner(startup_pass=False) + + _run_partition_loops(runner, session, loops=2) + assert not _is_fired(apdr, session) + + runner._partition_next_resync_at = 0.0 + _run_partition_loops(runner, session) + assert _is_fired(apdr, session) + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +def test_startup_pass_evaluates_pending_partition_dag_runs_without_a_signal( + dag_maker: DagMaker, session: Session +): + [apdr] = _make_n_satisfied_apdrs( + consumer_dag_id="startup-consumer", + asset=Asset(name="startup-asset"), + partition_keys=["k"], + session=session, + dag_maker=dag_maker, + ) + _delete_wakes(session) + + _run_partition_loops(_make_partition_runner(), session) + + assert _is_fired(apdr, session) + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +def test_wake_row_written_while_its_run_is_evaluated_survives(dag_maker: DagMaker, session: Session): + [apdr] = _make_blocked_apdrs( + consumer_dag_id="late-wake-consumer", partition_keys=["k"], session=session, dag_maker=dag_maker + ) + runner = _make_partition_runner(startup_pass=False) + evaluate = SchedulerJobRunner._evaluate_partition_dag_runs + late_wake_ids: list[int] = [] + + def evaluate_while_a_producer_writes(self, pending_apdrs, *, session): + with create_session(scoped=False) as producer: + late = AssetPartitionDagRunWake( + apdr_id=apdr.id, target_dag_id=apdr.target_dag_id, reason=PartitionWakeReason.KEY_LOG + ) + producer.add(late) + producer.flush() + late_wake_ids.append(late.id) + return evaluate(self, pending_apdrs, session=session) + + with mock.patch.object( + SchedulerJobRunner, + "_evaluate_partition_dag_runs", + autospec=True, + side_effect=evaluate_while_a_producer_writes, + ): + _run_partition_loops(runner, session) + + assert session.scalars(select(AssetPartitionDagRunWake.id)).all() == late_wake_ids + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +def test_partition_signals_of_dags_that_cannot_run_wait_without_holding_back_other_dags( + dag_maker: DagMaker, session: Session +): + """The requests of Dags that cannot run are kept, and do not take the loop from those of other Dags.""" + paused = [ + _make_n_satisfied_apdrs( + consumer_dag_id=f"paused-consumer-{i}", + asset=Asset(name=f"paused-asset-{i}"), + partition_keys=["k"], + session=session, + dag_maker=dag_maker, + producer_prefix=f"paused-producer-{i}", + )[0] + for i in range(2) + ] + session.execute(update(DagModel).where(DagModel.dag_id.like("paused-consumer-%")).values(is_paused=True)) + [running] = _make_n_satisfied_apdrs( + consumer_dag_id="running-consumer", + asset=Asset(name="running-asset"), + partition_keys=["k"], + session=session, + dag_maker=dag_maker, + producer_prefix="running-producer", + ) + runner = _make_partition_runner(startup_pass=False, per_loop=1) + + _run_partition_loops(runner, session) + assert _is_fired(running, session) + assert not any(_is_fired(apdr, session) for apdr in paused) + assert _read_wakes(session) == [ + (paused[0].id, "paused-consumer-0", "key_log"), + (paused[1].id, "paused-consumer-1", "key_log"), + ] + + session.execute(update(DagModel).where(DagModel.dag_id.like("paused-consumer-%")).values(is_paused=False)) + session.commit() + _run_partition_loops(runner, session, loops=2) + assert all(_is_fired(apdr, session) for apdr in paused) + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +@mock.patch("airflow.jobs.scheduler_job_runner._PARTITION_MAX_ORPHANED_WAKE_ROWS", 1) +def test_wake_rows_of_runs_that_fired_or_were_deleted_are_deleted(dag_maker: DagMaker, session: Session): + fired, deleted = _make_blocked_apdrs( + consumer_dag_id="orphan-consumer", partition_keys=["k1", "k2"], session=session, dag_maker=dag_maker + ) + deleted_id = deleted.id + session.execute( + update(AssetPartitionDagRun) + .where(AssetPartitionDagRun.id == fired.id) + .values(created_dag_run_id=session.scalar(select(DagRun.id).limit(1)), pending_partition_key=None) + ) + session.execute(delete(AssetPartitionDagRun).where(AssetPartitionDagRun.id == deleted_id)) + session.commit() + assert len(_read_wakes(session)) == 2 + runner = _make_partition_runner(startup_pass=False) + + # One row per loop, the oldest first. + _run_partition_loops(runner, session) + assert _read_wakes(session) == [(deleted_id, "orphan-consumer", "key_log")] + + _run_partition_loops(runner, session) + assert _read_wakes(session) == [] + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +@pytest.mark.parametrize("reusing_dag_can_run", [True, False]) +def test_wake_row_of_a_reused_run_id_does_not_hold_back_the_run_that_reuses_it( + dag_maker: DagMaker, session: Session, reusing_dag_can_run: bool +): + """SQLite gives a deleted run's id to the next run, maybe of another Dag; its old wake rows remain.""" + [apdr] = _make_n_satisfied_apdrs( + consumer_dag_id="reusing-consumer", + asset=Asset(name="reusing-asset"), + partition_keys=["k"], + session=session, + dag_maker=dag_maker, + ) + _make_blocked_apdrs(consumer_dag_id="old-owner", partition_keys=[], session=session, dag_maker=dag_maker) + # The deleted run's row is older than the row of the run that took its id, so it is read first. + session.add(AssetPartitionDagRunWake(id=-1, apdr_id=apdr.id, target_dag_id="old-owner", reason="key_log")) + session.commit() + assert _read_wakes(session) == [ + (apdr.id, "old-owner", "key_log"), + (apdr.id, "reusing-consumer", "key_log"), + ] + session.execute( + update(DagModel) + .where(DagModel.dag_id == "reusing-consumer") + .values(is_paused=not reusing_dag_can_run) + ) + session.commit() + + _run_partition_loops(_make_partition_runner(startup_pass=False), session) + + assert _is_fired(apdr, session) is reusing_dag_can_run + # The old owner's row is dropped either way; the reusing run's row waits while its Dag cannot run. + assert _read_wakes(session) == ([] if reusing_dag_can_run else [(apdr.id, "reusing-consumer", "key_log")]) + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +def test_timetable_change_wakes_every_pending_run_of_the_dag(dag_maker: DagMaker, session: Session): + blocked = _make_blocked_apdrs( + consumer_dag_id="widened-consumer", + partition_keys=[f"k{i}" for i in range(5)], + session=session, + dag_maker=dag_maker, + ) + runner = _make_partition_runner(startup_pass=False, per_loop=2) + _run_partition_loops(runner, session, loops=3) + assert _read_wakes(session) == [] + + _widen_to_either_asset("widened-consumer", session=session) + assert _read_wakes(session) == [(None, "widened-consumer", "dag_changed")] + + # One loop starts the fan-out over the Dag's pending runs; then two runs fire per loop. + _run_partition_loops(runner, session, loops=4) + assert all(_is_fired(apdr, session) for apdr in blocked) + assert _read_wakes(session) == [] + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +def test_pausing_a_dag_stops_the_partition_fan_out_over_its_pending_runs( + dag_maker: DagMaker, session: Session +): + blocked = _make_blocked_apdrs( + consumer_dag_id="paused-mid-fan-out", + partition_keys=[f"k{i}" for i in range(6)], + session=session, + dag_maker=dag_maker, + ) + runner = _make_partition_runner(startup_pass=False, per_loop=2) + _run_partition_loops(runner, session, loops=3) + _widen_to_either_asset("paused-mid-fan-out", session=session) + _run_partition_loops(runner, session, loops=2) + assert sum(_is_fired(apdr, session) for apdr in blocked) == 2 + + session.execute(update(DagModel).where(DagModel.dag_id == "paused-mid-fan-out").values(is_paused=True)) + session.commit() + _run_partition_loops(runner, session, loops=2) + assert sum(_is_fired(apdr, session) for apdr in blocked) == 2 + + session.execute(update(DagModel).where(DagModel.dag_id == "paused-mid-fan-out").values(is_paused=False)) + session.commit() + _run_partition_loops(runner, session, loops=4) + assert all(_is_fired(apdr, session) for apdr in blocked) + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +def test_many_dag_level_partition_signals_do_not_hold_back_a_key_log_signal( + dag_maker: DagMaker, session: Session +): + """More Dag-level requests than twice the budget, with their fan-outs running, leave a key-log request in the read.""" + dag_count = 11 + for i in range(dag_count): + _make_blocked_apdrs( + consumer_dag_id=f"fanned-out-{i}", + partition_keys=["k1", "k2"], + session=session, + dag_maker=dag_maker, + ) + runner = _make_partition_runner(startup_pass=False, per_loop=5) + _run_partition_loops(runner, session, loops=5) + assert _read_wakes(session) == [] + session.add_all( + AssetPartitionDagRunWake(target_dag_id=f"fanned-out-{i}", reason=PartitionWakeReason.DAG_CHANGED) + for i in range(dag_count) + ) + session.commit() + # Starts the fan-outs, which then share one slot per loop. + _run_partition_loops(runner, session) + + [signalled] = _make_n_satisfied_apdrs( + consumer_dag_id="signalled-consumer", + asset=Asset(name="signalled-asset"), + partition_keys=["k"], + session=session, + dag_maker=dag_maker, + producer_prefix="signalled-producer", + ) + _run_partition_loops(runner, session) + + assert _is_fired(signalled, session) + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +@mock.patch("airflow.jobs.scheduler_job_runner._PARTITION_FAN_OUT_STARTS_PER_LOOP", 1) +def test_running_partition_fan_out_does_not_hide_the_dag_level_signal_of_another_dag( + dag_maker: DagMaker, session: Session +): + """A Dag-level row stays until its fan-out is done, so the read takes one more such row per fan-out.""" + _make_blocked_apdrs( + consumer_dag_id="long-fan-out", + partition_keys=[f"k{i}" for i in range(6)], + session=session, + dag_maker=dag_maker, + ) + [widened] = _make_blocked_apdrs( + consumer_dag_id="widened-later", partition_keys=["k"], session=session, dag_maker=dag_maker + ) + _delete_wakes(session) + session.add( + AssetPartitionDagRunWake(target_dag_id="long-fan-out", reason=PartitionWakeReason.DAG_CHANGED) + ) + session.commit() + runner = _make_partition_runner(startup_pass=False, per_loop=1) + # Starts the fan-out over six runs, of which it then evaluates one per loop. + _run_partition_loops(runner, session) + + _widen_to_either_asset("widened-later", session=session) + _run_partition_loops(runner, session, loops=2) + + assert _is_fired(widened, session) + assert "long-fan-out" in runner._partition_fan_out + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +@mock.patch("airflow.jobs.scheduler_job_runner._PARTITION_FAN_OUT_STARTS_PER_LOOP", 1) +def test_partition_fan_outs_start_with_the_dag_level_signal_that_waited_longest( + dag_maker: DagMaker, session: Session +): + for dag_id in ("waited-longest", "asked-later"): + _make_blocked_apdrs( + consumer_dag_id=dag_id, partition_keys=["k"], session=session, dag_maker=dag_maker + ) + _delete_wakes(session) + for dag_id in ("waited-longest", "asked-later"): + session.add(AssetPartitionDagRunWake(target_dag_id=dag_id, reason=PartitionWakeReason.DAG_CHANGED)) + session.commit() + runner = _make_partition_runner(startup_pass=False) + + _run_partition_loops(runner, session) + + assert "waited-longest" in runner._partition_fan_out + assert "asked-later" not in runner._partition_fan_out + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +@pytest.mark.parametrize("schedule", ["asset", "alias"]) +def test_pending_runs_of_a_dag_that_is_no_longer_partitioned_are_dropped( + dag_maker: DagMaker, session: Session, schedule: str +): + """A Dag scheduled on aliases only stamps the same empty fingerprint whether it is partitioned or not.""" + if schedule == "asset": + [apdr] = _make_blocked_apdrs( + consumer_dag_id="unpartitioned-consumer", + partition_keys=["k"], + session=session, + dag_maker=dag_maker, + ) + else: + with dag_maker( + dag_id="unpartitioned-consumer", + schedule=PartitionedAssetTimetable(assets=AssetAlias("alias-only")), + session=session, + ): + EmptyOperator(task_id="hi") + # An alias gives the fingerprint no asset to describe. + apdr = AssetPartitionDagRun( + target_dag_id="unpartitioned-consumer", + partition_key="k", + pending_partition_key="k", + rollup_fingerprint={}, + ) + session.add(apdr) + session.commit() + runner = _make_partition_runner(startup_pass=False) + _run_partition_loops(runner, session) + + with DAG(dag_id="unpartitioned-consumer", schedule="@daily", start_date=DEFAULT_DATE) as dag: + EmptyOperator(task_id="hi") + sync_dag_to_db(dag, session=session) + session.commit() + _run_partition_loops(runner, session, loops=2) + + assert session.get(AssetPartitionDagRun, apdr.id) is None + assert _read_wakes(session) == [] + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +@pytest.mark.parametrize("activator", ["scheduler", "dag_processor"]) +def test_activating_an_asset_wakes_the_pending_runs_frozen_on_it( + dag_maker: DagMaker, session: Session, activator: str +): + asset = Asset(name="frozen-asset") + [apdr] = _make_n_satisfied_apdrs( + consumer_dag_id="frozen-consumer", + asset=asset, + partition_keys=["k"], + session=session, + dag_maker=dag_maker, + ) + bundle_name = session.scalar(select(DagModel.bundle_name).where(DagModel.dag_id == "frozen-consumer")) + assert bundle_name is not None + _set_asset_active(name=asset.name, uri=asset.uri, session=session, active=False) + runner = _make_partition_runner(startup_pass=False) + _run_partition_loops(runner, session) + assert not _is_fired(apdr, session) + assert _read_wakes(session) == [] + + if activator == "scheduler": + runner._update_asset_orphanage(session=session) + else: + with DAG(dag_id="asset-redeclarer", schedule=None, start_date=DEFAULT_DATE) as redeclarer: + EmptyOperator(task_id="hi", outlets=[asset]) + # The Dag processor parses each file in a session of its own. + with create_session(scoped=False) as parse_session: + SerializedDAG.bulk_write_to_db(bundle_name, None, [redeclarer], session=parse_session) + session.commit() + try: + assert _read_wakes(session) == [(None, "frozen-consumer", "asset_activated")] + + _run_partition_loops(runner, session, loops=2) + assert _is_fired(apdr, session) + finally: + # dag_maker cleans up only the Dags it made. + session.execute(delete(DagModel).where(DagModel.dag_id == "asset-redeclarer")) + session.commit() + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +def test_partition_startup_pass_leaves_a_dag_level_signal_for_a_dag_that_cannot_run( + dag_maker: DagMaker, session: Session +): + [apdr] = _make_n_satisfied_apdrs( + consumer_dag_id="paused-at-startup", + asset=Asset(name="paused-at-startup-asset"), + partition_keys=["k"], + session=session, + dag_maker=dag_maker, + ) + _delete_wakes(session) + session.execute(update(DagModel).where(DagModel.dag_id == "paused-at-startup").values(is_paused=True)) + session.commit() + runner = _make_partition_runner() + + _run_partition_loops(runner, session) + assert not _is_fired(apdr, session) + assert _read_wakes(session) == [(None, "paused-at-startup", "resync")] + # The next pass finds the Dag still paused, and its Dag-level row still there. + runner._partition_next_resync_at = 0.0 + _run_partition_loops(runner, session) + assert _read_wakes(session) == [(None, "paused-at-startup", "resync")] + + session.execute(update(DagModel).where(DagModel.dag_id == "paused-at-startup").values(is_paused=False)) + session.commit() + _run_partition_loops(runner, session, loops=2) + assert _is_fired(apdr, session) + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +def test_partition_resync_leaves_a_dag_level_signal_past_the_rows_a_running_fan_out_read( + dag_maker: DagMaker, session: Session +): + """A running fan-out deletes the Dag-level rows it read once it is done with the runs it read then.""" + dag_ids = ["fanning-out-1", "fanning-out-2", "fanning-out-3"] + for dag_id in dag_ids: + _make_blocked_apdrs( + consumer_dag_id=dag_id, partition_keys=["k1", "k2"], session=session, dag_maker=dag_maker + ) + _delete_wakes(session) + session.add_all( + AssetPartitionDagRunWake(target_dag_id=dag_id, reason=PartitionWakeReason.DAG_CHANGED) + for dag_id in dag_ids + ) + session.commit() + runner = _make_partition_runner(startup_pass=False, per_loop=2) + # Starts the three fan-outs. The next loop's fan-out window holds only the first two, so pausing + # fanning-out-3 does not drop its fan-out. + _run_partition_loops(runner, session) + session.execute(update(DagModel).where(DagModel.dag_id == "fanning-out-3").values(is_paused=True)) + session.commit() + # A periodic pass that serves fanning-out-3 first. + runner._partition_resync_passes = 1 + runner._partition_next_resync_at = 0.0 + rng = mock.create_autospec(random.Random, instance=True) + rng.randrange.return_value = 2 + runner._partition_rng = rng + + _run_partition_loops(runner, session) + assert (None, "fanning-out-3", "resync") in _read_wakes(session) + + # A run that no wake row asks for, as only a pass would find it. + session.execute(update(DagModel).where(DagModel.dag_id == "fanning-out-3").values(is_paused=False)) + session.commit() + for name in ("a", "b"): + late = _produce_and_register_asset_event( + dag_id=f"fanning-out-3-late-producer-{name}", + asset=Asset(name=f"fanning-out-3-{name}"), + partition_key="k3", + session=session, + dag_maker=dag_maker, + ) + session.execute(delete(AssetPartitionDagRunWake).where(AssetPartitionDagRunWake.apdr_id == late.id)) + session.commit() + + _run_partition_loops(runner, session, loops=10) + assert _is_fired(late, session) + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +def test_partition_resync_keeps_a_share_of_each_loop_while_signals_fill_it( + dag_maker: DagMaker, session: Session +): + _make_blocked_apdrs( + consumer_dag_id="busy-consumer", + partition_keys=[f"k{i}" for i in range(25)], + session=session, + dag_maker=dag_maker, + ) + unsignalled = _make_n_satisfied_apdrs( + consumer_dag_id="resynced-consumer", + asset=Asset(name="resynced-asset"), + partition_keys=["k1", "k2"], + session=session, + dag_maker=dag_maker, + ) + session.execute( + delete(AssetPartitionDagRunWake).where(AssetPartitionDagRunWake.target_dag_id == "resynced-consumer") + ) + session.commit() + # A share of one run per loop, given to the Dags of the pass in turn: the pass starts at + # busy-consumer, whose next run is past the ones the signals read takes, then resynced-consumer. + runner = _make_partition_runner(per_loop=10) + rng = mock.create_autospec(random.Random, instance=True) + rng.randrange.return_value = 0 + runner._partition_rng = rng + + _run_partition_loops(runner, session, loops=2) + + assert sum(_is_fired(apdr, session) for apdr in unsignalled) == 1 + assert any(dag_id == "busy-consumer" for _, dag_id, _ in _read_wakes(session)) + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +def test_retried_loop_evaluates_the_partition_resync_runs_its_rolled_back_attempt_took( + dag_maker: DagMaker, session: Session +): + [apdr] = _make_n_satisfied_apdrs( + consumer_dag_id="retried-consumer", + asset=Asset(name="retried-asset"), + partition_keys=["k"], + session=session, + dag_maker=dag_maker, + ) + _delete_wakes(session) + runner = _make_partition_runner() + create_dag_runs = SchedulerJobRunner._create_dag_runs + attempts: list[int] = [] + + def fail_once(self, dag_models, session): + attempts.append(len(attempts)) + if len(attempts) == 1: + raise OperationalError("SELECT 1", {}, Exception("deadlock")) + return create_dag_runs(self, dag_models, session) + + with mock.patch.object(SchedulerJobRunner, "_create_dag_runs", autospec=True, side_effect=fail_once): + _run_partition_loops(runner, session) + + assert len(attempts) == 2 + assert _is_fired(apdr, session) + assert ( + session.scalar(select(func.count()).select_from(DagRun).where(DagRun.dag_id == "retried-consumer")) + == 1 + ) + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +def test_partition_resync_read_that_fails_is_read_again_by_the_retried_loop( + dag_maker: DagMaker, session: Session +): + [apdr] = _make_n_satisfied_apdrs( + consumer_dag_id="read-again-consumer", + asset=Asset(name="read-again-asset"), + partition_keys=["k"], + session=session, + dag_maker=dag_maker, + ) + _delete_wakes(session) + runner = _make_partition_runner() + read = SchedulerJobRunner._read_pending_partition_dag_runs + attempts: list[int] = [] + + def fail_once(*, session): + attempts.append(len(attempts)) + if len(attempts) == 1: + raise OperationalError("SELECT 1", {}, Exception("connection lost")) + return read(session=session) + + with mock.patch.object( + SchedulerJobRunner, "_read_pending_partition_dag_runs", autospec=True, side_effect=fail_once + ): + _run_partition_loops(runner, session) + + assert len(attempts) == 2 + assert _is_fired(apdr, session) + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +@conf_vars({("scheduler", "asset_partition_resync_interval"): "60"}) +@mock.patch("airflow.jobs.scheduler_job_runner.stats", autospec=True) +def test_partition_resync_runs_again_once_the_interval_has_passed_since_the_last_pass_started( + mock_stats, dag_maker: DagMaker, session: Session +): + first, second = _make_n_satisfied_apdrs( + consumer_dag_id="periodic-consumer", + asset=Asset(name="periodic-asset"), + partition_keys=["k1", "k2"], + session=session, + dag_maker=dag_maker, + ) + _delete_wakes(session) + read = SchedulerJobRunner._read_pending_partition_dag_runs + with ( + mock.patch("airflow.jobs.scheduler_job_runner.time.monotonic", autospec=True) as clock, + mock.patch.object( + SchedulerJobRunner, "_read_pending_partition_dag_runs", autospec=True, side_effect=read + ) as reads, + ): + clock.return_value = 1000.0 + # One run per loop, so that the startup pass is still running once the next one is due. + runner = _make_partition_runner(per_loop=1) + _run_partition_loops(runner, session) + clock.return_value = 1061.0 + _run_partition_loops(runner, session) + assert _is_fired(first, session) + assert _is_fired(second, session) + assert reads.call_count == 1 + + # Due since 1060, so the next loop starts a pass, over no pending run; the one after is due at 1122. + clock.return_value = 1062.0 + _run_partition_loops(runner, session) + assert reads.call_count == 2 + [later] = _make_n_satisfied_apdrs( + consumer_dag_id="periodic-consumer-later", + asset=Asset(name="periodic-asset-later"), + partition_keys=["k"], + session=session, + dag_maker=dag_maker, + producer_prefix="periodic-producer-later", + ) + _delete_wakes(session) + clock.return_value = 1121.0 + _run_partition_loops(runner, session) + assert not _is_fired(later, session) + assert reads.call_count == 2 + + mock_stats.reset_mock() + clock.return_value = 1123.0 + _run_partition_loops(runner, session) + assert _is_fired(later, session) + assert reads.call_count == 3 + + mock_stats.incr.assert_any_call("scheduler.asset_partitions.resync_fired", 1, tags={"pass": "periodic"}) + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +@mock.patch("airflow.jobs.scheduler_job_runner._PARTITION_FAN_OUT_STARTS_PER_LOOP", 1) +@mock.patch("airflow.jobs.scheduler_job_runner.stats", autospec=True) +def test_partition_runs_a_dag_level_row_asked_for_are_not_counted_as_found_by_the_resync( + mock_stats, dag_maker: DagMaker, session: Session +): + """The first loop reads the Dag-level row of one Dag, and leaves the row of the other for a later loop.""" + dag_ids = ["requested-consumer", "requested-later-consumer"] + blocked = [ + apdr + for dag_id in dag_ids + for apdr in _make_blocked_apdrs( + consumer_dag_id=dag_id, partition_keys=["k1", "k2"], session=session, dag_maker=dag_maker + ) + ] + runner = _make_partition_runner(startup_pass=False, per_loop=2) + _run_partition_loops(runner, session, loops=2) + for dag_id in dag_ids: + _widen_to_either_asset(dag_id, session=session) + # A periodic pass that runs while the fan-outs of the Dag-level rows do. + runner._partition_resync_passes = 1 + runner._partition_next_resync_at = 0.0 + rng = mock.create_autospec(random.Random, instance=True) + rng.randrange.return_value = 0 + runner._partition_rng = rng + + _run_partition_loops(runner, session, loops=4) + + assert all(_is_fired(apdr, session) for apdr in blocked) + assert [ + c for c in mock_stats.incr.mock_calls if c.args[0] == "scheduler.asset_partitions.resync_fired" + ] == [] + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +@mock.patch("airflow.jobs.scheduler_job_runner.stats", autospec=True) +def test_partition_resync_counts_a_run_only_when_no_running_fan_out_holds_it( + mock_stats, dag_maker: DagMaker, session: Session +): + """A fan-out evaluates the pending runs it read when it started; its Dag-level row asks for those only.""" + *_, held = _make_blocked_apdrs( + consumer_dag_id="fanned-out-consumer", + partition_keys=[f"k{i}" for i in range(4)], + session=session, + dag_maker=dag_maker, + ) + _delete_wakes(session) + session.add( + AssetPartitionDagRunWake(target_dag_id="fanned-out-consumer", reason=PartitionWakeReason.DAG_CHANGED) + ) + session.commit() + runner = _make_partition_runner(startup_pass=False, per_loop=2) + # Starts the fan-out over the four runs. + _run_partition_loops(runner, session) + # Satisfied once the fan-out has started, with no wake row: a run the fan-out holds, and a later one. + for name, key in (("b", "k3"), ("a", "late"), ("b", "late")): + satisfied = _produce_and_register_asset_event( + dag_id=f"fanned-out-consumer-producer-{key}-{name}", + asset=Asset(name=f"fanned-out-consumer-{name}"), + partition_key=key, + session=session, + dag_maker=dag_maker, + ) + late = satisfied + session.execute( + delete(AssetPartitionDagRunWake).where(AssetPartitionDagRunWake.apdr_id.in_([held.id, late.id])) + ) + session.commit() + runner._partition_resync_passes = 1 + runner._partition_next_resync_at = 0.0 + rng = mock.create_autospec(random.Random, instance=True) + rng.randrange.return_value = 0 + runner._partition_rng = rng + + _run_partition_loops(runner, session, loops=2) + assert _is_fired(held, session) + assert not _is_fired(late, session) + assert "fanned-out-consumer" in runner._partition_fan_out + + _run_partition_loops(runner, session) + + assert _is_fired(late, session) + assert [ + c for c in mock_stats.incr.mock_calls if c.args[0] == "scheduler.asset_partitions.resync_fired" + ] == [mock.call("scheduler.asset_partitions.resync_fired", 1, tags={"pass": "periodic"})] + + +def test_partition_sweep_holds_only_the_runs_left_to_evaluate(): + sweep = _PartitionSweep() + sweep.add("dag", [5, 1, 3]) + sweep.commit(done={"dag": {3}}, dropped=(), served=0) + + assert [apdr_id for apdr_id in range(7) if sweep.holds("dag", apdr_id)] == [1, 5] + assert not sweep.holds("other-dag", 1) + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +@pytest.mark.parametrize("flag", ["is_paused", "is_draining", "is_stale"]) +def test_partition_run_of_a_dag_that_stops_running_during_the_loop_is_not_created( + dag_maker: DagMaker, session: Session, flag: str +): + [apdr] = _make_blocked_apdrs( + consumer_dag_id="stopping-consumer", partition_keys=["k"], session=session, dag_maker=dag_maker + ) + runner = _make_partition_runner(startup_pass=False) + _run_partition_loops(runner, session) + _widen_to_either_asset("stopping-consumer", session=session) + # Starts the fan-out over the Dag's pending runs. + _run_partition_loops(runner, session) + read_eligibility = SchedulerJobRunner._read_dag_eligibility + + def stop_the_dag_after_the_check(dag_ids, *, session): + can_run = read_eligibility(dag_ids, session=session) + session.execute(update(DagModel).where(DagModel.dag_id == "stopping-consumer").values({flag: True})) + return can_run + + with mock.patch.object( + SchedulerJobRunner, "_read_dag_eligibility", autospec=True, side_effect=stop_the_dag_after_the_check + ): + _run_partition_loops(runner, session) + + assert not _is_fired(apdr, session) + assert _read_wakes(session) == [(None, "stopping-consumer", "dag_changed")] + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +def test_partition_fan_out_keeps_a_share_of_each_loop_while_signals_fill_it( + dag_maker: DagMaker, session: Session +): + _make_blocked_apdrs( + consumer_dag_id="busy-consumer", + partition_keys=[f"k{i}" for i in range(20)], + session=session, + dag_maker=dag_maker, + ) + widened = _make_blocked_apdrs( + consumer_dag_id="widened-consumer", + partition_keys=["k1", "k2", "k3"], + session=session, + dag_maker=dag_maker, + ) + session.execute( + delete(AssetPartitionDagRunWake).where(AssetPartitionDagRunWake.target_dag_id == "widened-consumer") + ) + session.commit() + _widen_to_either_asset("widened-consumer", session=session) + runner = _make_partition_runner(startup_pass=False, per_loop=10) + # Starts the fan-out, while ten of the busy Dag's requests take the loop. + _run_partition_loops(runner, session) + + _run_partition_loops(runner, session) + + assert sum(_is_fired(apdr, session) for apdr in widened) == 1 + assert any(dag_id == "busy-consumer" for _, dag_id, _ in _read_wakes(session)) + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +def test_idle_partition_claim_issues_one_query_and_writes_nothing(dag_maker: DagMaker, session: Session): + # The first definition of a partitioned Dag asks to evaluate its pending runs, of which there are + # none, so the loop that reads the row is done with it; the requests of a paused Dag wait for it. + with DAG( + dag_id="first-definition", + schedule=PartitionedAssetTimetable(assets=Asset(name="first-definition-asset")), + start_date=DEFAULT_DATE, + ) as dag: + EmptyOperator(task_id="hi") + sync_dag_to_db(dag, session=session) + try: + with dag_maker(dag_id="paused-dag", schedule=None, session=session): + EmptyOperator(task_id="hi") + session.execute(update(DagModel).where(DagModel.dag_id == "paused-dag").values(is_paused=True)) + session.add(AssetPartitionDagRunWake(apdr_id=1, target_dag_id="paused-dag", reason="key_log")) + session.add(AssetPartitionDagRunWake(target_dag_id="paused-dag", reason="dag_changed")) + session.commit() + assert sorted(_read_wakes(session), key=str) == [ + (1, "paused-dag", "key_log"), + (None, "first-definition", "dag_changed"), + (None, "paused-dag", "dag_changed"), + ] + runner = _make_partition_runner() + _run_partition_loops(runner, session) + assert _read_wakes(session) == [(1, "paused-dag", "key_log"), (None, "paused-dag", "dag_changed")] + + # The signal read alone, so nothing is written. + with assert_queries_count(1): + runner._create_dagruns_for_partitioned_asset_dags(session=session) + finally: + # dag_maker cleans up only the Dags it made. + session.rollback() + session.execute(delete(DagModel).where(DagModel.dag_id == "first-definition")) + session.commit() + + +@pytest.mark.db_test +@pytest.mark.usefixtures("clear_asset_partition_rows") +def test_clean_up_partition_wakes_drops_rows_of_deleted_dags_and_merges_rows_of_dags_that_cannot_run( + dag_maker: DagMaker, session: Session +): + for dag_id, is_paused in (("cannot-run", True), ("can-run", False)): + with dag_maker(dag_id=dag_id, schedule=None, session=session): + EmptyOperator(task_id="hi") + session.execute(update(DagModel).where(DagModel.dag_id == dag_id).values(is_paused=is_paused)) + session.add_all( + AssetPartitionDagRunWake(apdr_id=apdr_id, target_dag_id=dag_id, reason="key_log") + for dag_id in ("deleted", "cannot-run", "can-run") + for apdr_id in (1, 2, 3) + ) + session.commit() + + _make_partition_runner()._clean_up_partition_wakes() + + assert sorted(_read_wakes(session), key=str) == sorted( + [ + (None, "cannot-run", "coalesced"), + (1, "can-run", "key_log"), + (2, "can-run", "key_log"), + (3, "can-run", "key_log"), + ], + key=str, + ) + + +@pytest.mark.db_test +@pytest.mark.usefixtures("clear_asset_partition_rows") +@pytest.mark.parametrize( + ("batches", "expected"), + [ + pytest.param(20, [(None, "stale-consumer", "coalesced")], id="within-a-firing"), + # One batch finds no row of a deleted Dag; each of the other two merges 500 rows into one. + pytest.param( + 3, + [(apdr_id, "stale-consumer", "key_log") for apdr_id in range(1001, 1201)] + + [(None, "stale-consumer", "coalesced")] * 2, + id="past-the-batches-of-a-firing", + ), + ], +) +def test_clean_up_partition_wakes_merges_a_large_backlog_of_a_dag_in_batches( + dag_maker: DagMaker, session: Session, batches: int, expected: list[tuple[int | None, str, str]] +): + with dag_maker(dag_id="stale-consumer", schedule=None, session=session): + EmptyOperator(task_id="hi") + session.execute(update(DagModel).where(DagModel.dag_id == "stale-consumer").values(is_stale=True)) + session.execute( + insert(AssetPartitionDagRunWake), + [ + {"apdr_id": apdr_id, "target_dag_id": "stale-consumer", "reason": "key_log"} + for apdr_id in range(1, 1201) + ], + ) + session.commit() + + with mock.patch("airflow.jobs.scheduler_job_runner._PARTITION_WAKE_CLEANUP_BATCHES", batches): + _make_partition_runner()._clean_up_partition_wakes() + + assert _read_wakes(session) == expected + + +@pytest.mark.db_test +@pytest.mark.usefixtures("clear_asset_partition_rows") +@mock.patch("airflow.jobs.scheduler_job_runner._PARTITION_WAKE_CLEANUP_BATCHES", 3) +def test_clean_up_partition_wakes_merges_the_dags_that_waited_longest_first( + dag_maker: DagMaker, session: Session +): + dag_ids = [f"paused-{i}" for i in range(4)] + for dag_id in dag_ids: + with dag_maker(dag_id=dag_id, schedule=None, session=session): + EmptyOperator(task_id="hi") + session.execute(update(DagModel).where(DagModel.dag_id.in_(dag_ids)).values(is_paused=True)) + session.add_all( + AssetPartitionDagRunWake(apdr_id=apdr_id, target_dag_id=dag_id, reason="key_log") + for dag_id in dag_ids + for apdr_id in (1, 2) + ) + session.commit() + runner = _make_partition_runner() + + # A firing of three batches: one finds no row of a deleted Dag, two merge the first Dags. + runner._clean_up_partition_wakes() + merged = {dag_id for _, dag_id, reason in _read_wakes(session) if reason == "coalesced"} + assert merged == {"paused-0", "paused-1"} + # A new request each makes them qualify again, behind the Dags that were not merged yet. + session.add_all( + AssetPartitionDagRunWake(apdr_id=3, target_dag_id=dag_id, reason="key_log") for dag_id in merged + ) + session.commit() + + runner._clean_up_partition_wakes() + + assert {dag_id for _, dag_id, reason in _read_wakes(session) if reason == "coalesced"} == set(dag_ids) + assert [dag_id for _, dag_id, _ in _read_wakes(session)].count("paused-2") == 1 + assert [dag_id for _, dag_id, _ in _read_wakes(session)].count("paused-3") == 1 + + +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +@mock.patch("airflow.jobs.scheduler_job_runner.stats", autospec=True) +def test_partition_claim_reports_its_metrics_once_the_loop_commits( + mock_stats, dag_maker: DagMaker, session: Session +): + [unsignalled] = _make_n_satisfied_apdrs( + consumer_dag_id="resync-found-consumer", + asset=Asset(name="resync-found-asset"), + partition_keys=["k"], + session=session, + dag_maker=dag_maker, + ) + _delete_wakes(session) + [waiting] = _make_blocked_apdrs( + consumer_dag_id="still-waiting", partition_keys=["k"], session=session, dag_maker=dag_maker + ) + session.execute( + update(AssetPartitionDagRunWake).values(created_at=timezone.utcnow() - timedelta(minutes=10)) + ) + # The request of a run that is gone is older, but no run waits on it. + session.add( + AssetPartitionDagRunWake( + apdr_id=waiting.id + 1000, + target_dag_id="still-waiting", + reason="key_log", + created_at=timezone.utcnow() - timedelta(days=1), + ) + ) + session.commit() + runner = _make_partition_runner() + + def get_partition_metric_calls() -> list: + return [c for c in mock_stats.mock_calls if c.args and "asset_partitions" in str(c.args[0])] + + runner._create_dagruns_for_partitioned_asset_dags(session=session) + assert get_partition_metric_calls() == [] + + runner._commit_partition_loop() + + mock_stats.incr.assert_any_call("scheduler.asset_partitions.evaluated", 2) + mock_stats.incr.assert_any_call("scheduler.asset_partitions.resync_fired", 1, tags={"pass": "startup"}) + [lag] = [c for c in mock_stats.gauge.mock_calls if c.args[0] == "scheduler.asset_partitions.wake_lag"] + assert 600 <= lag.args[1] < 3600 + [duration] = [c for c in mock_stats.timing.mock_calls if "asset_partitions" in c.args[0]] + assert duration.args[0] == "scheduler.asset_partitions.resync_pass_duration" + assert _is_fired(unsignalled, session) + assert not _is_fired(waiting, session) + + mock_stats.reset_mock() + _run_partition_loops(runner, session) + assert get_partition_metric_calls() == [mock.call.gauge("scheduler.asset_partitions.wake_lag", 0)] + + +@conf_vars({("scheduler", "asset_partition_resync_interval"): "0"}) +def test_asset_partition_resync_interval_must_be_positive(): + with pytest.raises(ValueError, match="asset_partition_resync_interval must be greater than 0"): + SchedulerJobRunner(job=Job(job_type=SchedulerJobRunner.job_type), executors=[MockExecutor()]) + + +@pytest.mark.backend("postgres", "mysql") +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +def test_partition_fan_out_keeps_a_pending_run_another_transaction_holds( + dag_maker: DagMaker, session: Session +): + held, free = _make_blocked_apdrs( + consumer_dag_id="held-consumer", partition_keys=["k1", "k2"], session=session, dag_maker=dag_maker + ) + runner = _make_partition_runner(startup_pass=False) + _run_partition_loops(runner, session) + _widen_to_either_asset("held-consumer", session=session) + + with create_session(scoped=False) as holder: + holder.execute( + select(AssetPartitionDagRun.id).where(AssetPartitionDagRun.id == held.id).with_for_update() + ) + _run_partition_loops(runner, session, loops=3) + assert _is_fired(free, session) + assert not _is_fired(held, session) + assert _read_wakes(session) == [(None, "held-consumer", "dag_changed")] + + _run_partition_loops(runner, session, loops=2) + assert _is_fired(held, session) + assert _read_wakes(session) == [] + + +@pytest.mark.backend("postgres") +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +def test_partition_claim_skips_the_wake_rows_another_transaction_holds(dag_maker: DagMaker, session: Session): + """Another scheduler can delete the same orphan and Dag-level rows; the loop does not wait for it.""" + [fired] = _make_blocked_apdrs( + consumer_dag_id="held-orphan-consumer", partition_keys=["k"], session=session, dag_maker=dag_maker + ) + session.execute( + update(AssetPartitionDagRun) + .where(AssetPartitionDagRun.id == fired.id) + .values(created_dag_run_id=session.scalar(select(DagRun.id).limit(1)), pending_partition_key=None) + ) + with dag_maker(dag_id="no-pending-runs", schedule=None, session=session): + EmptyOperator(task_id="hi") + session.add(AssetPartitionDagRunWake(target_dag_id="no-pending-runs", reason="dag_changed")) + session.commit() + held = sorted(session.scalars(select(AssetPartitionDagRunWake.id)).all()) + assert len(held) == 2 + runner = _make_partition_runner(startup_pass=False) + + with create_session(scoped=False) as holder: + holder.execute( + select(AssetPartitionDagRunWake.id).where(AssetPartitionDagRunWake.id.in_(held)).with_for_update() + ) + # Outside the retries of the scheduler loop, whose next attempt would run without the + # timeout, so that a lock wait fails the claim. + session.execute(text("SET LOCAL lock_timeout = '1s'")) + runner._create_dagruns_for_partitioned_asset_dags(session=session) + session.commit() + runner._commit_partition_loop() + assert sorted(session.scalars(select(AssetPartitionDagRunWake.id)).all()) == held + + _run_partition_loops(runner, session) + assert _read_wakes(session) == [] + + +@pytest.mark.backend("postgres", "mysql") +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +def test_two_schedulers_claim_disjoint_pending_partition_dag_runs(dag_maker: DagMaker, session: Session): + apdrs = _make_n_satisfied_apdrs( + consumer_dag_id="ha-consumer", + asset=Asset(name="ha-asset"), + partition_keys=["k1", "k2", "k3", "k4"], + session=session, + dag_maker=dag_maker, + ) + first, second = (_make_partition_runner(startup_pass=False, per_loop=2) for _ in range(2)) + + with create_session(scoped=False) as other: + claimed_first = { + apdr.id for apdr in first._claim_partition_dag_runs(_PartitionLoop(None), session=session).apdrs + } + claimed_second = { + apdr.id for apdr in second._claim_partition_dag_runs(_PartitionLoop(None), session=other).apdrs + } + other.rollback() + session.rollback() + + assert len(claimed_first) == 2 + assert len(claimed_second) == 2 + assert claimed_first.isdisjoint(claimed_second) + assert claimed_first | claimed_second == {apdr.id for apdr in apdrs} + + +@pytest.mark.backend("postgres", "mysql") +@pytest.mark.need_serialized_dag +@pytest.mark.usefixtures("clear_asset_partition_rows") +def test_producer_does_not_wait_on_the_partition_claim(dag_maker: DagMaker, session: Session): + [apdr] = _make_blocked_apdrs( + consumer_dag_id="claimed-consumer", partition_keys=["k"], session=session, dag_maker=dag_maker + ) + runner = _make_partition_runner(startup_pass=False) + claim = runner._claim_partition_dag_runs(_PartitionLoop(None), session=session) + assert [claimed.id for claimed in claim.apdrs] == [apdr.id] + + # A connection that is closed rather than pooled, so that its lock timeout does not outlive the test. + engine = create_engine(settings.get_engine().url, **settings.prepare_engine_args(pool_class=NullPool)) + try: + with sessionmaker(engine).begin() as producer: + if engine.dialect.name == "postgresql": + producer.execute(text("SET LOCAL lock_timeout = '1s'")) + else: + producer.execute(text("SET SESSION innodb_lock_wait_timeout = 1")) + producer.add( + PartitionedAssetKeyLog( + asset_id=1, + asset_event_id=1, + asset_partition_dag_run_id=apdr.id, + source_partition_key="k", + target_dag_id="claimed-consumer", + target_partition_key="k", + ) + ) + producer.flush() + finally: + engine.dispose() + session.rollback() + + assert len(_read_wakes(session)) == 2 # --------------------------------------------------------------------------- diff --git a/airflow-core/tests/unit/migrations/test_0157_add_asset_partition_dag_run_wake.py b/airflow-core/tests/unit/migrations/test_0157_add_asset_partition_dag_run_wake.py new file mode 100644 index 0000000000000..7f4d416b18b47 --- /dev/null +++ b/airflow-core/tests/unit/migrations/test_0157_add_asset_partition_dag_run_wake.py @@ -0,0 +1,88 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. + +from __future__ import annotations + +from importlib import import_module +from uuid import uuid4 + +import pytest +import sqlalchemy as sa +from alembic.migration import MigrationContext +from alembic.operations import Operations +from sqlalchemy.pool import NullPool + +from airflow import settings + +pytestmark = pytest.mark.db_test + +migration = import_module("airflow.migrations.versions.0157_3_5_0_add_asset_partition_dag_run_wake") +TABLE = "asset_partition_dag_run_wake" + + +def _run_migration(engine, migrate): + with engine.begin() as connection, Operations.context(MigrationContext.configure(connection)): + migrate() + + +@pytest.fixture +def empty_database(tmp_path): + source = settings.engine + name = f"partition_wake_{uuid4().hex}" + if source.dialect.name == "sqlite": + url = source.url.set(database=str(tmp_path / "partition_wake.db")) + else: + url = source.url.set(database=name) + with source.connect().execution_options(isolation_level="AUTOCOMMIT") as admin: + admin.exec_driver_sql(f"CREATE DATABASE {name}") + engine = sa.create_engine(url, poolclass=NullPool) + try: + yield engine + finally: + engine.dispose() + if source.dialect.name != "sqlite": + with source.connect().execution_options(isolation_level="AUTOCOMMIT") as admin: + admin.exec_driver_sql(f"DROP DATABASE {name}") + + +def test_upgrade_adds_the_wake_table_and_downgrade_drops_it(empty_database): + _run_migration(empty_database, migration.upgrade) + + inspector = sa.inspect(empty_database) + assert [column["name"] for column in inspector.get_columns(TABLE)] == [ + "id", + "apdr_id", + "target_dag_id", + "reason", + "created_at", + ] + assert {index["name"]: index["column_names"] for index in inspector.get_indexes(TABLE)} == { + "idx_apdr_wake_apdr_id": ["apdr_id"], + "idx_apdr_wake_target_dag_id": ["target_dag_id", "apdr_id"], + } + assert inspector.get_foreign_keys(TABLE) == [] + if empty_database.dialect.name == "sqlite": + # As the model declares it, so that SQLite never gives a deleted row's id to a new one. + with empty_database.connect() as connection: + sql = connection.scalar( + sa.text("SELECT sql FROM sqlite_master WHERE name = :name"), {"name": TABLE} + ) + assert "AUTOINCREMENT" in sql + + _run_migration(empty_database, migration.downgrade) + + assert not sa.inspect(empty_database).has_table(TABLE) diff --git a/airflow-core/tests/unit/models/test_asset.py b/airflow-core/tests/unit/models/test_asset.py index 84d27a3f1eac4..f0922f1c66ddc 100644 --- a/airflow-core/tests/unit/models/test_asset.py +++ b/airflow-core/tests/unit/models/test_asset.py @@ -17,18 +17,24 @@ from __future__ import annotations +import sqlite3 + import pytest -from sqlalchemy import select, update +from sqlalchemy import delete, event, select, update +from sqlalchemy.exc import IntegrityError +from airflow import settings from airflow.assets.manager import AssetManager from airflow.models.asset import ( AssetAliasModel, AssetEvent, AssetModel, + AssetPartitionDagRunWake, DagScheduleAssetAliasReference, DagScheduleAssetNameReference, DagScheduleAssetReference, DagScheduleAssetUriReference, + PartitionedAssetKeyLog, TaskOutletAssetReference, expand_alias_to_assets, remove_references_to_deleted_dags, @@ -36,12 +42,20 @@ from airflow.models.dag import DagModel from airflow.models.dynamic_region import DynamicRegion from airflow.models.taskinstance import TaskInstance, clear_task_instances +from airflow.models.variable import Variable from airflow.providers.standard.operators.empty import EmptyOperator from airflow.sdk import Asset, AssetAlias from airflow.serialization.definitions.assets import SerializedAssetAlias from airflow.utils.state import TaskInstanceState from tests_common.test_utils.dag import sync_dags_to_db +from tests_common.test_utils.db import ( + clear_db_apdr, + clear_db_assets, + clear_db_dags, + clear_db_pakl, + clear_db_variables, +) pytestmark = pytest.mark.db_test @@ -268,3 +282,145 @@ def _simulate_soft_dag_deletion(dag_id): _simulate_soft_dag_deletion("test2") remove_references_to_deleted_dags(session=session) assert set(session.execute(select_stmt)) == set() + + +class TestPartitionDagRunWakes: + """The wake rows that ask the scheduler to evaluate pending partition Dag runs.""" + + @pytest.fixture(autouse=True) + def _clear_db(self): + clear_db_dags() + clear_db_assets() + clear_db_apdr() + clear_db_pakl() + clear_db_variables() + yield + clear_db_dags() + clear_db_assets() + clear_db_apdr() + clear_db_pakl() + clear_db_variables() + + @staticmethod + def _build_key_log(apdr_id: int) -> PartitionedAssetKeyLog: + return PartitionedAssetKeyLog( + asset_id=1, + asset_event_id=1, + asset_partition_dag_run_id=apdr_id, + source_partition_key="k", + target_dag_id="consumer", + target_partition_key="k", + ) + + @staticmethod + def _read_wakes(session) -> list[tuple[int | None, str, str]]: + return [ + tuple(row) + for row in session.execute( + select( + AssetPartitionDagRunWake.apdr_id, + AssetPartitionDagRunWake.target_dag_id, + AssetPartitionDagRunWake.reason, + ).order_by(AssetPartitionDagRunWake.id) + ) + ] + + def test_key_log_writes_its_wake_row_in_its_transaction(self, session): + session.add(self._build_key_log(7)) + session.flush() + assert self._read_wakes(session) == [(7, "consumer", "key_log")] + + session.rollback() + assert self._read_wakes(session) == [] + + def test_partial_flush_writes_wake_rows_only_for_the_key_logs_it_inserts(self, session): + first, second = self._build_key_log(1), self._build_key_log(2) + session.add_all([first, second]) + + session.flush([first]) + assert self._read_wakes(session) == [(1, "consumer", "key_log")] + + session.flush() + assert self._read_wakes(session) == [(1, "consumer", "key_log"), (2, "consumer", "key_log")] + + def test_flush_without_key_logs_writes_no_wake_row(self, session): + statements: list[str] = [] + + def _record(conn, cursor, statement, parameters, context, executemany): + statements.append(statement) + + event.listen(settings.engine, "before_cursor_execute", _record) + try: + session.add(AssetModel(name="no-key-log", uri="s3://no-key-log")) + session.flush() + finally: + event.remove(settings.engine, "before_cursor_execute", _record) + + assert statements + assert not any("asset_partition_dag_run_wake" in statement for statement in statements) + + def test_key_logs_of_a_failed_flush_get_no_wake_row(self, session): + session.add(Variable(key="taken", val="v")) + session.commit() + # Unrelated mappers flush in name order, so the key log is inserted before the variable that fails. + session.add(self._build_key_log(1)) + session.add(Variable(key="taken", val="v")) + with pytest.raises(IntegrityError): + session.flush() + session.rollback() + + session.add(AssetModel(name="after-the-failure", uri="s3://after-the-failure")) + session.flush() + + assert self._read_wakes(session) == [] + + @pytest.mark.parametrize( + "schedule", + [ + pytest.param(lambda asset: [asset], id="asset"), + pytest.param(lambda asset: [Asset.ref(name=asset.name)], id="name-ref"), + pytest.param(lambda asset: [Asset.ref(uri=asset.uri)], id="uri-ref"), + pytest.param(lambda asset: [AssetAlias("activated-alias")], id="alias"), + ], + ) + def test_activation_wakes_the_dags_scheduled_on_the_asset(self, dag_maker, session, schedule): + asset = Asset(name="activated", uri="s3://activated") + with dag_maker(dag_id="scheduled-on-it", schedule=schedule(asset), session=session): + EmptyOperator(task_id="t") + with dag_maker(dag_id="scheduled-elsewhere", schedule=[Asset(name="other")], session=session): + EmptyOperator(task_id="t") + with dag_maker(dag_id="producer", schedule=None, session=session): + EmptyOperator(task_id="t", outlets=[asset]) + asset_model = session.scalar(select(AssetModel).where(AssetModel.name == asset.name)) + alias_model = session.scalar(select(AssetAliasModel).where(AssetAliasModel.name == "activated-alias")) + if alias_model is not None: + alias_model.assets.append(asset_model) + session.execute(delete(AssetPartitionDagRunWake)) + session.flush() + + AssetPartitionDagRunWake.add_for_activated_assets([asset_model], session=session) + + assert self._read_wakes(session) == [(None, "scheduled-on-it", "asset_activated")] + + @pytest.mark.backend("sqlite") + def test_activation_of_many_assets_stays_within_the_bind_parameter_limit_of_old_sqlite( + self, dag_maker, session + ): + assets = [Asset(name=f"many-{i}", uri=f"s3://many/{i}") for i in range(300)] + with dag_maker(dag_id="scheduled-on-the-first", schedule=[assets[0]], session=session): + EmptyOperator(task_id="t") + with dag_maker(dag_id="producer-of-many", schedule=None, session=session): + EmptyOperator(task_id="t", outlets=assets) + session.execute(delete(AssetPartitionDagRunWake)) + session.flush() + asset_models = session.scalars(select(AssetModel).where(AssetModel.name.like("many-%"))).all() + assert len(asset_models) == 300 + # The default limit of SQLite before 3.32. + dbapi_connection = session.connection().connection.dbapi_connection + limit = dbapi_connection.setlimit(sqlite3.SQLITE_LIMIT_VARIABLE_NUMBER, 999) + try: + AssetPartitionDagRunWake.add_for_activated_assets(asset_models, session=session) + finally: + dbapi_connection.setlimit(sqlite3.SQLITE_LIMIT_VARIABLE_NUMBER, limit) + + assert self._read_wakes(session) == [(None, "scheduled-on-the-first", "asset_activated")] diff --git a/airflow-core/tests/unit/models/test_serialized_dag.py b/airflow-core/tests/unit/models/test_serialized_dag.py index 7b47f70fdf2ba..451c84a53173b 100644 --- a/airflow-core/tests/unit/models/test_serialized_dag.py +++ b/airflow-core/tests/unit/models/test_serialized_dag.py @@ -32,9 +32,16 @@ import airflow.models.serialized_dag as serialized_dag_module from airflow._shared.observability.metrics.base_stats_logger import StatsLogger from airflow.dag_processing.dagbag import DagBag -from airflow.models.asset import AssetActive, AssetAliasModel, AssetModel +from airflow.models.asset import ( + AssetActive, + AssetAliasModel, + AssetModel, + AssetPartitionDagRun, + AssetPartitionDagRunWake, +) from airflow.models.dag import DagModel from airflow.models.dag_version import DagVersion +from airflow.models.dagcode import DagCode from airflow.models.deadline_alert import DeadlineAlert as DAM from airflow.models.serialized_dag import SerializedDagModel as SDM from airflow.providers.standard.operators.bash import BashOperator @@ -43,6 +50,7 @@ from airflow.sdk import DAG, Asset, AssetAlias, task as task_decorator from airflow.sdk.definitions.callback import AsyncCallback from airflow.sdk.definitions.deadline import DeadlineAlert, DeadlineReference +from airflow.sdk.definitions.timetables.assets import PartitionedAssetTimetable from airflow.serialization.dag_dependency import DagDependency from airflow.serialization.definitions.dag import SerializedDAG from airflow.serialization.serialized_objects import DagSerialization, LazyDeserializedDAG @@ -107,12 +115,14 @@ def setup_test_cases(self, request, monkeypatch): db.clear_db_dags() db.clear_db_runs() db.clear_db_serialized_dags() + db.clear_db_apdr() # The module reads the config once at import time, so conf_vars alone leaves the already # bound constant untouched and the compressed branch never runs. monkeypatch.setattr(serialized_dag_module, "_COMPRESS_SERIALIZED_DAGS", request.param) with conf_vars({("core", "compress_serialized_dags"): str(request.param)}): yield db.clear_db_serialized_dags() + db.clear_db_apdr() def _write_example_dags(self): example_dags = make_example_dags(example_dags_module) @@ -1232,6 +1242,143 @@ def test_write_dag_hashes_the_dag_once_when_updating_in_place(self, testing_dag_ assert serdag.dag_hash != original_hash assert serdag.dag_hash == SDM.hash(serdag.data) + @staticmethod + def _build_partitioned_dag(dag_id: str, assets, *, tasks: tuple[str, ...] = ("task1",)) -> DAG: + dag = DAG(dag_id=dag_id, schedule=PartitionedAssetTimetable(assets=assets)) + for task_id in tasks: + EmptyOperator(task_id=task_id, dag=dag) + return dag + + @staticmethod + def _read_dag_level_wakes(session, dag_id: str) -> list[str]: + return session.scalars( + select(AssetPartitionDagRunWake.reason).where( + AssetPartitionDagRunWake.target_dag_id == dag_id, AssetPartitionDagRunWake.apdr_id.is_(None) + ) + ).all() + + def _sync_partitioned_dag(self, dag_id: str, session, *, with_task_instances: bool = False) -> None: + scheduler_dag = sync_dag_to_db( + self._build_partitioned_dag(dag_id, Asset("a") & Asset("b")), session=session + ) + if with_task_instances: + scheduler_dag.create_dagrun( + run_id="test1", + run_after=DEFAULT_DATE, + state=DagRunState.QUEUED, + logical_date=None, + data_interval=None, + partition_key="k", + triggered_by=DagRunTriggeredByType.TEST, + run_type=DagRunType.MANUAL, + ) + # The first definition of a partitioned Dag asks for the evaluation of its pending runs. + assert self._read_dag_level_wakes(session, dag_id) == ["dag_changed"] + session.execute(delete(AssetPartitionDagRunWake)) + session.commit() + + @pytest.mark.parametrize("with_task_instances", [False, True], ids=["in-place", "new-version"]) + def test_write_dag_wakes_partition_dag_runs_when_the_timetable_changes( + self, testing_dag_bundle, session, with_task_instances + ): + dag_id = "partition_timetable_changed" + self._sync_partitioned_dag(dag_id, session, with_task_instances=with_task_instances) + + widened = self._build_partitioned_dag(dag_id, Asset("a") | Asset("b")) + SDM.write_dag(LazyDeserializedDAG.from_dag(widened), bundle_name="testing", session=session) + session.commit() + + assert self._read_dag_level_wakes(session, dag_id) == ["dag_changed"] + assert session.scalar(select(func.count()).where(SDM.dag_id == dag_id)) == ( + 2 if with_task_instances else 1 + ) + + def test_write_dag_does_not_wake_partition_dag_runs_when_only_tasks_change( + self, testing_dag_bundle, session + ): + dag_id = "partition_tasks_changed" + self._sync_partitioned_dag(dag_id, session) + + more_tasks = self._build_partitioned_dag(dag_id, Asset("a") & Asset("b"), tasks=("task1", "task2")) + assert SDM.write_dag(LazyDeserializedDAG.from_dag(more_tasks), bundle_name="testing", session=session) + session.commit() + + assert self._read_dag_level_wakes(session, dag_id) == [] + + @pytest.mark.parametrize("has_pending_run", [True, False]) + def test_write_dag_wakes_the_pending_runs_of_a_dag_that_is_no_longer_partitioned( + self, testing_dag_bundle, session, has_pending_run + ): + dag_id = "partition_turned_off" + self._sync_partitioned_dag(dag_id, session) + if has_pending_run: + session.add( + AssetPartitionDagRun(target_dag_id=dag_id, partition_key="k", pending_partition_key="k") + ) + session.commit() + + unpartitioned = DAG(dag_id=dag_id, schedule=None) + EmptyOperator(task_id="task1", dag=unpartitioned) + SDM.write_dag(LazyDeserializedDAG.from_dag(unpartitioned), bundle_name="testing", session=session) + session.commit() + + assert self._read_dag_level_wakes(session, dag_id) == (["dag_changed"] if has_pending_run else []) + + def test_write_dag_wakes_the_pending_runs_left_to_a_dag_written_for_the_first_time( + self, testing_dag_bundle, session + ): + """A Dag deleted and created again finds the pending runs that its earlier definition left behind.""" + dag_id = "partition_dag_created_again" + session.add(AssetPartitionDagRun(target_dag_id=dag_id, partition_key="k", pending_partition_key="k")) + session.commit() + + dag = DAG(dag_id=dag_id, schedule=None) + EmptyOperator(task_id="task1", dag=dag) + sync_dag_to_db(dag, session=session) + session.commit() + + assert self._read_dag_level_wakes(session, dag_id) == ["dag_changed"] + + @pytest.mark.parametrize( + ("with_task_instances", "code_write"), + [(False, "update_source_code"), (True, "write_code")], + ids=["in-place", "new-version"], + ) + def test_write_dag_adds_the_wake_row_before_the_writes_that_follow_the_definition( + self, testing_dag_bundle, session, with_task_instances, code_write + ): + """The Dag processor commits the definition even when a write after it fails, so the row must be in.""" + dag_id = "partition_code_write_fails" + self._sync_partitioned_dag(dag_id, session, with_task_instances=with_task_instances) + + widened = self._build_partitioned_dag(dag_id, Asset("a") | Asset("b")) + with mock.patch.object(DagCode, code_write, autospec=True, side_effect=ValueError("broken")): + with pytest.raises(ValueError, match="broken"): + SDM.write_dag(LazyDeserializedDAG.from_dag(widened), bundle_name="testing", session=session) + + assert [wake.reason for wake in session.new if isinstance(wake, AssetPartitionDagRunWake)] == [ + "dag_changed" + ] + + def test_row_read_again_after_a_rollback_serves_a_definition_written_in_place_since( + self, testing_dag_bundle, session + ): + dag_id = "decoded_data_after_rollback" + dag = DAG(dag_id=dag_id) + EmptyOperator(task_id="task1", dag=dag) + sync_dag_to_db(dag, session=session) + session.commit() + serdag = session.scalar(select(SDM).where(SDM.dag_id == dag_id)) + assert serdag.dag.task_ids == ["task1"] + + EmptyOperator(task_id="task2", dag=dag) + with create_session(scoped=False) as writer: + SDM.write_dag(LazyDeserializedDAG.from_dag(dag), bundle_name="testing", session=writer) + session.rollback() + + assert sorted(serdag.dag.task_ids) == ["task1", "task2"] + assert session.scalar(select(SDM).where(SDM.dag_id == dag_id)) is serdag + def test_deadline_uuid_regeneration_keeps_hash_consistent(self, testing_dag_bundle, session): """Regenerated deadline UUIDs rewrite dag.data, so the stored hash must describe the rewrite.""" dag_id = "test_hash_deadline_regenerated" diff --git a/airflow-core/tests/unit/utils/test_db_cleanup.py b/airflow-core/tests/unit/utils/test_db_cleanup.py index 9202e7c2fdcc6..7556fc14a032d 100644 --- a/airflow-core/tests/unit/utils/test_db_cleanup.py +++ b/airflow-core/tests/unit/utils/test_db_cleanup.py @@ -1326,6 +1326,7 @@ def test_no_models_missing(self): "dag_schedule_asset_uri_reference", # leave alone for now "task_outlet_asset_reference", # leave alone for now "asset_dag_run_queue", # self-managed + "asset_partition_dag_run_wake", # self-managed, by the scheduler "asset_event_dag_run", # foreign keys "task_instance_note", # foreign keys "dag_run_note", # foreign keys diff --git a/devel-common/src/tests_common/test_utils/db.py b/devel-common/src/tests_common/test_utils/db.py index 221fd2cbb3b70..b062e4ada7c57 100644 --- a/devel-common/src/tests_common/test_utils/db.py +++ b/devel-common/src/tests_common/test_utils/db.py @@ -466,10 +466,13 @@ def clear_db_pakl(): def clear_db_apdr(): if not AIRFLOW_V_3_2_PLUS: return - from airflow.models.asset import AssetPartitionDagRun + from airflow.models import asset with create_session() as session: - session.execute(delete(AssetPartitionDagRun)) + session.execute(delete(asset.AssetPartitionDagRun)) + # Wake rows outlive their run, and SQLite reuses run ids once the table is empty. + if wake := getattr(asset, "AssetPartitionDagRunWake", None): + session.execute(delete(wake)) @_retry_db diff --git a/docs/spelling_wordlist.txt b/docs/spelling_wordlist.txt index 1c34f312e1dd5..be3172f100c33 100644 --- a/docs/spelling_wordlist.txt +++ b/docs/spelling_wordlist.txt @@ -1530,6 +1530,7 @@ responder reStructuredText resultset resumable +resync retryable Reusability rfc diff --git a/shared/observability/src/airflow_shared/observability/metrics/metrics_template.yaml b/shared/observability/src/airflow_shared/observability/metrics/metrics_template.yaml index cfe767fd4d3fa..3363e31dd58e4 100644 --- a/shared/observability/src/airflow_shared/observability/metrics/metrics_template.yaml +++ b/shared/observability/src/airflow_shared/observability/metrics/metrics_template.yaml @@ -256,6 +256,25 @@ metrics: legacy_name: "-" name_variables: [] + - name: "scheduler.asset_partitions.evaluated" + description: "Number of pending partition-driven Dag runs the scheduler evaluated: the runs that a key + log, a timetable change or an asset activation asked it to evaluate, and the runs of its resync passes." + type: "counter" + legacy_name: "-" + name_variables: [] + + - name: "scheduler.asset_partitions.resync_fired" + description: "Number of partition-driven Dag runs created for pending runs that only a resync pass found, + with no request to evaluate them. Metric with pass tagging: ``startup`` for the pass a scheduler runs + when it starts, which also evaluates the runs left from before an upgrade, and ``periodic`` for the + later passes. A periodic count above 0 means that a change reached no request: key logs written without + one, for example by components that still run an older Airflow version or by a bulk insert; an alias + newly linked to an asset that was already active; or a result that changed while the timetable, the key + logs and the active assets did not, as with a custom partition mapper that reads the clock." + type: "counter" + legacy_name: "-" + name_variables: ["pass"] + - name: "scheduler.loop_exceptions" description: "Number of times the scheduler loop exited with an unhandled exception. Metric with exception_class tagging." @@ -407,6 +426,15 @@ metrics: legacy_name: "-" name_variables: [] + - name: "scheduler.asset_partitions.wake_lag" + description: "Seconds since the oldest waiting request to evaluate a pending partition-driven Dag run + of a Dag that can run (not paused, draining or stale) was written, as one scheduler last read it; 0 + once none is waiting. A value that keeps growing means the scheduler is asked to evaluate more runs + than it evaluates per loop." + type: "gauge" + legacy_name: "-" + name_variables: [] + - name: "asset.triggered_dagruns" description: "Number of Dag runs triggered by an asset update" type: "counter" @@ -913,6 +941,13 @@ metrics: legacy_name: "-" name_variables: [] + - name: "scheduler.asset_partitions.resync_pass_duration" + description: "Milliseconds one scheduler took to evaluate every pending partition-driven Dag run once, in + the resync pass it runs when it starts or in a periodic one." + type: "timer" + legacy_name: "-" + name_variables: [] + - name: "scheduler.scheduler_loop_duration" description: "Milliseconds spent running one scheduler loop" type: "timer"