Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 25 additions & 0 deletions airflow-core/docs/authoring-and-scheduling/assets.rst
Original file line number Diff line number Diff line change
Expand Up @@ -702,6 +702,31 @@ uses ``StartOfDayMapper`` (which expects a timestamp-like key), no downstream
partition match can be produced, so the downstream Dag is not triggered for
that key.

While partitions wait for their upstream events, Airflow keeps pending
partition-driven Dag runs for them. The scheduler evaluates a pending run, and creates
its Dag run once the condition is met, when something happens that can change the
result: an asset event reaches the run, the Dag's timetable changes, or an asset that
the Dag is scheduled on becomes active. A run that keeps waiting, such as a rollup whose
window is not complete yet, is not evaluated again until then, so it cannot hold the
others back. Each scheduler also evaluates every pending run once when it starts, and
then once every :ref:`[scheduler] asset_partition_resync_interval
<config:scheduler__asset_partition_resync_interval>`. These passes catch asset events
that were recorded without a request for evaluation, for example by components that
still run an older Airflow version during an upgrade. The
``scheduler.asset_partitions.wake_lag`` metric reports how long the oldest request has
waited, and ``scheduler.asset_partitions.resync_fired`` counts the Dag runs that only
such a pass found.

This relies on the evaluation of a pending run depending only on the Dag's timetable,
the partition keys that reached the run, which assets are active, and how the Dag's
asset references and aliases resolve. A new link from an alias to an asset that is
already active is picked up only by the next pass. Custom partition mappers, rollup
windows and timetables must keep to this: for example, a mapper that reads the clock,
or fails now and then, has its runs evaluated again only by the next pass. A run whose
evaluation fails because of the Dag itself, for example because its mapper raises, is
not evaluated again on every loop either: only when an asset event reaches it, the Dag's
timetable changes, an asset that the Dag is scheduled on becomes active, or a pass runs.

Inside partitioned Dag runs, access the resolved partition through
``dag_run.partition_key``. When the consumer's partition mapper can
resolve the key to a ``datetime``, that value is also available as
Expand Down
4 changes: 3 additions & 1 deletion airflow-core/docs/migrations-ref.rst
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,9 @@ Here's the list of all the Database Migrations that are executed via when you ru
+-------------------------+------------------+-------------------+--------------------------------------------------------------+
| Revision ID | Revises ID | Airflow Version | Description |
+=========================+==================+===================+==============================================================+
| ``d8c6f3a129b0`` (head) | ``687eb253de24`` | ``3.5.0`` | Add durable Dag processor work claims. |
| ``6b0e4c1f2a37`` (head) | ``d8c6f3a129b0`` | ``3.5.0`` | Add asset_partition_dag_run_wake. |
+-------------------------+------------------+-------------------+--------------------------------------------------------------+
| ``d8c6f3a129b0`` | ``687eb253de24`` | ``3.5.0`` | Add durable Dag processor work claims. |
+-------------------------+------------------+-------------------+--------------------------------------------------------------+
| ``687eb253de24`` | ``a8b48633fa60`` | ``3.5.0`` | Add Dag processor publication receipts and source revisions. |
+-------------------------+------------------+-------------------+--------------------------------------------------------------+
Expand Down
28 changes: 28 additions & 0 deletions airflow-core/newsfragments/76706.significant.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
Pending partition-driven Dag runs are evaluated only when their inputs change

The scheduler no longer evaluates the oldest pending partition-driven Dag runs on every loop. It
evaluates a pending run when an asset event reaches it, when its Dag's timetable changes, or when an
asset that its Dag is scheduled on becomes active. Each scheduler also evaluates every pending run once
when it starts, and then once every ``[scheduler] asset_partition_resync_interval`` (3600 seconds by
default).

The evaluation of a pending run must depend only on the Dag's timetable, the partition keys that reached
the run, which assets are active, and how the Dag's asset references and aliases resolve. A custom
partition mapper, rollup window or timetable that reads the clock, or that fails now and then, has its
runs evaluated again only by the next pass. So do runs whose key logs are written in bulk rather than
through a session, and runs that a new link from an alias to an asset that is already active satisfies.

Upgrade the schedulers after the other components, or restart them once every component runs this
version: components that run an older version do not ask for an evaluation, so their asset events are
picked up by the next pass.

* Types of change

* [ ] Dag changes
* [x] Config changes
* [ ] API changes
* [ ] CLI changes
* [x] Behaviour changes
* [ ] Plugin changes
* [ ] Dependency changes
* [ ] Code interface changes
Original file line number Diff line number Diff line change
Expand Up @@ -278,9 +278,9 @@ def next_run_assets(
return NextRunAssetsResponse.model_validate(model_data)

# Partitioned Dags: enrich with per-asset received/required counts and rollup flag.
# FIFO matches the scheduler's pending-APDR processing order
# (``_create_dagruns_for_partitioned_asset_dags``), so the "next run" the UI
# surfaces is the same one the scheduler will fire next.
# This shows the pending APDR that has waited longest. It is not necessarily the
# next run: the scheduler (``_create_dagruns_for_partitioned_asset_dags``) fires
# whichever pending APDR is satisfied, whatever its age.
pending_apdr = session.execute(
select(AssetPartitionDagRun.id, AssetPartitionDagRun.partition_key)
.where(
Expand Down
14 changes: 14 additions & 0 deletions airflow-core/src/airflow/config_templates/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -2984,6 +2984,20 @@ scheduler:
type: integer
example: ~
default: "1000"
asset_partition_resync_interval:
description: |
How often (in seconds) the scheduler evaluates every pending partition-driven Dag run once,
whether or not anything asked for it. Otherwise the scheduler evaluates a pending run only
when a key log is added to it, when its Dag's timetable changes, or when an asset its Dag is
scheduled on is activated. This pass also evaluates key logs that were written without such a
request: for example, by components that still run an older Airflow version during a rolling
upgrade, or by code that inserts key logs in bulk rather than through a session. Each scheduler
also runs one pass when it starts. A pass starts when the previous one is complete and this
interval has passed since that one started.
version_added: 3.5.0
type: integer
example: ~
default: "3600"
use_job_schedule:
description: |
Set to ``False`` to stop the scheduler creating Dag runs on its own, whether a Dag is due
Expand Down
24 changes: 22 additions & 2 deletions airflow-core/src/airflow/dag_processing/collection.py
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@
from typing import TYPE_CHECKING, Any, NamedTuple, TypeVar

import structlog
from sqlalchemy import delete, false, func, insert, or_, select, tuple_, update
from sqlalchemy import delete, false, func, insert, inspect, or_, select, tuple_, update
from sqlalchemy.exc import OperationalError
from sqlalchemy.orm import Session, joinedload, load_only

Expand All @@ -44,6 +44,7 @@
AssetActive,
AssetAliasModel,
AssetModel,
AssetPartitionDagRunWake,
DagScheduleAssetAliasReference,
DagScheduleAssetNameReference,
DagScheduleAssetReference,
Expand All @@ -70,6 +71,7 @@
from airflow.serialization.enums import Encoding
from airflow.serialization.serialized_objects import BaseSerialization, DagSerialization, LazyDeserializedDAG
from airflow.triggers.base import BaseEventTrigger
from airflow.utils.helpers import chunks
from airflow.utils.retries import MAX_DB_RETRIES, run_with_db_retries
from airflow.utils.sqlalchemy import build_upsert_stmt, get_dialect_name, with_row_locks
from airflow.utils.types import DagRunType
Expand Down Expand Up @@ -1207,7 +1209,9 @@ def sync_assets(self, *, session: Session) -> dict[tuple[str, str], AssetModel]:
orm_assets: dict[tuple[str, str], AssetModel] = {
(am.name, am.uri): am
for am in session.scalars(
select(AssetModel).where(tuple_(AssetModel.name, AssetModel.uri).in_(self.assets))
select(AssetModel)
.where(tuple_(AssetModel.name, AssetModel.uri).in_(self.assets))
.options(joinedload(AssetModel.active))
)
}
for key, model in orm_assets.items():
Expand Down Expand Up @@ -1268,8 +1272,24 @@ def activate_assets_if_possible(self, models: Iterable[AssetModel], *, session:
from sqlalchemy.dialects.sqlite import insert as sqlite_insert

stmt = sqlite_insert(AssetActive).on_conflict_do_nothing()
models = list(models)
if values := [{"name": m.name, "uri": m.uri} for m in models]:
session.execute(stmt, values)
# An inactive asset may hold key logs of pending partition Dag runs, which activating it can
# satisfy. sync_assets loads whether each existing asset is active; the assets it creates have
# no key logs, and their activation state is not loaded.
if inactive := [m for m in models if inspect(m).attrs.active.loaded_value is None]:
active = {
pair
for chunk in chunks(inactive, 400)
for pair in session.execute(
select(AssetActive.name, AssetActive.uri).where(
tuple_(AssetActive.name, AssetActive.uri).in_([(m.name, m.uri) for m in chunk])
)
).tuples()
}
if activated := [m for m in inactive if (m.name, m.uri) in active]:
AssetPartitionDagRunWake.add_for_activated_assets(activated, session=session)

def add_dag_asset_references(
self,
Expand Down
Loading
Loading