Skip to content

Send each queued stub task the artifact it is bound to - #74137

Draft
jason810496 wants to merge 5 commits into
jason/core-taskhandler-refactor/15-task-handler-artifact-reffrom
jason/core-taskhandler-refactor/16-send-task-handler-artifact
Draft

jason810496 wants to merge 5 commits into
jason/core-taskhandler-refactor/15-task-handler-artifact-reffrom
jason/core-taskhandler-refactor/16-send-task-handler-artifact

Conversation

@jason810496

@jason810496 jason810496 commented Oct 3, 2026 •

Copy link
Copy Markdown
Member

Stack (bottom to top): #73970, #73971, #73972, #73973, #73974, #73975, #73976, #73977, #74030, #74031, #74032, #74033, #74067, #74135, #74136, #74137, #74138, #74139, #74140, #74141

Why

The worker reads no database, so a stub task can run the artifact the Dag processor bound it to only if its workload names that artifact (ADR-0013 "Objects on the wire"). The previous PR added that reference to ExecuteTask. This PR has the scheduler fill it: after it picks the task instances to queue, it reads the bindings of the ones on a routed queue and sends each task its artifact. Without [sdk] queue_to_coordinator the scheduler reads no binding, so behavior is unchanged.

What changes

# The scheduler, once per scheduling loop, after it picks the task instances to queue
refs = get_task_handler_artifact_refs(queued_tis, session=session)
# {("etl", "load"): TaskHandlerArtifactRef(bundle_info=BundleInfo(name="go-task-handlers"), rel_path="bin/etl")}
workload = ExecuteTask.make(ti, generator=..., task_handler_artifact=refs.get((ti.dag_id, ti.task_id)))

The scheduler now needs the same [sdk] configuration as the Dag processor, including the task handler Dag bundles in its [dag_processor] dag_bundle_config_list. It needs no artifact files or language runtime for that.

[dag_processor]
dag_bundle_config_list = [..., {"name": "go-task-handlers", "classpath": "airflow.dag_processing.bundles.local.LocalDagBundle", "kwargs": {"path": "/opt/airflow/go-task-handlers"}}]

[sdk]
coordinators = {"go": {"classpath": "airflow.sdk.coordinators.executable.ExecutableCoordinator", "kwargs": {"task_handler_bundle_name": "go-task-handlers"}}}
queue_to_coordinator = {"golang": "go"}
  • The lookup. get_task_handler_artifact_refs(task_instances, *, session) in models/lang_sdk_task_handler.py keeps the task instances whose queue queue_to_coordinator routes to a coordinator (a None queue routes as "default", like the worker), then reads their (dag_id, task_id) bindings joined to their artifacts with one plain SELECT and no row locks. ExecuteTask.make takes the result as task_handler_artifact.

  • In the scheduler. _critical_section_enqueue_task_instances calls it right after _select_task_instances_to_queue and passes the result to every _enqueue_task_instances_with_queued_state call, so the read covers only the task instances that were queued, for all executors at once. It needs only columns the selected task instances already hold.

    Situation Extra queries per scheduling loop
    Nothing queued 0
    No [sdk] queue_to_coordinator, or an [sdk] the scheduler cannot read 0
    No queued task on a routed queue 0
    Any queued task on a routed queue 1, whatever the number of tasks or executors
  • The reference. It always names the artifact's bundle, with no version. The worker already reads a reference to the task's own Dag bundle at the version the run uses, so a pinned run keeps its version and the Dag bundle's version_data is never sent twice. A binding records its Dag's bundle at the last parse, which can lag behind the Dag's current bundle, so the scheduler does not compare the two: the worker decides whether a reference names the task's own bundle.

  • The scheduler never fails a task. A routed task without a binding (a Python task, a task of a Dag defined in a Lang SDK, or a stub task the Dag processor has not bound yet) is queued without a reference. A stub task on a queue no coordinator serves is not looked up at all.

  • A scheduler that cannot read [sdk] logs a warning on each scheduling loop that queues a task and queues every task without a reference, whatever the failure. That covers a value that is not valid JSON, a value of the wrong JSON type such as an array for coordinators, a queue routed to an unknown coordinator, a dag_bundle_config_list that cannot be parsed, and a task_handler_bundle_name that is not a Dag bundle in the scheduler's dag_bundle_config_list, which is rejected the same way wherever [sdk] is read.

  • dag.test(use_executor=True) looks up the binding of the task it sends, so the workload of a stub task a Dag processor has bound names its artifact.

  • Not changed. The ECS and Batch adoption commands (their relaunch command carries an empty token today) and Kubernetes generate_pod_command_args (display only) build workloads outside the scheduler loop and send no reference.

  • Known limits. Until the next PR, a binding can name a file its bundle no longer has: the file was renamed before the next parse, or is missing at the version a pinned run uses. The try then fails before the runtime starts, with no reason in the task log, and retries apply.

  • Docs. The [sdk] reference, the Language SDK guide, the Go, Java and TypeScript pages and their READMEs say the scheduler needs the configuration. ADR-0013 Flow 2 describes the read, and its failure handling says the scheduler never fails a task.

How to test

uv run --project airflow-core --with-editable shared/secrets_masker pytest airflow-core/tests/unit/models/test_lang_sdk_task_handler.py airflow-core/tests/unit/jobs/test_scheduler_job.py airflow-core/tests/unit/models/test_dag.py -q
uv run --project airflow-core --with-editable shared/secrets_masker pytest airflow-core/tests/unit/executors/ -q
breeze run --backend postgres pytest --with-db-init airflow-core/tests/unit/models/test_lang_sdk_task_handler.py airflow-core/tests/unit/jobs/test_scheduler_job.py -k "TestGetTaskHandlerArtifactRefs or TestSchedulerJobTaskHandlerArtifacts or critical_section_enqueue or enqueue_task_instances"
breeze run --backend mysql pytest --with-db-init airflow-core/tests/unit/models/test_lang_sdk_task_handler.py airflow-core/tests/unit/jobs/test_scheduler_job.py -k "TestGetTaskHandlerArtifactRefs or TestSchedulerJobTaskHandlerArtifacts or critical_section_enqueue or enqueue_task_instances"
prek run generate-supervisor-schemas-snapshot check-go-sdk-generated-drift --all-files

Ran:

  • airflow-core model, scheduler and dag.test tests: 758 passed, 4 skipped (a Postgres or MySQL only test, two long-running tests, and a test that needs row-level locking).
  • airflow-core executors/: 259 passed, 1 skipped (non-fork spawning only).
  • The new model tests and scheduler tests, plus the existing critical section and enqueue tests, on Postgres and on MySQL through breeze: 36 passed on each. The lookup for 10 task instances uses lang_sdk_task_handler_pkey and lang_sdk_task_handler_artifact_pkey on Postgres, and PRIMARY as a range scan then an eq_ref join on MySQL, both on tables seeded with 1000 bindings and 500 artifacts, with seq scans disabled on Postgres.
  • prek run --from-ref <parent> --stage pre-commit passed. mypy-airflow-core and mypy-task-sdk passed with --all-files. The schema snapshot and Go SDK generated files are unchanged, since no supervisor message changed.
EXPLAIN statement for the lookup of 10 task instances
EXPLAIN
SELECT lang_sdk_task_handler.dag_id, lang_sdk_task_handler.task_id,
       lang_sdk_task_handler_artifact.bundle_name, lang_sdk_task_handler_artifact.relative_fileloc
FROM lang_sdk_task_handler
JOIN lang_sdk_task_handler_artifact ON lang_sdk_task_handler_artifact.id = lang_sdk_task_handler.artifact_id
WHERE (lang_sdk_task_handler.dag_id, lang_sdk_task_handler.task_id)
      IN (('explain_0', 'task_0'), ('explain_1', 'task_0'), ... ('explain_9', 'task_0'))

Each new test fails without its change. Tests that assert a negative (no reference, no query) fail when the lookup stops filtering on the routed queue.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code (Opus 5.5) following the guidelines

The scheduler has to send each stub task the artifact the Dag processor bound it to, because the worker reads nothing from the database. get_task_handler_artifact_refs reads those bindings for a batch of task instances in one query, keyed by Dag and task id. It looks up only task instances on a queue that [sdk] queue_to_coordinator routes to a coordinator, using the same "default" fallback as the worker, so it costs no query without that config. It always names the artifact's bundle without a version. An [sdk] configuration that cannot be read is logged and gives no reference, which includes a coordinator whose task_handler_bundle_name is not a configured Dag bundle on this host. ExecuteTask.make accepts the reference. Nothing calls either yet.
After the scheduler picks the task instances to queue, it reads the bindings of the routed ones once and puts each reference on that task's workload, so a worker can run the file the Dag processor validated. The read happens once per critical section, for every executor, and only when an [sdk] queue routes some queued task to a coordinator. A task without a binding, such as a Python task, a stub task on a queue no coordinator serves, or a stub task the Dag processor has not bound yet, is queued without a reference. The scheduler never fails a task for a missing binding. A scheduler that lacks the Dag processor's [sdk] configuration or its task handler Dag bundles sends no reference at all.
dag.test(use_executor=True) builds the task workload itself, outside the scheduler, so a stub task would reach a Lang-SDK coordinator without its artifact. It now looks up the binding of the task it sends. A stub task gets a reference only if a Dag processor has bound it, so a Dag that no Dag processor has parsed runs as before.
The scheduler now reads [sdk] queue_to_coordinator to find the queued tasks on a routed queue, and each routed coordinator's task_handler_bundle_name must be a Dag bundle in its [dag_processor] dag_bundle_config_list. The configuration reference, the Language SDK index and the Go, Java and TypeScript pages and READMEs say the scheduler needs the same [sdk] configuration as the Dag processor, but not the language runtime or the artifact files. A scheduler that cannot read the configuration logs a warning and sends no artifacts. The Lang SDK spec words the reference's bundle_info as the bundle name, with null for the task's own Dag bundle.
Flow 2 still described a binding loaded with the task instance query, and a scheduler that fails a stub task without a binding. It now describes the read that follows the selection of the task instances to queue, which covers only routed queues, costs one query and takes no row locks, and says the scheduler never fails a task. The reference always names the artifact's bundle without a version, and the worker, not the scheduler, decides that a reference names the task's own Dag bundle. Flow 3 says the worker reads such a bundle at the version the run uses, and the Consequences count a bundle name in every payload. The Consequences note that the scheduler reads [sdk] and the task handler Dag bundles, and that scheduling costs nothing without queue_to_coordinator. A workload without a reference also covers a scheduler that cannot read [sdk], which warns only on a loop that queues a task.
@jason810496
jason810496 added this pull request to stack #73978 October 3, 2026 07:56
@jason810496 jason810496 changed the title jason/core taskhandler refactor/16 send task handler artifact Send each queued stub task the artifact it is bound to Oct 3, 2026
@jason810496
jason810496 removed this pull request from stack #73978 October 6, 2026 02:27
@github-actions

Copy link
Copy Markdown
Contributor

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.

@github-actions github-actions Bot added the stale Stale PRs per the .github/workflows/stale.yml policy file label Nov 18, 2026
@jason810496 jason810496 removed the stale Stale PRs per the .github/workflows/stale.yml policy file label Nov 18, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant