Repository navigation
Send each queued stub task the artifact it is bound to - #74137
Draft
jason810496 wants to merge 5 commits into
Conversation
The scheduler has to send each stub task the artifact the Dag processor bound it to, because the worker reads nothing from the database. get_task_handler_artifact_refs reads those bindings for a batch of task instances in one query, keyed by Dag and task id. It looks up only task instances on a queue that [sdk] queue_to_coordinator routes to a coordinator, using the same "default" fallback as the worker, so it costs no query without that config. It always names the artifact's bundle without a version. An [sdk] configuration that cannot be read is logged and gives no reference, which includes a coordinator whose task_handler_bundle_name is not a configured Dag bundle on this host. ExecuteTask.make accepts the reference. Nothing calls either yet.
After the scheduler picks the task instances to queue, it reads the bindings of the routed ones once and puts each reference on that task's workload, so a worker can run the file the Dag processor validated. The read happens once per critical section, for every executor, and only when an [sdk] queue routes some queued task to a coordinator. A task without a binding, such as a Python task, a stub task on a queue no coordinator serves, or a stub task the Dag processor has not bound yet, is queued without a reference. The scheduler never fails a task for a missing binding. A scheduler that lacks the Dag processor's [sdk] configuration or its task handler Dag bundles sends no reference at all.
dag.test(use_executor=True) builds the task workload itself, outside the scheduler, so a stub task would reach a Lang-SDK coordinator without its artifact. It now looks up the binding of the task it sends. A stub task gets a reference only if a Dag processor has bound it, so a Dag that no Dag processor has parsed runs as before.
The scheduler now reads [sdk] queue_to_coordinator to find the queued tasks on a routed queue, and each routed coordinator's task_handler_bundle_name must be a Dag bundle in its [dag_processor] dag_bundle_config_list. The configuration reference, the Language SDK index and the Go, Java and TypeScript pages and READMEs say the scheduler needs the same [sdk] configuration as the Dag processor, but not the language runtime or the artifact files. A scheduler that cannot read the configuration logs a warning and sends no artifacts. The Lang SDK spec words the reference's bundle_info as the bundle name, with null for the task's own Dag bundle.
Flow 2 still described a binding loaded with the task instance query, and a scheduler that fails a stub task without a binding. It now describes the read that follows the selection of the task instances to queue, which covers only routed queues, costs one query and takes no row locks, and says the scheduler never fails a task. The reference always names the artifact's bundle without a version, and the worker, not the scheduler, decides that a reference names the task's own Dag bundle. Flow 3 says the worker reads such a bundle at the version the run uses, and the Consequences count a bundle name in every payload. The Consequences note that the scheduler reads [sdk] and the task handler Dag bundles, and that scheduling costs nothing without queue_to_coordinator. A workload without a reference also covers a scheduler that cannot read [sdk], which warns only on a loop that queues a task.
This was referenced Oct 3, 2026
This was referenced Oct 3, 2026
1 task done
Contributor
|
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions. |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stack (bottom to top): #73970, #73971, #73972, #73973, #73974, #73975, #73976, #73977, #74030, #74031, #74032, #74033, #74067, #74135, #74136, #74137, #74138, #74139, #74140, #74141
Why
The worker reads no database, so a stub task can run the artifact the Dag processor bound it to only if its workload names that artifact (ADR-0013 "Objects on the wire"). The previous PR added that reference to
ExecuteTask. This PR has the scheduler fill it: after it picks the task instances to queue, it reads the bindings of the ones on a routed queue and sends each task its artifact. Without[sdk] queue_to_coordinatorthe scheduler reads no binding, so behavior is unchanged.What changes
The scheduler now needs the same
[sdk]configuration as the Dag processor, including the task handler Dag bundles in its[dag_processor] dag_bundle_config_list. It needs no artifact files or language runtime for that.The lookup.
get_task_handler_artifact_refs(task_instances, *, session)inmodels/lang_sdk_task_handler.pykeeps the task instances whose queuequeue_to_coordinatorroutes to a coordinator (aNonequeue routes as"default", like the worker), then reads their(dag_id, task_id)bindings joined to their artifacts with one plainSELECTand no row locks.ExecuteTask.maketakes the result astask_handler_artifact.In the scheduler.
_critical_section_enqueue_task_instancescalls it right after_select_task_instances_to_queueand passes the result to every_enqueue_task_instances_with_queued_statecall, so the read covers only the task instances that were queued, for all executors at once. It needs only columns the selected task instances already hold.[sdk] queue_to_coordinator, or an[sdk]the scheduler cannot readThe reference. It always names the artifact's bundle, with no version. The worker already reads a reference to the task's own Dag bundle at the version the run uses, so a pinned run keeps its version and the Dag bundle's
version_datais never sent twice. A binding records its Dag's bundle at the last parse, which can lag behind the Dag's current bundle, so the scheduler does not compare the two: the worker decides whether a reference names the task's own bundle.The scheduler never fails a task. A routed task without a binding (a Python task, a task of a Dag defined in a Lang SDK, or a stub task the Dag processor has not bound yet) is queued without a reference. A stub task on a queue no coordinator serves is not looked up at all.
A scheduler that cannot read
[sdk]logs a warning on each scheduling loop that queues a task and queues every task without a reference, whatever the failure. That covers a value that is not valid JSON, a value of the wrong JSON type such as an array forcoordinators, a queue routed to an unknown coordinator, adag_bundle_config_listthat cannot be parsed, and atask_handler_bundle_namethat is not a Dag bundle in the scheduler'sdag_bundle_config_list, which is rejected the same way wherever[sdk]is read.dag.test(use_executor=True)looks up the binding of the task it sends, so the workload of a stub task a Dag processor has bound names its artifact.Not changed. The ECS and Batch adoption commands (their relaunch command carries an empty token today) and Kubernetes
generate_pod_command_args(display only) build workloads outside the scheduler loop and send no reference.Known limits. Until the next PR, a binding can name a file its bundle no longer has: the file was renamed before the next parse, or is missing at the version a pinned run uses. The try then fails before the runtime starts, with no reason in the task log, and retries apply.
Docs. The
[sdk]reference, the Language SDK guide, the Go, Java and TypeScript pages and their READMEs say the scheduler needs the configuration. ADR-0013 Flow 2 describes the read, and its failure handling says the scheduler never fails a task.How to test
Ran:
dag.testtests: 758 passed, 4 skipped (a Postgres or MySQL only test, two long-running tests, and a test that needs row-level locking).executors/: 259 passed, 1 skipped (non-fork spawning only).lang_sdk_task_handler_pkeyandlang_sdk_task_handler_artifact_pkeyon Postgres, andPRIMARYas a range scan then aneq_refjoin on MySQL, both on tables seeded with 1000 bindings and 500 artifacts, with seq scans disabled on Postgres.prek run --from-ref <parent> --stage pre-commitpassed.mypy-airflow-coreandmypy-task-sdkpassed with--all-files. The schema snapshot and Go SDK generated files are unchanged, since no supervisor message changed.EXPLAIN statement for the lookup of 10 task instances
Each new test fails without its change. Tests that assert a negative (no reference, no query) fail when the lookup stops filtering on the routed queue.
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines