Repository navigation
Fix dag.test() failing for task ids with non-ASCII characters - #74472
Open
per-andersen wants to merge 3 commits into
Open
per-andersen wants to merge 3 commits into
per-andersen wants to merge 3 commits into
Conversation
|
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contributors' Guide
|
Contributor
|
Thanks for the the fix. non-blocking thought: the wrapper also normalizes |
fat-catTW
approved these changes
Oct 9, 2026
per-andersen
force-pushed
the
dagtest-fails-for-tasks-whose-task_id-contains-non-ascii-characters
branch
from
October 9, 2026 19:12
2e6f578 to
57d5a31
Compare
Author
|
Thanks for the review! Updated:
|
2 tasks done
fat-catTW
suggested changes
Oct 15, 2026
per-andersen
force-pushed
the
dagtest-fails-for-tasks-whose-task_id-contains-non-ascii-characters
branch
from
October 18, 2026 11:40
1afe394 to
df3827d
Compare
fat-catTW
approved these changes
Oct 18, 2026
fat-catTW
left a comment
Contributor
There was a problem hiding this comment.
Thanks for the update!
LGTM!
pierrejeambrun
enabled auto-merge (squash)
November 3, 2026 12:53
auto-merge was automatically disabled
November 7, 2026 13:39
Head branch was pushed to by a user without write access
per-andersen
force-pushed
the
dagtest-fails-for-tasks-whose-task_id-contains-non-ascii-characters
branch
from
November 7, 2026 13:39
df3827d to
90e1711
Compare
per-andersen
force-pushed
the
dagtest-fails-for-tasks-whose-task_id-contains-non-ascii-characters
branch
3 times, most recently
from
November 11, 2026 18:49
cfbaed6 to
11f6f56
Compare
per-andersen
force-pushed
the
dagtest-fails-for-tasks-whose-task_id-contains-non-ascii-characters
branch
2 times, most recently
from
November 11, 2026 20:39
3da944e to
94f4fb5
Compare
A task id like "Købsmoms" passes task id validation, but dag.test() failed as soon as such a task pushed an XCom. In-process execution sends requests through httpx.WSGITransport, which puts the decoded URL path into PATH_INFO as a plain str. a2wsgi follows PEP 3333 and expects UTF-8 bytes decoded as latin-1, so any non-ASCII character in the path raised UnicodeDecodeError or UnicodeEncodeError and the task failed. Requests over a real HTTP connection are not affected.
The in-process Execution API transport now uses httpx2, and test_app.py no longer imports httpx after main was merged in.
The in-process transport moved from httpx to httpx2.
per-andersen
force-pushed
the
dagtest-fails-for-tasks-whose-task_id-contains-non-ascii-characters
branch
from
November 12, 2026 16:40
94f4fb5 to
71a5da4
Compare
Author
|
Do you need anything further from me? Or can this fix be merged to main? |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
dag.test()fails for any task whosetask_idcontains a non-ASCII character (for exampleKøbsmoms) as soon as the task pushes an XCom, even thoughvalidate_keyaccepts these ids.dag.test()serves the Execution API in-process: the Task SDK client talks to a2wsgi'sASGIMiddlewarethroughhttpx.WSGITransport. httpx puts the percent-decoded path intoPATH_INFOas a plainstr, while a2wsgi follows PEP 3333 and decodesPATH_INFOas UTF-8 bytes carried in a latin-1 string. A non-ASCII path segment therefore fails ina2wsgi.asgi.build_scope, withUnicodeDecodeErrorfor characters such asøandUnicodeEncodeErrorfor characters outside Latin-1 such asタ.This PR subclasses a2wsgi's
ASGIMiddlewareinInProcessExecutionAPI.transportso thatPATH_INFOandSCRIPT_NAMEare re-encoded to PEP 3333 form before a2wsgi builds the ASGI scope. Only in-process execution is affected; over a real HTTP connection the ASGI server decodes the path correctly.closes: #74469
Testing:
test_in_process_execution_api_handles_non_ascii_pathandtest_in_process_execution_api_handles_non_ascii_script_name, which fail without this change and pass with it.dag.test()in Breeze: onmainthe XCom push fails withUnicodeDecodeErrorand the Dag run fails; with this change the Dag run succeeds.prekpre-commit and manual stages pass for the changed files.Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Opus 5.5) following the guidelines