Skip to content

Task SDK: send error response on IPC decode failure - #76555

Merged
shahar1 merged 1 commit into
apache:mainfrom
meel-ayush:fix-task-sdk-supervisor-ipc-deadlock
Nov 13, 2026
Merged

shahar1 merged 1 commit into
apache:mainfrom
meel-ayush:fix-task-sdk-supervisor-ipc-deadlock

Conversation

@meel-ayush

Copy link
Copy Markdown
Contributor

Description

When handle_requests() fails to deserialize or validate an incoming request frame — due to a missing discriminator, schema mismatch, or malformed payload — it was logging the exception and issuing continue. No response frame was sent back.

The task runner's CommsDecoder.send() blocks on socket.recv(4) waiting for the 4-byte length header of the reply. With nothing coming, the child process deadlocks permanently. The supervisor's heartbeat loop keeps running normally, so the scheduler's zombie detector never fires. The task stays stuck in RUNNING and holds worker capacity until manually killed.

The fix catches the exception as e and replies with ErrorResponse(error=ErrorType.GENERIC_ERROR) before continue, matching the existing pattern used for ServerResponseError and unhandled exceptions later in the same method. The send is wrapped in suppress(Exception) so a secondary socket error during the reply cannot crash the generator loop itself.

closes: #76550

Tests

  • Added test_handle_requests_decoding_error_sends_error_response in task-sdk/tests/task_sdk/execution_time/test_supervisor.py, parametrized over an unknown message type and a message missing required fields. Each case asserts the response frame carries a GENERIC_ERROR, the task runner side raises AirflowRuntimeError, and the generator loop stays alive for the next request.

Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

When handle_requests() fails to deserialize or validate an incoming request frame, it logged the exception and issued continue. The task runner's recv(4) call was left blocking with nothing ever coming back, deadlocking the child process indefinitely. Meanwhile the supervisor kept heartbeating the API server, so the scheduler's zombie detector never fired — the task would stay stuck in RUNNING until someone killed it manually. Catching the exception and replying with ErrorResponse(GENERIC_ERROR) before continue lets the task runner raise AirflowRuntimeError and exit cleanly.
@meel-ayush
meel-ayush force-pushed the fix-task-sdk-supervisor-ipc-deadlock branch from 7210fd3 to bbec2f4 Compare November 10, 2026 10:36

@bingqin2 bingqin2 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested on main at da7dab3 on macOS. supervisor.py and test_supervisor.py haven't changed between that commit and 4d1bc90.

I also checked this with a real task process. After startup the task sends a frame the supervisor can't decode, then waits for the reply. I tried two frames: {"type": "NonExistentMessageType"}, and {"type": "GetVariable"} without key.

  • On main, in both cases the task process is still blocked in recv 5 s later, as described in #76550.
  • With this change, it immediately gets an AirflowRuntimeError with GENERIC_ERROR and exits.

The new tests fail on main and pass with this change. The rest of test_supervisor.py passes, except 4 tests that fail the same way on main in my environment (they see an extra "Secrets backends loaded for worker" log line). ruff and mypy pass on the changed files.

@meel-ayush

Copy link
Copy Markdown
Contributor Author

Thanks for taking the time to test this with a real task process, @bingqin2 Really glad to hear both invalid message cases unblock cleanly now.

Appreciate the review and approval!

@shahar1
shahar1 enabled auto-merge (squash) November 13, 2026 13:04
@shahar1
shahar1 merged commit f8c7ead into apache:main Nov 13, 2026
104 checks passed
@boring-cyborg

boring-cyborg Bot commented Nov 13, 2026

Copy link
Copy Markdown

Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions.

@meel-ayush
meel-ayush deleted the fix-task-sdk-supervisor-ipc-deadlock branch November 13, 2026 14:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Task SDK: Supervisor IPC deadlocks child task runner on unvalidated request frames

4 participants