Repository navigation
Task SDK: send error response on IPC decode failure - #76555
Conversation
f7ebb04 to
7210fd3
Compare
When handle_requests() fails to deserialize or validate an incoming request frame, it logged the exception and issued continue. The task runner's recv(4) call was left blocking with nothing ever coming back, deadlocking the child process indefinitely. Meanwhile the supervisor kept heartbeating the API server, so the scheduler's zombie detector never fired — the task would stay stuck in RUNNING until someone killed it manually. Catching the exception and replying with ErrorResponse(GENERIC_ERROR) before continue lets the task runner raise AirflowRuntimeError and exit cleanly.
7210fd3 to
bbec2f4
Compare
bingqin2
left a comment
There was a problem hiding this comment.
Tested on main at da7dab3 on macOS. supervisor.py and test_supervisor.py haven't changed between that commit and 4d1bc90.
I also checked this with a real task process. After startup the task sends a frame the supervisor can't decode, then waits for the reply. I tried two frames: {"type": "NonExistentMessageType"}, and {"type": "GetVariable"} without key.
- On main, in both cases the task process is still blocked in
recv5 s later, as described in #76550. - With this change, it immediately gets an
AirflowRuntimeErrorwithGENERIC_ERRORand exits.
The new tests fail on main and pass with this change. The rest of test_supervisor.py passes, except 4 tests that fail the same way on main in my environment (they see an extra "Secrets backends loaded for worker" log line). ruff and mypy pass on the changed files.
|
Thanks for taking the time to test this with a real task process, @bingqin2 Really glad to hear both invalid message cases unblock cleanly now. Appreciate the review and approval! |
|
Awesome work, congrats on your first merged pull request! You are invited to check our Issue Tracker for additional contributions. |
Description
When
handle_requests()fails to deserialize or validate an incoming request frame — due to a missing discriminator, schema mismatch, or malformed payload — it was logging the exception and issuingcontinue. No response frame was sent back.The task runner's
CommsDecoder.send()blocks onsocket.recv(4)waiting for the 4-byte length header of the reply. With nothing coming, the child process deadlocks permanently. The supervisor's heartbeat loop keeps running normally, so the scheduler's zombie detector never fires. The task stays stuck inRUNNINGand holds worker capacity until manually killed.The fix catches the exception as
eand replies withErrorResponse(error=ErrorType.GENERIC_ERROR)beforecontinue, matching the existing pattern used forServerResponseErrorand unhandled exceptions later in the same method. The send is wrapped insuppress(Exception)so a secondary socket error during the reply cannot crash the generator loop itself.closes: #76550
Tests
test_handle_requests_decoding_error_sends_error_responseintask-sdk/tests/task_sdk/execution_time/test_supervisor.py, parametrized over an unknown message type and a message missing required fields. Each case asserts the response frame carries aGENERIC_ERROR, the task runner side raisesAirflowRuntimeError, and the generator loop stays alive for the next request.Was generative AI tooling used to co-author this PR?