Repository navigation
fix(stt): detect half-open websockets in OpenAI, ElevenLabs and Inworld STT - #7912
ninglinLiu wants to merge 4 commits into
Conversation
…rld STT plugins Add heartbeat=30.0 to ws_connect and handle WSMsgType.ERROR by raising retryable APIConnectionError in OpenAI realtime, ElevenLabs, and Inworld STT plugins. Without heartbeat, a silently dropped socket (half-open TCP connection, no FIN/RST) is never noticed: recv_task parks on ws.receive() forever and the reconnect loop, which is exception-driven, never runs. The heartbeat closes the socket when a ping goes unanswered, and that surfaces as WSMsgType.ERROR. ws.exception() is the only place the reason survives, so we chain it to the APIConnectionError. Add regression tests for each plugin proving they notice a silent websocket via heartbeat and raise retryable APIConnectionError within bounded time. Fixes livekit#7900 Co-authored-by: NingLin Liu <x07514475479@gmail.com>
Run ruff format on 4 files per upstream CI requirements: - livekit-plugins-elevenlabs/livekit/plugins/elevenlabs/stt.py - tests/test_plugin_openai_stt_halfopen.py - tests/test_plugin_elevenlabs_stt_halfopen.py - tests/test_plugin_inworld_stt_halfopen.py Restructured the Inworld test's _stream function to keep the type: ignore[arg-type] comment on the http_session argument line where mypy will apply it, while satisfying ruff format requirements. Co-authored-by: NingLin Liu <x07514475479@gmail.com>
|
Ran the probe from #7900 against this branch, and it works:
So it now matches the other fixed plugins at about 45s. I'm happy to follow up with the shared helper and a shorter heartbeat across all the STT plugins if that's the direction you want, so it can land separately from this one. |
There was a problem hiding this comment.
let's move the tests into the main plugin tests.. vs a dedicated file for halfopen.
also there are a ton of duplications of helper functions/etc in these tests.
There was a problem hiding this comment.
Done in edc69d4. The tests now live in each plugin's existing test file (test_plugin_openai_stt_context.py, test_plugin_elevenlabs_stt.py, and a new test_plugin_inworld_stt.py since Inworld had no STT test file), and the fake sockets, session and wait_until helper are shared from tests/fake_ws.py instead of being copied three times. The _halfopen files are gone.
|
Thanks @ninglinLiu for the effort here! We're handling WebSocket ERROR frames centrally in the liveness monitor in #8147, so closing this in favor of that. |
Problem
If a streaming STT websocket goes half-open (the provider, or a NAT in between, disappears without a FIN/RST), the OpenAI realtime, ElevenLabs and Inworld STT streams wait on
ws.receive()forever. Nothing raises, so the stream never reconnects andstt.FallbackAdapternever fails over; the agent just stops hearing the user (#7900).Deepgram, Soniox, Telnyx, xAI and Simplismart were already fixed the same way (#7206, #7357, #7359, #7364, #7369).
Fix
Same pattern as those merged fixes, applied to the three remaining plugins:
heartbeat=30.0tows_connectinopenai/stt.py,elevenlabs/stt.pyandinworld/stt.py. aiohttp defaultsheartbeattoNone, so without it an unanswered socket is never noticed.WSMsgType.ERRORin the receive loop as a retryableAPIConnectionError, chained fromws.exception(). That is where aiohttp reports the missed pong. It already returns quietly when we are closing the socket ourselves. Before this change, OpenAI/ElevenLabs logged it as an "unexpected message type" and continued, and Inworld logged it and returned without triggering a reconnect.I kept the change inline per plugin, the same way as the existing fixes, and kept the 30s interval they use. The issue asks whether a shared helper in
livekit.agents.utilsand/or a shorter interval would be better. I'm happy to rework this PR either way if you prefer.Tests
Unit tests live in each plugin's existing test file, sharing one set of fake aiohttp sockets and session in
tests/fake_ws.py:tests/test_plugin_openai_stt_context.pytests/test_plugin_elevenlabs_stt.pytests/test_plugin_inworld_stt.py(new; Inworld had no STT test file yet)Each checks that the socket is opened with
heartbeat=30.0, and that a heartbeat-timeoutERRORframe ends the receive loop and leads to a reconnect without spinning.Fixes #7900