Repository navigation
fix(vad): close the open speech segment when input ends - #8295
ANDROIDHASSAN wants to merge 4 commits into
Conversation
VADStream.end_input() flushed before closing, so the segment open at the end of the input was reset without an END_OF_SPEECH and its audio never reached the consumer. RecognizeStream.end_input() also sends a flush, and StreamAdapter passed it on to the VAD, so the adapter lost that utterance the same way. end_input now closes the input without a reset, and both VADs emit the closing END_OF_SPEECH, including the samples short of an inference window. StreamAdapter no longer resets the VAD on a flush. Inference STT skips the VAD finalize once input ended, as its own session.finalize covers it, and Microsoft AI flushes first since consume() commits that tail itself.
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
| # the session.finalize sent at input end already covers this segment | ||
| if ws.closed or input_ended.is_set(): | ||
| return |
There was a problem hiding this comment.
🔴 Earlier speech turns lose finalization
When input ends before VAD catches up, _vad_finalize_task drops speech ends from earlier turns. input_ended also blocks their finalizations, merging distinct turns into one transcript.
Learn more
The sender queues audio to the VAD and then send_task sets input_ended without waiting for VAD inference. If input is pushed faster than inference, earlier END_OF_SPEECH events remain queued after this flag becomes set. The finalization listener now abandons those events, although they mark separate turns within the audio sent before input ended. The final session.finalize cannot recover those individual boundaries.
Example: Push speech, enough silence to end it, then another speech segment and call end_input() immediately. If the VAD processes the silence after input_ended is set, the first segment receives no separate finalize.
Recommended fix: Keep finalizing VAD boundaries that occurred before the end of the audio stream, and suppress only the synthetic closing event for the last open segment. Coordinate VAD processing with input-end finalization so messages arrive in audio order.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
The guard only runs after input_ended.is_set(), and send_task sets that after the last audio frame has gone out. So any finalize it skips would have landed behind all the audio, where it can no longer mark a boundary between turns.
Measured with a mock gateway and a VAD model slower than the frames are pushed: speech, silence, speech, then end_input() before the VAD catches up.
- guard removed:
session.create, input_audio x20, session.finalize x2 - guard kept:
session.create, input_audio x20, session.finalize x1
Both finalizes in the first run come after the last input_audio, so the second one adds no boundary. Finalizes from turns the VAD closes while audio is still flowing are sent as before, because input_ended is not set yet. No code change here.
…loses-speech # Conflicts: # tests/test_vad_buffer_resize.py
The rime STT tests merged in livekit#8013 drive the adapter with a VAD that closes a segment on each flush, including the one end_input sends. Dropping that flush broke them. end_input sends its flush again. Silero and inference VAD treat a flush that is the last item before the input closes as the end of input and close the open segment; any earlier flush still resets. StreamAdapter forwards a flush unless it is the one end_input sent.
…loses-speech # Conflicts: # tests/test_inference_stt_connection.py
Problem
When input ends while the user is still speaking, the open segment is lost.
VADStream.end_input()callsflush(), and Silero and inference VAD treat every flush as a hard reset that drops accumulated speech. So noEND_OF_SPEECHwas emitted for that segment and its audio never reached the consumer.RecognizeStream.end_input()also sends a flush, andStreamAdapterpassed it on tovad_stream.flush(), so the adapter reset the VAD before its ownend_input(). Fixing the VAD alone does not fix the adapter. #5687 added the reset to the VADs and the adapter inherited it: before that PR Silero ignored the sentinel.Measured on this PR's first commit (d9dc66d) with the Silero plugin and the real ONNX model on silero-vad's
tests/data/test.wav(59.94s, 16 kHz),min_speech_duration=0.1,min_silence_duration=0.75, input ended mid-speech:mainrecognize()got[7.584s]recognize()got[7.584s, 21.19s]recognize()got[7.584s, 30.868s, 5.61s]So 21 seconds of speech was never transcribed in the first case.
Fix
VADStream.end_input()is unchanged: it still sends a flush, then closes the input. VADs outside this repo that close a segment on a flush (like the test VAD intests/test_plugin_rime_stt.py) keep working.END_OF_SPEECHfor a segment still open, with the samples short of a full inference window included inframes. Any earlier flush still resets._copy_speech_bufferand the new_send_end_of_speechmoved out of thewhileloop so the end of_main_taskcan reach them, which is most of the diff in those two files.StreamAdapterpasses a flush on to the VAD unless it is the oneend_input()sent.session.finalizeonce input has ended: the one it sends at input end already covers the segment, and without this it would send two.vad_stream.flush()beforeend_input(), which keeps its behaviour onmain:consume()commits the rest itself, with the samples past the last inference window.Tests
tests/test_vad_buffer_resize.py, all against both Silero and inference VAD with a stub model, assert the exact event, samples, durations and frames:main(10 in total): the open segment is closed, trailing silence stays out ofspeech_duration, the tail short of a window is kept (16 kHz and 48 kHz), andStreamAdapterreturns the final transcript.main, guarding behaviour that must not change: no extra event after a segment already closed, andflush()right beforeend_input()still drops the segment.tests/test_inference_stt.py::test_input_ending_in_open_speech_sends_one_session_finalize(moved there after #8510 folded the connection tests into that module; it now runs against the in-memory gateway undervirtual_time, with the VAD's predict on the loop thread so the clock cannot outrun it) passes onmain, where no event closes the segment. Withinference/stt.pyfrommainand this PR's VAD change kept, it fails: a secondsession.finalizeis sent.Removing the end-of-input check from the two VADs fails 10 of these tests; removing it from
StreamAdapterfails both adapter tests.ruff format,ruff check,make type-checkanduv run pytest --unit --audio_eot -qpass on 71beef3; after mergingmain, d1d2ab5: 6245 passed, 4 skipped, includingtests/test_plugin_rime_stt.py.tests/test_vad.py(silero plugin category): 7 of 8 pass locally withtests/long.mp3fetched from git-lfs;test_plugin_checkpoint_matches_inference_vadneeds another lfs model file I don't have.Scope
end_input(): direct users ofStreamAdapter, and the STT plugins that drive a VAD.AgentSession's defaultstt_nodedoes not callend_input()on the recognize stream, so an utterance cut off by a session closing is unchanged. That is from reading the code, not an end-to-end run.flush()keeps its meaning everywhere, including on the adapter._stop_speaking()and, for realtime-only models, commits the audio buffer (send errors are already swallowed while closing). Mistral emitsEND_OF_SPEECH. Speechmatics finalizes, as its comment there already expects.samples_indexandspeech_durationstop at the last full inference window, whileframesruns to the last sample.aclose()stays silent, so cancelling a stream emits nothing.