[rosetta-livekit-bot]

Ports livekit/agents#7796 to the STT providers and voice pipeline available in agents-js.

The port adds capture-aware provider clocks, preserves verified acoustic speech boundaries on EOS events, and keeps intentionally untimed providers untimed. It also cancels stale input reads across internal reconnects so the next connection cannot lose its first frame.

Validation

• pnpm build
• pnpm lint (passes with existing warnings)
• pnpm format:check
• pnpm --filter @livekit/agents api:check
• Port-specific suites: 358 passed, 3 skipped
• Full touched-package run: 4,951 passed, 22 skipped; remaining failures are unrelated environment/baseline failures (missing Git LFS Silero ONNX asset, host CPU-accounting assertions, an existing TCP backpressure assertion, and an OpenAI test using the environment's incompatible base URL)
• cue-cli validation was unavailable because cue-cli is not installed in this environment

Source diff coverage

1. livekit-agents/livekit/agents/inference/stt.py: adapted to agents/src/inference/stt.ts; JS schema/event differences, same empty-final and last-word EOS behavior.
2. livekit-agents/livekit/agents/stt/multi_speaker_adapter.py: not applicable; agents-js has no multi-speaker adapter.
3. livekit-agents/livekit/agents/voice/audio_recognition.py: adapted to agents/src/voice/audio_recognition.ts; preserves an ended turn boundary in the JS state machine.
4. livekit-plugins/livekit-plugins-assemblyai/livekit/plugins/assemblyai/stt.py: ported to plugins/assemblyai/src/stt.ts.
5. livekit-plugins/livekit-plugins-aws/livekit/plugins/aws/stt.py: not applicable; no JS AWS plugin.
6. livekit-plugins/livekit-plugins-azure/livekit/plugins/azure/stt.py: ported to plugins/azure/src/stt.ts.
7. livekit-plugins/livekit-plugins-baseten/livekit/plugins/baseten/stt.py: ported to plugins/baseten/src/stt.ts.
8. livekit-plugins/livekit-plugins-cartesia/livekit/plugins/cartesia/_recognize_streams/legacy_recognize_stream.py: adapted to the shared clock used by plugins/cartesia/src/stt.ts; JS only has the newer untimed turns protocol, so billed duration remains excluded from EOS.
9. livekit-plugins/livekit-plugins-deepgram/livekit/plugins/deepgram/stt.py: ported to plugins/deepgram/src/stt.ts.
10. livekit-plugins/livekit-plugins-deepgram/livekit/plugins/deepgram/stt_v2.py: ported to plugins/deepgram/src/stt_v2.ts.
11. livekit-plugins/livekit-plugins-elevenlabs/livekit/plugins/elevenlabs/stt.py: ported to plugins/elevenlabs/src/stt.ts.
12. livekit-plugins/livekit-plugins-fireworksai/livekit/plugins/fireworksai/stt.py: not applicable; no JS FireworksAI plugin.
13. livekit-plugins/livekit-plugins-gladia/livekit/plugins/gladia/stt.py: not applicable; no JS Gladia plugin.
14. livekit-plugins/livekit-plugins-google/livekit/plugins/google/stt.py: not applicable; JS has Gemini Live STT, not the changed Cloud Speech protocol.
15. livekit-plugins/livekit-plugins-gradium/livekit/plugins/gradium/stt.py: not applicable; no JS Gradium plugin.
16. livekit-plugins/livekit-plugins-inworld/livekit/plugins/inworld/stt.py: ported to plugins/inworld/src/stt.ts.
17. livekit-plugins/livekit-plugins-mistralai/livekit/plugins/mistralai/stt.py: ported to plugins/mistralai/src/stt.ts.
18. livekit-plugins/livekit-plugins-nvidia/livekit/plugins/nvidia/stt.py: not applicable; no JS NVIDIA plugin.
19. livekit-plugins/livekit-plugins-openai/livekit/plugins/openai/stt.py: ported to plugins/openai/src/stt.ts.
20. livekit-plugins/livekit-plugins-reson8/livekit/plugins/reson8/stt.py: not applicable; no JS Reson8 plugin.
21. livekit-plugins/livekit-plugins-rtzr/livekit/plugins/rtzr/stt.py: not applicable; no JS RTZR plugin.
22. livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt.py: not applicable; source hunk is comment-only.
23. livekit-plugins/livekit-plugins-sarvam/livekit/plugins/sarvam/stt_streaming.py: not applicable; JS lacks Sarvam realtime/manual endpointing.
24. livekit-plugins/livekit-plugins-sarvam/tests/test_speech_timing.py: not applicable; fixture for the absent realtime stream.
25. livekit-plugins/livekit-plugins-simplismart/livekit/plugins/simplismart/stt.py: not applicable; no JS SimpliSmart plugin.
26. livekit-plugins/livekit-plugins-smallestai/livekit/plugins/smallestai/stt.py: not applicable; no JS SmallestAI plugin.
27. livekit-plugins/livekit-plugins-soniox/livekit/plugins/soniox/stt.py: ported to plugins/soniox/src/stt.ts and plugins/soniox/src/_internal.ts.
28. livekit-plugins/livekit-plugins-speechmatics/livekit/plugins/speechmatics/stt.py: not applicable; no JS Speechmatics plugin.
29. livekit-plugins/livekit-plugins-xai/livekit/plugins/xai/stt.py: ported to plugins/xai/src/stt.ts.
30. tests/stt_reconnect.py: adapted into capture-clock tests and provider recovery tests using Vitest/WebSocket infrastructure.
31. tests/test_audio_recognition_turn_detection.py: adapted to existing voice timing tests.
32. tests/test_inference_stt_start_of_speech.py: adapted to agents/src/inference/stt.test.ts.
33. tests/test_meta_stt.py: adapted to existing plugins/meta/src/stt.test.ts coverage; Meta remains explicitly untimed and does not expose processed duration as EOS.
34. tests/test_multi_speaker_adapter.py: not applicable; no JS adapter.
35. tests/test_multi_speaker_primary_speaker.py: not applicable; no JS primary-speaker adapter.
36. tests/test_plugin_assemblyai_stt.py: ported to AssemblyAI STT tests.
37. tests/test_plugin_aws_stt.py: not applicable; absent plugin.
38. tests/test_plugin_azure_stt.py: ported to plugins/azure/src/stt.test.ts.
39. tests/test_plugin_baseten_stt.py: adapted to the shared capture clock and existing Baseten recovery tests.
40. tests/test_plugin_cartesia_stt.py: adapted to existing Cartesia tests; current protocol remains intentionally untimed.
41. tests/test_plugin_deepgram_stt.py: ported to Deepgram v1/v2/recovery tests.
42. tests/test_plugin_elevenlabs_stt.py: ported to plugins/elevenlabs/src/stt.test.ts.
43. tests/test_plugin_fireworksai_stt.py: not applicable; absent plugin.
44. tests/test_plugin_gladia_stt.py: not applicable; absent plugin.
45. tests/test_plugin_gnani_stt.py: not applicable; absent plugin.
46. tests/test_plugin_google_stt.py: not applicable; changed Cloud Speech protocol is absent.
47. tests/test_plugin_gradium_stt.py: not applicable; absent plugin.
48. tests/test_plugin_inworld_stt.py: ported to plugins/inworld/src/stt_recovery.test.ts.
49. tests/test_plugin_mistralai_stt.py: ported to Mistral production behavior and existing Mistral suite; server turn EOS remains untimed.
50. tests/test_plugin_nabrah_stt.py: not applicable; absent plugin.
51. tests/test_plugin_nvidia_stt.py: not applicable; absent plugin.
52. tests/test_plugin_openai_stt_context.py: ported to OpenAI recovery/timing tests.
53. tests/test_plugin_reson8_stt.py: not applicable; absent plugin.
54. tests/test_plugin_rtzr_stt.py: not applicable; absent plugin.
55. tests/test_plugin_sarvam_stt_streaming.py: not applicable; absent realtime stream.
56. tests/test_plugin_simplismart_stt.py: not applicable; absent plugin.
57. tests/test_plugin_smallestai_stt.py: not applicable; absent plugin.
58. tests/test_plugin_soniox_stt.py: ported to plugins/soniox/src/stt.test.ts.
59. tests/test_plugin_speechmatics_stt.py: not applicable; absent plugin.
60. tests/test_plugin_xai_stt.py: ported to xAI recovery/timing tests.
61. tests/test_stt.py: adapted to package-local JS tests; agents-js has no centralized live-provider pytest matrix.
62. tests/test_stt_stream_timing.py: adapted to core capture-clock tests in agents/src/stt/stt.test.ts.
63. tests/test_transcription_delay_anchor.py: adapted to agents/src/voice/audio_recognition_transcription_delay_anchor.test.ts.
64. tests/utils/stt.py: adapted through package-local Vitest helpers and EOS assertions rather than a Python pacing helper.

The source PR depends on the capture-aware provider clock from livekit/agents#7795. agents-js lacked that infrastructure, so this PR ports it in agents/src/stt/stt.ts and wires every target provider's internal reconnect path to it.

Source: livekit/agents#7796

---

Ported from livekit/agents#7796

Original PR description

Report speech-end timestamps for live-tested STT configurations and retain clock fixes across providers. Sarvam realtime keeps untimed EOS with corrected manual ordering. Untimed transcripts that follow EOS retain the current turn’s boundary.

Paced tests follow @Malik1234567891011's proposal. AssemblyAI gateway parameter forwarding is tracked separately in livekit/agent-gateway#1827.

Stacked on livekit/agents#7795. Addresses AGT-3678 (Linear) and livekit/agents#7651 (@AndrazP's diagnosis). Reconnect handling follows @longcw's review.

@davidzhao’s review identified delayed local VAD timestamps; his Gradium and consumer findings were also reproduced. The fixes remove synthetic padding from Gradium timing, use captured audio time for OpenAI and Mistral local VAD, and retain EOS across untimed finals.

Plugin status

Changes are relative to livekit/agents#7795, which supplies the shared clock fix. Only changed production paths are listed. EOS checks use the paced stream tests.

Live checks ran on 2026-10-30; Gradium, OpenAI, and Mistral were rechecked on 2026-11-01. Reconnect checks use real provider connections and verify the original audio timeline after reconnection. OpenAI also reuses a live pooled socket; Sarvam checks nonzero session offsets. RTZR's English-fixture WER remains exempt; its speech timing passed. Reson8’s internal-reconnect probe adds a 1 s delay before reopening to avoid its connection limit; the socket-failure probe uses the normal retry path.

Plugin / path	Clock fix in this PR	EOS reporting in this PR	Live test validation
AssemblyAI (direct)	Unchanged	Updated: final word end	Passed: Universal-3.6 Pro and Universal-3.5 Pro EOS
AWS	Fixed: idle reconnect	Deferred	Passed: idle-timeout reconnect and transcript timing
Azure Speech SDK	Fixed: reconnect	Updated: recognized speech boundary	Passed: reconnect and timed EOS
Baseten Whisper	Fixed: reconnect	Deferred	Passed: reconnect, socket failure, and transcript timing
Cartesia ink-whisper (legacy)	Fixed: reconnect	Updated: word end on EOS; transcript spans unchanged	Passed: reconnect and timed EOS
Deepgram Nova	Fixed: reconnect	Updated: word end / last_word_end	Passed: internal reconnect, socket failure, and timed EOS
Deepgram Flux	Fixed: reconnect	Updated: word end	Passed: reconnect and timed EOS
ElevenLabs realtime	Fixed: reconnect	Updated: committed word timing	Passed: Scribe v2 reconnect and timed EOS with timestamps and server VAD
Fireworks	Fixed: reconnect	Deferred	Not live validated: no available streaming endpoint
Gladia	Fixed: reconnect	Updated: utterance end	Passed: reconnect and timed EOS
Google	Fixed: reconnect	Updated: speech event / transcript boundary	Passed: reconnect and timed EOS with voice activity events
Gradium	Fixed: reconnect and synthetic silence offsets	Updated: end_text.stop_s	Passed: US endpoint reconnect and repeated timed EOS with default vad_flush=True
Inworld	Fixed: reconnect	Updated: requested word timestamps	Passed: inworld-stt-1 reconnect and timed EOS
Mistral realtime	Fixed: connection reset and local VAD capture timing	Updated: local VAD speech end	Passed: Voxtral 2602 and Voxtral 3 reconnect and timed EOS with local VAD; native Voxtral 3 EOS remains untimed
NVIDIA Riva	Fixed: word timestamp units (milliseconds to seconds)	Unchanged; untimed EOS	Passed: English and multilingual Parakeet streaming and word timing
OpenAI realtime	Fixed: reconnect, pooled connection offsets, and local VAD capture timing	Updated: server / local VAD speech end	Passed: model-change reconnect, live socket reuse, and server/local VAD EOS
Reson8	Fixed: internal reconnect	Unchanged; untimed EOS	Passed: reconnect, socket failure, transcript timing, and shutdown
RTZR	Fixed: segment reconnect and word offsets	Updated: last word end	Passed: segment reconnect and native Korean speech timing
Sarvam realtime	Fixed: session offsets	Manual ordering fixed; remains untimed	Passed: provider timestamp offsets, VAD ordering, and manual-to-VAD switch
SimpliSmart	Fixed: reconnect	Unchanged; emits no EOS	Not live validated: no available streaming deployment
SmallestAI	Fixed: reconnect	Deferred	Passed: Pulse reconnect and transcript timing
Soniox	Fixed: reconnect	Updated: source token end	Passed: stt-rt-v5 reconnect and timed EOS
Speechmatics	Unchanged	Updated: final segment end	Passed: Linden-1 timed EOS
xAI	Fixed: reconnect and timestamp offsets	Updated: word timing	Passed: reconnect and timed EOS
LiveKit inference: word-timed models	Unchanged	Updated: final word end	Passed: Nova-3, AssemblyAI Universal Streaming, and xAI timed EOS

Initial prompt and agent context

Model: GPT-6 Astra

Can you investigate 3678? I believe this is a STT plugin reporting gap.

Can you create a PR to address the EOS speech end time reporting gap?

yes, we need to fix all of them. and verify them one by one please.

sorry, I think I was in a wrong thread. 11labs is not special from other plugins in this thread.

okay, let's add that. But let's verify every changed plugin before committing

let's skip baseten and azure changes for now

Clarification for Azure-related exclusions:

Only microsoft_ai

so for plugins we don't have credentials, we skip the live verification. but we double check the docs and implementation to make sure it is reasonable.

Latest scope:

we should reduce the end time reporting to the ones we validated live, and keep clock fixes

and also double check the ones mentioned in the original issue.

Final validation scope:

can you add a column to the table to show live test validated, even if it is just clock fixing?

SLNG is sunsetting so we can drop that changes too. we should drop plugins that are both unchanged and deffered.

[changeset-bot]

🦋 Changeset detected

Latest commit: c253209

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 41 packages

Name	Type
@livekit/agents	Patch
@livekit/agents-plugin-assemblyai	Patch
@livekit/agents-plugin-azure	Patch
@livekit/agents-plugin-baseten	Patch
@livekit/agents-plugin-deepgram	Patch
@livekit/agents-plugin-elevenlabs	Patch
@livekit/agents-plugin-inworld	Patch
@livekit/agents-plugin-mistralai	Patch
@livekit/agents-plugin-openai	Patch
@livekit/agents-plugin-soniox	Patch
@livekit/agents-plugin-xai	Patch
@livekit/agents-plugin-anam	Patch
@livekit/agents-plugin-anthropic	Patch
@livekit/agents-plugin-bey	Patch
@livekit/agents-plugin-cartesia	Patch
@livekit/agents-plugin-cerebras	Patch
@livekit/agents-plugin-did	Patch
@livekit/agents-plugin-fishaudio	Patch
@livekit/agents-plugin-google	Patch
@livekit/agents-plugin-hume	Patch
@livekit/agents-plugin-krisp	Patch
@livekit/agents-plugin-lemonslice	Patch
@livekit/agents-plugin-liveavatar	Patch
@livekit/agents-plugin-livekit	Patch
@livekit/agents-plugin-meta	Patch
@livekit/agents-plugin-minimax	Patch
@livekit/agents-plugin-mistral	Patch
@livekit/agents-plugin-neuphonic	Patch
@livekit/agents-plugin-perplexity	Patch
@livekit/agents-plugin-phonic	Patch
@livekit/agents-plugin-protoface	Patch
@livekit/agents-plugin-resemble	Patch
@livekit/agents-plugin-rime	Patch
@livekit/agents-plugin-runway	Patch
@livekit/agents-plugin-sarvam	Patch
@livekit/agents-plugin-silero	Patch
@livekit/agents-plugin-speechify	Patch
@livekit/agents-plugin-synthesia	Patch
@livekit/agents-plugin-tavus	Patch
@livekit/agents-plugins-test	Patch
@livekit/agents-plugin-trugen	Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

[devin-ai-integration]

Devin Review found 2 potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

[devin-ai-integration]

🟡 Soniox timestamps restart after reconnect

When Soniox reconnects, options.startTimeOffset freezes before the next frame establishes the provider clock. Later transcripts and speech ends use the old offset, placing them on the wrong audio timeline.

(Refers to this code)

Learn more

Soniox passes an options object into processMessage, which uses its startTimeOffset for every emitted transcript and speech end. The options object is constructed before #sendAudio calls input.next(). That read is what updates the capture-aware offset in TimedInputQueue, so on a reconnect it retains the previous connection's offset for the entire new session.

Example: A first connection consumes audio at offset 0, then drops. Audio continues for 10 seconds before Soniox reconnects. The second connection's first frame advances stream.startTimeOffset near 10, but the listener still timestamps a word ending at 0.4 as 0.4 instead of 10.4.

Recommended fix: Read this.startTimeOffset when processing each Soniox message, after the input sender consumes the first frame, rather than caching it during #runWS setup. Preserve the existing fallback for messages received before any frame.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[devin-ai-integration]

🟡 VAD timestamps restart after reconnect

When client VAD reconnects, event.timestamp resets with the new VAD stream while startTimeOffset tracks the input timeline. The absolute speech-end time uses both anchors and remains correct.

---

Was this helpful? React with 👍 or 👎 to provide feedback.