fix(stt): anchor transcript timestamps on the session input start by chenghao-mou · Pull Request #7795 · livekit/agents

chenghao-mou · 2026-10-13T21:22:41Z

The default stt_node offsets provider timestamps from the session start, but the recognizer converted them with the pipeline's first audio frame. On a pipeline created after a handoff, the mismatch pushed stopped_speaking_at into the future, and the clamp to now replaced it with the arrival time (cause 2 in #7651, AGT-3678).

Streams now absorb the delay to their first frame and re-anchor after a reconnect, so start_time_offset is exact on the caller's timeline. The default node records its base on the pipeline and the recognizer adds that same base. Custom nodes keep their current behavior.

Diagnosis and the first-frame approach come from @AndrazP's report. #7694 takes a broader approach to all three causes; END_OF_SPEECH without a timestamp and mic gaps after the first frame remain open here.

Addresses AGT-3678

Initial prompt and agent context

Model: Claude Fable 5.1

can you review my current changes?

Later messages that changed the request:

Astra can't fix this, can you help it? We need to be mindful about changes because custom node might already report session relative timestamps

maybe we add the gap between stream creation and first frame into start_time_offset?

okay create a PR for this please

The default stt_node offsets provider timestamps from the session start, but the recognizer converted them with the pipeline's first audio frame. On a pipeline created after a handoff the two bases differ by the session's age, so stopped_speaking_at landed in the future and was clamped to the arrival time. The stream now absorbs the delay between its creation and its first frame, and re-anchors on the next frame after a reconnect, so its offset is exact on the caller's timeline. The default node records the base it measured from on the pipeline, and the recognizer adds that same base. A custom node that forwards provider timestamps untouched is anchored on the pipeline's creation, when it opens its stream. Addresses AGT-3678 Co-authored-by: Andraž Pajtler <1963904+AndrazP@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>

…irst-audio based Two review findings. A frame queued while a connection fails is the first audio the next connection sends, but the offset was only anchored on push, so that frame kept the old connection's offset. The stream now anchors on the first frame each attempt consumes, with the capture time recorded at push. A node that builds events itself, relative to its first frame, was anchored on the pipeline's creation. An undeclared offset now means the stream's timeline starts with its first audio, so such streams keep a zero offset and the pipeline is stamped by the first frame pushed, as before this change. Only the default node, which declares a session-relative offset, is anchored on the session input start.

A resampler can hold audio across pushes, so its first output can carry samples from an earlier frame. Stamping it with the push that released it anchored the offset late by the time between the two pushes. Track the capture time of the oldest held sample and stamp each resampled frame from there.

A resampler can release audio from several pushes at once. Stamping the later outputs by adding durations erased the gap between those pushes, so a retry starting on one of them was anchored early. Track the held input as capture spans and advance through them per output frame.

Keep the change small. The offset is anchored on the first frame pushed to each attempt; a frame still queued when a connection fails is left as a TODO. Resampled frames are no longer stamped individually, so resampler buffering is a millisecond-level approximation.

…fset helpers

The stream encoded one timeline in six fields and turned the caller's origin into an offset and back. Move the timeline into RecognizeStream._InputChan, which sees every push and every read. The channel stores the origin and anchors the offset when a connection reads its first frame. That frame was captured at the last push minus the audio still queued. This fixes the case where a connection fails before it reads: the retry now anchors on the oldest queued frame, not on the next push. A value set through the setter now means that the timeline began that many seconds ago, also after the first frame.

The push-audio stub left input_started_at as a truthy MagicMock, so no test reached the stamping branch. Reset it to None and assert it again. State the pipeline anchor contract next to the field.

The anchor took the last push minus the queued audio. A pause between queued frames is not in that sum, so the anchor moved forward by the pause. Also keep the push time of each queued frame. A frame is pushed after it is captured, so both values are upper bounds on its capture start: take the earlier one. It is exact across a pause and for a burst of buffered audio.

A leg was seeded only when it was created. If the caller set the offset after the first leg started, that leg kept a zero offset. Seed a leg that was not seeded when the first frame is forwarded to it. The outer stream has read that frame, so its timeline has a start.

The module docstring said a custom node is anchored on the pipeline's creation. It counts from its first audio, as on main.

longcw

looks good to me overall

longcw · 2026-10-19T06:53:05Z

+
+        def restart_provider_clock(self) -> None:
+            """Anchor the offset again on the next frame read, for a new connection."""
+            self._first_frame_at = None


if a plugin reconnect inside _run, e.g. for the update_options, it restarts the provider's clock but nothing calls restart_provider_clock, would this be a problem?

good point. I am adding them in the #7796 so each plugin resets when it reconnects internally

devin-ai-integration

Devin Review found 1 new potential issue.

devin-ai-integration · 2026-10-30T12:06:41Z

Cartesia's receiver

+                # _run has stopped reading, so the next frame read is the next connection's
+                self._input_ch.restart_provider_clock()
                if not e.retryable or max_retries == 0:


🔴 Idle STT sessions exhaust reconnects

When an idle provider repeatedly closes healthy connections, _num_retries reaches its limit without any recognition events. Cartesia marks only transcript or turn events as progress, so a silent participant's stream stops transcribing after several reconnects.

Learn more

The streaming retry counter tracks consecutive failures, but an idle participant can have a healthy connection without any speech events. Cartesia's receiver resets progress only for transcripts; its turn-detecting receiver resets only for turn events. After each idle socket closes, _main_task increments the counter even when the connection lasted longer than the connect timeout. Once it reaches max_retry, the stream fails and later speech is not transcribed.

Example: A caller stays silent while Cartesia closes its socket after three minutes. Four otherwise healthy socket lifetimes with no transcripts exhaust the default three retries, ending transcription before the caller speaks.

Recommended fix: Retain a retry reset for an attempt that outlived the connection timeout, alongside _mark_provider_progress resets. Keep the finite budget for consecutive short-lived connection failures.

Was this helpful? React with 👍 or 👎 to provide feedback.

Port #7795 and #7796 onto the shared STT connection coordinator, with #7739 that they build on. Each provider connection restarts the input clock. RTZR detects idle in the send loop, and Gladia and Reson8 re-anchor each session on its first frame. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Fold the end-of-input tests into tests/test_plugin_sarvam_stt.py, following livekit#8510, and re-anchor the provider clock with restart_provider_clock() on reconnect, following livekit#7795. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Fold the reconnect tests into the reorganized tests/test_plugin_sarvam_stt.py, following livekit#8510, and re-anchor the provider clock with restart_provider_clock() on reconnect, following livekit#7795. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Upstream livekit#7795 anchors start_time_offset on the capture time of the first frame read, so frames pushed in a burst move it back by the queued audio.

chenghao-mou requested a review from a team as a code owner October 13, 2026 21:22

chenghao-mou added the review_effort:high Needs broad context and substantial validation label Oct 13, 2026

This comment was marked as resolved.

Sign in to view

chenghao-mou marked this pull request as draft October 13, 2026 21:57

chenghao-mou mentioned this pull request Oct 13, 2026

fix: report speech end timestamps across stt plugins #7796

Merged

This comment was marked as resolved.

Sign in to view

chenghao-mou added this pull request to stack #7798 October 13, 2026 22:52

chenghao-mou marked this pull request as ready for review October 13, 2026 22:58

chenghao-mou mentioned this pull request Oct 13, 2026

STT turn detection: stopped_speaking_at ignores word timestamps, and is wrong after a handoff or a gap in mic audio #7651

Open

chenghao-mou and others added 15 commits October 14, 2026 13:33

docs(stt): state the timestamp contract on stt_node and rename the of…

abe2a94

…fset helpers

refactor(stt): keep one offset helper and write the setter inline

025dee8

refactor(stt): rename the offset helper to _capture_start_offset_from

faff357

refactor(stt): seed the fallback leg with a walrus

dc10a8d

chore(stt): assign the queued-frame TODO

8253d79

test(voice): restore the input_started_at stamping check

728e81b

The push-audio stub left input_started_at as a truthy MagicMock, so no test reached the stamping branch. Reset it to None and assert it again. State the pipeline anchor contract next to the field.

test(stt): describe the first-audio anchor for custom nodes

3fd2c0e

The module docstring said a custom node is anchored on the pipeline's creation. It counts from its first audio, as on main.

chenghao-mou force-pushed the chenghao/fix/AGT-3678-stt-timestamp-anchor branch from 4e59347 to 3fd2c0e Compare October 14, 2026 12:33

longcw reviewed Oct 19, 2026

View reviewed changes

chenghao-mou mentioned this pull request Oct 20, 2026

fix: report speech end times across other stt plugins #7988

Closed

longcw approved these changes Oct 21, 2026

View reviewed changes

Malik1234567891011 mentioned this pull request Oct 23, 2026

Deepgram Flux + explicit VAD: underreports transcription_delay #7960

Open

chore: merge main into stt timestamp anchor

e1489ca

devin-ai-integration Bot reviewed Oct 30, 2026

View reviewed changes

test: update reson8 fixture for input audio clock

c12361b

chenghao-mou merged commit 6234faa into main Nov 2, 2026
33 of 35 checks passed

chenghao-mou deleted the chenghao/fix/AGT-3678-stt-timestamp-anchor branch November 2, 2026 20:53

chenghao-mou mentioned this pull request Nov 2, 2026

fix(stt): apply speech timestamp fixes onto the connection refactor #8333

Open

rosetta-livekit-bot Bot mentioned this pull request Nov 2, 2026

fix: report speech end timestamps across STT plugins livekit/agents-js#2928

Open

Navigation Menu

Repository navigation

fix(stt): anchor transcript timestamps on the session input start - #7795

fix(stt): anchor transcript timestamps on the session input start#7795
chenghao-mou merged 17 commits into
mainfrom
chenghao/fix/AGT-3678-stt-timestamp-anchor

chenghao-mou commented Oct 13, 2026 •

edited

Loading

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

longcw left a comment

Uh oh!

longcw Oct 19, 2026

Uh oh!

chenghao-mou Oct 20, 2026

Uh oh!

devin-ai-integration Bot left a comment

Uh oh!

devin-ai-integration Bot Oct 30, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

chenghao-mou commented Oct 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

This comment was marked as resolved.

Uh oh!

longcw left a comment

Choose a reason for hiding this comment

Uh oh!

longcw Oct 19, 2026

Choose a reason for hiding this comment

Uh oh!

chenghao-mou Oct 20, 2026

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

Uh oh!

devin-ai-integration Bot Oct 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chenghao-mou commented Oct 13, 2026 •

edited

Loading