[15054538509]

Fixes #7651

With turn_detection="stt", stopped_speaking_at, transcription_delay, end_of_turn_delay and the derived e2e_latency were off on every turn, for the reasons in the report. They sit on one missing piece: the framework mapped provider timestamps with a single constant (end_time + _input_started_at) that only holds when the stream was created at session start and never saw a gap in the mic audio.

Changes

• A pushed-audio clock on the pipeline. _STTPipeline anchors itself when it is created and records the lag of the audio pushed into it 鈥?one entry per plateau, a 50 ms tolerance absorbs scheduling jitter. wall_time(position) maps a position back to the wall clock with the plateau it falls in: audio captured before a gap keeps its wall time, everything after the gap is shifted by the gap, a plateau recorded later never moves audio pushed before it, an exact plateau boundary keeps the earlier plateau, and a plateau with a negative lag (the silence commit_user_turn flushes into the stt) maps the audio after it back onto real time.
• Timestamps are laid out on the pushed-audio timeline. The offset a stream is seeded with is now the pipeline's pushed-audio position instead of a wall-clock read: wall-clock time includes the audio that never reached the pipeline (a gap in the mic, frames dropped while a stream was down), which the clock then adds back a second time. On a handoff the fallback to the session start is gone as well, which is what counted the session time twice. FallbackAdapter continues each leg at the forwarded audio duration for the same reason, so a leg created after a gap keeps the input timeline without counting the gap twice.
• A provider-derived metric: _stopped_speaking_at. In stt mode the last final/preflight transcript's word end, and an END_OF_SPEECH that does carry timing, update it. An END_OF_SPEECH without timing does not: it still moves the endpointing anchor to its arrival (that is how the anchor delays the commit there), but the metrics keep reporting where the user actually stopped talking instead of turning the delay into 0. _compute_end_of_turn_metrics prefers that value, and falls back to the endpointing anchor when it is missing or predates the turn's segment.

The endpointing anchor, the commit timing and vad turn detection are unchanged; in vad mode _stopped_speaking_at is never set.

Tests

tests/test_stt_pipeline_wall_clock.py (new) covers the clock: the anchor is stamped before the first frame, the position mapping, the gap behaviour, the exact boundary, the silence flush and the plateau collapsing. tests/test_transcription_delay_anchor.py covers the tracked value: an untimestamped END_OF_SPEECH no longer overwrites it, preflights count, a new segment clears it, and the word end behind a mic gap keeps its capture time. tests/test_end_of_turn_metrics.py covers the precedence and the stale-value fallback. tests/test_stt_fallback.py covers the leg delta continuing at the forwarded audio, and the affected stubs in the existing suites gained the new attribute.

Ran locally: the new and updated tests pass, together with the related audio-recognition/session suites (223 + 143 tests), ruff check, ruff format --check and mypy on the touched modules.

Verification

The reproduction script from the issue (fake mic, fake Soniox-like STT, no API keys) against this branch, reporting stopped_speaking_at minus the true last-word end:

scenario	main	this branch
plain	+0.59..0.61 s	+0.012 s
plain --eos-timestamp	+0.004 s	+0.006 s
task (handoff)	+0.59..0.61 s	+0.013..0.014 s
input_gap --eos-timestamp	+0.005 s then dropped metrics	+0.006..0.008 s, metrics present
input_gap	+5.59 s	+0.010..0.016 s

Commit timing stays as before in the first two scenarios (the anchor still moves to the arrival of an untimestamped END_OF_SPEECH); in the handoff and gap scenarios it now waits the endpointing delay after the real word end rather than committing immediately.

Two earlier revisions of this branch were corrected by Devin Review: removing the seeded offset in the mapping broke a stream restarted by the pump, and the clock's own lag double counted the gap once the offset was wall-clock based. The fix now sits in the offset seeding, the clock and the reset-aware plateau lookup.

[devin-ai-integration]

Devin Review found 2 new potential issues.

1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)

[devin-ai-integration]

🔴 Stream offsets double-count input gaps

After a fallback or recreated stream follows an input gap, _provider_wall_time adds that gap twice. start_time_offset includes elapsed gaps, while wall_time adds _arrival_lags; speech metrics move forward by the gap.

Learn more

Provider timestamps from the default STT node include the stream's start_time_offset. That offset is wall-clock time since the pipeline anchor, not pushed-audio duration. A stream recreated by _stt_pump receives a new offset, and a fallback leg receives an equivalent elapsed-time offset in FallbackAdapter. The pipeline's arrival lag separately records wall-clock gaps in pushed audio, so mapping the offset timestamp through that lag counts each earlier input gap twice.

Example: The pipeline starts at 0 seconds, receives 5 seconds of audio over 10 wall-clock seconds, then recreates its stream. The new stream offset is 10 seconds and its first word ends 0.5 seconds later. wall_time(10.5) adds the recorded 5-second lag and returns 15.5 seconds, although the word ended at 10.5 seconds.

Recommended fix: Keep provider timestamps and lag lookup on one coordinate system. Track each active stream's pushed-audio position separately from its wall-clock start_time_offset, then normalize event timestamps to the pipeline's cumulative pushed-audio position before calling wall_time. Cover both _stt_pump recreation and FallbackAdapter leg switching after an earlier microphone gap.

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[15054538509]

Fixed. The offset a stream is seeded with is now the pipeline's pushed-audio position (_input_duration) instead of a wall-clock read, so the gap is no longer part of it, and FallbackAdapter continues each leg at the forwarded audio duration instead of a wall-clock delta. A stream recreated after a gap now keeps the input timeline with the gap added exactly once by the clock. Covered by test_stt_stream_fallback_offset_continues_at_the_forwarded_audio and the input_gap scenarios of the report's reproduction script.

[devin-ai-integration]

🟡 Gap boundaries shift preceding speech

When a word ends exactly at a post-gap plateau boundary, bisect_right assigns the new lag. _provider_wall_time moves that pre-gap word forward by the entire gap.

Learn more

Each lag entry is keyed to the start position of the frame that established the new plateau. An event end timestamp at that exact position describes audio ending immediately before that frame, so it belongs to the preceding plateau. bisect_right instead selects the new plateau at equality.

Example: Frames cover positions 0.0–0.3, followed by a five-second gap and a new frame starting at 0.3. A word ending at 0.3 maps with the five-second lag, although all its audio preceded the gap.

Recommended fix: Use bisect_left so an exact boundary selects the previous entry. The existing max(index - 1, 0) still handles the first plateau.

Suggested change

	index = bisect.bisect_right(
	self._arrival_lags, audio_position, key=lambda entry: entry[0]
	)
	index = bisect.bisect_left(
	self._arrival_lags, audio_position, key=lambda entry: entry[0]
	)

---

Was this helpful? React with 👍 or 👎 to provide feedback.

[15054538509]

Fixed as suggested: bisect_left, so an event ending exactly at a plateau's first position keeps the previous plateau. Covered by test_word_ending_exactly_at_a_gap_boundary_keeps_the_earlier_plateau.