feat: add multi-turn answering-machine detection by chenghao-mou · Pull Request #6202 · livekit/agents

chenghao-mou · 2026-06-24T11:03:21Z

AMD handles screening, voicemail, and IVR across turns using rolling history and successful DTMF calls. WAIT suppresses replies during announcements; stale predictions are discarded.

A category FSM recommends next stages; another category corrects the stage. AMD owns deadlines and reply authorization and supports realtime replies. The classifier prompt is tuned on scripted and real calls across several models, and the default classifier is now google/gemma-4-31b-it. With both session STT and an AMD STT, the classifier sees each turn's two transcripts as JSON, so a transcript that arrives late from one STT does not empty the turn. A turn whose classification times out or fails is uncertain in the current stage. See the AMD spec.

Likely human after a menu or screener. After a hold or a screener, a short greeting such as "Hi. You've reached Sam." read as human and ended AMD before the voicemail. In the screening and IVR stages, a first human now counts as uncertain (likely human), and the next turn decides; the agent still replies, and the classifier sees its words. AMD's own STT finalized each sentence about 0.1 to 0.2 s after the session committed it, so a turn carried the previous turn's text; a turn without AMD text now waits up to 300 ms for its final. Until the next turn confirms the person, AMD stays active, so handoffs and end_call wait one more exchange.

Customer-handled stages. Set screening_instructions, voicemail_instructions, or ivr_instructions to None to complete AMD when the call enters that stage, with no reply. For example, to hang up at the first voicemail prediction:

async with AMD(session, voicemail_instructions=None) as detector:
    result = await detector.execute()
if result.category is AMDCategory.MACHINE_VM:
    ctx.shutdown("voicemail")

Remote sessions. The session host now sends machine-screening and wait as their own categories, each prediction's turn ID, stage, and inference time, and the amd_completed result. This needs livekit-protocol>=1.1.31 (livekit/protocol#1880).

Decision models. llm also accepts a decision model that supports choice. It gets the classifier prompt and the seven categories as one choice; AMD skips menu extraction with it. On the six eval splits, Luna this way passed 70% of scenarios with 3 critical errors, against 72% and 3 for the livekit/agents-private#511 adapter.

async with AMD(session, llm=inference.DecisionModel("openai/gpt-6-luna-decisions")) as detector:
    result = await detector.execute()

Simulated calls (100 composed flows per run: screeners, IVR menus, holds, voicemail, and people; before at cad29d3bf, after with this change on the same base):

Metric	Before	After
Flows completed ↑	80%	97%
IVR calls ↑	75%	95%
Judge passed ↑	76%	90%
Long silence calls ↓	1%	0%
AMD accuracy ↑	57%	90%
Critical errors ↓	42%	10%
Human calls ↑	97%	100%
Voicemail calls ↑	27%	82%
Other machine calls ↑	49%	87%
Human F1 ↑	0.59	0.88
Human TTG p50 ↓	0.36 s	0.40 s

Each call counts under what it reached: a person, a voicemail, or only machines (closed lines, menus that hung up, declined screeners). Critical errors: a machine taken for a person, or a person sent to voicemail or machine-unavailable.

Addresses AGT-3038. Adopts @karan-dhir's screening proposal (source issue) and @eashwar-mp's rolling-history guidance.

eashwar-mp

Nice work, this lands right where we need it. Sharing a few things from running outbound AMD (screeners + voicemail) in production that might save some pain, especially around trusting the LLM for the screening category. Happy to dig into any of these.

Classify at client-side EOT across screening, voicemail, and IVR turns. Keep reply control in AgentSession and expose separate prediction and completion events. Race optional STT for AMD without changing Agent text. Addresses AGT-3038 Co-authored-by: karan-dhir <karan@woflow.com> Co-authored-by: eashwar-mp <eashwar.perumal@salesai.com>

Machine stages require 1.5 seconds of continuous participant silence before predictions or replies can be released. Human and initial uncertain results keep normal EOT timing. Apply the same rule to machine-stage fallbacks and cancel pending releases when speech resumes. Addresses AGT-3038

Keep AMD reply gating intact after the turn-completion tracing refactor. Preserve AMD turn IDs and trace cleanup. Addresses AGT-3038

Select the default AMD models when llm or stt is omitted and LiveKit Cloud credentials are available. Explicit None inherits the active Agent LLM or session transcript. Addresses AGT-3038

The detector mixed stage policy, turn bookkeeping, and asyncio timers in one class, so every rule needed an event loop to test. Move policy into _AMDFSM with injected clocks and returned events; the detector keeps I/O. Delete the unused legacy classifier, move classification to a tool call, and give AgentActivity _is_busy and _cancel_pending_replies so AMD stops reading its private fields.

Turn state was a set of nullable fields whose legal combinations were implicit, so deadline handling matched floats and idle checks probed three fields. Each turn now has one phase (idle, inferring, holding) and one deadline, and the FSM applies due deadlines in order. Replace the six-way category handler table with one stage transition, return a ReplyDecision instead of overloading AMDCategory, type reasons as AMDReason, and build the classifier request as a model. Remove the unused should_wait and detection_delay fields.

The detector held its agent, model, completion future, and open transcript as Optionals that were None only before __aenter__, so every use asserted. Group them in a _Run created on enter, with one guard at the boundary. Task and listener failures now complete the run as internal_error instead of inference_error, so callers can tell a model failure from a bug. Split transcript_ready so superseding older work and building the new request are separate steps.

Reduce coupling between AMD policy and speech scheduling. Keep each turn's reply guard valid after AMD detaches, including while a customer hook runs. Addresses AGT-3038

Remove the public user_turn_committed event and session turn counter. AMD returns a guard bound to each turn before the customer hook runs, so pending replies keep their original decision after newer turns arrive. Addresses AGT-3038

Route model and fallback predictions through the same release path. Inline turn handling and document the FSM and prediction reasons. Keep saved decisions intact across late results and silent supersession. Addresses AGT-3038

Use TurnHooks for committed user input, reply authorization, and accepted agent output. Keep reply tool preparation separate so preemptive attempts do not commit an agent turn. Return turn-bound hooks at commit so delayed customer callbacks retain their AMD decision. Addresses AGT-3038

The FSM kept a phase per turn and patched predictions across turns to wake waiters, so stale work lingered and empty turns needed special cases. Keep one inference and one hold. A new transcript supersedes both, an empty turn reuses them through a pointer, and a late model result retargets a held REUSED prediction instead of being dropped. Speech after the last commit freezes a hold until the next speech end or commit. finish() no longer fabricates predictions; waiters exit on completion. The detector arms one timer for the FSM's next deadline. Menu extraction and voicemail tracking are FSM decisions. UserStateChangedEvent gains speech_timestamp so created_at keeps the event-creation convention. Rename AMD.notify_dtmf_sent to on_dtmf_event. Addresses AGT-3038

Keep category transitions in the FSM and manage inference, deadlines, and reply authorization in AMD. Add WAIT, bounded classifier history, and successful DTMF context. Discard stale predictions and restore interruption settings after detection. Preserve speech timestamps, roll back partial setup, and cover pipeline and realtime replies with regression tests. Addresses AGT-3038

Create owned model clients after entry validation and close them when setup fails. Keep provider and listener exception content out of AMD logs and DTMF tool errors. Addresses AGT-3038

Keep event-creation time in created_at. Direct speech-timing consumers to speech_timestamp, with created_at as the fallback when the boundary is unknown. This applies to sessions with or without AMD. Addresses AGT-3038

Successful DTMF sends keep their result in history and wait for the next prompt. Use ToolResult from #7340 so the tool controls reply generation.

A pipeline reply starts generating when the prediction arrives. A realtime reply waits for playback authorization before it asks the model, so its latency follows the machine silence hold.

After a hold or a screener, a short greeting such as "Hi. You've reached Sam." was classified human, and AMD completed on it. A voicemail behind a menu or screener then ended as human, and the agent lost the DTMF tool. - In the screening and IVR stages, a first human prediction is likely human: it counts as uncertain, and the next turn decides. The agent can still reply, without stage instructions. The next request labels the previous prediction "uncertain (likely human)" and shows the agent's words, so a reply to the agent reads as a person. - AMD's own STT finalized each sentence about 0.1 to 0.2 s after the session committed it, so its text landed in the next turn. A turn without AMD text now waits up to 300 ms for that final.

Without session STT, a turn that waited for its late AMD final could get a reused prediction from the pending request, then a second classification when the final arrived. A prediction is never revised now: the late text goes into classifier history, and the next request covers it. A final that arrived before the turn's task started waiting was lost. The turn now takes its late-final wait at commit. The usage doc moves from examples/telephony/amd.md to spec/amd.md as a spec checked against the code. CONTEXT.md is folded into its Terms section.

To hang up on voicemail, a caller had to watch amd_prediction: execute() returns only after the message and the voicemail idle timeout. Setting screening_instructions, voicemail_instructions, or ivr_instructions to None now makes that a customer-handled stage: AMD completes when the call enters it. As with machine-unavailable, AMD skips that turn's reply and cancels pending replies. human_instructions=None keeps the Agent's own instructions. The spec defines the stage in section 9.5.

livekit-protocol 1.1.31 (livekit/protocol#1880) adds the AMD v2 fields. Screening and wait now map to their own categories, not AMD_UNKNOWN. Each prediction carries its turn ID, stage, and inference duration, and the amd_completed result reaches the session host with its last prediction.

Port the AMD v2 detector, stage FSM, classifier calls, AMD STT, and the turn hooks in AgentSession, AgentActivity, and AudioRecognition from livekit/agents#6202 at cad29d3bf. Remove the v1 AMD. Addresses AGT-3801

Now that per-response realtime tools (#5310 port) and ToolResult (#7340 port) are below this branch, finish the parts of livekit/agents#6202 that need them: - generateReply passes the turn-hook tools per response, as Python does when the hook changes the tool set. - A realtime generation runs tool calls against its reply's tools and gives its tool follow-up the same tools. - sendDtmfEvents returns ToolResult with replyRequired false and throws ToolError without provider text when a publish fails. Addresses AGT-3801

The session adds an agent reply when its speech ends. A participant turn committed during that speech went into AMD's classifier history first, so later classifications saw the reply after the turn that followed it. AMD now inserts the reply marker at the reply's start time. The docs also say that a realtime reply starts generating only after the machine silence hold, so its model latency adds to the wait. Syncs livekit/agents#6202 commits 6d30d20b1 and 33b4893f8.

After a menu or a screener, a short greeting such as "Hi. You've reached Sam." can come from a person or a voicemail box. In those stages, the first human prediction now counts as uncertain, and the next turn decides. The agent still replies, without stage instructions, and the classifier then sees the agent's words. AMD's own STT finalizes a sentence 0.1 to 0.2 s after the session commits it, so its text landed in the next turn. A turn without AMD text now waits up to 300 ms for that final. A late final never revises a prediction. In the tests, the AMD harness answers a trailing system message, as Python's FakeLLM does. The test hook's edit sorts after the user message, as Python's float clock orders it. Syncs livekit/agents#6202 commits 8157c2e13 and a01e8ebfe.

To hang up on voicemail, a caller had to watch amd_prediction: execute() returns only after the message and the voicemail idle timeout. Setting screeningInstructions, voicemailInstructions, or ivrInstructions to null now makes that a customer-handled stage: AMD completes when the call enters it. As with machine-unavailable, AMD skips that turn's reply and cancels pending replies. humanInstructions: null keeps the Agent's own instructions. `undefined` still selects the default instructions, as it does for every AMD option, and `null` turns a stage's reply off, as `stt: null` does. Syncs livekit/agents#6202 commit 6afd595ce.

@livekit/protocol 1.53.3 (livekit/protocol#1880) adds the AMD v2 fields. Screening and wait now map to their own categories, not AMD_UNKNOWN. Each prediction carries its turn ID, stage, and inference duration, and the amd_completed result reaches the session host with its last prediction. Python's RemoteSession emits every event under its oneof name. The JS RemoteSession dispatches each event by name, so it gains an amd_completed event. Syncs livekit/agents#6202 commit dcdf6078f.

AMD's llm now also accepts a DecisionModel that supports choice decisions. It gets the same content as an LLM, as one choice: CLASSIFY_PROMPT as the prompt, the seven categories as options, and the request state and classifier history as the input text. The options have no descriptions: with each category's short definition attached, Luna read a virtual assistant as a person. A decision model cannot write a menu, so AMD skips menu extraction with one. The approach follows the decision-model adapters in livekit/agents-private#511.

# Conflicts: # uv.lock

The flag once let the classifier unlock a stage correction. Since AMD only recommends transitions, the FSM derives it, and nothing reads it: not the classifier request, the remote session, or logs. The protocol dropped it too (livekit/protocol#1880).

Port the AMD v2 detector, stage FSM, classifier calls, AMD STT, and the turn hooks in AgentSession, AgentActivity, and AudioRecognition from livekit/agents#6202 at cad29d3bf. Remove the v1 AMD. Addresses AGT-3801

Now that per-response realtime tools (#5310 port) and ToolResult (#7340 port) are below this branch, finish the parts of livekit/agents#6202 that need them: - generateReply passes the turn-hook tools per response, as Python does when the hook changes the tool set. - A realtime generation runs tool calls against its reply's tools and gives its tool follow-up the same tools. - sendDtmfEvents returns ToolResult with replyRequired false and throws ToolError without provider text when a publish fails. Addresses AGT-3801

The session adds an agent reply when its speech ends. A participant turn committed during that speech went into AMD's classifier history first, so later classifications saw the reply after the turn that followed it. AMD now inserts the reply marker at the reply's start time. The docs also say that a realtime reply starts generating only after the machine silence hold, so its model latency adds to the wait. Syncs livekit/agents#6202 commits 6d30d20b1 and 33b4893f8.

After a menu or a screener, a short greeting such as "Hi. You've reached Sam." can come from a person or a voicemail box. In those stages, the first human prediction now counts as uncertain, and the next turn decides. The agent still replies, without stage instructions, and the classifier then sees the agent's words. AMD's own STT finalizes a sentence 0.1 to 0.2 s after the session commits it, so its text landed in the next turn. A turn without AMD text now waits up to 300 ms for that final. A late final never revises a prediction. In the tests, the AMD harness answers a trailing system message, as Python's FakeLLM does. The test hook's edit sorts after the user message, as Python's float clock orders it. Syncs livekit/agents#6202 commits 8157c2e13 and a01e8ebfe.

To hang up on voicemail, a caller had to watch amd_prediction: execute() returns only after the message and the voicemail idle timeout. Setting screeningInstructions, voicemailInstructions, or ivrInstructions to null now makes that a customer-handled stage: AMD completes when the call enters it. As with machine-unavailable, AMD skips that turn's reply and cancels pending replies. humanInstructions: null keeps the Agent's own instructions. `undefined` still selects the default instructions, as it does for every AMD option, and `null` turns a stage's reply off, as `stt: null` does. Syncs livekit/agents#6202 commit 6afd595ce.

@livekit/protocol 1.53.3 (livekit/protocol#1880) adds the AMD v2 fields. Screening and wait now map to their own categories, not AMD_UNKNOWN. Each prediction carries its turn ID, stage, and inference duration, and the amd_completed result reaches the session host with its last prediction. Python's RemoteSession emits every event under its oneof name. The JS RemoteSession dispatches each event by name, so it gains an amd_completed event. Syncs livekit/agents#6202 commit dcdf6078f.

AMD's llm option now also accepts a DecisionModel that supports choice decisions. It gets the same content as an LLM, as one choice: CLASSIFY_PROMPT as the prompt, the seven categories as options, and the request state and classifier history as the input text. The options have no descriptions: with each category's short definition attached, Luna read a virtual assistant as a person. A decision model cannot write a menu, so AMD skips menu extraction with one. Syncs livekit/agents#6202 commit 371b13995.

The flag once let the classifier unlock a stage correction. Since AMD only recommends transitions, the FSM derives it, and nothing reads it: not the classifier request, the remote session, or logs. The protocol dropped it too (livekit/protocol#1880). The model no longer returns correction fields, so the test that AMD ignores them goes too. Syncs livekit/agents#6202 commits 7856a5de4 and 1464df2f1.

main folded per-PR test files (#8510) and fixed callee hangups in the old AMD classifier (#8242). This branch replaces that classifier, and its detector already completes with reason participant_disconnected, so the old classifier.py and main's tests/test_amd.py are dropped. The branch's edits to the folded test files move to tests/test_audio_recognition.py and tests/test_agent_interruptions.py.

main now keeps one test module per source module (#8510). The nine tests/test_amd_*.py files and tests/amd_test_utils.py become regions of tests/test_amd.py. The realtime session helper is now running_realtime, because the detector tests already define running.

chenghao-mou force-pushed the chenghao/feat/AGT-3038-call-screening-support-in-amd branch from 3173918 to 14d36f4 Compare July 24, 2026 12:00

dexhunter mentioned this pull request Jul 25, 2026

fix(amd): start SIP detection after answer #6386

Closed

chenghao-mou force-pushed the chenghao/feat/AGT-3038-call-screening-support-in-amd branch from 94f4e05 to 7dfc05f Compare August 8, 2026 21:51

cursor Bot mentioned this pull request Aug 9, 2026

docs: daily engineering digest for 2026-08-09 Seventhen/agents#54

Draft

eashwar-mp reviewed Aug 23, 2026

View reviewed changes

chenghao-mou force-pushed the chenghao/feat/AGT-3038-call-screening-support-in-amd branch from 7dfc05f to 7d3110f Compare September 11, 2026 11:59

chenghao-mou added the review_effort:high Needs broad context and substantial validation label Sep 11, 2026

chenghao-mou changed the title ~~feat(amd): support call screening~~ feat: add multi-turn answering-machine detection Sep 11, 2026

chenghao-mou added 12 commits September 11, 2026 16:25

chore: merge main into amd

6fc0231

Keep AMD reply gating intact after the turn-completion tracing refactor. Preserve AMD turn IDs and trace cleanup. Addresses AGT-3038

fix: restore amd model auto-selection

57943de

Select the default AMD models when llm or stt is omitted and LiveKit Cloud credentials are available. Explicit None inherits the active Agent LLM or session transcript. Addresses AGT-3038

chore: checkpoint amd refactor (wip)

f8c9892

refactor: separate amd reply control from agent activity

ad5c61e

Reduce coupling between AMD policy and speech scheduling. Keep each turn's reply guard valid after AMD detaches, including while a customer hook runs. Addresses AGT-3038

refactor: keep turn tracking inside amd

beafe1b

Remove the public user_turn_committed event and session turn counter. AMD returns a guard bound to each turn before the customer hook runs, so pending replies keep their original decision after newer turns arrive. Addresses AGT-3038

refactor: simplify amd prediction flow

5b1f92d

Route model and fallback predictions through the same release path. Inline turn handling and document the FSM and prediction reasons. Keep saved decisions intact across late results and silent supersession. Addresses AGT-3038

chenghao-mou mentioned this pull request Sep 16, 2026

feat: add multi-turn answering-machine detection chenghao-mou/agents#1

Open

chenghao-mou marked this pull request as ready for review September 18, 2026 10:15

chenghao-mou requested a review from a team as a code owner September 18, 2026 10:15

This comment was marked as resolved.

Sign in to view

chenghao-mou added 4 commits September 18, 2026 11:36

fix: clean up amd setup and sanitize failures

19b6b14

Create owned model clients after entry validation and close them when setup fails. Keep provider and listener exception content out of AMD logs and DTMF tool errors. Addresses AGT-3038

docs: explain user state timestamp migration

3a28eb7

Keep event-creation time in created_at. Direct speech-timing consumers to speech_timestamp, with created_at as the fallback when the boundary is unknown. This applies to sessions with or without AMD. Addresses AGT-3038

feat: support reply_required in tool results (#7340)

45db6d3

fix: suppress automatic replies after sending DTMF

6f3e628

Successful DTMF sends keep their result in history and wait for the next prompt. Use ToolResult from #7340 so the tool controls reply generation.

chenghao-mou added 2 commits October 28, 2026 15:21

docs(amd): say that realtime replies generate after the silence hold

33b4893

A pipeline reply starts generating when the prediction arrives. A realtime reply waits for playback authorization before it asks the model, so its latency follows the machine silence hold.

chenghao-mou mentioned this pull request Oct 28, 2026

fix(thunderphone): use _shorten_id after the openai_utils rename #8229

Merged

chenghao-mou added 2 commits October 29, 2026 11:14

chenghao-mou mentioned this pull request Nov 4, 2026

chore(protocol): bump livekit-protocol/protocol to 8da979b livekit/python-sdks#921

Merged

chenghao-mou added 4 commits November 5, 2026 14:04

Merge remote-tracking branch 'origin/main' into t3/amd-eval-set

26ef965

# Conflicts: # uv.lock

test(amd): drop the test for removed correction fields

1464df2

chenghao-mou mentioned this pull request Nov 5, 2026

feat(amd): tag AMD's gateway requests with a client feature header #8402

Open

chenghao-mou added this pull request to stack #8403 November 5, 2026 14:43

chenghao-mou added 3 commits November 10, 2026 14:18

Merge remote-tracking branch 'origin/main' into t3/amd-eval-set

cde23b8

Navigation Menu

Repository navigation

feat: add multi-turn answering-machine detection - #6202

feat: add multi-turn answering-machine detection#6202
chenghao-mou wants to merge 51 commits into
mainfrom
chenghao/feat/AGT-3038-call-screening-support-in-amd

chenghao-mou commented Jun 24, 2026 •

edited

Loading

Uh oh!

eashwar-mp left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

chenghao-mou commented Jun 24, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

eashwar-mp left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

This comment was marked as resolved.

Uh oh!

This branch has not been deployed

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

chenghao-mou commented Jun 24, 2026 •

edited

Loading