Repository navigation
feat(amd): tag AMD's gateway requests with a client feature header - #8402
chenghao-mou wants to merge 3 commits into
Conversation
AMD's classifier and IVR menu calls and its own STT stream now send X-LiveKit-Client-Feature: AMD when they reach LiveKit Inference, so the gateway can measure AMD traffic apart from the agent's own. The tag is per request: the agent's next turn on a shared inference.LLM stays untagged, and plugin models never get the header. inference.STT.stream() takes extra_headers for this. inference.LLM.chat() now merges a turn's extra_headers with the model's own instead of replacing them, and stamps SDK headers on that copy. Addresses AGT-3866
…nto chenghao/feat/AGT-3866-amd-client-feature The base branch merged main, which folded the per-PR test files (#8510) and dropped tests/test_inference_stt_start_of_speech.py. The one-line edit this branch made to that file goes with it: the test it touched no longer exists.
tests/test_amd_client_feature.py was a per-PR test file. The classifier and transcriber tests move to their regions of tests/test_amd.py. The inference STT header test moves to tests/test_inference_stt.py and now uses that module's in-memory gateway, which records connection headers.
| extra_kwargs: NotGivenOr[dict[str, Any]] = ( | ||
| {"extra_headers": {HEADER_CLIENT_FEATURE: CLIENT_FEATURE}} | ||
| if _reaches_only_inference(model) | ||
| else NOT_GIVEN | ||
| ) |
There was a problem hiding this comment.
🟡 Decision-model classifications remain untagged
When AMD uses inference.DecisionModel, classify calls evaluate instead of the tagged chat path. The gateway receives no AMD header for those classifications.
Learn more
AMD can classify with an inference decision model instead of an LLM. That path calls evaluate, bypassing the new tagged chat call. The gateway request uses DecisionModel._headers, which does not include the AMD feature header.
Example: Pass inference.DecisionModel("typesafe/jev") to AMD. Its /decisions classification request reaches the gateway without X-LiveKit-Client-Feature: AMD, while an inference.LLM classification includes it.
Recommended fix: Add a per-evaluation header override for inference decision models and pass the AMD feature only from AMD's decision-model branch; preserve the untagged default for ordinary evaluations.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if isinstance(self._model, inference.STT): | ||
| self._stream = self._model.stream( | ||
| conn_options=self._conn_options, | ||
| extra_headers={HEADER_CLIENT_FEATURE: _inference.CLIENT_FEATURE}, | ||
| ) | ||
| else: | ||
| self._stream = self._model.stream(conn_options=self._conn_options) |
There was a problem hiding this comment.
🟡 Fallback STT requests lose AMD attribution
When AMD uses stt.FallbackAdapter around inference STTs, AMDTranscriber skips the tagged stream path. The adapter opens each inference stream without extra_headers, so none of its gateway requests carry AMD attribution.
Learn more
AMD creates an optional separate STT stream to transcribe calls. A fallback STT is a valid streaming STT, but it fails the top-level inference.STT check. FallbackRecognizeStream then opens each inference stream without forwarding any feature headers.
Example: Configure AMD with stt.FallbackAdapter([inference.STT("deepgram/nova-3"), inference.STT("cartesia/ink-whisper")]). Both the primary gateway connection and any fallback connection omit the AMD feature header.
Recommended fix: Carry a per-stream feature header through the fallback adapter to inference STT legs without sending it to third-party STT implementations; preserve ordinary fallback streams' existing defaults.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if call_headers or model_headers: | ||
| extra["extra_headers"] = {**(call_headers or {}), **(model_headers or {})} |
There was a problem hiding this comment.
🟡 Model header overrides AMD request label
When model-level extra_headers sets X-LiveKit-Client-Feature, LLM.chat overwrites AMD's per-call value. The gateway attributes classification and menu requests to the model's configured feature instead of AMD.
Learn more
AMD supplies its feature name per call through the structured request. Inference LLMs also accept persistent extra_headers. The new merge copies per-call headers first and then overwrites duplicate keys with persistent headers, so a configured feature value wins.
Example: Configure an inference LLM with extra_kwargs={"extra_headers": {"X-LiveKit-Client-Feature": "Agent"}} and pass it to AMD. AMD's "AMD" request header becomes "Agent" on the wire.
Recommended fix: Give the per-call feature header precedence for AMD calls while preserving the intended precedence of other persistent headers; add a test covering duplicate feature keys.
Was this helpful? React with 👍 or 👎 to provide feedback.
AMD's classifier, IVR menu, and STT requests look like the agent's own Inference traffic, so AMD adoption is not measurable. They now send
X-LiveKit-Client-Feature: AMD, per request and only to LiveKit Inference (also an all-InferenceFallbackAdapter); the gateway records it (livekit/agent-gateway#1904).inference.LLM.chat()now merges a turn'sextra_headerswith the model's own instead of dropping them.Stacked on #6202. Addresses AGT-3866
Initial prompt and agent context
Model: Claude Opus 5.5
Follow-ups that changed the request: