Skip to content

feat(google-genai-live): Gemini Live instrumentor (TH-8243) - #255

Merged
nik13 merged 6 commits into
devfrom
feat/th-8243-gemini-live
Oct 28, 2026
Merged

nik13 merged 6 commits into
devfrom
feat/th-8243-gemini-live

Conversation

@nik13

@nik13 nik13 commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds traceAI-google-genai-live in python/frameworks/google-genai-live/ for Gemini Live sessions opened by a customer's Python process using the official SDK's client.aio.live.connect(...) on the Gemini Developer API. TH-8243, parent TH-8103. Companion docs: future-agi/docs#938.

  • Wraps AsyncLive.connect and AsyncSession.send_client_content, send_realtime_input, send_tool_response, and receive. Does not wrap deprecated send / start_stream, ordinary generation, LiveKit or Pipecat.
  • One session trace: root → setup, user/model turns, tool children and reconnect spans when the application resumes with its stored handle.
  • Audio sessions use a parentless CONVERSATION root, voice-call timings/turn metadata and transcripts when content capture permits. Text-only sessions use CHAIN.
  • Model and the API's latest reported usage are copied to model-turn spans; usage messages are also events, not summed. Tool calls receive an LLM parent even when the tool request arrives before model output.
  • Uses standard traceAI TraceConfig / FI_HIDE_* controls, not a package-specific metadata-only flag. Raw error text requires both content directions enabled. API keys, raw audio, resumption handles and stacktraces are excluded; connection URL is path only.

Exact-head acceptance — 2026-10-28

PASS at 341d22b11e0ebae838793341a7ed62049a6a124c. Current dev was merged into the PR normally; no integration code changes were needed for this refresh.

Endpoint: native Gemini Developer API WebSocket, wss://generativelanguage.googleapis.com/ws/google.ai.generativelanguage.v1beta.GenerativeService.BidiGenerateContent. This run used neither CLIProxy nor Vertex. The model list was fetched with the temporary key, and gemini-2.5-flash-native-audio-latest was selected from models advertising bidiGenerateContent.

Consumer build: wheel built from this exact head and installed into a fresh Python 3.12.10 environment with google-genai==2.31.0, published fi-instrumentation-otel==1.1.0, OpenTelemetry SDK 1.45.1 and websockets 16.1.1. The README's examples/voice_tool.py was copied unchanged into the consumer project and run with the selected model: one short typed weather question, a real get_weather function call/response, and an audio answer with output transcription. No audio file was saved or exported.

Flow Future AGI Cloud trace Result
README weather tool session 3d61a5a7-8f7c-4814-83b5-c1fa371a0f50 CONVERSATION → setup + user turn + LLM turn → tool; all OK; answer “18C and sunny.”; transcript present
Both FI_HIDE_INPUTS and FI_HIDE_OUTPUTS dd7f9509-8825-4688-a416-6c5a0a20a05a Successful same hierarchy with model/usage/tool metadata retained; content and transcript omitted; unique prompt/system/tool marker absent from the full retrieved payload
FI_HIDE_INPUTS alone 05f0e85f-25c8-41b0-ac4f-31b0418cee8a Input direction omitted; answer/tool output remains visible; root transcript omitted
FI_HIDE_OUTPUTS alone 8065533d-c198-472e-a65d-78ff3e18a57d Output direction omitted; input remains visible; root transcript omitted
Invalid Live model d97ab816-5deb-46e1-9815-f2a58cb0af3e SDK APIError, close 1008; ERROR session + ERROR setup, setup_rejected, no invented turns or token counts
Invalid model with both hide flags 711c7751-9a23-493b-8864-d42e88ba5f60 Same error metadata; system instruction and raw exception message omitted

Direct export to Future AGI Cloud and read-back verified all six traces: unique span IDs, correct parent relationships, requested model, positive API-reported LLM usage, transcript/tool content in default mode, directional privacy and real setup errors. Default-session LLM usage was 538 prompt / 59 response tokens, matching the server's latest usage metadata. The actual temporary Gemini key and both FI key values were absent from every retrieved payload, as were local paths, stacktraces and raw audio.

A single key restricted to generativelanguage.googleapis.com was created in fagi-omega-staging for both Gemini integration tests, then deleted and verified absent from active keys. generativelanguage.googleapis.com and apikeys.googleapis.com were both restored to their original disabled states. The earlier CLIProxy Live-route blocker is superseded for acceptance by these successful native Developer API sessions.

Tests and CI

  • Installed-wheel tests with google-genai 2.31.0 — 48 passed, 1 skipped (version fence), one vendor experimental-token warning.
  • Exact-head package/core Python CI — SUCCESS, including package (google-genai-live).
  • Exact-head integration CI — SUCCESS.

Supported surface and limits

  • Alpha integration; Gemini Live is preview and the model list changes. Dependency range is google-genai>=2.28,<3; this refresh used 2.31.0. Python range is >=3.10,<3.14, matching the FI instrumentation dependency.
  • Vertex AI Live and ephemeral tokens are explicitly warned and excluded. Node @google/genai, google-genai 3.x, deprecated send methods and audio-recording attachments are outside scope.
  • Session resumption is application-driven: this instrumentor traces a reconnect with a handle it stored; it does not reconnect the application. Cancellation, interruptions, goAway and resumption are covered by the package's scripted-SDK test suite; this native refresh exercised short clean sessions and rejected setup.
  • Root transcripts require both content directions enabled. Single-direction privacy is intentionally directional: text present in a tool output may remain visible with only inputs hidden, and text supplied as input may remain visible with only outputs hidden.
  • call.talk_ratio is omitted unless both speech durations are measured. The typed-input sessions in this refresh do not claim a user-audio timing ratio.
  • No cost in currency is computed by this package; no package publishing is part of this refresh.

Remaining acceptance blockers: none. This refresh does not merge the PR.

🤖 Generated with Claude Code

New package traceAI-google-genai-live: wraps google-genai AsyncLive.connect and the
AsyncSession send_client_content / send_realtime_input / send_tool_response / receive
methods (Gemini Developer API, google-genai >=2.28,<3) so one Live session is one trace:
setup, user turns, model turns with model and tokens, tool calls with cancellation,
interruptions, goAway, and a resume with the stored handle as a reconnect span in the same
trace. Audio sessions get a parentless CONVERSATION root with call.* and
conversation.transcript* so Future AGI lists them as calls; text sessions are CHAIN.

Audio bytes, API keys and resumption handles never land on a span. Text content is on by
default; metadata_only / TraceConfig hide_inputs / hide_outputs drop it. Vertex Live,
ephemeral tokens, Live.send and Live.start_stream are not wrapped.

Tests run the real SDK against a scripted socket, including frames recorded from real
gemini-3.8-live sessions on 2026-10-15.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
nik13 and others added 2 commits October 15, 2026 21:58
…ming tests

Drop the package-specific metadata_only option. Content capture now follows the
standard traceAI TraceConfig convention only: text on by default; FI_HIDE_INPUTS /
FI_HIDE_OUTPUTS (or TraceConfig(hide_inputs/hide_outputs)) drop a direction, and the
*_MESSAGES / *_TEXT variants do the same because all Live content is message text.
The example, run_e2e.sh, README and changelog use the env vars.

Tests: privacy tests run through the env vars and TraceConfig; new cases for the
message/text switches. The non-blocking-reader check allows half the 0.5 s export time
instead of 50 ms, and the flush-once check waits for the flush thread to start, so both
hold on a loaded machine. Unused imports removed from test_fence.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@nik13
nik13 marked this pull request as ready for review October 21, 2026 22:19
nik13 and others added 3 commits October 27, 2026 20:37
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
Co-Authored-By: Claude Code <noreply@anthropic.com>
@nik13
nik13 merged commit d8fe7bf into dev Oct 28, 2026
9 checks passed
@nik13
nik13 deleted the feat/th-8243-gemini-live branch October 28, 2026 06:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant