Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions python/frameworks/google-genai-live/CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
# Changelog

## 0.1.0 (alpha)

- New package: tracing for Gemini Live sessions opened with `client.aio.live.connect` on the
Gemini Developer API (google-genai `>=2.28,<3`; tested on 2.31.0 and 2.28.0 with
`gemini-3.8-live`). Gemini Live is a preview API: a field rename upstream is a floor and
fixture update here, not a silent fallback. google-genai 3.0 removes `Live.send` /
`Live.start_stream`, which this package never wrapped; pin below 3.
- One trace per session: setup, user turns, model turns (model, tokens, transcription), tool
calls with cancellation, interruptions, `goAway`, and a resume with the stored handle as a
`gemini.live.reconnect` span in the same trace.
- Audio sessions carry `fi.span.kind=CONVERSATION` on a parentless root plus `call.*` and
`conversation.transcript*`, so Future AGI lists them as calls. Text-only sessions are `CHAIN`.
- Privacy: audio bytes, API keys and resumption handles never land on a span. Text content is
on by default, as in every traceAI instrumentor; the standard `FI_HIDE_INPUTS` /
`FI_HIDE_OUTPUTS` (and `*_MESSAGES` / `*_TEXT`) variables or `TraceConfig(...)` drop it.
No package-specific privacy switch. Raw error text is content; stack traces are never exported.
- Not supported: Vertex AI Live, ephemeral tokens, the deprecated `send` / `start_stream`,
LiveKit / Pipecat sessions, Node.
148 changes: 148 additions & 0 deletions python/frameworks/google-genai-live/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,148 @@
# traceAI-google-genai-live

Future AGI tracing for **Gemini Live** sessions that your Python process opens itself with
the official SDK: `client.aio.live.connect(...)` on the Gemini Developer API.

One Live session is one trace. Inside it you get the setup, each user turn, each model turn
(model, tokens, transcription), each tool call, interruptions, cancellations, `goAway`, and
a reconnect that resumes the session with the handle this process stored. Audio sessions are
listed as calls in Future AGI's voice views.

Status: **alpha**. Gemini Live is a preview API.

| Tested | Version |
|---|---|
| google-genai | **2.31.0** (latest, 2026-10-14) and 2.28.0 (floor) |
| Live model | `gemini-3.8-live` (Gemini Developer API, real sessions on 2026-10-15) |
| Python | 3.10 - 3.13 (`fi-instrumentation-otel` declares `<3.14`) |

## Install

```bash
pip install traceAI-google-genai-live "google-genai>=2.28,<3"
```

## Use

```python
import os

from fi_instrumentation import register
from fi_instrumentation.fi_types import ProjectType
from google import genai
from traceai_google_genai_live import instrument

trace_provider = register(project_type=ProjectType.OBSERVE, project_name="my-voice-agent")
instrument(tracer_provider=trace_provider)

client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
async with client.aio.live.connect(model="gemini-3.8-live",
config={"response_modalities": ["AUDIO"],
"input_audio_transcription": {},
"output_audio_transcription": {}}) as session:
... # your send_realtime_input / send_client_content / receive / send_tool_response loop
```

`GoogleGenAILiveInstrumentor().instrument(tracer_provider=...)` is the same thing.
A complete runnable example with a tool call is in [`examples/voice_tool.py`](examples/voice_tool.py).

Options:

| Option | Default | Effect |
|---|---|---|
| `config=TraceConfig(...)` or the standard `FI_HIDE_*` environment variables | content on | `hide_inputs` / `FI_HIDE_INPUTS` drops the input side (system instruction, user text and transcription, tool arguments); `hide_outputs` / `FI_HIDE_OUTPUTS` drops the output side (model text and transcription, tool results). `*_input_messages` / `*_input_text` (and the output pair) do the same, because all Live content is message text. Both sides off leaves metadata only: model, voice name, timings, tool names, flags, close codes and token counts. The conversation transcript and raw error text need both sides on. |
| `session_id`, `user_id` | none | Defaults for every session. `fi_instrumentation.using_session(...)` / `using_user(...)` around `connect` override them per call. |
| `resume_window_seconds` | `30` | How long a session whose connection ended after `goAway`, or died, waits for a reconnect with its stored handle before the root span closes. |

## What you get

```
gemini.live.session CONVERSATION (audio) / CHAIN (text-only); parentless when CONVERSATION
├─ gemini.live.setup url.path (no host, no query), system instruction (content)
├─ gemini.live.user_turn transcription or text (content); gemini.turn_start / gemini.turn_boundary
├─ gemini.live.model_turn LLM: model, llm.token_count.*, gemini.usage.*, transcription (content)
│ └─ gemini.live.tool tool name, call id, args and result (content); cancelled / unanswered flags
├─ gemini.live.reconnect gap between the dropped connection and the resumed setup
└─ gemini.live.tool_response a response for a cancelled (ignored) or unknown (unmatched, ERROR) call id
```

Root attributes: `gen_ai.system=google_genai`, `gen_ai.request.model`, `session.id`
(+ `gemini.session_id_source` = `customer` | `server` | `local`), `user.id` only if you set
one, `gemini.response_modalities`, `gemini.voice_name`, `gemini.tool_names`,
`gemini.realtime_input_count`, `gemini.go_away`, `gemini.time_left`, `gemini.reconnect_count`.

For audio sessions the root also carries the voice-call fields: `call.duration`,
`call.total_turns` (omitted, not 0, when no turn closed), `call.talk_ratio` (model speech
seconds / user speech seconds; omitted unless both were measured: user speech from the
server's `voiceActivity` offsets, model speech from raw PCM length at the declared rate), and
`conversation.transcript` / `fi.conversation.transcript` plus flattened
`conversation.transcript.{i}.message.role|content` rows when content capture is on.

How server events map:

- `generationComplete` sets `gemini.generation_complete` and does not close the model turn.
`turnComplete` closes it. `interrupted` closes it at once (`gemini.interrupted=true`).
- `send_client_content` during an open model turn closes that turn with
`gemini.close_reason=client_content`.
- A user turn opens on the server's `voiceActivity` `ACTIVITY_START` (or your
`activity_start`, a realtime `text=` input, or the first input transcription) and closes on
`ACTIVITY_END` (`gemini.turn_boundary=activity_end`), falling back to `turnComplete`
(`gemini.turn_boundary=turn_complete`). `send_client_content` is its own user turn.
- `toolCall` opens a model turn if none is open (real sessions send the call with no model
output before it, then `turnComplete` + usage for that generation), and one tool span per
function call. The span closes on your `send_tool_response` with the same id.
`toolCallCancellation` closes it with `gemini.cancelled=true` and no result.
- `usageMetadata` is copied onto the open model turn (latest message wins on that span, and
every message is also a span event). Nothing is summed. In real `gemini-3.8-live` sessions
each message covered one generation (its prompt count includes the context so far).
- Audio frames sent with `send_realtime_input` are counted, never spanned.

## Failures

| Case | Trace |
|---|---|
| Setup rejected (bad model, bad key) | One ERROR root (`setup_rejected`) with the setup span; the SDK's `APIError` reaches your code unchanged. No turns, no `call.total_turns`. |
| Socket dies (no `goAway`) | Open spans close ERROR `socket_closed`, `gemini.close_code` on the root, one flush. No reconnect by this package. |
| `goAway` | `gemini.go_away` / `gemini.time_left` on the root; the root stays open. |
| Reconnect with the stored handle | Same trace, one `gemini.live.reconnect` span. Failure: root ERROR `resume_failed`. |
| Reconnect with a handle this process did not store | New trace (`gemini.resumed_unknown_handle=true`). |
| Task cancelled | Open spans and the root close ERROR `task_cancelled`. |
| Exception from `receive()` | Recorded on the root, re-raised unchanged. |
| Export queue full | Spans are dropped by the batch processor; the socket reader is never blocked. |

## Privacy

- Default: text on, audio off, as in every traceAI instrumentor. Audio bytes (`realtimeInput`
and `modelTurn` inline data) never land on a span.
- Metadata only: `FI_HIDE_INPUTS=true FI_HIDE_OUTPUTS=true` (or
`TraceConfig(hide_inputs=True, hide_outputs=True)`). There is no package-specific switch.
- The API key is never copied. The connection URL is recorded as its path only.
- The session resumption handle stays in memory and is never a span attribute or a log line.
- Raw error text is content: it is recorded only when both content directions are on. Stack
traces are never exported.

## Not supported

- Vertex AI Live (`vertexai=True`): one warning, not traced.
- Ephemeral tokens (`auth_tokens/...`, `BidiGenerateContentConstrained`): one warning, not traced.
- The deprecated `session.send` and `session.start_stream` are not wrapped. Use
`send_client_content`, `send_realtime_input`, `send_tool_response` and `receive`.
- `generate_content` / `generate_content_stream`: use `traceAI-google-genai`.
- Gemini Live inside LiveKit or Pipecat: use `traceAI-livekit` / `traceAI-pipecat`.
- Node `@google/genai`.
- google-genai 3.x (the version check refuses it).
- No audio recording attachment, no cost in currency (the platform estimates cost from model
and tokens where it has a price).

## Tests

```bash
pip install -e . pytest pytest-asyncio
pytest
```

The tests run the real google-genai SDK against a scripted socket (`tests/conftest.py`).
`tests/fixtures/real_20261015_*.json` are server frames recorded from real
`gemini-3.8-live` sessions (audio replaced by silence, resumption handles replaced).
`tests/_voice_tab.py` is a local stand-in for the shared TH-8339 voice-tab assertion helper,
which had not merged; replace it with the shared helper when it lands.
26 changes: 26 additions & 0 deletions python/frameworks/google-genai-live/examples/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
# Example: a traced Gemini Live voice session

`voice_tool.py` opens one Live session on `gemini-3.8-live`, asks for the weather (typed, or
spoken from a 16 kHz 16-bit mono PCM file), answers the model's `get_weather` tool call, and
lets the model answer out loud. With `instrument()` the whole session is one Future AGI trace.

```bash
pip install traceAI-google-genai-live "google-genai>=2.28,<3"
export GEMINI_API_KEY=... FI_API_KEY=... FI_SECRET_KEY=...
# export FI_BASE_URL=... # only for a self-hosted Future AGI collector
python voice_tool.py --project gemini-live-demo
python voice_tool.py --pcm question.pcm # spoken question
FI_HIDE_INPUTS=true FI_HIDE_OUTPUTS=true python voice_tool.py --marker X # no content leaves the process
python voice_tool.py --resume # drop the socket, resume with the handle: still one trace
python voice_tool.py --model not-a-model # setup rejected: one ERROR root
```

A spoken question on macOS:

```bash
say -o q.aiff "What is the weather in Paris right now?"
afconvert -f WAVE -d LEI16@16000 -c 1 q.aiff q.wav
python -c "import wave; w=wave.open('q.wav'); open('question.pcm','wb').write(w.readframes(w.getnframes()))"
```

`run_e2e.sh` runs the scenarios above in a fresh virtualenv built from this checkout.
24 changes: 24 additions & 0 deletions python/frameworks/google-genai-live/examples/run_e2e.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
#!/usr/bin/env bash
# End-to-end run of traceAI-google-genai-live against a real Gemini Live model and a Future AGI
# collector. Needs GEMINI_API_KEY, FI_API_KEY, FI_SECRET_KEY (and FI_BASE_URL for a local
# collector). Each scenario is one short real Live session.
# usage: [PYTHON=python3.13] examples/run_e2e.sh [project-name] [google-genai version]
# Python 3.10-3.13 (fi-instrumentation-otel declares <3.14).
set -euo pipefail
HERE="$(cd "$(dirname "$0")" && pwd)"
PKG="$(dirname "$HERE")"
PROJECT="${1:-gemini-live-e2e}"
GENAI="${2:-2.31.0}"
WORK="$(mktemp -d)"
trap 'rm -rf "$WORK"' EXIT
"${PYTHON:-python3}" -m venv "$WORK/venv"
"$WORK/venv/bin/pip" -q install "$PKG" "google-genai==$GENAI"
export FI_PROJECT_NAME="$PROJECT"
PY="$WORK/venv/bin/python"
EX="$HERE/voice_tool.py"
MARK="FI-MARKER-$RANDOM$RANDOM"
echo "1. typed question + tool"; "$PY" "$EX"
echo "2. drop + resume (one trace)"; "$PY" "$EX" --resume
echo "3. content hidden ($MARK)"; FI_HIDE_INPUTS=true FI_HIDE_OUTPUTS=true "$PY" "$EX" --marker "$MARK"
echo "4. setup rejected (expect error)"; "$PY" "$EX" --model gemini-not-a-model-live || true
echo "Done. In Future AGI open project $PROJECT: 4 traces; search for $MARK must return nothing."
131 changes: 131 additions & 0 deletions python/frameworks/google-genai-live/examples/voice_tool.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,131 @@
"""Trace one Gemini Live voice session with a tool call in Future AGI.

pip install traceAI-google-genai-live "google-genai>=2.28,<3"
export GEMINI_API_KEY=... FI_API_KEY=... FI_SECRET_KEY=...
python voice_tool.py # typed question, spoken answer
python voice_tool.py --pcm question.pcm # spoken question (16 kHz, 16-bit mono PCM)
FI_HIDE_INPUTS=true FI_HIDE_OUTPUTS=true python voice_tool.py # metadata only, no content

Options used by the package's end-to-end run:
--marker TEXT put TEXT in the system instruction, the question and the tool result
--resume drop the socket after the first answer and resume with the stored handle
--model NAME Live model (default gemini-3.8-live; the list is preview, re-read it)
--modality TEXT ask for TEXT responses (native-audio models refuse this)
--user-id ID attach your own user id to the call (the wire carries none)
"""

import argparse
import asyncio
import os

from fi_instrumentation import register, using_user
from fi_instrumentation.fi_types import ProjectType
from google import genai
from google.genai import errors, types

from traceai_google_genai_live import instrument

parser = argparse.ArgumentParser()
parser.add_argument("--project", default=os.getenv("FI_PROJECT_NAME", "gemini-live-demo"))
parser.add_argument("--model", default="gemini-3.8-live")
parser.add_argument("--pcm")
parser.add_argument("--marker", default="")
parser.add_argument("--resume", action="store_true")
parser.add_argument("--modality", default="AUDIO")
parser.add_argument("--user-id")
args = parser.parse_args()

# 1. Future AGI first, then the instrumentor. Content capture follows the standard
# FI_HIDE_INPUTS / FI_HIDE_OUTPUTS environment variables (or pass config=TraceConfig(...)).
trace_provider = register(project_type=ProjectType.OBSERVE, project_name=args.project)
instrument(tracer_provider=trace_provider)

WEATHER = {
"name": "get_weather",
"description": "Current weather for a city.",
"parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]},
}


def get_weather(city: str) -> dict:
return {"city": city, "forecast": "18C and sunny", "note": args.marker}


def live_config(handle=None) -> dict:
return {
"response_modalities": [args.modality],
"input_audio_transcription": {},
"output_audio_transcription": {},
"system_instruction": "You are a terse voice assistant. Always use get_weather for weather. "
+ args.marker,
"tools": [{"function_declarations": [WEATHER]}],
"session_resumption": {"handle": handle} if handle else {},
}


async def stream_pcm(session, path: str) -> None:
pcm = open(path, "rb").read()
chunk = 3200 # 100 ms of 16 kHz 16-bit mono
for i in range(0, len(pcm), chunk):
await session.send_realtime_input(audio=types.Blob(data=pcm[i:i + chunk], mime_type="audio/pcm;rate=16000"))
await asyncio.sleep(0.1)
for _ in range(15): # trailing silence so server VAD ends the user turn
await session.send_realtime_input(audio=types.Blob(data=b"\x00" * chunk, mime_type="audio/pcm;rate=16000"))
await asyncio.sleep(0.1)
await session.send_realtime_input(audio_stream_end=True)


async def answer_turns(session, state: dict) -> None:
"""Receive until the model has spoken an answer (a turn with audio), answering tool calls."""
while True:
spoke = False
async for msg in session.receive():
if msg.session_resumption_update and msg.session_resumption_update.new_handle:
state["handle"] = msg.session_resumption_update.new_handle
if msg.data:
spoke = True
if msg.server_content and msg.server_content.output_transcription:
print("model:", msg.server_content.output_transcription.text, flush=True)
if msg.tool_call:
await session.send_tool_response(function_responses=[
types.FunctionResponse(id=fc.id, name=fc.name, response=get_weather(**(fc.args or {})))
for fc in msg.tool_call.function_calls])
if spoke:
return


async def main() -> None:
client = genai.Client(api_key=os.environ["GEMINI_API_KEY"])
question = "What is the weather in Paris right now? " + args.marker
state: dict = {}
async with client.aio.live.connect(model=args.model, config=live_config()) as session:
if args.pcm:
sender = asyncio.create_task(stream_pcm(session, args.pcm))
else:
await session.send_client_content(turns={"role": "user", "parts": [{"text": question}]})
sender = None
await asyncio.wait_for(answer_turns(session, state), 60)
if sender:
await sender
if args.resume:
# Simulate a dropped connection: the next receive fails, as it would on a network drop.
await session.close()
try:
async for _ in session.receive():
pass
except errors.APIError as exc:
print("connection dropped:", exc.code, flush=True)
if args.resume and state.get("handle"):
async with client.aio.live.connect(model=args.model, config=live_config(state["handle"])) as session:
await session.send_client_content(turns={"role": "user", "parts": [{"text": "Which city did I ask about?"}]})
await asyncio.wait_for(answer_turns(session, state), 60)


try:
if args.user_id:
with using_user(args.user_id):
asyncio.run(main())
else:
asyncio.run(main())
finally:
trace_provider.force_flush()
Loading
Loading