Repository navigation
[ADK BigQuery Plugin] Event trace_id fractures between INVOCATION_STARTING and AGENT_STARTING #4645
Description
There is a bug in the bigquery_agent_analytics_plugin where the trace_id for a single invocation splits halfway through the turn.
Early lifecycle events (USER_MESSAGE_RECEIVED and INVOCATION_STARTING) are logged with one
trace_id (fallback to invocation_id), while subsequent events (starting at AGENT_STARTING and continuing through LLM/Tool interactions) are logged under a completely different, newly generated trace_id.
Impact: This breaks downstream joins and observability.
A trace_id is meant to be the single, unbroken thread that ties a user's action to the database logs. By falling back to the invocation_id early, but generating an OTel trace ID late, the plugin breaks the data model in half. You cannot easily GROUP BY trace_id to evaluate an entire turn without writing complex queries holding the invocation_id together.
You can verify the data corruption in BigQuery using the following query on the destination table:
WITH SequencedEvents AS (
SELECT
timestamp,
session_id,
invocation_id,
event_type,
trace_id,
LEAD(event_type) OVER(PARTITION BY invocation_id ORDER BY timestamp ASC) as next_event,
LEAD(trace_id) OVER(PARTITION BY invocation_id ORDER BY timestamp ASC) as next_trace_id
FROM `YOUR_PROJECT.YOUR_DATASET.agent_events`
WHERE event_type IN ('INVOCATION_STARTING', 'AGENT_STARTING')
)
SELECT *
FROM SequencedEvents
WHERE event_type = 'INVOCATION_STARTING'
AND next_event = 'AGENT_STARTING'
AND trace_id != next_trace_id
ORDER BY timestamp DESC
Activity
caohy1988 commentedon Feb 28, 2026
Root Cause
When OTel IS configured but no ambient span exists (e.g. Agent Engine, custom runners), early events (USER_MESSAGE_RECEIVED, INVOCATION_STARTING) fell
back to invocation_id as trace_id, while AGENT_STARTING got a new OTel hex trace_id from push_span("agent") — causing trace_id fracture within a single
invocation.
Additionally, push_span() always created independent root spans (each with its own trace_id), so even when a parent span existed on the internal stack,
child spans wouldn't share the same trace_id.
Changes to bigquery_agent_analytics_plugin.py
parent span context without attaching to the ambient OTel context (preserving Span Hierarchy and Duration Inconsistency in BigQueryAgentAnalyticsPlugin #4561 isolation).
- If stack already has entries → no-op
- If ambient OTel span is valid → attach_current_span() (reuse runner's span)
- Otherwise → push_span("invocation") (create new root span)
Changes to test file
- test_trace_id_continuity_no_ambient_span — The core [ADK BigQuery Plugin] Event trace_id fractures between INVOCATION_STARTING and AGENT_STARTING #4645 regression test
- test_trace_id_continuity_with_ambient_span — Verifies ambient span is reused
- test_invocation_root_span_isolated_across_turns — Verifies each turn gets a unique trace_id
fix: trace_id continuity and o11y alignment for BQ plugin (google#4645)
caohy1988 commentedon Feb 28, 2026
Trace-ID fix details (#4645)
Root cause: When OTel is configured but the Runner's ambient span is absent (Agent Engine, custom runners), early events fell back to
invocation_idwhilepush_span("agent")created a new OTel span with a different hex trace_id.Fix:
TraceManager.ensure_invocation_span()— called inon_user_message_callbackandbefore_run_callbackbefore any events fire. Attaches the ambient span if valid, or creates a new root span.push_span()— now creates child spans under existing stack parent viatrace.set_span_in_context(), so all spans share one trace_id._resolve_ids()(replaces_resolve_span_ids()) — 3-layer priority for trace_id/span_id/parent_span_id:EventDataoverrides (post-pop callbacks likeafter_run_callback)EventData.trace_id_override— captured before popping the invocation span inafter_run_callback, ensuring INVOCATION_COMPLETED shares the same trace_id.feat: Enhance BQ plugin with fork safety, auto views, and trace conti…
haiyuan-eng-google commentedon Mar 2, 2026
fixed
fix(bigquery-analytics): stop exporting plugin-owned OTel spans (goog…
fix(bigquery-analytics): stop exporting plugin-owned OTel spans (goog…
docs(bqaa): drop GitHub-issue references from docstrings/comments
feat: Enhance BQ plugin with fork safety, auto views, and trace conti…