Emit gen_ai.* attributes on APM spans by mz1119 · Pull Request #12450 · DataDog/dd-trace-java

mz1119 · 2026-09-10T15:02:09Z

What Does This Do

Emits the scalar gen_ai.* attributes onto APM spans, making them indexed APM tags:

meta: gen_ai.operation.name, gen_ai.request.model, gen_ai.provider.name, gen_ai.application.name, gen_ai.conversation.id
metrics: gen_ai.usage.{input,output,total,cache_read_input,cache_write_input,reasoning_output}_tokens

GenAiApmTags reads the values back off the span's _ml_obs_tag.* / _ml_obs_metric.* keys at finish time and is called from both emission sites: manual SDK spans (DDLLMObsSpan.finish) and auto-instrumentation (OpenAiDecorator.doBeforeFinish). Since LLMObsSpanMapper only serializes _ml_obs_* keys, the unprefixed tags land on the APM span only.

Alongside them it sets the internal _dd.llmobs.artificial_gen_ai_tags tag, marking the attributes as tracer-written rather than user-annotated. Without it the backend processor treats the APM span as a user-annotated gen_ai span and creates a duplicate LLM Observability span.

Message bodies (input, output, tool definitions, retrieval documents) stay off the APM span and keep coming from the LLMObs track. Port of DataDog/dd-trace-py#20083, DataDog/dd-trace-py#20411 and DataDog/dd-trace-js#10261.

With LLM Observability disabled

The openai instrumentation still traces, so it still emits the attributes it can resolve. It does not build the LLMObs tag set in that configuration, so OpenAiDecorator passes operation (from openai.request.endpoint), model and ml_app to GenAiApmTags.apply directly, and the token counts it reads off every response are routed through TokenUsage to gen_ai.usage.* instead of _ml_obs_metric.*. gen_ai.conversation.id is the one attribute with no APM-side source, so it is absent. This matches dd-trace-js.

Motivation

The APM trace UI merges these keys from the LLMObs track client-side, so the values render but aren't indexed. You can't filter, facet, or monitor on model, provider, or token usage in APM.

Testing

testing streaming and non-streaming, llmobs enabled and disabled: 4 traces show up in query

showing up in query

embedding

🤖 Generated with Claude Code

Write the scalar gen_ai.* subset of an LLM Observability span onto the APM span, so model, provider, application, conversation and token usage become indexed, searchable APM tags. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

datadog-datadog-us1-prod · 2026-09-10T15:17:32Z

🎯 Code Coverage (details)
• Patch Coverage: 94.74%
• Overall Coverage: 59.21% (+0.00%)

_{This comment will be updated automatically if new data arrives.

🔗 Commit SHA: 3556486 | Docs | Give us feedback!}

dd-octo-sts · 2026-09-10T15:28:47Z

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite	Status
Startup	🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results

Scenario	Candidate	master	Δ (95% CI of mean)
startup:insecure-bank:iast:Agent	14.00 s	13.91 s	[-0.0%; +1.3%] (no difference)
startup:insecure-bank:tracing:Agent	12.91 s	12.98 s	[-1.3%; +0.1%] (no difference)
startup:petclinic:appsec:Agent	17.12 s	16.94 s	[+0.1%; +2.0%] (maybe worse)
startup:petclinic:iast:Agent	16.93 s	17.08 s	[-1.7%; -0.2%] (maybe better)
startup:petclinic:profiling:Agent	16.65 s	16.97 s	[-3.2%; -0.6%] (maybe better)
startup:petclinic:sca:Agent	16.87 s	16.78 s	[-0.3%; +1.4%] (no difference)
startup:petclinic:tracing:Agent	16.16 s	16.10 s	[-0.7%; +1.4%] (no difference)

Commit: 35564866 · CI Pipeline · Benchmarking Platform UI

Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

mz1119 · 2026-09-10T18:26:33Z

@codex review

chatgpt-codex-connector · 2026-09-10T18:26:52Z

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-11T17:28:18.683154Z	`1527dca`	Draft marked ready
🔒 Security Review	✅ Completed 2026-09-11T17:29:41.733301Z	`1527dca`	Draft marked ready

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

chatgpt-codex-connector · 2026-09-10T18:31:01Z

🛡️ Codex Security Review · Automatically triggered

Security review completed. No security issues were found in this pull request.

Reviewed commit: f1d46cfe83

View security finding report

Only the user who started this review can view the report in Codex.

ℹ️ About Codex security reviews in GitHub

This is an experimental Codex feature. Security reviews are triggered when:

You comment "@codex security review"
A regular code review gets triggered (for example, "@codex review" or when a PR is opened), and you’re opted in so security review runs alongside code review

Once complete, Codex will leave suggestions, or a comment if no findings are found.

chatgpt-codex-connector · 2026-09-10T18:31:04Z

Codex Review: Didn't find any major issues. Swish!

Reviewed commit: f1d46cfe83

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

The openai-java instrumentation keeps tracing when LLM Observability is disabled, and the operation, model, provider and ml_app are all known on that path. Token usage and conversation id are not computed there, so they stay unreported. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The repository does not accept new .groovy files. The module already has a JUnit forked-test base for LLMObs configurations, so the coverage moves there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

datadog-datadog-us1-prod

More details

The finish-time mapping reads only scalar LLM Observability tags. It keeps message bodies off APM spans and limits usage metrics to LLM and embedding spans.

Was this helpful? React 👍 or 👎

_{Open Bits AI session}

_{🤖 Datadog Autotest · Commit 1527dca · What is Autotest? · @DataDog review to ask questions · Any feedback? Reach out in #autotest}

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

ncybul · 2026-09-11T18:16:56Z

@codex

chatgpt-codex-connector · 2026-09-11T18:20:31Z

Codex Review: Didn't find any major issues. Keep it up!

Reviewed commit: faac2ec6f8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Both tests in the class produce an openai.request span, so taking the first one written let a late trace from the sibling test be asserted on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

The attributes are derived from the LLMObs tag set, which the openai decorators do not build with LLM Observability off. Drop the fallback that resolved operation, model and ml_app from APM tags instead, and collapse GenAiApmTags back to a single apply(span). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Sets _dd.llmobs.artificial_gen_ai_tags alongside the gen_ai.* attributes so the backend processor does not treat them as a user annotation and create a duplicate LLM Observability span. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

This reverts commit eeeafa8.

The openai decorators read the token counts off every response, but only tagged them behind the LLMObs guard, so the disabled path carried the gen_ai scalars without any usage. Extract the counts above the guard and route them through TokenUsage, which writes the LLMObs metric when enabled and the gen_ai.usage.* APM metric when not. _ml_obs_metric.* stays behind the guard. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

mz1119 · 2026-09-21T19:56:09Z

wait until backend change is merged tuesday before merging.

ncybul

Just a couple small questions but other than that LGTM!

vandonr

looks good overall, 2 minor comments

Matches the other three decorators, which all hand their usage object to a private static withUsage(AgentSpan, ...). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

dd-octo-sts · 2026-09-23T15:34:42Z

/merge

gh-worker-devflow-routing-ef8351 · 2026-09-23T15:34:46Z

View all feedbacks in Devflow UI.

2026-09-23 15:34:46 UTC ℹ️ Start processing command /merge

2026-09-23 15:34:49 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).

2026-09-23 16:39:58 UTC ℹ️ MergeQueue: This merge request was merged

Emit gen_ai.* attributes on APM spans

f1d46cf

Write the scalar gen_ai.* subset of an LLM Observability span onto the APM span, so model, provider, application, conversation and token usage become indexed, searchable APM tags. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

mz1119 added type: feature Enhancements and improvements comp: mlobs ML Observability (LLMObs) tag: ai generated Largely based on code generated by an AI or LLM labels Sep 10, 2026

mz1119 and others added 2 commits September 11, 2026 10:26

Rewrite the LLMObs-disabled openai test in Java

1527dca

The repository does not accept new .groovy files. The module already has a JUnit forked-test base for LLMObs configurations, so the coverage moves there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

mz1119 marked this pull request as ready for review September 11, 2026 17:23

mz1119 requested review from a team as code owners September 11, 2026 17:23

mz1119 requested review from vandonr and removed request for a team September 11, 2026 17:23

datadog-datadog-us1-prod Bot reviewed Sep 11, 2026

View reviewed changes

Trim comments and docstrings

faac2ec

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

mz1119 mentioned this pull request Sep 11, 2026

feat(llmobs): emit gen_ai.* APM tags when LLMObs is disabled DataDog/dd-trace-js#10300

Closed

ncybul approved these changes Sep 11, 2026

View reviewed changes

Comment thread internal-api/src/main/java/datadog/trace/api/llmobs/GenAiApmTags.java Outdated

Comment thread internal-api/src/main/java/datadog/trace/api/llmobs/GenAiApmTags.java Outdated

Comment thread ...instrumentation/openai-java/openai-java-3.0/src/test/groovy/ChatCompletionServiceTest.groovy

mz1119 and others added 3 commits September 11, 2026 16:32

Address review: single apply method and shared tag reader

324b5bb

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Select the openai.request span by endpoint in the LLMObs-disabled test

a33c1c9

Both tests in the class produce an openai.request span, so taking the first one written let a late trace from the sibling test be asserted on. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Merge branch 'master' into max.zhang/llmobs-gen-ai-apm-tags

c4a4b4c

mz1119 mentioned this pull request Sep 14, 2026

Emit gen_ai.usage.* token metrics with LLM Observability disabled #12489

Closed

mz1119 and others added 6 commits September 16, 2026 16:33

Use a literal lowercase value for the artificial gen_ai tags marker

3a4ee32

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Revert "Emit no gen_ai.* attributes when LLM Observability is disabled"

b02a0fe

This reverts commit eeeafa8.

Merge branch 'master' into max.zhang/llmobs-gen-ai-apm-tags

621441e

ncybul approved these changes Sep 22, 2026

View reviewed changes

Comment thread internal-api/src/main/java/datadog/trace/api/llmobs/GenAiApmTags.java

Comment thread ...java/openai-java-3.0/src/main/java/datadog/trace/instrumentation/openai_java/TokenUsage.java

vandonr approved these changes Sep 23, 2026

View reviewed changes

Comment thread dd-java-agent/agent-llmobs/src/main/java/datadog/trace/llmobs/domain/DDLLMObsSpan.java Outdated

Comment thread ...nai-java-3.0/src/main/java/datadog/trace/instrumentation/openai_java/EmbeddingDecorator.java Outdated

mz1119 and others added 2 commits September 23, 2026 10:44

Extract the embedding token usage into a withUsage method

49689b0

Matches the other three decorators, which all hand their usage object to a private static withUsage(AgentSpan, ...). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

Drop the self-explanatory comment on the gen_ai tagging in finish()

3556486

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

vandonr approved these changes Sep 23, 2026

View reviewed changes

mz1119 enabled auto-merge September 23, 2026 15:09

mz1119 added this pull request to the merge queue Sep 23, 2026

github-merge-queue Bot removed this pull request from the merge queue due to failed status checks Sep 23, 2026

gh-worker-dd-mergequeue-cf854d Bot merged commit afd9b0c into master Sep 23, 2026
606 checks passed

gh-worker-dd-mergequeue-cf854d Bot deleted the max.zhang/llmobs-gen-ai-apm-tags branch September 23, 2026 16:39

github-actions Bot added this to the 1.67.0 milestone Sep 23, 2026

Conversation

mz1119 commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What Does This Do

With LLM Observability disabled

Motivation

Testing

Uh oh!

datadog-datadog-us1-prod Bot commented Sep 10, 2026 • edited by datadog-datadog-prod-us1-2 Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dd-octo-sts Bot commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🟢 Java Benchmark SLOs — All performance SLOs passed

Uh oh!

mz1119 commented Sep 10, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codex Review Summary

Uh oh!

chatgpt-codex-connector Bot commented Sep 10, 2026

🛡️ Codex Security Review · Automatically triggered

Uh oh!

chatgpt-codex-connector Bot commented Sep 10, 2026

Uh oh!

datadog-datadog-us1-prod Bot left a comment

Choose a reason for hiding this comment

Uh oh!

ncybul commented Sep 11, 2026

Uh oh!

chatgpt-codex-connector Bot commented Sep 11, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mz1119 commented Sep 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ncybul left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

vandonr left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

dd-octo-sts Bot commented Sep 23, 2026

Uh oh!

gh-worker-devflow-routing-ef8351 Bot commented Sep 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

mz1119 commented Sep 10, 2026 •

edited

Loading

datadog-datadog-us1-prod Bot commented Sep 10, 2026 •

edited by datadog-datadog-prod-us1-2 Bot

Loading

dd-octo-sts Bot commented Sep 10, 2026 •

edited

Loading

chatgpt-codex-connector Bot commented Sep 10, 2026 •

edited

Loading

mz1119 commented Sep 21, 2026 •

edited

Loading

gh-worker-devflow-routing-ef8351 Bot commented Sep 23, 2026 •

edited

Loading