feat(tracing)!: JSON-encode observation and trace metadata values

[niklassemmler]

Problem

Per-key metadata attributes (langfuse.observation.metadata.<key>) are built in the dict branch of _flatten_and_serialize_metadata. Today str and int values are passed through as-is and everything else goes through _serialize (JSON). That causes two problems:

1. Ints beyond int64 make the OTLP encoder fail for that key ("Failed to encode key ...: Value out of range"), so the key is silently dropped. Ints beyond 2^53 lose precision on the (JS) server.
2. The server will start JSON-decoding per-key metadata values for newer SDK versions. With raw strings, the string "123" or "true" would decode to a number/bool, so strings and numbers would be conflated.

Fix

Every value in the dict branch is now sent as json.dumps(value, cls=EventSerializer), strings included. Non-dict metadata on the bare langfuse.observation.metadata attribute gets the same encoding.

• Values are written with separators=(",", ":"), ensure_ascii=False, so the output is byte-identical with JS JSON.stringify (compact separators, non-ASCII kept as is), same as feat(tracing)!: JSON-serialize propagated metadata values #1932.
• EventSerializer already turns ints outside the JS-safe range into JSON strings and NaN/Infinity into strings, so nothing extra is needed for those.
• EventSerializer.encode catches every exception and returns a placeholder JSON string, so a value that fails to serialize can't drop the other keys. No extra per-value handling added.
• None values are still skipped (logged at debug level, same message as langfuse-js fix(media): allow setting IO media via decorator update #1005), so update(metadata={"k": None}) doesn't overwrite an earlier value.
• The non-dict branch (bare langfuse.observation.metadata attribute) is now JSON-encoded too, strings included, matching langfuse-js fix(media): allow setting IO media via decorator update #1005. Before, it went through _serialize, which passes str through raw, so metadata="foo" sent foo from Python and "foo" from JS. None is still not written. Input/output and the other _serialize callers are unchanged.
• The type="trace" prefix of the same function gets the same encoding. On prepare-v5-release no production code calls it with "trace" (create_trace_attributes takes no metadata); only a unit test does.
• _flatten_and_serialize_metadata_values (propagated trace metadata via propagate_attributes) is not touched here. feat(tracing)!: JSON-serialize propagated metadata values #1932 (already in prepare-v5-release) covers it.
• OpenAI, LangChain and run_experiment only pass metadata dicts in. Nothing in the SDK reads these attributes back, so no integration code changed.

This is one of three small PRs replacing the abandoned single-JSON-blob approach in #1944 (LFE-17153).

Wire values

Python value	Before	After
"hello"	hello (str)	"hello"
"123"	123 (str)	"123"
"true"	true (str)	"true"
5	5 (OTel int)	5 (str)
True	true (OTel bool)	true (str)
1.5	1.5 (str)	1.5 (str)
2**70	key dropped by the OTLP encoder	"1180591620717411303424"
float("nan")	"NaN"	"NaN"
[1, "a", None]	[1, "a", null]	[1,"a",null]
{"a": {"b": [1]}}	{"a": {"b": [1]}}	{"a":{"b":[1]}}
datetime(..., tz=UTC)	"2024-01-02T03:04:05Z"	"2024-01-02T03:04:05Z"
None	skipped	skipped

Non-dict metadata on the bare langfuse.observation.metadata key, matching langfuse-js #1005:

Python value	Before	After
"foo"	foo	"foo"
5	5	5
[1, "a", None]	[1, "a", null]	[1,"a",null]
None	not written	not written

Wire format change

• All per-key metadata attributes, and the bare langfuse.observation.metadata attribute for non-dict metadata, are now JSON strings (same as langfuse-js fix(media): allow setting IO media via decorator update #1005). Custom span exporters and mask_otel_spans callbacks now see e.g. '"hello"' instead of 'hello', and "5" / "true" strings instead of native int/bool attributes.
• This relies on the Langfuse server JSON-decoding per-key metadata values for newer SDK versions. Without that, the server would store string values with their quotes ("hello" including the quotes) and numbers/bools as strings. This PR must ship together with or after that server change.

E2E / live-provider tests that depend on server-side decoding

I did not change these expectations. They read observation metadata back through the API and will pass only once the server decodes the JSON values:

• tests/e2e/test_core_sdk.py: test_concurrency (metadata["count"] == i, int), test_create_generation_complex (metadata["tags"] == ["yo"]; JSON before and after, unaffected), test_update_generation, test_update_span, test_end_generation_with_data, test_end_span_with_data, test_kwargs (string values such as "value", "whatsapp")
• tests/e2e/test_decorators.py: test_nested_observations, test_nested_observations_with_non_parentheses_decorator, test_concurrent_decorator_executions, test_decorators_langchain, test_decorated_class_and_instance_methods, test_async_nested_openai_chat_stream (observation metadata["key"] / someKey), plus the multiproject tests that read obs.metadata.get("level" / "async" / "type")
• tests/e2e/test_media.py::test_replace_media_reference_string_in_object and tests/e2e/test_decorators.py::test_media (dict value; JSON before and after, unaffected)
• tests/live_provider/test_langchain.py (metadata["is_langchain_root"] is True, bool) and tests/live_provider/test_openai.py (metadata["someKey"] == "someResponse")

tests/e2e/test_core_sdk.py: test_create_numeric_score, test_create_boolean_score, test_create_categorical_score and test_create_text_score set non-dict metadata (metadata="test") on a generation but never read it back, so they don't depend on the decoding.

Trace-level metadata assertions (trace.metadata[...]) in e2e go through propagate_attributes and are out of scope here (see #1932).

CI follow-up (2f2596f)

The server decodes per-key metadata only for Python SDK major >= 5 (langfuse/langfuse#18436, not released yet), so on this 4.x branch nothing is decoded in CI. Three e2e tests failed because of that:

• test_concurrency wrote str(i) and asserted an int. It now writes the int, which reads back the same way with or without decoding.
• test_run_experiment_on_local_dataset / test_run_experiment_on_langfuse_dataset: the experiment items API doesn't parse metadata values on read, so values came back as '"Euro capitals"'. They now compare against raw_metadata_value(...) (tests/support/utils.py). That helper expects the JSON string below SDK major 5 and the plain value from 5 on, so the strict assertions return automatically with the version bump.

Verification

• uv run --frozen ruff check .: passed
• uv run --frozen ruff format --check .: only tests/unit/test_media.py would be reformatted. This PR doesn't touch that file and it's the same on main.
• uv run --frozen mypy langfuse --no-error-summary: passed
• uv run --frozen pytest -n auto --dist worksteal tests/unit (after rebasing onto prepare-v5-release): 773 passed, 2 skipped, 18 errors. All 18 errors are in tests/unit/test_prompt.py ("Langfuse client is not initialized"), which also happens locally on main.
• New tests in tests/unit/test_otel.py::TestMetadataHandling were written first and failed before the fix: per-type encoding, big int surviving real OTLP encode_spans, None keeping the earlier value on update, and the trace prefix. test_non_dict_metadata_is_json_encoded covers string, int and list as non-dict metadata.
• Skipped: e2e and live-provider suites. They need a Langfuse server with the decoding change (see above).

🤖 Written by Claude (an AI agent) on behalf of Niklas.

🤖 Generated with Claude Code

Confidence Score: 2/5

Fix the lost run scores, repeated evaluations, and stale-output scoring before merging, and satisfy the required import layout.

What we checked:

• Can one bad value drop other keys?: EventSerializer.encode catches encoding errors and returns a JSON string. Each metadata key is encoded separately.

Summary

The diff JSON-encodes each observation metadata value and preserves existing values when updates contain None. It also changes experiment tracking, batch evaluation, exporter defaults, supported Python versions, and integration tests.

• Experiment run scores can disappear with custom samplers.
• Cursor-based resume can score an observation twice.
• Batch reads omit the update times needed to choose the newest duplicate row.
• New imports must move to module level to satisfy the repository rule.

niklassemmler acknowledged the metadata wire-format change and its server-decoding dependency; those were not reported again.

Diagram

%%{init: {'theme': 'neutral'}}%%
flowchart LR
    A[Observation metadata] --> B[Encode each value as JSON]
    B --> C[OpenTelemetry attributes]
    C --> D[Langfuse exporter]
    E[Experiment items] --> F[Item spans]
    F --> D
    E --> G[Run evaluators]
    G --> H[Sampling check]
    H --> I[Run scores]
    J[Existing observations] --> K[Cursor pages]
    K --> L[Choose duplicate rows]
    L --> M[Evaluators and scores]
    K --> N[Resume token]
    N --> K

Loading

_{Reviews (1) · Last reviewed commit: "fix(tracing): JSON-encode metadata attri..." · Reviewed by Greptile}

[github-actions]

@claude review

[claude]

Claude Code Review

No review was started: this request came from a bot account. Manual reviews can only be requested by someone with write access to this repository. Ask a maintainer to comment @claude review, or have your automation post the comment from a user account with write access.

_{Tip: disable this comment in your organization's Code Review settings.}

[claude]

Beyond the inline finding, I also checked the lack of a version/feature gate for the new JSON-quoted wire format against self-hosted servers that haven't added the corresponding decoding change — the PR description explicitly discloses this as an intentional, coordinated-rollout constraint ("must ship together with or after that server change"), not an overlooked gap, so I'm not raising it separately.

Extended reasoning...

The diff changes _flatten_and_serialize_metadata in langfuse/_client/attributes.py to json.dumps every per-key metadata dict value and to skip None values, with matching test updates across four unit test files; it touches span/trace attribute export (no auth/crypto) but does affect how exported string attributes are later scanned for base64 media by span_exporter.py. I verified the confirmed inline finding by tracing _is_base64_data_uri in span_exporter.py, which requires value.startswith("data:") — after json.dumps a base64 data-URI string gets wrapped in literal quotes, so the startswith check fails and the payload ships inline instead of being extracted as media, matching the reported bug. Given that confirmed regression, a human should review before merging.

[niklassemmler]

@claude review

🤖 Written by Claude (an AI agent) on behalf of Niklas

[claude]

Code review found no new issues

No new issues were found in this update; 1 finding from earlier reviews is still open above.

[cursor]

At Hassieb's request, I added these on top of Niklas's changes:

• Merged the latest prepare-v5-release (with fix(tracing): keep metadata within the span attribute limit #1957 and fix(tracing): mask dict metadata per key when the mask function fails #1956) and updated their tests to expect JSON-encoded values.
• Fixed media extraction for top-level data-URI strings in JSON-encoded metadata. This was Claude's thread, now resolved.
• propagate_attributes(metadata=...) now uses the same encoding as observation metadata, so "123" stays a string. Big ints and NaN become JSON strings, and the 200-character limit applies after encoding.
• Added a str fast path that skips EventSerializer. Its output is byte-identical, pinned by a test against JSON.stringify.

The PR description doesn't mention the propagated-metadata, media and fast-path changes yet.

[cursor]

@claude review

[claude]

Claude Code Review

No review was started: this request came from a bot account. Manual reviews can only be requested by someone with write access to this repository. Ask a maintainer to comment @claude review, or have your automation post the comment from a user account with write access.

_{Tip: disable this comment in your organization's Code Review settings.}