Repository navigation
Conversation
The B3 debug flag arrives either as the sampling state "d" in the single header or as x-b3-flags: 1 in the multi header form. Both were collapsed into the SAMPLED trace flag and the debug state itself was discarded, so nothing downstream could tell a debug trace from an ordinary sampled one. Keep the collapse into SAMPLED and additionally record the debug state under a context key, which the inject side can then read. The two forms are kept apart: a sampling state of "d" means debug only in the single header, since x-b3-sampled is not a carrier for it, even though this propagator has always accepted "d" there as a request to sample.
Inject only ever emitted a sampling decision of 1 or 0, so a debug trace arriving at a service left it as an ordinary sampled one and the debug state was lost for the rest of the trace. Read the debug state recorded by extract and re-emit it: the sampling state "d" in the single header form, and x-b3-flags: 1 in the multi header form. Debug implies an accept decision, so x-b3-sampled is set to 1 alongside it, which is what the Java implementation emits as well. x-b3-flags joins the fields of B3MultiFormat, since inject can now set it and a reused carrier has to have it cleared.
extract takes an optional context to extend, and it only ever set the debug flag, never cleared it. Extracting a carrier that carries no debug flag into a context that already held one therefore returned a context still marked for debug, and injecting it put the debug flag on the wire for a trace that was never marked for it. Record the debug state of the carrier being extracted, whether or not it is set, so the state the inject side reads is the state the carrier actually carried.
Two paths the suite did not reach. Instrumentation activates an extracted context by attaching it and then calls inject with no context argument, and every test so far passed the context to inject explicitly. test_flags_and_sampling named a combination of x-b3-flags and x-b3-sampled that it never built, leaving it byte for byte the same carrier as test_flags, so the case where the two headers disagree had no test at all.
test_flags and test_flags_and_sampling used to prove that x-b3-flags: 1 sets TraceFlags.SAMPLED on the extracted span context. Rewriting them to assert the debug flag replaced that with an assertion the inject side satisfies from the debug state alone, so deleting the flags clause from extract left the whole suite green. Assert the trace flag on the extracted parent as well as the debug flag on the injected carrier.
The B3 specification allows a sampling state to be sent on its own, so "b3: d" and "x-b3-flags: 1" are both complete carriers that mark a trace for debug without naming one. extract returned early on the missing ids and threw the flag away, and the service then started a root trace and injected an ordinary sampled decision. Store the flag on that path, and only when no ids were supplied at all, so a carrier whose ids are present but unparsable still leaves the context alone.
Every test that reached inject used a carrier with ids, so the store on the early-return path was only ever asserted through the context and never through what goes on the wire.
Nothing failed when either comparison was relaxed to a presence check, so an x-b3-flags of "0" read as debug and a single header accept state of "1" read as debug, and the suite stayed green both times.
Pull request dashboard statusWaiting on reviewers · refreshed 2026-11-09 14:14 UTC Review the latest changes. Status above doesn't look right?
|
chiragjangiir
left a comment
There was a problem hiding this comment.
Code Review: Preserve and re-propagate the B3 debug flag
1. Context & Motivation
The Zipkin B3 propagation specification supports a debug flag (X-B3-Flags: 1 or d in single-header format) which indicates that sampling should be forced regardless of typical rate limits. When spans crossed Python services using B3MultiFormat or B3SingleFormat, the debug flag was frequently lost upon re-injection into downstream headers.
2. Technical Evaluation
- Context Extraction & Storage: The debug state is properly extracted into trace state / baggage or trace flags bitmask.
- Header Formatting: Both single-header (
b3: {trace_id}-{span_id}-d) and multi-header (X-B3-Flags: 1) serialization paths have been updated to preserve the debug state across outbound HTTP calls. - Interoperability: Maintains strict adherence to B3 specs where the sample decision (
1vs0) is suppressed or overridden when debug (dorFlags: 1) is active.
3. Test Suite
- Comprehensive matrix tests covering:
- Multi-header extraction with
X-B3-Flags: 1 - Single-header extraction with debug flag
- Downstream propagation to verify neither header format loses the debug bit.
- Multi-header extraction with
Clean and compliant fix for distributed tracing fidelity.
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
B3 extraction can retain stale debug state after sampling-only decisions and incorrectly enable debug for malformed carriers with empty identifiers.
1 open issue
What changed in this PR
This PR preserves received B3 debug state in the OpenTelemetry context so it can be propagated to downstream services.
Changes:
- Store debug state during extraction and re-emit it in single- and multi-header formats.
- Add regression tests for debug propagation, sampling-only carriers, context handling, and header fields.
- Document the fix in the changelog.
| File | Description |
|---|---|
propagator/opentelemetry-propagator-b3/tests/test_b3_format.py |
Adds debug extraction and propagation tests. |
propagator/opentelemetry-propagator-b3/src/opentelemetry/propagators/b3/__init__.py |
Preserves debug state and updates injected headers and declared fields. |
.changelog/5870.fixed |
Records the B3 debug propagation fix. |
Review effort: Balanced
Contributor
|
Thanks for the fix! I had Codex perform a quick review and it found a few issues that are valid, but unrelated to your changes. You're welcome to fix them in this PR, or we can fix them in a follow-up:
|
Contributor
|
I reviewed the changes to context propagation in The handling of the debug flag aligns with the Zipkin B3 specification:
Tested the extraction and injection edge cases locally; trace context propagation remains consistent across hops. |
This comment has been minimized.
This comment has been minimized.
The branch that keeps a debug flag sent with no identifiers asked
whether trace_id and span_id still held their sentinels, and the
multi header extraction gave them those sentinels for a header that
arrived empty as well as for one that never arrived. A carrier of
{"x-b3-traceid": "", "x-b3-spanid": "", "x-b3-flags": "1"} therefore
reached that branch and modified the supplied context, which the
extract contract at
https://opentelemetry.io/docs/specs/otel/context/api-propagators/#extract
does not allow for a carrier it cannot parse.
The branch is now taken only when the carrier supplied a sampling
state and no identifiers at all, which the extraction records as it
reads the headers, before any sentinel is substituted.
A carrier whose identifiers are present already overwrites whatever debug flag an earlier carrier left in the context, because extract stores the state it read whether or not it is a debug state. A carrier that sends a sampling state on its own only ever stored a debug state, so extracting b3: 0, or an x-b3-sampled of its own, into a context that already carried a debug flag left that flag in place and inject went on emitting it. A sampling state sent alone is a complete carrier, so it now overwrites the stored flag the same way, with the state it read.
https://github.com/openzipkin/b3-propagation#debug-flag says "Debug implies an accept decision, so don't also send the X-B3-Sampled header", and the multi header inject sent both. Sending both was a deliberate deviation, taken to agree with B3PropagatorInjectorMultipleHeaders in opentelemetry-java and to keep the accept decision readable by a receiver that handles x-b3-sampled and ignores x-b3-flags, and a reviewer asked for the b3 encoding instead. x-b3-flags of "1" is now the whole sampling state of an injected debug trace, and x-b3-sampled is injected for every other trace. Both headers stay in fields, because either one may be written and a reused carrier has to have both cleared.
Contributor
Author
Right, addressed all those comments in this PR ✌️ |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The B3 propagator accepted a received debug flag and then threw it away.
On extract, a sampling state of
din the single header form, andX-B3-Flags: 1in the multi header form, were both collapsed into theSAMPLEDtrace flag, and the debug state itself was not stored anywhere. On inject, the sampling decision was always re-emitted as1or0. A debug trace therefore stopped being a debug trace at the first service that used this propagator, and the operator who set the flag at the edge lost it for the rest of the request chain.The B3 extract requirements in
specification/context/api-propagators.mdsay "MUST preserve a debug trace flag, if received, and propagate it with subsequent requests. Additionally, an OpenTelemetry implementation MUST set the sampled trace flag when the debug flag is set." The second sentence was already satisfied. The first was not.extractnow records the debug state of the carrier under a context key, which is whereio.opentelemetry.extension.trace.propagation.B3Propagatorkeeps it, andinjectreads it back:B3SingleFormatemits the sampling stated.B3MultiFormatemitsX-B3-Flags: 1, together withX-B3-Sampled: 1.X-B3-FlagsjoinsB3MultiFormat.fields, because inject can now set it and a reused carrier has to have it cleared.Three decisions in this change are worth a reviewer's attention.
X-B3-Sampledis not sent for a debug trace. https://github.com/openzipkin/b3-propagation carries the note "Debug implies an accept decision, so don't also send theX-B3-Sampledheader", andB3MultiFormat.injectfollows it:X-B3-Flags: 1is the whole sampling state of an injected debug trace, andX-B3-Sampledis injected for every other trace. An earlier revision of this change sent both, to agree withB3PropagatorInjectorMultipleHeadersin opentelemetry-java and to keep the accept decision readable by a receiver that handlesX-B3-Sampledand ignoresX-B3-Flags. A reviewer asked for the b3 encoding instead. Both headers stay inB3MultiFormat.fields, because either one may be written and a reused carrier has to have both cleared.The two header forms are kept apart on extract. A sampling state of
dmeans debug only in the single header, becauseX-B3-Sampledis not a carrier for the debug flag. This propagator has always accepteddinX-B3-Sampledas a request to sample, and that behaviour is unchanged.A debug flag that arrives with no identifiers is kept. The B3 specification allows a sampling state to be sent on its own, so
b3: dandX-B3-Flags: 1are both complete carriers that mark a trace for debug without naming one.extractreturns early when the identifiers are missing, which threw that flag away too, and the service then started a root trace and injected an ordinary sampled decision. The state the carrier sent is now stored on that path, and only when the carrier supplied a sampling state and no identifiers at all. An identifier that arrives empty counts as supplied, so{"x-b3-traceid": "", "x-b3-spanid": "", "x-b3-flags": "1"}leaves the context untouched, as does any other carrier whose identifiers are present but unparsable. A sampling state sent alone overwrites a debug flag an earlier carrier left in the context, the same way a carrier with identifiers already did, sob3: 0clears one rather than leaving it to be injected again.B3PropagatorExtractorSingleHeaderandB3PropagatorExtractorMultipleHeadersin opentelemetry-java both return before their debug check here, so this part goes further than that implementation. Under the SDK default sampler the root span that follows is recorded, so the injected decision and the local one agree.One consequence of keeping the state on the
Contextrather than on theSpanContextis worth knowing.Tracer.use_spanattachesset_value(_SPAN_KEY, span)against the current context, so a caller that passes an extracted context only asstart_as_current_span(context=...), without attaching it, loses every non-span entry of that context, and the debug flag is then not re-emitted. This is not specific to this change: W3C baggage is lost by the same call in the same way. Instrumentation that attaches the extracted context, which is what_start_internal_or_server_spanin opentelemetry-python-contrib does, re-emits the flag correctly, and there is a test for that path.Type of change
tox -e py313-test-opentelemetry-propagator-b3passes 94 tests,tox -e lint-opentelemetry-propagator-b3scores 10.00/10, and everytox -e precommithook passes.tox -e typecheckreports two errors, both in the Python 3.14 standard librarystring/templatelib.py, and both are present on the base commit without these changes;[tool.pyright].includeinpyproject.tomldoes not list this package, so the run does not examine the changed file.New tests cover extraction of the debug flag from both header forms, extraction from a sampling state sent with no identifiers in both forms, the absence of a debug flag on an ordinary sampled trace,
X-B3-Flags: 0and a single header sampling state of1not being read as debug,X-B3-Sampled: dnot being read as debug, a debug flag left in a context by an earlier carrier being cleared by a later one, re-propagation on inject from both forms, re-propagation onto the first local span after a debug flag received with no identifiers, re-propagation after the extracted context is attached, a received debug flag being injected in place of an unsampled span's decision,X-B3-Flags: 1winning overX-B3-Sampled: 0, a sampling state sent alone in either header form clearing a debug flag left by an earlier carrier, and a carrier whose identifiers are empty, or whose single header identifiers are unparsable, leaving the context untouched.Checked by mutation: deleting
or flags == "1"from the sampling decision, dropping the debug branch from either inject, removingX-B3-Flagsfromfields, makingextractstop clearing a stored flag, relaxing either debug comparison from an exact value to a presence check, reading an empty identifier as an absent one, dropping the requirement that a carrier supply a sampling state before its debug state is stored, widening the single header check for identifiers past one field, or restoringX-B3-SampledalongsideX-B3-Flagseach fails at least one test.