Skip to content

fix(otel): export sampled fallback roots early - #754

Open
zhongkechen wants to merge 15 commits into
mainfrom
fix/otel-fallback-root
Open

zhongkechen wants to merge 15 commits into
mainfrom
fix/otel-fallback-root

Conversation

@zhongkechen

@zhongkechen zhongkechen commented Oct 2, 2026 •

Copy link
Copy Markdown
Contributor

Fixes #748.

Related timing requirement: aws/aws-durable-execution-sdk-js#938.

Both OTel views now export DurableExecutionRoot when a sampled invocation resolves an SDK-owned fallback ancestor. The anchor is materialized at invocation start and flushed through normal invocation end, including PENDING and RETRY, so an execution that times out while suspended already has its ancestor. Complete remote parents are untouched, and unsampled invocations emit no synthetic root.

The span uses the existing ARN-scoped synthetic-parent ID, durable.execution.synthetic_root=true, and the stable execution-start timestamp for both start and end. Each sampled fallback invocation can re-export it to recover lost telemetry; first-invocation retries retain the same SDK-owned identity/timing/attributes. Workflow continues to carry duration and terminal outcome. The synthetic-root path uses a private one-span ID scope, consumed before processors run, so reentrant roots on the same cached tracer get independent IDs. The exported generator's existing use_ids signature and scope-wide trace override semantics remain unchanged. The normal configured tracer, provider resource, processors, and resolved sampler metadata are preserved. Provider resources/custom enrichment may vary across environments; this does not replace configured service identity or promise exactly-once export.

This branch includes main's approved invocation-worker lifecycle from #756. OTel 1.1 now requires core 2.1.0; publish the core first. CI validates built immutable core/OTel wheels and separately checks the released OTel 1.0 plugin with the new core. The old plugin retains its documented Invocation fallback limitations. There is no handler_context plugin API or opt-in marker.

Validation after merging main at ffc1b180: all 654 OTel tests pass in the workspace and against the actual built wheels, plus 12 current-pair and 12 released-plugin compatibility controls. All 34 core and 14 OTel Python source files in the wheels match the committed tree. The core package is byte-identical to main; OTel types (36 files), Hatch lint/format, commit lint and diff checks pass. Functional CI has now passed on a4fe1a9939b82029a027e8aee0b07d663d5c1694, including Cloud tests, Python package and OpenTelemetry Conformance Tests run 37974255509. The PR still awaits a valid approval.

Regression evidence covers both views across absent/invalid/partial remote context, sampling decisions, shared trace IDs, retry/resume, resource changes and buffered flushes. Real decorated handlers cover suspension/resume success and failure, completed-step suppression and a timeout without a terminal hook. The sampler-collision tests check SDK identity before processors and retain the original sampling result. The earlier complete 449-test minimum-OpenTelemetry 1.20 run and published TraceFlags.sampled source proof remain recorded in the review discussion; no new minimum-version matrix run is claimed here.

@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 2, 2026 21:02 — with GitHub Actions Active
@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 2, 2026 21:03 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 2, 2026 21:24 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 2, 2026 21:51 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 2, 2026 23:24 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 5, 2026 23:38 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@zhongkechen
zhongkechen had a problem deploying to ai-pr-review-runtime October 7, 2026 02:06 — with GitHub Actions Failure
@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 7, 2026 02:40 — with GitHub Actions Active
Comment thread .github/workflows/cloud-tests.yml
@github-actions

This comment has been minimized.

@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 7, 2026 03:09 — with GitHub Actions Active
Comment thread .github/workflows/cloud-tests.yml
@github-actions

This comment has been minimized.

@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 7, 2026 19:13 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 7, 2026 23:06 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@zhongkechen
zhongkechen requested a review from a team October 8, 2026 16:59
@zhongkechen
zhongkechen deployed to ai-pr-review-runtime October 9, 2026 18:36 — with GitHub Actions Active
@github-actions

github-actions Bot commented Oct 9, 2026

Copy link
Copy Markdown
Contributor

Codex AI review

No actionable findings. Residual risk: tests were not executed under the read-only review constraints, including the OpenTelemetry 1.20 compatibility matrix.

Reviewed commit a4fe1a9939b82029a027e8aee0b07d663d5c1694. Workflow run

This branch was successfully deployed

1 active deployment
ai-pr-review-runtime — a4fe1a99 Deployed Oct 9, 2026 by zhongkechen via ai-pr-review / Codex review / Generate Codex review #1520
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug]: export stable fallback synthetic roots, including before suspension

1 participant