Skip to content

Prevent Pekko callbacks from retaining request contexts - #12675

Merged
gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits into
masterfrom
andrea.marziali/pekko-fix
Oct 1, 2026
Merged

gh-worker-dd-mergequeue-cf854d[bot] merged 3 commits into
masterfrom
andrea.marziali/pekko-fix

Conversation

@amarziali

Copy link
Copy Markdown
Contributor

What Does This Do

Fix a continuation leak in Pekko HTTP async handlers.

When an application Future completed with the request context active, the future returned to Pekko could run framework callbacks under that context. A Pekko actor envelope could then capture the request continuation and retain it indefinitely if the envelope was discarded.

The fix:

  • returns a dedicated Promise to Pekko and completes it under the root context
  • copies the Scala Try in completion-priority mode so attached context cannot reach Pekko
  • ensures spans finish even when response decoration throws
  • centralizes the Scala Promise completion-priority configuration
  • adds deterministic regression tests for both propagation modes
  • removes the corresponding flaky-test annotation
flowchart LR
  A["Application Future completes<br/>(request context active)"]

  A -->|"old transform"| B["Pekko callback captures<br/>request continuation"]
  B --> C["Discarded actor envelope<br/>retains continuation"]

  A --> D["Finish request span"]
  D --> E["Attach root context"]
  E --> F["Complete Promise exposed to Pekko"]
  F --> G["Pekko callback runs<br/>without request context"]

  classDef broken fill:#ffd6d6,stroke:#c62828
  classDef healthy fill:#d8f3dc,stroke:#2e7d32
  class B,C broken
  class D,E,F,G healthy
Loading

The implementation is based on Alexey K.'s existing unmerged alexeyk/pekko-fix work and has been adapted to the current codebase.

Motivation

Closes #9396

Additional Notes

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@amarziali
amarziali requested review from a team as code owners September 29, 2026 07:45
@amarziali amarziali added the type: bug fix Bug fix label Sep 29, 2026
@amarziali
amarziali requested review from sarahchen6 and removed request for a team September 29, 2026 07:45
@amarziali amarziali added the inst: others All other instrumentations label Sep 29, 2026
@amarziali
amarziali requested review from ValentinZakharov and removed request for a team September 29, 2026 07:45
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-09-29T07:49:34.584345Z 936eb21 PR opened
🔒 Security Review ✅ Completed 2026-09-29T07:49:24.176490Z 936eb21 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@datadog-prod-us1-6

This comment has been minimized.

@datadog-prod-us1-6 datadog-prod-us1-6 Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Bits Code Review: PASS

More details

The Pekko-facing Promise is completed under the root context, while completion-priority mode copies the Scala Try before framework callbacks can inherit request context.

Was this helpful? React 👍 or 👎

Open Bits AI session

🤖 Bits Code Review · Commit 936eb21 · @DataDog review to ask questions

@dd-octo-sts

dd-octo-sts Bot commented Sep 29, 2026 •

Copy link
Copy Markdown
Contributor

🟢 Java Benchmark SLOs — All performance SLOs passed

Suite Status
Startup 🟢 pass

SLO thresholds are defined here based on automatically generated metrics. A warning is raised when results are within 5% of the threshold.

PR vs. master results
Scenario Candidate master Δ (95% CI of mean)
startup:insecure-bank:iast:Agent 14.03 s 13.96 s [-0.1%; +1.2%] (no difference)
startup:insecure-bank:tracing:Agent 12.94 s 13.01 s [-1.5%; +0.4%] (no difference)
startup:petclinic:appsec:Agent 17.13 s 16.78 s [+1.2%; +3.0%] (significantly worse)
startup:petclinic:iast:Agent 17.02 s 17.06 s [-1.0%; +0.6%] (no difference)
startup:petclinic:profiling:Agent 16.67 s 16.84 s [-2.4%; +0.4%] (no difference)
startup:petclinic:sca:Agent 17.01 s 17.05 s [-1.2%; +0.8%] (no difference)
startup:petclinic:tracing:Agent 16.15 s 16.17 s [-1.1%; +0.9%] (no difference)

Commit: 1dd66fee · CI Pipeline · Benchmarking Platform UI


Load and DaCapo benchmarks can be triggered manually in the GitLab pipeline. Results will appear in the Benchmarking Platform UI after completion.

@sarahchen6 sarahchen6 left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks reasonable, thanks! Btw - the warning benchmark results are from the known AgentMeter benchmark (job). I re-ran to confirm, and it's good now (job).

@dougqh

dougqh commented Sep 29, 2026

Copy link
Copy Markdown
Contributor

(Comment drafted by Claude on behalf of @dougqh.)

Question on scope: akka-http-10.0 has its own DatadogAsyncHandlerWrapper, and AkkaHttp2ServerInstrumentation still lists DatadogAsyncHandlerWrapper$1 and $2 as helpers, which suggests it still uses the same two-function transform pattern that this PR replaces for Pekko. If so, the Akka HTTP/2 async handler path would have the same continuation-retention leak.

Is leaving Akka out intentional, for example because it does not reproduce there or because you want to keep this PR small? If it is just a follow-up, it might be worth filing an issue so it does not get lost. I have not tried to reproduce the Akka leak, so this is a question rather than a confirmed bug.

@amarziali

Copy link
Copy Markdown
Contributor Author

Is leaving Akka out intentional, for example because it does not reproduce there or because you want to keep this PR small? If it is just a follow-up, it might be worth filing an issue so it does not get lost. I have not tried to reproduce the Akka leak, so this is a question rather than a confirmed bug.

Yes I had a look to akka without finding the same evidence. So I left for follow up. Here I just would like to correct what I observed that is failing in CI (this is actually a flaky test behaviour that we'd like to fix and the diagnostic gave us few hint to understand and propose a fix)

@amarziali
amarziali force-pushed the andrea.marziali/pekko-fix branch from 936eb21 to fdf889e Compare September 30, 2026 08:28
@amarziali

Copy link
Copy Markdown
Contributor Author

/merge

@gh-worker-devflow-routing-ef8351

gh-worker-devflow-routing-ef8351 Bot commented Oct 1, 2026 •

Copy link
Copy Markdown

View all feedbacks in Devflow UI.

2026-10-01 12:10:55 UTC ℹ️ Start processing command /merge


2026-10-01 12:10:59 UTC ℹ️ MergeQueue: pull request added to the queue

The expected merge time in master is approximately 1h (p90).


2026-10-01 13:21:37 UTC ℹ️ MergeQueue: This merge request was merged

@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot merged commit 60b0bbd into master Oct 1, 2026
606 checks passed
@gh-worker-dd-mergequeue-cf854d
gh-worker-dd-mergequeue-cf854d Bot deleted the andrea.marziali/pekko-fix branch October 1, 2026 13:21
@github-actions github-actions Bot added this to the 1.67.0 milestone Oct 1, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

inst: others All other instrumentations type: bug fix Bug fix

Projects

None yet

Development

Successfully merging this pull request may close these issues.

PekkoHttpServerInstrumentationAsyncHttp2Test."test exception" is flaky

5 participants