Skip to content

[SDK Sentinel] Fix staged workflow worker shutdown race - #3185

Open
sdk-sentinel-bot wants to merge 1 commit into
temporalio:mainfrom
sdk-sentinel-forks:automation/sdk-sentinel/ci-flake/java/41365239444-1-3e3db4300bbf
Open

sdk-sentinel-bot wants to merge 1 commit into
temporalio:mainfrom
sdk-sentinel-forks:automation/sdk-sentinel/ci-flake/java/41365239444-1-3e3db4300bbf

Conversation

@sdk-sentinel-bot

@sdk-sentinel-bot sdk-sentinel-bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Caution

This PR contains untrusted AI-generated code. Do not approve or run CI until a maintainer has reviewed the diff. SDK Sentinel verified that GitHub Actions remained approval-gated with zero executable jobs when this PR was opened.

Summary

Fix a race that let workflow-worker termination checks run before staged shutdown completed. The [first captured cloud attempt](https://github.com/temporalio/sdk-java/actions/runs/41173921213/job/124457428717) failed the autoscaling shutdownNow assertion, while the [same-commit rerun](https://github.com/temporalio/sdk-java/actions/runs/41173921213/job/124464560263) passed this test. No earlier equivalent run was captured. There are no public API, breaking, or server changes.

Root cause

Workflow-worker shutdown advances asynchronously from poller shutdown to poll-task-executor shutdown. awaitTermination could inspect the later executor before shutdown had been invoked on it, causing its wait to return immediately and expose a not-yet-terminated factory.

Fix

Track the complete staged shutdown future and wait for it within the caller's existing timeout budget before checking component executors. Add deterministic coverage that blocks the first stage and verifies later checks cannot run early.

Validation

The focused shutdown class and new regression test pass. Formatting and all test classes compile when sandbox-blocked submodule update tasks are excluded. The canonical validator is blocked by read-only Git submodule metadata, so normal validator and Linux/JDK 23 cloud CI remain required.

Validation status: validation-incomplete

  • Flake confidence: high
  • Fix confidence: high
  • Value: medium
  • Patch scope: standard — 2 changed files; 7.7 KiB
  • Local reproduction: not-reproduced — The unchanged in-memory focused class passed once; the exact credentialed cloud/JDK 23 mode was unavailable.
  • Regression coverage: pass-after-only — Added awaitTerminationWaitsForStagedShutdown, which deterministically holds the first shutdown stage incomplete and passed after the fix; it was not run against an unpatched checkout.
  • Unchanged baseline (Gradle formatting and test-source compilation) — passed
  • Independent candidate (Gradle formatting and test-source compilation) — did not pass (failure)

Investigator-run checks

  • ./gradlew --offline --no-daemon :temporal-sdk:test --tests "io.temporal.worker.shutdown.StickyWorkflowDrainShutdownTest" -x :temporal-serviceclient:initSubmodules -x :temporal-serviceclient:updateSubmodules — passed; 1 attempt(s). The unchanged focused class passed; natural reproduction did not occur.
  • ./gradlew --offline --no-daemon :temporal-sdk:cleanTest :temporal-sdk:test --tests "io.temporal.worker.shutdown.StickyWorkflowDrainShutdownTest" -x :temporal-serviceclient:initSubmodules -x :temporal-serviceclient:updateSubmodules — passed; 1 attempt(s). All four parameterized shutdown tests passed after the candidate.
  • ./gradlew --offline --no-daemon spotlessApply :temporal-sdk:test --tests "io.temporal.internal.worker.WorkflowWorkerTest.awaitTerminationWaitsForStagedShutdown" -x :temporal-serviceclient:initSubmodules -x :temporal-serviceclient:updateSubmodules — passed; 1 attempt(s). The deterministic staged-shutdown regression test passed after the candidate.
  • .ci-flake-runtime/input/validate.sh — failed; 2 attempt(s). Both attempts passed formatting through temporal-sdk, then failed when updateSubmodules could not write the sandboxed read-only .git/modules config. The trusted unchanged baseline was successful.
  • ./gradlew --offline --no-daemon spotlessCheck testClasses -x test -x :temporal-serviceclient:initSubmodules -x :temporal-serviceclient:updateSubmodules — passed; 2 attempt(s). Formatting and test-class compilation passed when only the sandbox-blocked submodule mutation tasks were excluded; the final attempt included both changed files.

Required target CI

  • Canonical .ci-flake-runtime/input/validate.sh in a checkout with writable submodule Git metadata
  • Continuous Integration / Unit test with cloud on Linux JDK 23
  • Focused temporal-sdk shutdown tests under both simple-maximum and autoscaling poller parameters

Residual risks

  • The exact cloud failure did not reproduce locally.
  • Canonical candidate validation remains incomplete because of the investigator sandbox, despite a successful unchanged baseline and passing equivalent checks.
  • The change makes awaitTermination spend part of its existing timeout on the staged shutdown future; cloud CI should confirm behavior across server capability modes.

Automation provenance

If this finding should not be fixed, apply the sdk-sentinel:false-positive label and close the PR. Sentinel will suppress the finding until its affected source changes.

SDK Sentinel recurrence history

  • Unique failed CI run attempts: 2
  • First occurrence: 2026-10-31 05:57 UTC
  • Latest occurrence: 2026-11-03 15:55 UTC
  • Recent occurrences:

@sdk-sentinel-bot
sdk-sentinel-bot requested a review from a team as a code owner November 2, 2026 00:35

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant