Repository navigation
[SDK Sentinel] Fix staged workflow worker shutdown race - #3185
Open
sdk-sentinel-bot wants to merge 1 commit into
Open
sdk-sentinel-bot wants to merge 1 commit into
sdk-sentinel-bot wants to merge 1 commit into
Conversation
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Caution
This PR contains untrusted AI-generated code. Do not approve or run CI until a maintainer has reviewed the diff. SDK Sentinel verified that GitHub Actions remained approval-gated with zero executable jobs when this PR was opened.
Summary
Fix a race that let workflow-worker termination checks run before staged shutdown completed. The [first captured cloud attempt](https://github.com/temporalio/sdk-java/actions/runs/41173921213/job/124457428717) failed the autoscaling
shutdownNowassertion, while the [same-commit rerun](https://github.com/temporalio/sdk-java/actions/runs/41173921213/job/124464560263) passed this test. No earlier equivalent run was captured. There are no public API, breaking, or server changes.Root cause
Workflow-worker shutdown advances asynchronously from poller shutdown to poll-task-executor shutdown.
awaitTerminationcould inspect the later executor before shutdown had been invoked on it, causing its wait to return immediately and expose a not-yet-terminated factory.Fix
Track the complete staged shutdown future and wait for it within the caller's existing timeout budget before checking component executors. Add deterministic coverage that blocks the first stage and verifies later checks cannot run early.
Validation
The focused shutdown class and new regression test pass. Formatting and all test classes compile when sandbox-blocked submodule update tasks are excluded. The canonical validator is blocked by read-only Git submodule metadata, so normal validator and Linux/JDK 23 cloud CI remain required.
Validation status:
validation-incompletehighhighmediumstandard— 2 changed files; 7.7 KiBnot-reproduced— The unchanged in-memory focused class passed once; the exact credentialed cloud/JDK 23 mode was unavailable.pass-after-only— Added awaitTerminationWaitsForStagedShutdown, which deterministically holds the first shutdown stage incomplete and passed after the fix; it was not run against an unpatched checkout.Gradle formatting and test-source compilation) — passedGradle formatting and test-source compilation) — did not pass (failure)Investigator-run checks
./gradlew --offline --no-daemon :temporal-sdk:test --tests "io.temporal.worker.shutdown.StickyWorkflowDrainShutdownTest" -x :temporal-serviceclient:initSubmodules -x :temporal-serviceclient:updateSubmodules— passed; 1 attempt(s). The unchanged focused class passed; natural reproduction did not occur../gradlew --offline --no-daemon :temporal-sdk:cleanTest :temporal-sdk:test --tests "io.temporal.worker.shutdown.StickyWorkflowDrainShutdownTest" -x :temporal-serviceclient:initSubmodules -x :temporal-serviceclient:updateSubmodules— passed; 1 attempt(s). All four parameterized shutdown tests passed after the candidate../gradlew --offline --no-daemon spotlessApply :temporal-sdk:test --tests "io.temporal.internal.worker.WorkflowWorkerTest.awaitTerminationWaitsForStagedShutdown" -x :temporal-serviceclient:initSubmodules -x :temporal-serviceclient:updateSubmodules— passed; 1 attempt(s). The deterministic staged-shutdown regression test passed after the candidate..ci-flake-runtime/input/validate.sh— failed; 2 attempt(s). Both attempts passed formatting through temporal-sdk, then failed when updateSubmodules could not write the sandboxed read-only .git/modules config. The trusted unchanged baseline was successful../gradlew --offline --no-daemon spotlessCheck testClasses -x test -x :temporal-serviceclient:initSubmodules -x :temporal-serviceclient:updateSubmodules— passed; 2 attempt(s). Formatting and test-class compilation passed when only the sandbox-blocked submodule mutation tasks were excluded; the final attempt included both changed files.Required target CI
Residual risks
Automation provenance
de77db89c65e164b14633744479cd25dc42f0d07If this finding should not be fixed, apply the
sdk-sentinel:false-positivelabel and close the PR. Sentinel will suppress the finding until its affected source changes.SDK Sentinel recurrence history
2026-10-31 05:57 UTC2026-11-03 15:55 UTC