Repository navigation
Fix stale IoTConsensus peers after restart during removal - #18892
Merged
CRZbulabula merged 2 commits intoOct 19, 2026
Merged
Conversation
d-wang-commit
marked this pull request as draft
October 15, 2026 09:46
d-wang-commit
marked this pull request as ready for review
October 16, 2026 11:11
Commit removal metadata before reconciling IoTConsensus V1 peer lists. Retry metadata updates and V1 resets up to three times before failing. Cover restart recovery and filter V1-only crash points in Ratis/V2 ITs.
d-wang-commit
force-pushed
the
fix-stale-peers-after-restart
branch
from
October 16, 2026 11:22
14b786c to
e68b2f9
Compare
Contributor
There was a problem hiding this comment.
🟡 Changes recommended
Retry exhaustion can abort enclosing DataNode-removal cleanup and leave removal targets registered in Removing.
1 open issue
What changed in this PR
Repairs stale IoTConsensus V1 membership after a replica restarts during peer removal.
Changes:
- Commits metadata before reconciling retained replicas, with bounded finalization retries.
- Adds restart, recovery, and replication regression tests.
- Updates localized diagnostics and protocol-specific crash-test expectations.
| File | Description |
|---|---|
| iotdb-core/consensus/src/test/java/org/apache/iotdb/consensus/iot/StalePeerListRestartReproductionTest.java | Reproduces stale membership and verifies explicit repair. |
| iotdb-core/confignode/src/test/java/org/apache/iotdb/confignode/procedure/impl/region/RestartDuringRemovePeerTest.java | Tests ordering, retries, recovery, and protocol gating. |
| iotdb-core/confignode/src/test/java/org/apache/iotdb/confignode/procedure/impl/region/RemovePeerRestartConsensusTest.java | Verifies reconciliation with real consensus instances. |
| iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/state/RemoveRegionPeerState.java | Appends the V1 reconciliation state. |
| iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/impl/region/RemoveRegionPeerProcedure.java | Adds reconciliation and bounded finalization retries. |
| iotdb-core/confignode/src/main/java/org/apache/iotdb/confignode/procedure/env/RegionMaintainHandler.java | Checks metadata results and retained-replica acknowledgments. |
| iotdb-core/confignode/src/main/i18n/zh/org/apache/iotdb/confignode/i18n/ProcedureMessages.java | Adds Chinese diagnostics. |
| iotdb-core/confignode/src/main/i18n/en/org/apache/iotdb/confignode/i18n/ProcedureMessages.java | Adds English diagnostics. |
| integration-test/src/test/java/org/apache/iotdb/confignode/it/regionmigration/pass/daily/ratis/IoTDBRegionMigrateConfigNodeCrashForRatisIT.java | Excludes the V1-only crash point. |
| integration-test/src/test/java/org/apache/iotdb/confignode/it/regionmigration/pass/daily/iotv2/stream/IoTDBRegionMigrateConfigNodeCrashIoTV2StreamIT.java | Excludes the V1-only crash point. |
| integration-test/src/test/java/org/apache/iotdb/confignode/it/regionmigration/pass/daily/iotv2/batch/IoTDBRegionMigrateConfigNodeCrashIoTV2BatchIT.java | Excludes the V1-only crash point. |
Review effort: Balanced
Finalize drained nodes or restore previous statuses during rollback. Retry failed cleanup without losing the original reconciliation failure. Cover retry exhaustion and delayed cleanup with ProcedureExecutor tests.
4 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Restart during peer removal
A surviving IoTConsensus V1 replica can restart after the removed peer's sync channel has been deleted but before ConfigNode commits the updated replica set. Its startup response still includes the removed peer, so it recreates that peer's dispatcher. Previously, completing the metadata update did not correct this recovered membership. Failed replication to the removed peer could then pin the minimum sync index and keep syncLag above zero.
Reconcile committed membership
Commit the replica-set removal before entering the new RESET_IOT_CONSENSUS_V1_PEER_LIST state. For IoTConsensus V1 DataRegions, send the final peer list to every retained replica through the existing resetPeerList RPC, including temporarily unavailable replicas. The existing DataNode handler waits for consensus startup before applying the list.
Metadata failures and unsuccessful V1 resets perform an initial attempt and up to three retries per state, sleeping five seconds between retries. Exhaustion marks the procedure as failed. The business state is persisted by the existing procedure framework and can resume after ConfigNode recovery; the in-memory cycle counter restarts after recovery. Existing state IDs are unchanged, and the new state is appended. No DataNode production code or RPC definitions are changed.
Ratis and IoTConsensus V2 crash ITs now exclude the V1-only state from their expected kill points. V1 retains coverage of the new state, and the assertion requiring all expected kill points to trigger is preserved.
Tests
Side effects and risks
This PR has:
Key changed/added classes (or packages if there are too many classes) in this PR