[CRZbulabula]

Problem

IoTDBRegionGroupLeaderDistributionIT fails intermittently in the cluster IT suites. In one
1C3D run all three of its test methods failed at once:

IoTDBRegionGroupLeaderDistributionIT.testBasicLeaderDistribution:135 expected:<3> but was:<2>
IoTDBRegionGroupLeaderDistributionIT.testCFDWithUnknownStatus:205
IoTDBRegionGroupLeaderDistributionIT.testCFDWithReadOnlyStatus:321

Root cause

Both problems are in the assertions, not in the balancer.

1. testBasicLeaderDistribution sampled the distribution exactly once, immediately after
getOrCreateDataPartitionTable. The RegionGroups have only just been created at that point, so
the leaders are still being elected and RouteBalancer is still migrating them. From the
ConfigNode log of the failing run, while all three DataNodes were already Running:

20:59:40,114 [LeaderBalancer] Try to change the leader of Region: TConsensusGroupId(type:SchemaRegion, id:0) to DataNode: 3
20:59:40,317 [LeaderBalancer] Try to change the leader of Region: TConsensusGroupId(type:DataRegion, id:1) to DataNode: 3
20:59:43,xxx testBasicLeaderDistribution -> Assert.assertEquals(3, leaderCounter.size()) // was 2

The single sample therefore observed a transient state, and a moment later the distribution was
fine.

2. testCFDWithUnknownStatus and testCFDWithReadOnlyStatus did retry (50 x 1s), but
asserted a fixed per-DataNode leader count (databaseNum / TEST_DATA_NODE_NUM, i.e. "exactly 2
leaders each"). That is not a state the leader balancer guarantees:

• it balances the System and Audit databases as well, while these assertions filter those rows
  out, so the counted subset is not the quantity being balanced;
• the number of RegionGroups is not necessarily divisible by the number of DataNodes.

In the observed run the cluster held 5 RegionGroups for 3 databases (4 DataRegionGroups + 1
SchemaRegionGroup); the Audit database owns the extra DataRegionGroup and the only
SchemaRegionGroup. Filtering the Audit database therefore leaves a subset whose even split is not
what the balancer optimizes, so the loop could not converge and failed only after the entire retry
budget.

Fix

• Extract the leader counting into countRegionLeaders and the waiting into
  awaitBalancedLeaderDistribution, replacing four copies of the same sampling block.
• Poll instead of sampling once.
• Assert the property the balancer actually provides: every expected DataNode holds at least one
  RegionGroup leader, and no two DataNodes differ by more than one leader. This is the invariant
  IoTDBMultiDBRegionGroupLeaderDistributionIT already uses.
• Report the observed distribution in the failure message instead of a bare AssertionError.
• The post-shutdown check reuses the helper with TEST_DATA_NODE_NUM - 1, so it additionally
  verifies that the leaders of the stopped DataNode really migrated: an unassigned leader keeps
  the count of leader-holding DataNodes below the expected size, which a fixed per-node count
  never checked.

For a converged cluster the behaviour is unchanged: 3 DataNodes holding 3 RegionGroups still has
to be 1/1/1.

Verification

• mvn test-compile -pl integration-test -am -DskipTests -P with-integration-tests — BUILD
  SUCCESS (39 modules).
• mvn spotless:apply -pl integration-test -P with-integration-tests — no diff.
• Reproduced the assertions against the failing CI run's cluster logs to confirm the retry budget
  was being spent waiting for a state that cannot be reached, rather than timing out on a merely
  slow balancer.