Repository navigation
Description
Split out of #36943 so the independent fixes there could land. The work is preserved on branch issue-36942-shard-rebalance-wip at f612b27ed8 — resume from there rather than rebuilding it.
It works
Measured on run 31203516115, with fail-fast disabled so every shard reported instead of the first failure cancelling the matrix:
IT shards 22.2 - 26.2m (was 38.9m)
Postman 18.6 - 28.8m (was 37.8m)
Bin-packing on measured per-class time balanced 557 integration classes to 18.4m each, spread 0.0m. Per-class times are stable run-to-run (115.2m vs 114.8m on a control set), so the balance should hold. Job wall times are noisy — an untouched shard swung +8m between runs — partly because runners vary by VM generation (D4ads_v5 / D4ds_v6 / D4ds_v7, all 4 vCPU / 15 GB).
Why it is not merged
3 of 7 integration shards and 5 of 9 Postman shards failed. Every failure was a test that only passed because of which suite-mates ran before it.
The Postman groups are dependency clusters, not labels. category-content grouped Category + ContentResourceV1 + Content_Resource because those collections share data. Rebalancing purely on time shattered every group. The GraphQL folder split failed the same way: "Page API - Testing 'page' field with inline fragments" needs setup performed by the Page API folder, which landed in the other shard.
Why it is still worth doing
Beyond the ~12m of wall clock, it is an effective detector of tests that depend on their neighbours. Every failure it surfaced was a real defect, not a false positive:
AWSS3PublishingEndPointthrew aNullPointerExceptionfrom inside a catch block whenever no request was bound to the thread — fixed in fix(ci): drop dead build artifact, guard an NPE in a catch block, and make integration tests report their failures #36943- The Language Variable content type is deletable when it should be
system— Language Variable content type is not markedsystemand can be deleted, breaking i18n site-wide #36958 RuleBundlerTestcreated fixtures in a static@DataProvider, evaluated at suite construction, ~65 classes before use — fixed in fix(ci): drop dead build artifact, guard an NPE in a catch block, and make integration tests report their failures #36943PublishingEndPointTestdepended on a leakedHttpServletRequestThreadLocal— fixed in fix(ci): drop dead build artifact, guard an NPE in a catch block, and make integration tests report their failures #36943
Outstanding, found but not yet fixed:
FieldUtilTest,ContentletAjaxTest— assume a suite-mate materialised theLanguagevariablecontent typeMenuResourceTest.test_get_translation— language-variable lookupWebAssetHelperIntegrationTest×3 — limited-user permissionsAjaxDirectorServletIntegrationTestPersonaAPITest— asserts an absolute global count (allPersonas should be 5, got:6), so any test that leaves a persona behind breaks it
Suggested approach when resumed
- Fix the tests above on their own, against the current shard layout, before re-sharding anything. Each is a real bug and lands independently.
- For Postman, treat existing groups as atomic. Rebalance by moving whole groups between shards, or by splitting a single oversized collection by file (as
default-splitwas done historically) — not by regrouping collections across clusters. - For integration, consider contiguous splits of the largest suites, which preserve each class's neighbours, before free-form repacking.
- Keep
fail-fast: falsewhile iterating — otherwise each ~40 minute run reveals only one shard's worth of coupling. Revert it before merging.
Note on measurement
.github/scripts/test-balance/find_swallowed.py landed in #36943. The packing scripts (parse_suites.py, pack_suites.py, pack_postman.py) are on the preserved branch.
The rebalance only pays off if both tails move: integration and Postman maxima were within a minute of each other (38.9m vs 37.8m), so fixing one alone changes wall clock by ~0.
Metadata
Metadata
Assignees
Type
Projects
- StatusShow more project fieldsNew
Activity
github-actions commentedon Nov 6, 2026
This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days.