Skip to content

CI: rebalance integration and Postman shards (deferred — needs test-independence work first) #36964

Open

Description

@wezell

Split out of #36943 so the independent fixes there could land. The work is preserved on branch issue-36942-shard-rebalance-wip at f612b27ed8 — resume from there rather than rebuilding it.

It works

Measured on run 31203516115, with fail-fast disabled so every shard reported instead of the first failure cancelling the matrix:

IT shards    22.2 - 26.2m   (was 38.9m)
Postman      18.6 - 28.8m   (was 37.8m)

Bin-packing on measured per-class time balanced 557 integration classes to 18.4m each, spread 0.0m. Per-class times are stable run-to-run (115.2m vs 114.8m on a control set), so the balance should hold. Job wall times are noisy — an untouched shard swung +8m between runs — partly because runners vary by VM generation (D4ads_v5 / D4ds_v6 / D4ds_v7, all 4 vCPU / 15 GB).

Why it is not merged

3 of 7 integration shards and 5 of 9 Postman shards failed. Every failure was a test that only passed because of which suite-mates ran before it.

The Postman groups are dependency clusters, not labels. category-content grouped Category + ContentResourceV1 + Content_Resource because those collections share data. Rebalancing purely on time shattered every group. The GraphQL folder split failed the same way: "Page API - Testing 'page' field with inline fragments" needs setup performed by the Page API folder, which landed in the other shard.

Why it is still worth doing

Beyond the ~12m of wall clock, it is an effective detector of tests that depend on their neighbours. Every failure it surfaced was a real defect, not a false positive:

Outstanding, found but not yet fixed:

  • FieldUtilTest, ContentletAjaxTest — assume a suite-mate materialised the Languagevariable content type
  • MenuResourceTest.test_get_translation — language-variable lookup
  • WebAssetHelperIntegrationTest ×3 — limited-user permissions
  • AjaxDirectorServletIntegrationTest
  • PersonaAPITest — asserts an absolute global count (allPersonas should be 5, got:6), so any test that leaves a persona behind breaks it

Suggested approach when resumed

  1. Fix the tests above on their own, against the current shard layout, before re-sharding anything. Each is a real bug and lands independently.
  2. For Postman, treat existing groups as atomic. Rebalance by moving whole groups between shards, or by splitting a single oversized collection by file (as default-split was done historically) — not by regrouping collections across clusters.
  3. For integration, consider contiguous splits of the largest suites, which preserve each class's neighbours, before free-form repacking.
  4. Keep fail-fast: false while iterating — otherwise each ~40 minute run reveals only one shard's worth of coupling. Revert it before merging.

Note on measurement

.github/scripts/test-balance/find_swallowed.py landed in #36943. The packing scripts (parse_suites.py, pack_suites.py, pack_postman.py) are on the preserved branch.

The rebalance only pays off if both tails move: integration and Postman maxima were within a minute of each other (38.9m vs 37.8m), so fixing one alone changes wall clock by ~0.

Activity

  1. github-actions commented on Nov 6, 2026

    @github-actions
    Contributor

    This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 30 days.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions