Skip to content

Apply bin packing algorithms to the setup of our parallel CI/CD test runners #393

Closed as not planned
Closed as not planned
Apply bin packing algorithms to the setup of our parallel CI/CD test runners#393
Labels
enhancementNew feature or requestr&d:polykey:supporting activitySupporting core activity

Description

@emmacasolin
emmacasolin
opened on Jun 29, 2022

Specification

We want to introduce test load balancing to better parallelise and speed up our CI/CD tests. Part of this task is determining the most efficient number of shards to use, since we want to have the lowest number of parallel runners with the highest rate test completion. Such a task falls under the scope of a bin packing problem, which is NP-hard, however there are approximations we can use to make the problem easier.

On top of simply choosing the best number of shards to use, we want the shards themselves to be evenly balanced so that every test runner finishes at approximately the same time. We can use cached timing information from previous test runs to help to make this decision.

Additional context

Tasks

  1. ...
  2. ...
  3. ...

Activity

  1. emmacasolin
    added
    enhancementNew feature or request
    on Jun 29, 2022

  2. CMCDragonkai commented on Jun 29, 2022

    @CMCDragonkai
    CMCDragonkai
    on Jun 29, 2022
    Member

    To do this, a custom test sequencer that overrides the shard method would need to be used. This experimented with prior in MatrixAI/TypeScript-Demo-Lib#65

  3. teebirdy
    added
    r&d:polykey:core activity 1Secret Vault Sharing and Secret History Management
    and removed
    r&d:polykey:core activity 1Secret Vault Sharing and Secret History Management
    on Jul 24, 2022
  4. CMCDragonkai
    added
    r&d:polykey:supporting activitySupporting core activity
    on Jul 10, 2023

  5. CMCDragonkai commented on Aug 16, 2024

    @CMCDragonkai
    CMCDragonkai
    on Aug 16, 2024
    Member

    Doesn't seem relevant anymore, testing is pretty quick now by spreading the workload between Polykey and Polykey-CLI.

    The sharding system is still being used (even in the migration to GitHub in https://github.com/MatrixAI/Orchestrator/issues/10), as can be seen by:

    scripts/build-platforms-generate.sh
10:# Using shards to optimise tests
14:# Number of parallel shards to split the test suite into
136:    - npm test -- --ci --coverage --shard="$CI_NODE_INDEX/$CI_NODE_TOTAL" --maxWorkers=50%
164:    - npm test -- --ci --coverage --shard="$CI_NODE_INDEX/$CI_NODE_TOTAL" --maxWorkers=50%

    However it's just relying on whatever jest's default sharing allocation algorithm is.

  6. CMCDragonkai
    closed this as not plannedon Aug 16, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestr&d:polykey:supporting activitySupporting core activity

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions