Repository navigation
Parallelise our CI/CD jobs #250
Description
Details from https://gitlab.com/MatrixAI/Engineering/Polykey/js-polykey/-/merge_requests/184#note_668294739
Using an ever longer delay is not a good way of doing these tests, they don't scale. Ideally we fix this up before going into prod.
I also noticed our tests taking longer now. Up to 30 minutes.
To speed up our testing, we can parallelise our testing. Now our jest tests are naturally parallelised by checking the number of cores the host environment has. But the CI/CD runners all have 1 CPU core:
If you’re using the GitLab.com 32 autoscaling shared runners, I believe the runners have 1CPU, 4GB RAM, and 22Gb of storage (16Gb available).
There should be a way for us to configure with more cores or other wise increase the number of parallel jobs by splitting the test job, which would end up using multiple runners (somewhat inefficient) due to duplication of resources, and then adding our own custom runner with more cores.
- https://docs.gitlab.com/runner/configuration/autoscale.html
- https://about.gitlab.com/blog/2021/01/20/using-run-parallel-jobs/
- https://docs.gitlab.com/ee/ci/yaml/#parallel
- https://docs.gitlab.com/ee/user/gitlab_com/#gitlab-cicd
A quick way to do this would to be create separate CI/CD jobs for each domain run. So npm test tests/X and npm test tests/Y might allow us to take advantage of gitlab CI/CD's splitting of each job in parallel. Only issue is the sharing of the Nix environment.
Tasks
- - Investigate dynamic parent child pipelines
- - Test splitting can be automatic by splitting a pipeline for each subdirectory under
tests/* - - For all the immediate files in
tests/*they can run under a single test pipeline
Activity
[-]Parallelise our CI/CD[/-][+]Parallelise our CI/CD jobs[/+]tegefaulkes commentedon Nov 17, 2021
It would be pretty simple to make multiple jobs for running tests. the question is how granular would we want the tests to be? I assume there is some cost to using more runners at a time.
I can do 1 test job per domain but really we just need to split the expensive ones up and keep or the trivial tests together.
CMCDragonkai commentedon Nov 17, 2021
Because it is open source, it costs free!
I suppose this has a relationship with optimising testing. So for tests that have complicated side effects that are centralised then it can be done separately. More info in #264.
CMCDragonkai commentedon Nov 19, 2021
It is being attempted here: #292
Although I think better grouping than
1/16. some way of tagging the job with the domain name so we can see it here.CMCDragonkai commentedon Dec 15, 2021
Being done in #292.
CMCDragonkai commentedon Jan 9, 2022
We are now automatically splitting all tests jobs by subdirectories. Any subdirectory in
testsends up as their own test job. This allows us future flexibility for further subdivision by just creating more subdirectories.Note that this works in a hierarchical way. Test modules in a parent directory get executed as their own job, but all subdirectories are separate jobs.
CMCDragonkai commentedon Jan 9, 2022
Doing this led to some discoveries regarding how gitlab CI/CD variables are used across pipelines. Recommend reading #292 (comment) for any future work/maintenance on this feature.