Repository navigation
CI/CD to build and sanity check docker container and release executables #231
Description
The final built executables coming out of vercel/pkg are not tested when uploaded as pre-release executables.
We should run a final test on those, maybe a simplified test that runs just the --version or --help page to ensure that the executable actually runs on the target environment. This can help us know whether the final packaged executables are actually working mostly.
This requires our CI/CD to expand to include all environments:
- Linux
- Windows - https://about.gitlab.com/blog/2020/01/21/windows-shared-runner-beta/
- Mac - @joshuakarp knows the details about this in Secret Vault Schema #4
We have these stages:
stages:
- check
- build
- release
This might work as an extra stage: qa to mean "quality assurance". And it needs to use raw environments (not based on our Nix image that we use to build things).
Tasks
- - Investigate adding an extra stage
qathat enables Linux, Windows, and Mac environment runners, that means this stage runs on an entirely different runner compared to check, build and release. - - Download the executables from the release point. This tests whether our release point is up and running. If the release point is down, the test should retry after some time, as this depends on network availability.
- - Run a sanity check over the executables such as
--helpand--versionto get the help page and the version page respectively. - - Ensure that this job is interruptible.
Regarding docker:
- - Create the docker image with
nix-build ./release.nix -A docker- test that this container works with
docker run -it name-version:hash. - example:
docker run -it matrixai_typescript-demo-lib-1.0.0:f3wf1wxvkdbbmrhxx9rhr5qw2skbrah6 - see TypeScript-Demo-Live README.md for more details
- the TMPDIR may be used by PK, if so, use this https://gist.github.com/CMCDragonkai/94e6550302aa7ebc175c06819927605a
- inspect the image contents for correctness by using this https://gist.github.com/CMCDragonkai/d7a652ada27289ab59e28bc914c942f6
- do we need a cacert and tzdata?
- cacert may be needed in case we want to contact external websites, but this is not a thing for the PK agents hosted on
testnet.polykey.io - tzdata shouldn't be necessary since we don't operate on any timezones, we assume OS time is UTC time
- cacert may be needed in case we want to contact external websites, but this is not a thing for the PK agents hosted on
- test that this container works with
Activity
CMCDragonkai commentedon Aug 29, 2021
Remember Gitlab provides many CI/CD runners. But I suspect this will require running a separate runner on the last stage. I'm not sure if the configuration allows this. But we must investigate this.
Also consider what additional costs there is, and whether it is free for Open Source, and we have replicated this to the Open Source group.
CMCDragonkai commentedon Aug 29, 2021
Should also do a hash check to ensure that the expected hashes of the released executables are in fact what we expect. That means during the release stage, we should be calculating the hashes from
nix-buildand then publishing those as well on the release page. Note that the hashes used in the naming currently are nix hashes which are hashes of the input derivation expression, not the build outputs.CMCDragonkai commentedon Aug 29, 2021
This maybe relevant too https://nixos.org/manual/nix/unstable/command-ref/new-cli/nix3-store-make-content-addressable.html
Checking the hash of the release is just about ensuring integrity but it doesn't ensure authenticity. A signature would ensure authenticity. However combining the hash with an "authentic" trusted source is sufficient to ensure authenticity and integrity. Of course end-to-end authenticity is enhanced with a proper signature applied.
To dog-food Polykey, it would make sense to do a code-sign from the Polykey organisation gestalt rather than GPG since we stopped using GPG.
CMCDragonkai commentedon Nov 1, 2021
The CI/CD check here should actually be done on the container image build. We should also do a check against the release executable builds while we're at it.
Note that the
release.nixis what has all the specifications. The currentdockerattribute bundles up thecacertpackage. For the testnet and mainnet. I believe thecacertwon't be required for NAT functionality as contacting external websites is only needed in theidentitiesdomain.If however we are going to join all of the testnet nodes as part of the Matrix AI gestalt and mainnet could be in a different gestalt, it may in fact be required. We can keep the dependency in just for now.
[-]CI/CD to sanity check on final built executables in all environments[/-][+]CI/CD to sanity check on docker container and release executables[/+]CMCDragonkai commentedon Nov 1, 2021
Useful gist notes about using docker and nix:
[-]CI/CD to sanity check on docker container and release executables[/-][+]CI/CD to build and sanity check docker container and release executables[/+]CMCDragonkai commentedon Nov 5, 2021
Pushing this Nov 11 finish from Nov 10 as an additional last sanity check should be done once #269 is done. But this can be started without #269.
CMCDragonkai commentedon Nov 5, 2021
Make sure to check that bundled files JSON and protobuf files are being put into the
./distdirectory properly. Except of course the./package.json.See: MatrixAI/TypeScript-Demo-Lib#34 (comment) for discussion and fix in TypeScript-Demo-Lib.
If there are future files that need to be copied over, it's also important to consider the
npm postbuildhook to copy over those non-transpiled files. These stages of "building" should be documented.CMCDragonkai commentedon Nov 7, 2021
Checking the build executables can be done in a number of ways automatically.
Application Emulation
The idea here is to directly execute the built artifacts in the same CI/CD container environment which currently is a "nix" container running on Linux. The advantage here is probably less resources being used, and all of this could be tested on our local development machines which are NixOS (or linux with Nix).
Disadvantage is that it's not a full test, we don't know for sure what might happen on the other platforms.
pkCLI via wineExample: https://jake-shadle.github.io/xwin/#2-specify-runner This guy uses WINE to test a cross-compiled Rust executable.
System Emulation/Virtualisation
This relies on virtualisation, such as using QEMU to run a full system. This is closer to the real deal, but more complex and expensive.
Atm, I don't believe the containers inside Gitlab CI/CD are allowed to execute QEMU so we don't automate it that way. Instead this will rely on the CI/CD system itself to provide these extra environments. Gitlab has MacOSX and Windows guests available to be used, and these can then run tests against our built executables. Note that this means the tests are not running inside the Nix container anymore. Which leads us to our next problem.
What kind of tests?
These tests that run against the final built executable are not like the tests we have currently inside our Jest testing. Jest itself is just a test runner, it uses Node executes JS at the end of the day. The JS itself right now is importing code directly from
srcand testing this. Even now ourtests/bintests don't actually do a child process execution. They are lighterweight by callingmainfunction directly and mocking the STDOUT/STDERR... etc. And even if they were to actually do a proper process exec, it's again done against thesrc/code, and not against the final built executables.So for testing the final executables, the tests have to shell out out and execute the commands and then testing what is the expected side-effects. This can be done in jest by using child process exec, and all we are doing is using the JS to run child processes.
If we use a library like https://github.com/raingerber/jest-shell-matchers, this allows us to create expectations and matchers that can execute external executables.
However there are 2 challenges with this:
tests/binto instead test external executables. Some of the tests can work without any change since all we do is swap out the usage ofmainwith executing an external executable.If abandon jest entirely, then we would need a new test harness, and then platform-specificity is an issue. For example if we were to write tests in a shell language like bash, this may work easily on Linux and MacOS, but not as easily in Windows which doesn't have bash. We could create platform specific ways for every platform, Linux and Mac can use bash, while Windows uses powershell, but this just ends up duplicating our
tests/bin2-times.We know that node and jest can work on Windows, Mac and Linux. But we won't be able to use Nix in most of these extra environments, since these are meant to be non-Nix platforms. So it's just a matter of installing
npmandnodeon those platforms and proceeding from there. I'm sure the CI/CD scripts have some suggested way of installing Node on those platforms. Windows may have chocolatey, and Mac may have homebrew. See the listed methods here: https://nodejs.org/fa/download/package-manager/Also Gitlab docs may have some suggested ways of doing things too.
Portability of
tests/binSo does this mean we have to write some new tests like
tests/bin-external? Or could we re-use some of the tests intests/bin?Surely we can swap out execution of
mainwith direct execution of the real thing right?That may be the case, but if we are using
jest-mock-process, then that won't work, since the real process is not using the Nodeprocessmocks.However if all of the mocking used by https://github.com/EpicEric/jest-mock-process is done within a wrapper function like
pkWithStdio, then it may be possible to swap that out with the real implementation in which case the side-effects are real, and STDOUT, STDERR and exit code can be captured as well. Then a parameter can be passed intotests/binto run the same tests on the external executable.I'm sure some tests may not work though, and this requires a careful review.
tegefaulkes commentedon Nov 8, 2021
I've done a little research into gitlabs CI/CD. It might be possible to test the built executables in native shell enviroments using a combination of tags, artefacts and dependencies.
We can us tags to specify a type of runner to run the job in. It seems like we can specify a runner that is a Linux bash shell or a windows Powershell this way.
Using job artefacts and dependencies we can ensure that the required built executable is copied between jobs.
I need to do some more research and testing for this but my impression is that it might be possible.
CMCDragonkai commentedon Nov 8, 2021
It would be best to first try it in the TS-Demo-Lib. You can create new branches and run the CI/CD manually there.
Also beware of Mac:
First thing would be non-NixOS executables and Windows which will the easiest to test.
32 remaining items
tegefaulkes commentedon Nov 17, 2021
Macos test is working now. just need to work out a small fix for it.
tegefaulkes commentedon Nov 17, 2021
All of the CI/CD tests are passing now!
The tests are just running Polykey without any arguments and expecting it to run with an exit code of 0. as it stands now we're just testing that commander runs. We want to do at minimum a test that starts a polykey agent and stops it.
tegefaulkes commentedon Nov 17, 2021
I added a flag to
pk agent startto do a test start and stop of polykey. I'm using this in the tests now.tegefaulkes commentedon Nov 18, 2021
Ok, right now the ground work has been laid out an we have basic tests running for the following enviroments.
npm run polykeyin nodeThe tests so far just run a quick test I made in
agent startthat starts and stopspolykeywith full logging. if it exits with a code 0 we assume the test succeeded.The docker test is just running the docker image with no flags for now so it's just printing out the help page.
I also expanded the test jobs out into multiple runners, so testing should complete quicker now.
tegefaulkes commentedon Nov 18, 2021
Note: It's possible to take the output from the build stage and use it in the publish stage. this should be addressed at some point.
tegefaulkes commentedon Nov 18, 2021
Also note,
Nix-buildfails when building locally but runs fine in the CI/CD. so far as I can tell it's failing due to the version ofnode-gyp-build. whennode-gyp-buildis4.2.3nix build patches the shebang and it runs fine. but if it's version4.3.0then nix doesn't patch the shebang and the build fails. As to why this is happening I am not sure and I'm running out if ideas to test.There are comments above where I am troubleshooting it.
joshuakarp commentedon Nov 29, 2021
Adjusting end date to Wednesday December 1 (from November 19 - delayed from refactoring work in #283), to give time for review and merge.
CMCDragonkai commentedon Dec 13, 2021
Should incorporate the cached compiler path: #296 (comment)
CMCDragonkai commentedon Dec 14, 2021
The networking tests has been changed since async start was integrated. I'm removing these changes so that I can see what the original code was meant to do and fix these problems. This is why it's not a good idea to use be using
beforeEachandafterEachfor things that is the unit under test. @tegefaulkes.CMCDragonkai commentedon Dec 14, 2021
@tegefaulkes all other tests which has this behaviour needs to be reverted back to the way it was before with the unit under test actually being in each test function.
CMCDragonkai commentedon Dec 14, 2021
I remember there was incorrect usage of getters in
ForwardProxyas well. Getters should be used only for mostly static properties that are readable but mutable by the inside. OtherwisegetProxyHost()is the correct way of doing this to indicate that this involved side-effects.CMCDragonkai commentedon Jan 9, 2022
This is now all working. Successful build and QA sanity check here: https://gitlab.com/MatrixAI/open-source/js-polykey/-/pipelines/444133744
Note that due to staging, and expected test failures from nodes and related functionality, right now master is expected to have a failed pipeline. However in qa-testing when skipping testing, all of this worked.
Furthermore, QA sanity check is very basic. All it is checking is whether the program itself can run. By running the help command. That's it. This is a good first step.
Subsequently we can reuse our test-suite that we are using in
tests/binto also apply it in the other platforms. This will require some abstractions to build later. But for now this is done. In production we'll just do manual testing in the other platforms later.CMCDragonkai commentedon Jan 9, 2022
In the process of working on this, we discovered issues regarding licenses and
--publicvs--public-packagesoption used bypkg. Worth understanding for future: #292 (comment)