Repository navigation
ci: run the samples and integration tests in their own jobs - #5809
Conversation
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #5809 +/- ##
==========================================
+ Coverage 76.12% 76.13% +0.01%
==========================================
Files 528 528
Lines 19579 19579
Branches 3829 3829
==========================================
+ Hits 14904 14906 +2
+ Misses 3792 3791 -1
+ Partials 883 882 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
d4f0e61 to
d673987
Compare
7cf192e to
13de6a8
Compare
d673987 to
7fa3c1e
Compare
13de6a8 to
1db0ebb
Compare
| src/Sentry.Android.AssemblyReader/Sentry.Android.AssemblyReader.csproj | ||
| tests: integration-test | ||
|
|
||
| steps: |
There was a problem hiding this comment.
The first 40 lines of code here is almost identical to
sentry-dotnet/.github/workflows/build.yml
Lines 645 to 695 in 1db0ebb
Maybe as a follow up PR that could be refactored so fixes/changes don't have to be copy/pasted (and more importantly, so they don't fall out of sync).
There was a problem hiding this comment.
Agreed, I'll pull that into a follow-up once the stack is in.
| include: | ||
| - os: xcode-27 | ||
| rid: macos | ||
| projects: >- |
There was a problem hiding this comment.
These project lists are repeated four times, and three of them are identical. Could they come from generated solution filters instead, the way the build legs pick theirs with slnf:? In scripts/generate-solution-filters-config.yaml:
groupConfigs:
integrationTestPackages:
- "**/Sentry.csproj"
- "**/Sentry.Extensions.Logging.csproj"
- "**/Sentry.Maui.csproj"
- "**/Sentry.Bindings.Android.csproj"
- "**/Sentry.Android.AssemblyReader.csproj"
- "**/Sentry.Bindings.Cocoa.csproj"
filterConfigs:
- outputPath: Sentry-CI-IntegrationTest-macOS.slnf
include:
groups: ["integrationTestPackages"]
- outputPath: Sentry-CI-IntegrationTest.slnf
include:
groups: ["integrationTestPackages"]
exclude:
groups: ["macOnly"]Then each matrix entry only needs slnf: Sentry-CI-IntegrationTest[-macOS].slnf, and the job can use it the way the legs do:
- uses: ./.github/actions/environment
with:
projects: ${{ matrix.slnf }}
- run: dotnet build ${{ matrix.slnf }} -c Release --nologo -v:minimal
- run: dotnet pack ${{ matrix.slnf }} -c Release --no-build --nologoThat keeps the list in one place, which test-solution-filters checks for drift, and it removes the PROJECTS env and both bash loops. Building then packing a filter with --no-build is what the macOS leg already does with Sentry.Maui, so it shouldn't hit the NU5026 problem.
There was a problem hiding this comment.
Nice, done. The job now builds and packs Sentry-CI-IntegrationTest-macOS.slnf and Sentry-CI-IntegrationTest.slnf, generated from your config, and the PROJECTS env and both loops are gone.
| if: matrix.rid == 'linux-arm64' || matrix.rid == 'linux-musl-arm64' || matrix.rid == 'macos' | ||
| uses: getsentry/github-workflows/sentry-cli/integration-test@959162c159b866489d3fabe93327f48f9b8cb7ee # 3.4.1 | ||
| env: | ||
| SKIP_MAUI_TESTS: ${{ runner.os == 'macOS' }} |
There was a problem hiding this comment.
This does make me a bit nervous.
The macOS leg now skips MAUI tests. They run in the new Integration test (macos) job on a different runner, which builds/packs its own packages.
That means that what we test isn't necessarily what we ship for MAUI anymore... GitHub can and do change what is installed on the runners between runs, so this isn't a hypothetical risk. I have had exactly the same code pass all the tests on one runner but fail on another, due to changes being rolled out progressively across the runners before.
I realise that's the crux of this build/performance improvement - it's also the reason we never explored doing this.
Note
We could do this... just worth being aware of the risks...
There was a problem hiding this comment.
Very fair point, thanks for flagging it! On release/* the macOS leg now runs the MAUI cases again, against the artifact craft publishes, so what we release is still tested as shipped. PRs and main keep the split. Does that cover it for you?
There was a problem hiding this comment.
That's a nice middle ground - doesn't slow us down most of the time.
7fa3c1e to
455b362
Compare
1db0ebb to
2f9296c
Compare
| # if: ${{ (matrix.rid != 'linux-musl-x64') && (matrix.rid != 'linux-musl-arm64') }} | ||
| # TODO: Re-enable once we have resolved https://github.com/getsentry/sentry-dotnet/issues/4788 | ||
| if: ${{ (matrix.rid != 'linux-musl-x64') }} | ||
| # TODO: Re-enable for linux-musl-x64 once we have resolved https://github.com/getsentry/sentry-dotnet/issues/4788 |
There was a problem hiding this comment.
This has been resolved. So either the comment is stale or we can drop the guard.
There was a problem hiding this comment.
True... I'll re-enable it and push on top of this PR (no impact on the total CI runtime). Thanks!
The app samples only check that they compile, but their Release builds trim, AOT-compile and package each app. In the macOS leg that's about 13.7 of the Build step's 28.3 task-minutes, and it's the whole tail of the step once the Cocoa SDK comes from the cache. On Windows, the MAUI sample's Android build is the last thing the Build step finishes. The Android, iOS, Mac Catalyst, macOS and MAUI samples now build in a separate job that starts with the .NET legs: all of them on macOS, the Android and MAUI samples on Linux, and the MAUI sample on Windows, which is the only build of its Windows target. The legs' solution filters no longer include them, and generated filters list them. The samples job's Cocoa SDK steps match the macOS leg's. So that msbuild.exe still builds an Android and a Windows app, the MSBuild job builds the MAUI sample after the solution filter. That took 180 to 211 seconds on three runs, and the job still ended about 10 minutes before the slowest check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The build legs built, ran the unit tests, and then ran the integration tests, one step after another. On macOS the MAUI cases in cli.Tests.ps1 took 8 to 10 of the leg's 13 to 15 integration test minutes, building three Release apps for Android, iOS and Mac Catalyst, and the Windows legs spent 12 to 16 minutes in the integration tests. The MAUI cases move to cli-maui.Tests.ps1, unchanged except that SKIP_MAUI_TESTS also skips them. A new integration-test job runs next to the legs: on macOS it runs the MAUI cases, which the macOS leg now skips, and on linux-x64, win-x64 and win-arm64 it runs every integration test file, which those legs now skip. Each entry builds and packs only the packages its tests use, from two generated solution filters: Sentry-CI-IntegrationTest-macOS.slnf, which adds the Cocoa bindings, and Sentry-CI-IntegrationTest.slnf. It packs with --no-build like the legs, because packing Sentry.Maui while building it on macOS fails with NU5026. These packages aren't the ones the macOS leg publishes, but CI_PUBLISHING_BUILD only adds the other desktop platforms' native libraries and sentry-cli binaries, which these tests don't use. On release branches the macOS leg runs the MAUI cases too, against the packages it uploads for craft, so the packages we publish are still the ones tested. The integration-test job runs them there as well, which keeps the jobs the same on every branch. On three runs with the other CI changes, the win-x64, linux-x64 and win-arm64 legs took 15.0, 11.2 and 17.9 minutes, and their integration jobs 18.5, 12.7 and 22.1. Before, the win-arm64 leg took 31 and was the slowest job. The new jobs passed the same 69 tests and skipped the same 35 as the legs did before. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bitsandfoxes
left a comment
There was a problem hiding this comment.
Other than the already disabled case in the comment LGTM!
2f9296c to
5cf10ba
Compare
#4789 turned them off on linux-musl-x64 while they hung there without an error, tracked in #4788. #4743 fixed the disk space problem behind it, and #4788 was closed in December 2025, but the guard stayed. The leg finishes in about 9 minutes without them, and the same tests take about 6 on linux-musl-arm64, so the leg still ends well before the slowest check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Part 3 of 6 for #5806, based on #5808. The build legs built, ran the unit tests and then ran the integration tests, one step after another, and the app samples made their Build step longer. This PR moves both into jobs that run next to the legs, one commit each.
Build the app samples in their own job. The app samples only check that they compile, but their Release builds trim, AOT-compile and package each app. On macOS that was about 13.7 of the Build step's 28.3 task-minutes.
Samplesjob builds all of them on macOS, the Android and MAUI samples on Linux, and the MAUI sample on Windows, which is the only build of its Windows target.generate-solution-filters-config.yaml.msbuild.exestill builds an Android and a Windows app, the MSBuild job builds the MAUI sample after the solution filter. That takes about 3 minutes and the job still ends well before the slowest check.Run the integration tests in their own jobs. A new
integration-testmatrix job runs next to the legs:cli.Tests.ps1tocli-maui.Tests.ps1, unchanged except that they also skip whenSKIP_MAUI_TESTSis set, as the macOS leg does.release/*branches the macOS leg runs the MAUI cases too, against the artifact craft publishes, so the packages we release are still the ones tested. Release runs take 8 to 10 minutes longer for it.Sentry-CI-IntegrationTest-macOS.slnf, which adds the Cocoa bindings, andSentry-CI-IntegrationTest.slnf. They aren't the packages the macOS leg publishes, butCI_PUBLISHING_BUILDonly adds the other desktop platforms' native libraries and sentry-cli binaries, which these tests don't use.On the stack's own three runs, #5812 with all 6 parts, the slowest check finished at 23.3, 25.5 and 24.3 minutes. On #5802, with the integration tests still in the legs, it was 32.4, and the win-arm64 leg was the slowest job at 31. On #5812, runs 1, 2 and 3:
The macOS MAUI job depends a lot on the Mac it gets: on run 2 its build took 5.3 minutes against 2.5, and its tests 16.3 against 7.2. The three runs of the throwaway #5805 looked the same: legs 15.0, 11.2 and 17.9, integration jobs 18.5, 12.7 and 22.1. #5802 didn't have the other parts' later fixes either, so not all of those 9 minutes come from this PR.
The linux-x64, win-x64 and win-arm64 integration jobs each passed the same 69 tests and skipped the same 35 as their legs did before, and on all three runs of #5812. The first measurements of each change are in #5783 and #5797.
#skip-changelog