Update x64 Helix test VMs to Azure Linux 4 by richlander · Pull Request #134819 · dotnet/runtime

richlander · 2026-09-28T21:32:24Z

Summary

Move runtime-owned x64 Helix container hosts from Azure Linux 3 to Azure Linux 4 while keeping standalone Helix test VMs on Azure Linux 3.

Surface	Previous VM	New VM
Container hosts	`AzureLinux.3.Amd64[.Open]`	`AzureLinux.4.Amd64[.Open]`
Standalone test VMs	`AzureLinux.3.Amd64[.Open]`	`AzureLinux.3.Amd64[.Open]`

Azure Linux 4 is used only to host containers. It is not selected as a bare test VM because the Azure Linux 4 package repositories do not currently provide libmsquic. The Azure Linux 4 Helix prerequisite container handles this by installing libmsquic from the Azure Linux 3 ms-oss feed.

Explicit Azure Linux 3 and Azure Linux 4 container test coverage is preserved; only the underlying Helix host moves to Azure Linux 4. Azure Pipelines build agents remain on build.azurelinux.3.amd64[.open] because Azure Linux 4 build images are not available in the build pools.

Azure Linux does not publish fixed end-of-support dates for these releases. See the Azure Linux release cadence and lifecycle.

No container tags or digests changed, so MCR image availability is unaffected.

Validation

git diff --check — passed
Parsed all four modified YAML files with Ruby YAML.load_file — passed
Confirmed no standalone AzureLinux.4.Amd64[.Open] queue values remain
Confirmed every remaining Azure Linux 4 reference is a compound container-host mapping
Confirmed no build.azurelinux.4.amd64 references remain

The default runtime pipeline covers the primary container-host mappings. runtime-extra-platforms covers the changed hosts backing Fedora, Debian, openSUSE, and Alpine edge coverage.

References

Note

This PR description was generated with GitHub Copilot.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 01f6ee32-8a73-4e38-8dff-885a73cb671a

richlander · 2026-09-28T21:32:29Z

/azp run runtime-extra-platforms

Note

This CI trigger comment was generated with GitHub Copilot.

azure-pipelines · 2026-09-28T21:32:36Z

No pipelines are associated with this pull request.

azure-pipelines · 2026-09-28T21:33:25Z

Azure Pipelines: Successfully started running 8 pipeline(s). 8 pipeline(s) were filtered out due to trigger conditions. There may be pipelines that require an authorized user to comment /azp run to run.

dotnet-policy-service · 2026-09-28T21:33:40Z

Tagging subscribers to this area: @dotnet/runtime-infrastructure
See info in area-owners.md if you want to be subscribed.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 01f6ee32-8a73-4e38-8dff-885a73cb671a

richlander · 2026-09-28T23:12:18Z

/azp run runtime-extra-platforms

azure-pipelines · 2026-09-28T23:12:39Z

Azure Pipelines: Successfully started running 1 pipeline(s).

richlander · 2026-09-28T23:28:04Z

I'm not sure why I started extra-platforms when I didn't change that definition. Has been a while since I've worked on infra!

vcsjones · 2026-09-29T14:57:53Z

There are legit crypto failures in here that are unfortunately getting swept up in another known build issue.

I will look at them.

richlander · 2026-09-30T00:39:20Z

Thanks @vcsjones!

vcsjones · 2026-10-02T14:51:16Z

@richlander it looks like the crypto failures are gone, but there is still a failure around QUIC not being available when it is expected to be.

richlander · 2026-10-14T20:56:02Z

Starting a re-run. Let's see if there are more QUIC failures. May need help @wfurt

richlander · 2026-10-15T15:58:44Z

@wfurt -- looks like the typical issue with QUIC.

at System.Net.Quic.Tests.MsQuicPlatformDetectionTests.SupportedLinuxPlatforms_IsSupportedIsTrue() in /_/src/libraries/System.Net.Quic/tests/FunctionalTests/MsQuicPlatformDetectionTests.cs:line 66
   at System.Reflection.IntrinsicInvokeHelper.Instance_Void_0(IntPtr fn, Object obj, IntPtr* _, RuntimeType __) in /_/src/coreclr/System.Private.CoreLib/src/System/Reflection/IntrinsicInvokeHelper.cs:line 837
   at System.Reflection.MethodBaseInvoker.InvokeWithSharedThunk(Object obj, IntPtr* args) in /_/src/coreclr/System.Private.CoreLib/src/System/Reflection/MethodBaseInvoker.CoreCLR.cs:line 30
   at System.Reflection.RuntimeMethodInfo.Invoke(Object obj, BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture) in /_/src/libraries/System.Private.CoreLib/src/System/Reflection/RuntimeMethodInfo.cs:line 134

https://dev.azure.com/dnceng-public/public/_build/results?buildId=1635749&view=ms.vss-test-web.build-test-results-tab&runId=45412400&paneView=debug&resultId=147510

Use Azure Linux 4 only as the host for container-backed x64 Helix queues until the bare VM image provides the required test dependencies. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: aeaf3e2c-381c-4017-896b-c3805ac77d94

richlander · 2026-10-15T16:39:31Z

After some research, I see the problem is the bare VMs. I'm going to switch those back to Azure Linux 3. We can switch them to Azure Linux 4 when MSQUIC is in the Azure Linux 4 feed, which doesn't appear to be the case. I would expect it to be here: https://packages.microsoft.com/azurelinux/4/prod/base/x86_64/Packages/l/.

richlander · 2026-10-15T21:01:12Z

/azp run runtime-extra-platforms

azure-pipelines · 2026-10-15T21:01:36Z

Azure Pipelines: Successfully started running 1 pipeline(s).

hoyosjs

Provided no tests fail in ways that hint at kernel issues, LGTM

richlander · 2026-10-20T21:52:30Z

/ba-g Failures are unrelated

## Summary Update Fedora x64 Helix coverage from Fedora 44 to Fedora 45. | Surface | Slot | Previous | New | | --- | --- | --- | --- | | Linux x64 Helix | Latest | Fedora 44 | Fedora 45 | Fedora 44 remains supported until June 2, 2027. Fedora 45 follows Fedora's N-2 lifecycle policy; its fixed EOL date is not yet published and is expected around November 2027. The exact `fedora-45-helix-amd64` container tag is available in the MCR registry and is pinned to `sha256:256fcd76faa9078632f0b8c68374f2e9ee45242b9ee7f68bda5f0a51fc28bc90`. This PR intentionally preserves the current Azure Linux 3 container host. #134819 updates the host independently, so the PRs may conflict depending on merge order. ## Validation - `git diff --check` - passed - Parsed both modified YAML files with Python `yaml.safe_load` - passed - Confirmed no Fedora 44 references remain in the Helix queue definitions - passed - Confirmed the Fedora 45 container tag and pinned digest against MCR - passed Fedora x64 coverage is carried by `runtime-extra-platforms`, so that pipeline is requested for this PR. ## References - [OS onboarding guide](https://github.com/dotnet/runtime/blob/main/docs/project/os-onboarding.md) - [.NET OS support tracking](dotnet/core#9638) - [Fedora release lifecycle](https://docs.fedoraproject.org/en-US/releases/lifecycle/) > [!NOTE] > This PR description was generated with GitHub Copilot. --------- Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 6d6ff904-3e8f-4ec8-902d-1df0da3380b9

## Summary Native ARM64 **container-host** equivalent of #134819: move the physical Helix host queues from Azure Linux 3 to Azure Linux 4 without moving standalone test VMs or changing payload distro images. | Host routing | Previous host | New host | Mappings | | --- | --- | --- | --- | | Public | `AzureLinux.3.Arm64.Open` | `AzureLinux.4.Arm64.Open` | 8 | | Internal | `AzureLinux.3.Arm64` | `AzureLinux.4.Arm64` | 3 | The 11 host-only replacements are limited to four YAML files: | File under `eng/pipelines/` | Preserved payload/alias coverage | Mappings | | --- | --- | --- | | `coreclr/templates/helix-queues-setup.yml` | Alpine 3.24 ARM64, public/internal | 2 | | `libraries/helix-queues-setup.yml` | Ubuntu 26.04 ARM64, Azure Linux 3.0 ARM64, Alpine 3.24 ARM64 | 3 | | `installer/helix-queues-setup.yml` | Ubuntu 26.04 ARM64 and Alpine 3.24 ARM64 | 2 | | `helix-platforms.yml` | Alpine ARM64 latest/oldest, public/internal aliases | 4 | Every descriptive alias, registry/image tag/digest, distro/version/architecture identity, lane condition, queue variable/group and lane count remains unchanged. In particular, `(AzureLinux.3.0.ArmArch.Open)` and its `azurelinux-3.0-helix-arm64v8` image remain Azure Linux 3 coverage; only their container host moves. All 14 ARM32-on-ARM64 container mappings and both standalone CoreCLR ARM64 VM mappings retain their existing hosts. Build agents, cross-build images, SuperPMI, GitHub Actions and other platforms are unchanged. The standalone AL4 VM `libmsquic` prerequisite is not bypassed by this change. Related to **#136313, H1**. This PR does not close the tracking issue or declare H1, S1 or S2 complete. ## Validation Static checks on commit `8869ad3d623ff1910b927f74e46adf995debac4b`: - `git diff HEAD^ HEAD --check` - passed. - `ruby -ryaml -e 'ARGV.each { |path| YAML.load_file(path); puts "PASS #{path}" }' eng/pipelines/coreclr/templates/helix-queues-setup.yml eng/pipelines/helix-platforms.yml eng/pipelines/installer/helix-queues-setup.yml eng/pipelines/libraries/helix-queues-setup.yml` - passed; all four files parsed. - An ad hoc Ruby/Open3 full-file invariant check compared each committed file with its parent after applying only the inventoried native ARM64 host-component replacements - passed. It asserted the exact four-file set, per-file replacement counts, eight public/three internal mappings, unchanged image/alias/condition bytes, and unchanged 14 ARM32/two standalone ARM64 mappings. - GitHub's comparison against `main` reports one commit, four changed files and 11 additions/11 deletions. Refreshed the live [Helix queue catalog](https://helix.dot.net/api/2019-06-17/info/queues) on 2026-10-23: | Queue | Architecture | IsAvailable | IsInternalOnly | | --- | --- | --- | --- | | `azurelinux.4.arm64` | ARM64 | true | true | | `azurelinux.4.arm64.open` | ARM64 | true | false | Queue availability does **not** establish capacity or native workload/container/kernel compatibility. No product build, native Helix execution, Docker activity, manual CI trigger/retry or Helix submission was performed for this metadata-only change. ## Required native CI coverage and existing gaps Standard CI selects only part of the changed mappings; it must not be treated as complete coverage of this PR: - Default `runtime` includes CoreCLR-selected Alpine ARM64 coverage and libraries Azure Linux 3 ARM64 coverage. Confirm actual execution on the new hosts, including the applicable public/internal routes. - `runtime-extra-platforms` selects Ubuntu ARM64 libraries coverage, but its current platform matrix does **not** list `linux_musl_arm64`. - The libraries Alpine ARM64 branch needs `includeAllPlatforms=true` coverage, such as the scheduled rolling libraries outerloop path. A normal extra-platforms run alone does not exercise it. - Current callers of the installer queue template select no ARM64 platforms. The changed installer ARM64 mappings therefore require separately arranged native ARM64 execution; Alpine additionally requires `isExtraPlatformsBuild=true` or `includeAllPlatforms=true`. - No direct YAML consumers were found for the central Alpine ARM64 latest/oldest aliases. Updating those definitions is not evidence that they executed. No coverage wiring is expanded in this PR. Native execution evidence and merge remain pending; static checks and queue availability do not complete the host migration. Reference: [OS onboarding guide](https://github.com/dotnet/runtime/blob/main/docs/project/os-onboarding.md). > [!NOTE] > This PR description was generated with GitHub Copilot. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: 96a5c84b-34cb-4531-b8cf-e6502299ec1d

Update x64 VMs to Azure Linux 4

9de6c05

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 01f6ee32-8a73-4e38-8dff-885a73cb671a

github-actions Bot added the area-Infrastructure label Sep 28, 2026

github-project-automation Bot added this to Runtime Infra Sep 28, 2026

dotnet-policy-service Bot assigned richlander Sep 28, 2026

Keep build VMs on Azure Linux 3

b4193e5

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 01f6ee32-8a73-4e38-8dff-885a73cb671a

richlander changed the title ~~Update x64 VMs to Azure Linux 4~~ Update x64 Helix test VMs to Azure Linux 4 Sep 28, 2026

richlander mentioned this pull request Sep 30, 2026

Update Fedora Helix coverage to 45 #134908

Merged

Merge branch 'main' into update-azurelinux-4-x64-vms

f033cc9

This was referenced Oct 2, 2026

System.Net.Security.Tests are failing on CI #135031

Closed

[ci-scan] Test failure: SslStream_TargetHostName_Succeeds reports zero callbacks #135057

Closed

[ci-scan] Test failure: SOS.Tests.ObjectGcHelperTests.DumpAlc_ResolvesDefaultLoadContext #135075

Closed

Merge branch 'main' into update-azurelinux-4-x64-vms

33bbc5c

richlander requested a review from hoyosjs October 15, 2026 15:51

hoyosjs approved these changes Oct 20, 2026

View reviewed changes

richlander merged commit 34d257a into dotnet:main Oct 20, 2026
227 of 234 checks passed

github-project-automation Bot moved this to Done in Runtime Infra Oct 20, 2026

dotnet-maestro Bot mentioned this pull request Oct 21, 2026

[main] Source code updates from dotnet/runtime dotnet/dotnet#10348

Merged

dotnet-milestone-bot Bot added this to the 12.0-preview1 milestone Oct 21, 2026

This was referenced Oct 22, 2026

Tracking: Migrate Ubuntu 22.04/24.04 infrastructure on main to Azure Linux 4 or Ubuntu 26.04 #136313

Open

Move ARM64 Helix container hosts to Azure Linux 4 #136387

Merged

Conversation

richlander commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Validation

References

Uh oh!

richlander commented Sep 28, 2026

Uh oh!

azure-pipelines Bot commented Sep 28, 2026

Uh oh!

azure-pipelines Bot commented Sep 28, 2026

Uh oh!

dotnet-policy-service Bot commented Sep 28, 2026

Uh oh!

richlander commented Sep 28, 2026

Uh oh!

azure-pipelines Bot commented Sep 28, 2026

Uh oh!

richlander commented Sep 28, 2026

Uh oh!

vcsjones commented Sep 29, 2026

Uh oh!

richlander commented Sep 30, 2026

Uh oh!

vcsjones commented Oct 2, 2026

Uh oh!

richlander commented Oct 14, 2026

Uh oh!

richlander commented Oct 15, 2026

Uh oh!

richlander commented Oct 15, 2026

Uh oh!

richlander commented Oct 15, 2026

Uh oh!

azure-pipelines Bot commented Oct 15, 2026

Uh oh!

hoyosjs left a comment

Choose a reason for hiding this comment

Uh oh!

richlander commented Oct 20, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

richlander commented Sep 28, 2026 •

edited

Loading