You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Frontier CI: build retry deletes build/venv, reinstalls from a compute node, and records a false cluster-wide PyPI outage #1813
Frontier compute nodes have no internet, and the build runs inside a SLURM job. The venv is fetched on the login node first (Fetch Dependencies, --deps-only) so the compute-node build never has to reach PyPI. The retry path breaks that contract.
.github/scripts/retry-build.sh reacts to a failed first attempt with rm -rf build, which deletes build/venv. Attempt 2 then runs ./mfc.sh ... on the compute node, sees no venv, and tries to uv install from PyPI, which cannot succeed there:
+ echo ' Build failed — nuking build directory before retry...'
+ rm -rf build
+ ./mfc.sh test -v --dry-run -a -j 8 --gpu acc
mfc: OK > Created a Python virtual environment (venv).
mfc: (venv) Using uv for fast installation...
├─▶ Failed to fetch: `https://pypi.org/simple/hatchling/`
├─▶ error sending request for url (https://pypi.org/simple/hatchling/)
mfc: ERROR > (venv) Installation failed.
+ bash .github/scripts/classify-build-failure.sh build-build-gpu-acc.log frontier
classify-build-failure.sh matches that Failed to fetch ... pypi line and records a cluster-wide outage (ci-outage.sh mark frontier "PyPI/uv dependency install failed during build"). For the next 20 minutes every Frontier job on every PR exits 78 without submitting anything, and each of them re-reads the marker rather than the code.
The real failure is hidden: the job that had the compile error is reported as a dependency outage, and the compile diagnostics are only in the attempt-1 part of its log.
The marker is refreshed by every later job that hits the same path, so one bad build on a busy day keeps Frontier CI dark well past 20 minutes.
Two fixes, either is enough:
retry-build.sh: keep build/venv when nuking (remove build/staging, build/install, and the CMake cache instead), so attempt 2 never needs PyPI. On Frontier the venv was fetched on the login node and is still valid.
classify-build-failure.sh: only mark an outage when the PyPI failure happened on attempt 1 (or on a login node). A fetch failure on a compute node is expected and says nothing about the cluster.
Not related to the code under test; MFC itself is unchanged by the retry.
Scan of today's Frontier jobs: the breaker was tripped twice by this path, at 18:01 UTC (branch ci/gpu-fault-diagnostics, attempt 1 failed with "Failed to install the syscheck target ... operation timed out") and at 19:14 UTC (probe/cce-acc, attempt 1 failed on a compile error). Every other Frontier failure today is a job exiting 78 on the marker with zero tests run.
Activity
sbryngelson commentedon Sep 3, 2026
Scan of today's Frontier jobs: the breaker was tripped twice by this path, at 18:01 UTC (branch ci/gpu-fault-diagnostics, attempt 1 failed with "Failed to install the syscheck target ... operation timed out") and at 19:14 UTC (probe/cce-acc, attempt 1 failed on a compile error). Every other Frontier failure today is a job exiting 78 on the marker with zero tests run.
ci: remove the cluster-wide outage breaker
sbryngelson commentedon Sep 4, 2026
The fix now lives in #1800 rather than #1814, which is closed — its two commits were cherry-picked because the two changes interact.
Both halves of this are addressed there:
retry_buildno longer removesbuild/venv, so attempt 2 does not try to reinstall from a compute node that has no route to PyPI.Leaving this open until #1800 lands.