TESTING: Make the test failures fatal and resolve all of them - #1371
Merged
Merged
Conversation
…iles are now differentiated between the jobs.
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #1371 +/- ##
==========================================
- Coverage 69.37% 69.36% -0.02%
==========================================
Files 6122 6122
Lines 486495 486711 +216
Branches 23268 23268
==========================================
+ Hits 337512 337584 +72
- Misses 148545 148689 +144
Partials 438 438
Continue to review full report in Codecov by Harness.
|
The merge kept both sides of the LAPACK_Test_Summary block, so nothing configured at all: add_test refused the second, duplicate, test name. Keep this branch's copy, which passes --fail-on-error and the job id, and take the mention of LAPACKE from master's comment. Moving the input check into the helper also dropped the rule that a BLAS driver reading an input file must not have its stdout redirected: it takes the name of its summary file from that input and opens it on unit 6 itself, so the redirect opened the one file twice and xBLAT2 died on the second open. Only xBLAT1, which has no input, needs it. Also drop a reference to the preprocessor macro the error-exit tests no longer use. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ACSimon33
force-pushed
the
fatal_test_failures
branch
from
September 9, 2026 14:47
e1e72bc to
637aa38
Compare
langou
previously approved these changes
Sep 11, 2026
nagfor rejects the SQRT( -ONE ) that creates a NaN, and the LAPACKE test build regenerates the ?errcxx drivers without the -Onopropagate that TESTING/LIN applies, so every nagfor job has failed to build since the LAPACKE LIN tests were merged. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every STOP in the test drivers was either bare or carried a character stop code, and both exit with status 0. That is not an accident of the implementation: the standard recommends the exit status be taken from an integer stop code, and recommends zero when the stop code is of type character or absent. So a driver that abandoned its run - an unrecognised subprogram name, an input file it could not parse, a failed ALLOCATE, or its own reference arithmetic evaluating wrongly - reported success, and ctest and lapack_testing.py, which both do check the status, saw a passing test. The 177 fatal paths now use STOP 1. A digit string as the stop code is FORTRAN 77, and STOP keeps normal termination, so all units are still flushed and closed; ERROR STOP would need Fortran 2008 and only recommends flushing. The 24 STOPs that end a program normally are left as they are. The ALLOCATE checks printed their message as the stop code, which sends it to standard error where lapack_testing.py never sees it. They now write it to the output unit before stopping. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--fail-if-empty only looked at the grand total, so one driver that wrote its header and then ran nothing passed unnoticed as long as its siblings kept the total above zero. That is the shape of a driver whose input file has stopped lining up with what it reads: the run is not an error by any measure the script applies, the tests simply are not there. Report every analyzed output file whose test count is zero, and let --fail-if-empty fail on it as well. Measured over a clean build, the smallest per-file count is 6 tests and none is zero, so this does not fire on a healthy run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ACSimon33
force-pushed
the
fatal_test_failures
branch
from
September 12, 2026 21:52
a14a48c to
3c87871
Compare
ACSimon33
marked this pull request as ready for review
September 13, 2026 12:12
Collaborator
Author
The missing lines are the abnormal exit branches from the testing framework ( |
ACSimon33
marked this pull request as draft
September 13, 2026 12:34
The drivers lost their _lapacke segment when the testing CMakeLists was reworked, so --run could not find any of them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
--fail-if-empty gated both "no results at all" and "some output file ran nothing", with no way to ask for one without the other. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Job ids carry the runner image version, so their dots split one job across two levels of the JUnit hierarchy. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The four ?chkdmd drivers printed FAILED and stopped with status 0, so ctest saw a clean run; the STOP that ends the program keeps status 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rewrite runs at configure time, so a copy outlived both edits to its source and the build flipping back to running the error exits. An input with no flag to turn off now gets no copy at all. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An unconditional else() clobbered -D, and a shared macOS build without USE_FLAT_NAMESPACE hits the same XERBLA override failure as Windows. The warning that replaced it also tested BUILD_TESTING before the option declaring it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A build directory with a space in it split them into extra arguments, which cmake_parse_arguments then discarded. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
BLAS/TESTING hand-rolled the same compiler-id test and flag the helper was added to hold, for its own set of drivers. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The four generalized-eigenvalue inputs were the only ones spelling it .TRUE., which the CMake rewrite carried a second pattern for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ACSimon33
marked this pull request as ready for review
September 13, 2026 14:34
Collaborator
Author
|
@langou All issues are resolved now. The code coverage is reduced slightly because of the unreachable |
langou
approved these changes
Sep 13, 2026
Contributor
|
Hi @ACSimon33, this is really nice work. Julien. |
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
The test suite could not fail. A driver that found a numerical error printed it and exited with status 0, and every fatal
STOPwas either bare or carried a character stop code, which also exits with 0 — soctestandlapack_testing.py, which both check the status, saw a passing run. This branch closes those paths and fixes what the CI then reported.Merge this last: the numerical failures it exposes are fixed on their own branches (see table below).
Failures now fail
LAPACK_Test_Summaryruns with--fail-on-error, so a numerical failure or error reported by any driver fails the run.STOPs becomeSTOP 1, across 45 files inBLAS/TESTING,CBLAS/testing,TESTING/EIGandTESTING/LIN.ALLOCATEchecks passed their message as the stop code, which goes to stderr where the script never sees it; they now write it to the output unit.Reports name their job
lapack_testing.py --junit-job NAMEprefixes the JUnit suite (job / suite) and class (job.class) names, exposed to CMake asLAPACK_TESTING_JUNIT_JOB_ID. Codecov keys a test case on those names alone, so the reports of ~40 jobs collapsed onto one another, and a failure did not say where it came from.Error-exit tests
XERBLAcannot replace the library's, so the first deliberate illegal argument killed the driver before any test ran (CBLAS already skips itsxerblatests for this).LAPACK_SKIP_ERROR_EXIT_TESTSplus the newlapack_test_input()feed such a build a copy of the input withTSTERRoff, written wherelapack_testing.pylooks for it so a hand-run driver uses the same inputctestdoes. This could be fixed bySET_XERBLA- override the error-handler without linker stuff #1407.?ec.inhad noTSTERRline at all —?chkeehard-coded.TRUE.. It is now read from the file, the line is added tosec.in/dec.in/cec.in/zec.in, and the input-format docs in[sdcz]chkee.Fare updated.The numerical failures, fixed elsewhere
macos-26-nagfor,sec/dec; closes #598ubuntu-26.04-gfortran, the 803-rowcnep/ced/znep/zedbaselineSTFSM1 of 7776, every x86-64 gfortran job; closes #679SHS/CHS1 of 2016, ARM gfortran and armflangubuntu-26.04-arm-flang,ubuntu-24.04-arm-armflang; item 3 of #732ubuntu-24.04-arm-armflangEarlier test failures cleared on the way here; all merged: #1337, #1338, #1340, #1342, #1343, #1344, #1346, #1347, #1355.