Ten non-Newtonian tests fail on amdflang gpu-omp (MI210): likely the #1588 bug class via static declare-target any_non_newtonian #1881

Summary

On master built alone (fbddfa9) with amdflang OpenMP offload on an AMD MI210, 10 non-Newtonian regression tests fail with tolerance mismatches. The pattern matches the bug class fixed for Re_size in #1588: a static declare target variable read inside device kernels in another translation unit. Here the variable is any_non_newtonian, which #1588's fix did not cover. The mechanism is a candidate and has not been tested; a discriminating test is proposed below.

Failing tests (upstream master alone, fbddfa9)

In the same run, 78EB6879 (1D -> 1 Fluid(s) -> Non-Newtonian -> nn=1.5) and three chemistry tests passed.

Environment

• AMD Instinct MI210 (gfx90a), one MPI rank per GPU, on the AMD AI & HPC Fund cluster
• amdflang from TheRock AFAR 23.2.1 (ROCm 7.13); ./mfc.sh build -j 16 --gpu mp --no-debug --no-single --mpi, then ./mfc.sh test --only <UUIDs>

What is established, and what is not

• Established: master alone (fbddfa9) fails these 10 on this GPU and compiler.
• Established: the same 10 also fail on a downstream branch on this GPU, and pass on CPU builds of that branch at the same commits.
• Not run: a CPU build of master alone.
• Consistent with codegen dependence: the Frontier (AMD) gpu-omp CI lane was green on master at dc0aec1, and one non-Newtonian test passes here. Fix silent viscosity loss on AMD flang GPU (host-capture Re_size in the Riemann solvers) #1588 described its Re_size failure as "unreliable and codegen-dependent", with some solver kernels affected and not others.

Candidate mechanism (not tested)

any_non_newtonian is a static declare target scalar (line numbers at master ec783a8):

• declared logical :: any_non_newtonian in src/simulation/m_global_parameters.fpp:165, under GPU_DECLARE(create=...) at :172;
• set at :848-851 and pushed with GPU_UPDATE(device=...) at :864, the conformant sync (see Harden GPU macros against silent map(to:) no-op on declare target statics (AMD path) #1613);
• read inside device code in other translation units: src/simulation/m_riemann_state.fpp :859, :888, :898, :908, :1069, :1103, :1114, :1125; src/simulation/m_hb_function.fpp:78; src/simulation/m_viscous.fpp :72, :1155; src/simulation/m_time_steppers.fpp:703.

This is the pattern #1588 fixed for Re_size. Its root cause there was a cross-translation-unit read of a static declare-target variable inside Riemann-solver kernels, and the upstream reports are ROCm/llvm-project#2890 and llvm/llvm-project#203711. The fix host-captured Re_size into Re_size_loc and passed it firstprivate; it covers Re_size only, and nothing captures any_non_newtonian. If a kernel reads it as .false., every non-Newtonian branch is skipped and the fluid integrates as Newtonian: wrong on GPU, correct on CPU. The per-fluid is_non_newtonian and hb_* arrays are allocatable, which is the kind #1589 does not flag.

Proposed discriminating test

Capture any_non_newtonian into a host-set local and pass it firstprivate in the kernels that read it, exactly as #1588 did for Re_size, then rerun the 10 tests on this platform.

• If they pass, the mechanism is confirmed, and the lint proposed in Harden AMD gpu-omp CI coverage + lint against silent declare-target regressions (follow-up to #1588) #1589 (item 3) should cover static declare-target scalars as well as arrays.
• If they still fail, the cause is elsewhere; the next check is whether is_non_newtonian and hb_* are current on the device where the kernels read them.

Related: #1588, #1589, #1613.