[AppleChen17]

Summary

Eight GPU loops in m_rhs, m_riemann_state, and m_weno copy or difference every variable of a field using collapse=4 over the variable and three spatial dimensions. This PR collapses only the three spatial loops and executes the variable loop sequentially inside each GPU thread.

On H200, this reduces grind time by 5.4–10.3% across three example cases, with 8.0–10.3% improvements on one GPU. Results were measured with NVHPC 25.9 and 26.3 on one GPU, and NVHPC 26.3 on two GPUs. Nsight Compute shows 59–67% fewer executed instructions in the changed kernels, with essentially unchanged DRAM traffic.

Outputs are bit-identical to master in our comparisons, and the full GPU and CPU test suites pass.

This PR is from the NTHU team ( Team NTHU SCC ) participating in the SC26 Student Cluster Competition.

Changes

The modification applies to the following routines:

File	Routine	Modified Operation
m_rhs.fpp	s_compute_rhs	Conservative-variable copy
m_rhs.fpp	s_compute_advection_source_term	x/y/z flux differences
m_riemann_state.fpp	s_finalize_riemann_solver, s_finalize_riemann_solver_hatR	Directional flux copies
m_weno.fpp	s_pack_weno_input_arr	WENO input packing

Each loop follows the same transformation (example from m_weno.fpp):

Before:

$:GPU_PARALLEL_LOOP(collapse=4)
do i = 1, v_size
    do l = ...
        do k = ...
            do j = ...
                v_rs_weno(j, k, l, i) = v_vf(i)%sf(j, k, l)

After:

$:GPU_PARALLEL_LOOP(collapse=3)
do l = ...
    do k = ...
        do j = ...
            $:GPU_LOOP(parallelism='[seq]')
            do i = 1, v_size
                v_rs_weno(j, k, l, i) = v_vf(i)%sf(j, k, l)

The numerical expressions and spatial loop order are preserved. In the three flux-difference loops, the inverse grid spacing is also computed outside the variable loop, avoiding repeated evaluation for each variable.

Why this is faster

With collapse=4, each (variable, cell) pair is a separate GPU iteration, requiring its own collapsed-index decomposition and spatial offset calculations. With collapse=3, each thread processes all variables at one spatial cell, allowing spatial index calculations to be reused across variables.

Nsight Compute profiling of the changed kernels on H200 (3D_performance_test, NVHPC 26.3) shows:

Metric	Change
Executed instructions	−59% to −67%
DRAM bytes read + written	Unchanged (within 0.13%)
Individual kernel duration	−40% to −49%

Nsys profiling further shows that the improvement comes from the modified kernels:

• 3D_performance_test: total GPU kernel time decreased by 10.2%, with 99.6% of the saving attributable to the eight changed kernels.
• 2D_shockdroplet: total GPU kernel time decreased by 11.5%. The six applicable changed kernels each improved by 36–44%, while the other kernels collectively remained nearly unchanged.

Environment

Component	Setting
MFC	master at 47568a0 (baseline) and this PR
GPU	NVIDIA H200 SXM
Compiler / MPI	nvfortran 26.3-0 and 25.9-0, CUDA 13.0, Open MPI 4.1.9 (HPC-X 2.25.1 / 2.24)
Build	./mfc.sh build --mpi --gpu acc --case-optimization, MFC_CUDA_CC=90
Flags (default)	-fast -O3 -acc -gpu=cc90 -Minline -Mfprelaxed -march=native
Run	./mfc.sh run <case.py> --case-optimization -n {1,2} -N 1 -b mpirun

Performance

Grind time (ns/gp/eq/rhs), master → this PR. Each entry is the mean of three interleaved runs per variant within the same allocation.

All runs use H200, double precision, and --case-optimization. Two-GPU runs use two MPI ranks on one node.

Example case	NVHPC 26.3, 1 GPU	NVHPC 25.9, 1 GPU	NVHPC 26.3, 2 GPUs
3D_performance_test (201³, WENO5, 30 steps)	0.41600 → 0.37876 (−8.95%)	0.41569 → 0.37808 (−9.05%)	0.25762 → 0.24042 (−6.68%)
3D_shockdroplet (201×101×101, WENO3, 1000 steps)	0.45487 → 0.41869 (−7.96%)	0.45143 → 0.41496 (−8.08%)	0.28420 → 0.26897 (−5.36%)
2D_shockdroplet (4800×1200, WENO3, 565 steps)	0.25092 → 0.22514 (−10.27%)	0.25056 → 0.22550 (−10.00%)	0.13561 → 0.12380 (−8.71%)

Maximum run-to-run spread was 1.24% (NVHPC 26.3, 1 GPU), 0.18% (NVHPC 25.9, 1 GPU), and 3.55% (NVHPC 26.3, 2 GPUs).

The cases are the examples/ cases unchanged except for a shorter t_step_stop (2D_shockdroplet is run with its --res 4 --mach 1.4 --cfl 0.2 option).

Testing

All comparisons use master at 47568a0 as the baseline.

• GPU full suite: NVHPC 26.3, OpenACC, H200 — 725 passed, 0 failed, 39 skipped.
• CPU full suite: NVHPC 26.3, Intel Xeon Platinum 8480+ — 722 passed, 0 failed, 39 skipped.
• Bit-identical outputs: Final restart files match master across the measured configurations, including case-optimized builds.
• No golden files changed.

CPU builds also use the reordered loops. Our shared-node CPU measurements were too noisy to establish a reliable performance difference; upstream CI benchmarks can provide additional coverage.

Contribution Policy

All contributions are expected to demonstrate:

• A clear understanding of the codebase
• Alignment with product direction
• Thoughtful reasoning behind changes
• Evidence of real-world usage or hands-on experience with the problem

If these expectations are not met, we would prefer to implement the changes ourselves rather than spend time reviewing low-effort submissions.

[github-actions]

Claude Code Review

Head SHA: d573124

Files changed:

• 3
• src/simulation/m_rhs.fpp
• src/simulation/m_riemann_state.fpp
• src/simulation/m_weno.fpp

Findings:
No issues found. Checked from the diff hunks only (the context directory was empty): m_rhs.fpp, m_riemann_state.fpp, m_weno.fpp.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 61.95%. Comparing base (24a9702) to head (277b517).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2087   +/-   ##
=======================================
  Coverage   61.95%   61.95%           
=======================================
  Files          86       86           
  Lines       22901    22901           
  Branches     3376     3376           
=======================================
  Hits        14189    14189           
  Misses       6206     6206           
  Partials     2506     2506           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[github-actions]

Lines of Code

File	Lines	Diff
src/simulation/m_rhs.fpp	1971	+4
src/simulation/m_riemann_state.fpp	1188	+3
src/simulation/m_weno.fpp	1322	+1

Directory	Lines	Diff
simulation	28563	+8
total	47622	+8

[sbryngelson]

i can't replicate your speedup on gh

[AppleChen17]

Thanks for testing this on GH!

All the performance numbers in the PR description were measured on H200 with --case-optimization. We ran a few additional experiments to better understand how the speedup varies across different configurations.

On one H200 SXM, comparing this PR's current head (277b517) against its master parent (961cde5), with three interleaved runs per variant in the same allocation, the grind time changed as follows (the final restart files matched master bit-for-bit in every run):

Setup	3D_performance_test	3D_shockdroplet	2D_shockdroplet --res 4
--case-optimization	−9.0%	−8.2%	−10.3%
Without --case-optimization	−4.0%	−4.6%	−5.3%

We observed a few factors that affect the measured speedup:

• Case optimization: Without --case-optimization, the absolute time saved per grid point remains similar, but the relative improvement is smaller because the overall grind time is higher. ( The Phoenix CI gpu-acc benchmarks on this PR (H200, 2 ranks, without case optimization) also show grind speedups of 1.01–1.07×. )
• Workload size: On one H200, the improvement decreases from about 9% at a 201³ grid to within measurement noise at 40³ and smaller grids (−0.6% at 40³, +0.9% at 32³).

These are observations from H200 only, so they may not explain what you see on GH, and we can't verify that directly.

If these factors don't seem to account for what you're seeing, we'd appreciate any details about your GH configuration and results that you could share ( case, grid size, compiler, case-optimization, grind time ... ) . We'd be happy to look into the difference further.