Repository navigation
Block-structured AMR with batched fine advance, load-balance and active-box infrastructure - #1628
Closed
sbryngelson wants to merge 1037 commits into
Closed
sbryngelson wants to merge 1037 commits into
sbryngelson wants to merge 1037 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR introduces an opt-in (“default-off”) family of performance/diagnostic features (load-weight and SFC partition diagnostics, weighted init-time decomposition, rank timing), plus major simulation capabilities (active-box RHS windowing and block-structured AMR) and corresponding post-processing support and documentation/validation updates.
Changes:
- Adds new runtime parameters and toolchain metadata/validation hooks for the experimental performance/AMR feature family.
- Extends the simulation code with new modules for active-box restriction, load-weight diagnostics, SFC partition reporting, rank timing, and AMR integration points (including restart/output plumbing).
- Updates post_process to read/write AMR fine-block overlays and adds/updates golden metadata plus documentation/indexing.
Reviewed changes
Copilot reviewed 82 out of 94 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| toolchain/mfc/params/descriptions.py | Adds user-facing descriptions for new experimental/performance parameters. |
| toolchain/mfc/params/definitions.py | Registers new parameters (AMR, hybrid sensors, load-balance diagnostics) and target applicability. |
| toolchain/mfc/lint_docs.py | Treats new validator checks as non-physics doc checks. |
| tests/F980C769/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ECABA006/golden-metadata.txt | Adds golden metadata for active-box test coverage. |
| tests/DD4CD8F3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/CC4213FD/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BD21A5C0/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/BCBA6E74/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/ACE05393/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/987D9025/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/852CCB81/golden-metadata.txt | Adds golden metadata for AMR-related golden tests. |
| tests/65C375B4/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/4DADE04B/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/454C565F/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/3A474BEE/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/2FC423D3/golden-metadata.txt | Adds golden metadata for a new/updated test. |
| tests/13945217/golden-metadata.txt | Adds golden metadata for moving-IB under AMR test coverage. |
| src/simulation/m_viscous.fpp | Clamps FD coefficient indexing to avoid ghost-region coefficient OOB in IB drag gradient evaluation. |
| src/simulation/m_time_steppers.fpp | Integrates active-box bounds into RK update loops and interleaves AMR fine-stage/subcycle operations. |
| src/simulation/m_start_up.fpp | Wires up new modules (rank timing, active-box, load-weight, SFC partition, AMR) into init/timestep/finalize and restart I/O. |
| src/simulation/m_sfc_partition.fpp | Adds analysis-only SFC tiling + weighted partition prediction and reporting. |
| src/simulation/m_rank_timing.fpp | Adds per-rank wall-time imbalance measurement helpers and reporting. |
| src/simulation/m_load_weight.fpp | Adds per-cell load-weight field construction and rank-level imbalance reporting. |
| src/simulation/m_hypoelastic.fpp | Refactors FD coefficient setup into a callable update routine (supporting AMR grid swaps). |
| src/simulation/m_global_parameters.fpp | Adds AMR working-state mirrors and slot selection helper plus defaults for new parameters. |
| src/simulation/m_data_output.fpp | Adds output/report hooks for load-weight, SFC partition, and rank-time diagnostics. |
| src/simulation/m_checker.fpp | Adds input validation/prohibits for active-box, hybrid sensors, load-balance, and AMR configurations. |
| src/simulation/m_active_box.fpp | Adds active-box initialization/growth and debug envelope checking. |
| src/simulation/m_acoustic_src.fpp | Adds AMR-aware handling of acoustic source support (bounding boxes and overlap abort). |
| src/post_process/m_start_up.fpp | Calls AMR fine-data reader and AMR overlay writer when amr is enabled. |
| src/post_process/m_global_parameters.fpp | Adds default-off amr flag for post_process overlay behavior. |
| src/post_process/m_data_output.fpp | Implements AMR fine-block overlay mesh/variables output (Silo/binary) and multimesh registration. |
| src/common/m_phase_change.fpp | Exposes per-cell Newton iteration count and threads it through relaxation to support load-weighting. |
| src/common/m_global_parameters_common.fpp | Adjusts start_idx lifecycle/allocation and makes load_weight_wrt visible to GPU macros. |
| src/common/m_derived_types.fpp | Introduces a simple t_box type used by new partitioning infrastructure. |
| src/common/m_box.fpp | Adds box/partition arithmetic helpers (equal/weighted splits, box-from-splits). |
| src/common/m_boundary_common.fpp | Skips BC buffer population during AMR fine advance to rely on coarse-driven ghost fill. |
| docs/module_categories.json | Registers new modules under documentation categories. |
| docs/documentation/readme.md | Adds AMR section link to the documentation index. |
| .typos.toml | Adds project-specific abbreviations to the spelling allowlist. |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## master #1628 +/- ##
==========================================
+ Coverage 62.82% 62.89% +0.06%
==========================================
Files 86 109 +23
Lines 22394 28029 +5635
Branches 3305 4306 +1001
==========================================
+ Hits 14070 17629 +3559
- Misses 6071 7170 +1099
- Partials 2253 3230 +977 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Member
Author
|
Upstream latent gap found during the MHD+AMR investigation (independent of this PR): Also for the record: MHD+AMR was attempted and re-gated on measured evidence rather than assumption — the coarse/fine seam is a continuous O(1) div(B) source that cleaning spreads but cannot remove (details in the amr.md support matrix row and commit ac203b1). |
Member
Author
|
2 tasks
fahnab666
added a commit
to fahnab666/mfc-nabid
that referenced
this pull request
Aug 29, 2026
…isting lint debt Merges the 20 commits added to sbryngelson/up-mega since ee7758b: the CCE descriptor and GPU_DECLARE fixes for the Frontier AMR abort, the NVHPC declare-ordering fix, restart format v2 (per-block owner/extents), amr_blocking_factor defaulting to 4, and the level-order clustering reductions. The delta is confined to AMR: m_amr, m_amr_regrid, m_amr_restart, m_constants, m_global_parameters, cmake/GPU.cmake, 14 AMR goldens, and two AMR-only toolchain hunks. No JWL, IBM, Riemann, or EOS source file is touched and no non-AMR golden changes. Merge was conflict-free; the local static-AMR GPU allocation cap for IBM survives intact. Also clears lint debt that predates this merge and blocked any commit: formatter line-wrapping in six JWL/IBM files, the duplicated rxn_val PROHIBITs in pre_process/m_checker.fpp (already enforced in case_validator.check_stiffened_eos, as its own docstring states), the stale hyperelasticity row in case.md (removed upstream), the missing sec-mhd anchor, and m_jwl/m_jwl_sources missing from module_categories.json.
…was too weak to see the degradation it exists to catch
…ing has ever set, whose dead brackets owned instrument rows that could never record (-56 LOC, all 70 AMR tests green)
…er-dispatch not per-byte, a quarter to a third of the gap -- two drafts withdrawn on review, the clustering knobs are inert, the cap-32 work saving belonged to a mesh nobody stepped
… gather consume is bit-identical and saves nothing; the census shows the pooled kernels are not slower, the per-block cost in the consume is non-kernel time; parked on task12/batched-gather
…pk/fill) around the per-block gather consume in both waves, same clock as [mpiwait]
…_OMP_PRESENT_ALLOCATABLE); m_amr.fpp opts in Without it amdflang maps every allocatable array of derived type a kernel touches on every launch, walking and re-attaching each component (~0.3 ms per launch for a 10-component array, linear in components; a per-element mapper generated for the type taxes every kernel in that compilation unit). m_amr opts in after an audit of its 95 kernels (bare module allocatables named: amr_cg, amr_cons_br, both allocated before first use; conditional ones are GPU_DECLARE'd). Not global: a kernel naming an unallocated allocatable variable or component aborts under present (m_variables_conversion's q_T_sf%sf). Step-loop wall -8% at cap 64, -18% at cap 32; restart files byte-identical; 70 AMR goldens; np=2 oracle. Ledger 82. Gated as 00a7c56/dede9e33 on origin/task13/host-profile (squashed here: the intermediate global form does not build).
…f allocatable derived-type arrays; per-file defaultmap(present:allocatable) opt-in for m_amr, -8%/-18% wall, bit-identical, gated
… in every form, null components do not; m_amr re-audited under the corrected rule (comments updated)
…locatable) (audited; the reflux/capture kernels carried 5.6k mapped args per rank-step)
…wrapped them into bare Fortran comment lines and a stray #)
…g at cap 64; the mapper walk was m_amr-specific and the launch-map ranking detects it but does not price it
…he ghost-fill and own-shell kernels, replacing eight per-launch copyin maps (same values, same order; bit-identical by construction)
…tered and confirmed on the ghost-fill launch (-29%/-40% gfill); wall deltas are MPI-wait movement, not claimed
…es and consume the F1 stage-fill wave for all owned blocks in one own-copy, one unpack and one ghost fill per wave (WIP: F1 wired, F2 per-block)
… consume (owner-side parent copy for co-located members, pooled unpack for the rest)
HONOR_IO_SNIPPET drops every mod the case already sets, so a test can keep its own parallel_io. BASE_CFG sets precision = 2 for every case, so the filter also deleted the --single guard's precision = 1 that the lines above it had just applied. All four honor_io_keys cases then asked post_process for double output on a single-precision build and failed validation: 'precision = 2 (double output) requires MFC built without --single'. precision is a build guard, not an IO choice, so it is exempt from the filter. Verified on the --single lane with --test-all: C26D81E1, 5EFB3277, 78314D65 and 0253D658 pass, and C26D81E1 still keeps its explicit parallel_io = F.
…e procedure s_amr_apply_reflux unrolled its batched loop nest three times into one procedure via the Fypp #:for over the face directions. CCE 19.0.0 cannot compile that, and it fails differently per target: the offload lanes abort at compile time with ftn-7991 "PDG node N requires copy" (pdgcs/v_pdg.c), while the CPU lane compiles, silently emits bad code, and segfaults the moment refluxing runs. Bisected on a single-file compile with the CI flags: one direction compiles clean, two (either 1,2 or 2,3) ICE, three ICE. It is the repetition, not any one construct, so the fix is one loop nest per procedure -- a driver plus Fypp-generated s_amr_apply_reflux_d1/d2/d3. Everything compiles at full optimization; no -Oipa0 exemption, which would cost occupancy on a hot kernel and which CMakeLists.txt already warns can break cross-file device-call registration on GPU builds. This is one root cause behind two apparently unrelated CI failures: the eight red CCE offload checks (test lanes, Case Opt, Bench) and the eighteen AMR tests segfaulting on the CCE CPU lane. The CPU half looked like a source bug and was not: -h bounds reports no violation, and a debug build passes, because there is no out-of-range access to find. A single-variable run on one node, everything else identical, pins it -- original routine: 5/5 segfault; split: 5/5 pass. Verified on Frontier, all 64 AMR/L0/active-box goldens: CCE gpu-mp 64/64, CCE gpu-acc 64/64, CCE cpu 64/64 with zero segfaults.
…C lane Three routines in m_amr_store read host-side arrays from inside a GPU region carrying default(present): s_amr_st_move_slot, s_amr_st_reserve and s_amr_br_load/_store used mbuf_lo/mbuf_hi (module integers in m_amr_state, never mapped) as loop bounds, and s_amr_br_load/_store_faces read the dummy arrays w_lo(d)/w_hi(d) inside the loop body. CCE's OpenACC runtime aborts on each with "find_in_present_table failed", while OpenMP offload maps them implicitly and never noticed. Scalars are firstprivate in both offload models, so hoisting needs no mapping and no update when the store resizes -- unlike adding the arrays to a GPU_DECLARE, which would add state to keep in sync. The bounds go through a new AMR_MBUF_BOUNDS macro so the reason is stated once. s_amr_reflux_apply_faces already did exactly this for its own copies of the same arguments, and says why; s_amr_br_*_faces now matches its sibling. This was invisible until now: the CCE OpenACC lanes never reached execution because the reflux ICE killed the build first. With that fixed, all 26 AMR tests the lane attempted failed on one signature, then 45/19 after the mbuf half. Verified on Frontier: CCE gpu-acc 64/64 with zero present-table errors, CCE gpu-mp 64/64.
is_buildable() returns False under --no-build, so every build() call in the test pre-build loop was a no-op: the loop computed each unique slug and built nothing. A variant the build phase had not produced then surfaced only once srun reached it, as "execve(): .../bin/syscheck: No such file or directory" naming a bare hash, hundreds of lines into the run and charged to the test rather than to the build that never happened. Under --no-build the loop now checks that each selected case's binaries exist and fails up front, naming the missing paths and the first case that needs each. Behaviour without --no-build is unchanged. This is the general fix for the AMD lane's missing eos_state_dependent variant: the next compile-time axis added to Case.get_fpp's _prepend() will fail loudly instead of silently. It caught three separate real mistakes while this branch was being fixed -- the missing variant itself, a --debug flag inherited from MFC's persisted build config, and a partial variant build -- each time turning a confusing late failure into an obvious immediate one.
The AMD GPU lane is the only one that does not enumerate build variants. Every other lane runs ./mfc.sh test --dry-run -a over the whole case set, which builds each unique slug it finds; 0857ace replaced that with a hardcoded "for v in base chem" so the two ~1 h amdflang device LTO links could run as concurrent jobs inside the 2 h walltime. That was complete when chemistry was the only always-on axis in Case.get_fpp's _prepend(). This branch moved eos_state_dependent from _case_optimization() into _prepend(), which applies to every target, so it now feeds the build slug of every target including syscheck. base (the empty case) bakes it False and chem selects on Chemistry, so nothing built the True side and the lane ran srun against binaries that were never produced. The correlation is exact: all 12 cases with a Mie-Gruneisen, JWL or Vinet fluid out of 831, and only those 12, failed. They share one slug per target, so any one of them builds the variant for all; -o eos=mie_gruneisen selects nine and is enough. Verified locally: with only base and chem built, those 12 report three missing binaries, including the gpu-mp-a0d86a439e/bin/syscheck path from the failing CI log; after the eos variant build, all present.
prebuild-case-optimization.sh builds syscheck, pre_process and post_process once from benchmarks[0] behind a done-marker and lets concurrent shards skip them. That serializes only the benchmarks[0] slug, so it is correct just while every benchmark hashes those three targets to that same slug -- if two cases in different shards shared some other slug, both shards would build that staging directory at once, unserialized. The slug moves with any compile-time axis in Case.get_fpp: chemistry and (as of this branch) eos_state_dependent from _prepend(), analytic ICs from the pre_process half. None of the five benchmarks trips any of them today, which is luck rather than design -- the suite's own hypoelasticity case uses Mie-Gruneisen while the benchmark of the same physics does not set an eos at all. Check it up front instead, in every shard so they all fail together rather than leaving the others to wait out the 5400 s marker timeout. Verified both ways: passes on the current five, and rejects a seeded Mie-Gruneisen benchmark naming all three targets.
./mfc.sh test writes this file when cases fail and deletes it again on a clean run, so a committed copy appears and disappears in every working tree depending on the outcome of the last run. 77c0bbe added it to the index on this branch. The existing /tests/*/** pattern covers tests/<uuid>/ but not tests/ itself, so it needs its own entry.
Nine upstream commits, 11 conflict hunks across 7 files. Most resolved as unions or by taking master's widened guards over this branch's allocation bounds; three needed more. m_rhs / s_compute_additional_physics_rhs. 0ac3a3f threads flux_src_n through as an explicit argument, but this branch rewrote that routine's body to read flux_src_rsx_vf (m_riemann_state) and consequently allocates flux_src_n only for hypo_nc_mode_dual_pass. Taking master's signature made every viscous and species-diffusion case dereference an unallocated array at the call site; the parameter is unused by this branch's body, so it is dropped. Note git also silently dropped master's flux_src_n_in declaration outside any conflict region -- no marker, caught only by the compiler. m_conduction. Fourier heat conduction accumulates into flux_src_n(E), which this branch neither allocates nor reads, so master's conduction tests would have crashed -- and merely allocating the array would have been worse, computing a flux nothing reads and failing on values instead. Ported onto the path this branch already uses for chemistry diffusion: the flux accumulates into flux_src_rsx_vf, which s_compute_additional_physics_rhs reads back and which m_riemann_state already zeroes per sweep under the guard master extended. m_conduction lives in src/simulation, so it uses the module array directly rather than m_chemistry's flat dummy -- m_chemistry is in src/common, cannot use m_riemann_state, and so pins src_lo to 1 to keep its lower bounds in agreement. src_lo must also start at mom%beg when conduction is on: it writes E alone, and E sits below adv%beg, so the adv-only band would put the accumulation out of range. lint_test_suite. Upstream's new linter forbids a test case needing a build no CI lane pre-builds -- the same failure this branch hit on the Frontier AMD gpu-omp lane, where eos_state_dependent (a _prepend compile-time constant, like chemistry) moves the slug of all twelve Mie-Gruneisen/JWL/Vinet cases. That lane's build is already extended with a third `eos` variant, so this teaches _allowed_slugs about it, exactly as the linter's own docstring prescribes for a lane whose build commands change. Verified on Frontier, merged tree: CCE gpu-acc 68/68 -- the 64 AMR/L0/active-box goldens plus master's four conduction cases. gpu-mp, gpu-acc and CPU all build clean.
Three upstream commits; the substantive one is dd7877d, which extracts m_eos and gives the EOS families a single registry. Six conflict hunks in four files, plus two changes in files that did not conflict at all. m_eos. Master moved s_compute_mixture_coefficients and its _dt companion out of m_variables_conversion into the new module, so the conflict reads as "delete this branch's copy" -- but this branch's copy carries the stiffened-gas fast path, and the relocated one does not. Taking the deletion alone would have dropped it silently: the result still compiles and still computes the right answer, just with the state-dependent EOS chain (s_phase_coefficients -> s_eos_coefficients -> s_reference_curve, a Newton loop) dragged into every conversion and Riemann kernel of a stiffened-gas build. Ported into the relocated routine instead; the _dt companion is identical and needed nothing. The any_state_dependent_eos handling in s_initialize_eos_module also arrives as master's runtime form, so it is restored to this branch's parameter semantics: unconditional PROHIBIT, never assigned, never pushed to the device. case.py. Master re-added the case-optimization block for any_state_dependent_eos; this branch bakes eos_state_dependent in every build from _prepend, so the block goes -- but master's EOS_STATE_DEPENDENT_VALUES registry replaces the hardcoded {3, 4, 5} here, so the family set has one owner (params/eos_families.py) rather than two copies. test_eos_selector. Master's new check requires every eos_* identifier under src/ to be a registry-derived selector or a declared non-family name. eos_state_dependent is the Fypp name _prepend bakes -- it carries whether ANY fluid's family is state-dependent, not which one -- so it joins eos_coeffs and eos_rk4_steps in that list rather than EOS_FAMILIES. The rest are unions: the public lists in m_variables_conversion, the use statements in post_process/m_data_output, and m_eos alongside this branch's modules in docs/module_categories.json. Verified on Frontier, merged tree: CCE gpu-acc 68/68 -- the 64 AMR/L0/active-box goldens plus master's four conduction cases -- and CPU builds clean. Note that a staging directory configured before m_eos.fpp existed fails its CMake reconfigure with a GLOB mismatch and an MPI detection error; clearing build/staging for the affected config is enough.
…NaN, OpenACC AMR PRESENT errors gpu-omp, every np>=2 test (illegal address at t_step 0), from 5146bc6: with any_state_dependent_eos a parameter, the CFL kernel in s_write_run_time_information reads an invalid q_prim_vf address on the second rank's GPU. That kernel alone now reads a runtime copy (any_state_dependent_eos_rt); every other kernel keeps the parameter, so HLLC grind is unchanged (1.16 ns both lanes). gpu-omp/acc, 3D cylindrical viscous NaN at t_step 1, from 7e014f2: s_compute_cylindrical_viscous_source_flux accesses one element past the end of flux_src_rsx_vf, spilling into the next variable's slice. flux_src_rsx_vf, vel_src_rsx_vf and Re_avg_rsx_vf now carry the same +1 spatial padding as mom_sp_rsx_vf. compute-sanitizer is clean. gpu-acc, 13 AMR tests (PRESENT clause not found): under -Minline, nvfortran folds AMR device routines into callers that pass host-array elements (amr_loc_of(...), old_ext(...), amr_slots(...)%m), and default(present) then demands those arrays. The 39 call sites now pass plain scalars, as every other device routine in MFC is called. Verified locally (NVHPC 25.11, A100): full suite 790 passed / 0 failed on both gpu-omp and gpu-acc; bench grind within noise of the PR tip.
"AMR -> 3D -> pinned max_grid_size above rank extent multi-level np=8" (A5DAD70D, 3afefa3) needs a 2x2x2 decomposition, but Phoenix GPU jobs requested --ntasks-per-node=4, so mpirun failed in syscheck with "not enough slots" on all three attempts, and fail-fast then cancelled the gpu-omp job. Frontier already requests 8. Test parallelism is derived from the GPU count, so nothing else changes. Verified on Phoenix (V100, gpu-v100, embers, -G2, CI's exact invocation with -- -c phoenix): with 8 tasks the test passes (job 13530448).
Conflict resolution: - m_ibm: keep AMR fine-marker state and master's corrected_gps; allocate corrected_gps at the widened marker bounds under AMR and rebuild it from the active ghost-point list after fine swap/restore/setup - fd coefficients: master's -fd_number:m+fd_number widening applied to the AMR m_alloc/n_alloc/p_alloc extents - post_process Silo output: master's precision loops, with db_real for the working-precision branch - simulation CFL: master's IB cell exclusion with any_state_dependent_eos_rt
Its sandiego.yaml mechanism is the only non-h2o2 one in the suite, so it forces a second chemistry build of every target just for this case, which takes too long to compile. It stays in the example skip list.
Every NVHPC gpu lane failed to compile, byte-identical on 23.11 through
26.3:
NVFORTRAN-S-0155-Compiler failed to translate accelerator region
(see -Minfo messages): Load of NULL symbol
(fypp/simulation/m_ibm.fpp.f90: 2153)
gpu-omp passes, so it is the OpenACC path only. num_gps is declare-target
and NVHPC's two-pass IPO inlines s_ibm_rebuild_corrected_gps into all
three of its call sites. Its second kernel carries firstprivate='[n_g]',
so at this call site firstprivate lands on a declare-target symbol and
the region fails to translate. The other two call sites already pass
plain locals (n_f, n_c), which is why only this one failed.
Verified: all 60 IBM tests pass (GNU/CPU).
Conflicts:
- m_riemann_solver_{hll,hllc,lf}.fpp: this branch re-indents the chemistry face-state block; master (MFlowCode#1919)
replaced its separate specific-heat and energy calls with get_mixture_caloric_state. Kept this branch's
structure and applied MFlowCode#1919's change to it.
- m_time_steppers.fpp: master's Dirichlet ramp update (MFlowCode#1931) runs before this branch's s_amr_stage_begin.
Brings in MFlowCode#1914 and MFlowCode#1930. Conflicts in m_ibm.fpp: - s_ibm_setup declarations: union of this branch's AMR sizing variables and MFlowCode#1930's max_num_gps_rank. - ghost-point overflow check: kept this branch's AMR-aware PROHIBIT; MFlowCode#1930's check guarded the same condition.
MFlowCode#1920 made s_ibm_correct_state's pb_in/mv_in required. Thread the arrays the fine-stage routines already receive through s_amr_ib_correct_fine; they are never read there, since amr prohibits qbmm.
The L0 tile path reaches s_amr_ib_correct_fine through s_l0_advance_stage_rk and s_amr_fine_stage_rk, which did not carry the arrays MFlowCode#1920 made required. Pass them the way the matching *_rhs routines already do.
sbryngelson
marked this pull request as draft
October 1, 2026 15:51
Conflicts with MFlowCode#1918 in m_chemistry and the HLL/HLLC/LF Riemann solvers: kept this branch's per-solver case-optimization guards and applied MFlowCode#1918's change inside them -- species locals are declared once at ${NUM_SPECIES}$ instead of an AMD_NUM_SPECIES_MAX / num_species pair.
# Conflicts: # src/simulation/m_riemann_solver_hllc.fpp # toolchain/mfc/test/test.py
Conflicts resolved by keeping both sides: damage_energy_cutoff with T_surface_min/max, s_mpi_allreduce_integer_min/max with s_mpi_allreduce_integer_sum_vec, the AMR checker block with master's if (ib) injection check, the union of m_ibm's public list and s_ibm_setup locals, and master's case_filter_labels chemistry selection plus the eos label in lint_test_suite. s_abort_on_bad_dt_cell is kept; master's s_apply_bodyforces copy is dropped because the branch moved it to m_body_forces.
…nnot bound The AMR STL gate checked geometry 12 only, so a 2D STL (geometry 5) passed. Moving bodies and dynamic regrid size blocks from s_amr_body_bbox, which bounds circle, rectangle, sphere, box and cylinder only; anything else (e.g. an airfoil, now drivable by prescribed kinematics after MFlowCode#1929) validated and then aborted mid-run.
…validation MFlowCode#1821's thermal and heterogeneous surface reconstruction runs inside s_ibm_correct_state, which the fine-block advance also calls, but it has no AMR test or validation. Fail closed in check_amr, where case-file gates live (lint_source keeps them out of m_checker).
Lines of Code
|
Member
Author
|
Closing this to clear my open-PR queue. The work is not abandoned and the branch is retained. Where it sat when closed
How to re-enable it Reopening needs no force-push: the branch is untouched and GitHub keeps this PR's full diff and discussion. The head is additionally archived in a local git bundle, so the commits survive even if the fork branch is ever pruned. This one needs real work to resume: it conflicts with master and has failing checks. It is also by far the largest of the set, so it would want splitting into reviewable pieces rather than going back up as a single 250-file PR. Now tracked by #2080 (labeled Closed with Claude Code. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Block-structured adaptive mesh refinement for MFC, with the load-balance and active-box infrastructure it was built on. All of it is default-off: a case without
amr = Truns the same code as master, and every golden file that exists on master is unchanged.src/simulation/m_amr*.fpp, 12.5k lines): a 2:1 refined block hierarchy of up toamr_max_levellevels over the base grid, Berger-Rigoutsos clustering on a density-gradient tag field, conservative restriction and conservative-linear prolongation with physics-specific closures, flux registers and refluxing, dynamic regrid on a fixed interval, restart. Blocks live in one device-resident flat store. Fine blocks advance in batches of equal shape (one RHS call per batch), every level in lock-step at the casedt. Every inter-rank transfer is a posted wave on one engine (m_amr_wave): keyed tags, per-peer aggregation, device-resident wire pools underrdma_mpi, and an identity-header audit underMFC_DEBUG.m_box,m_load_weight,m_sfc_partition,m_load_balance): partition arithmetic, a per-cell load-weight field, Morton space-filling-curve partitioning and tile migration for the base grid.m_active_box): restricts reconstruction, Riemann and RK windows to a light-cone-grown box around non-ambient flow.User documentation:
docs/documentation/amr.md. Internals:docs/documentation/amr_implementation.md. The performance campaign's records, design memos and what was learned:misc/amr_ledger/README.md.Physics support
The batched fine advance is the only fine advance. It runs on uniform Cartesian grids for the 5- and 6-equation models, MHD and relativistic MHD (1D), hypoelasticity and continuum damage, Lagrangian bubbles, chemistry (with and without species diffusion), IGR and static or prescribed-motion immersed bodies. Subcycling, stretched or cylindrical grids, QBMM, Euler bubbles, surface tension, phase change and moving particle clouds are rejected by the case validator with a named message, never silently. The per-block fine advance that served those (correct, golden-tested, about 1.2x slower per step) lives on
sbryngelson/MFC:amr-per-blockfor a follow-on PR.Correctness evidence
MFC_DEBUG, checked at unpack; a seeded-fault control proves the check can fail.Code shape (for reviewers of the AMR modules)
The AMR code is a chain of modules, lowest first, each using only the ones before it:
m_amr_state(all module-scope state,GPU_DECLAREs, leaf helpers),m_amr_distribution(rank decomposition, box arithmetic, SFC cut, ownership),m_amr_wave(the exchange engine: plan sides, pools, keyed tags, audit records, the wait),m_amr_store(flat-store slot machinery, bridge loaders),m_amr_exchange(the fill waves, seams, ghost fills),m_amr_frame(the grid-state swap),m_amr_transfer(prolong, restrict, reflux and their waves),m_amr_advance(the batched fine stage advance, IB/Lagrange fine services),m_amr_l0(level-0 tiling),m_amr_stage(the four hooks the RK stage loop calls);m_amris the init/finalize driver and re-exports them all.m_amr_regrid,m_amr_registers,m_amr_restartandm_amr_xchg_auditsit beside the chain;src/common/m_amr_restart_iois the restart layout shared with post_process.What the solver itself sees:
m_time_stepperscallss_amr_stage_begin/s_amr_stage_fine/s_amr_l0_stage_update/s_amr_step_fold;m_rhscarries the flux-register capture and the batched slab's reconstruction window;m_mpi_commongains the box-based equal split (byte-identical to the inline arithmetic) and a few reductions.m_riemann_solver_hllcshows +1408/-1371: that is the kernel restructuring the batched slab needed (per-cell bodies hoisted so one launch serves a slab of blocks), with every golden unchanged.The series of cleanups before review (each gated on the same goldens, CPU and GPU, byte-identical): one exchange engine for the seven waves; the fine advance reduced to the batched lock-step path; one gather mechanism (init and the regrid rebuild fill coarse patches with the same fill waves the step uses); a twelve-phase step budget under
rank_time_wrtin place of the campaign's instrumentation; the stage hooks pulled out ofm_time_steppers; the two largest routines split; dead machinery and stale comments removed; then a de-duplication pass (-1,300 lines) that gave the code one of each thing: one flux-register capture kernel for both register families, one L0 tile box router, one restriction body for its three targets, one stable key sort, one receive drain for the fill waves, one request routine per wire buffer kind, and anamr_dimdimension mask plus (x, y, z) Fypp lists in place of hand-copied 1D/2D/3D branches. Parameters are defined intoolchain/mfc/params/definitions.pyand constrained incase_validator.py; the batched-advance admissibility rules live there.Performance
Measured on one node of eight AMD MI250X GCDs, OpenMP offload, one rank per GCD, double precision in both codes. The comparable quantity between MFC and AMReX is each code's excess over its own uniform-grid run at the same number of advanced cells:
On a 400^3 base with two refinement levels, regrid every 20 steps, RK3, an advected density blob (AMReX:
Tests/GPU/CNSon the matched deck), same node, same session, three reps per arm, control and treatment interleaved:The instrument's floor is about 0.03 s/step (control-to-control on the same node). Relative to each code's own solver, MFC's AMR machinery inflates its step by 1.78x and AMReX's by 1.86x. MFC's AMR step is slower than AMReX's in absolute terms (0.96 vs 0.82 s/step) because its base scheme (WENO5 with mapped weights and monotonicity preservation, HLLC, the 5-equation model) costs 1.7x per cell, which is a scheme choice and not an AMR term. The ratio depends on the node: the same binary reads up to 1.9x on nodes with throttling GPUs or dead InfiniBand ports, so every figure above was read with its control on the same node.
Weak scaling from 8 to 32 GPUs at fixed work per rank is 1.28x per doubling against AMReX's 1.23x on matched cross-node rungs; the growth is the base grid's cross-node halo, not the regrid family.
Where the remaining 0.42 s/step goes: about a fifth is fine-solver inflation (ghost planes per block, batch padding, dispatch floor), about half is MPI wait set by the two heaviest ranks at roughly fourteen rendezvous per step, and the rest is regrid, restriction, gather and reflux work. The levers that were tried, what each bought, and what was found to be false are in
misc/amr_ledger/README.md.Known issues
amron this branch (validator-rejected); the per-block advance that supported all but surface tension is onamr-per-blockfor a follow-on PR.ftn-7991 "PDG node N requires copy"(pdgcs/v_pdg.c), while the CPU lane compiles and silently emits bad code that segfaults as soon as refluxing runs.s_amr_apply_refluxtherefore keeps one nest per procedure (s_amr_apply_reflux_d1/d2/d3, Fypp-generated from one body), which compiles correctly at full optimization on all three lanes -- no-Oipa0exemption. Folding them back together reintroduces both failures at once. Bounds checking finds nothing and a debug build passes, because there is no out-of-range access in the source.eos_state_dependentis a compile-time constant in every build (Case.get_fpp's_prepend, likechemistry), so a Mie-Gruneisen, JWL or Vinet fluid moves the build slug of every target. The Frontier AMD gpu-omp lane pre-builds a fixed variant list rather than enumerating them (each amdflang device link is ~1 h), so it gains a thirdeosvariant, and upstream'slint_test_suitegate knows about it../mfc.sh test --no-buildnow verifies that each selected case's binaries exist instead of silently skipping the build, so a future compile-time axis fails in the lint gate rather than as anexecve()on a missing path deep in a cluster run.