You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Moving STL/airfoil IB deadlocks at setup on more than one rank: s_compute_centroid_offset is a collective called from a rank-local loop #1895
A moving immersed boundary with geometry 4, 5, 11 or 12 (airfoil / STL model) hangs forever in s_ibm_setup on any run with more than one rank. Nothing is printed; the job sits at "Transforming model" until the walltime expires. It took two 28-minute hangs and a live backtrace to see what it was, because the mesh size does not change the behaviour.
Cause
s_ibm_setup (src/simulation/m_ibm.fpp:117-120 on master):
do i =1, num_ibs
if (patch_ib(i)%moving_ibm /=0) call s_compute_centroid_offset(i)
$:GPU_UPDATE(device='[patch_ib(i)]')
end do
s_compute_centroid_offset (m_ibm.fpp:1215) reduces the centre of mass over MPI_COMM_WORLD:
On a multi-rank run num_ibs is not the global patch count. m_start_up.fpp:1306-1323 compacts patch_ib down to the patches whose centroid lies in this rank's neighbourhood, so num_ibs differs from rank to rank and is zero on every rank far from the body. Those ranks never enter the loop, never reach the allreduce, and the body's ranks block in it forever.
Geometries that return before any collective (cuboid, sphere, cylinder, …) are unaffected, which is why this only shows up for STL and airfoil bodies.
Why CI does not see it
The only multi-rank IB cases in the suite use analytic geometries. A moving STL on more than one rank appears to be untested.
Fix
Loop the global patch list and have every rank call the routine for every global patch that needs an offset, in the same order — ranks that do not track the patch contribute zero to each reduction but still join it. That is what we run on our branch (a collective-safe variant taking the global id, with the local index negative on ranks that do not track the patch); the change is confined to the loop bound and the routine's signature.
Related, from the same debugging session: #1863 and the four issues filed alongside this one, all in the moving-IB-across-ranks path.
Found with Claude Code while running a flapping STL wing on OLCF Frontier.
Activity