Skip to content

Moving STL/airfoil IB deadlocks at setup on more than one rank: s_compute_centroid_offset is a collective called from a rank-local loop #1895

Closed

Description

@sbryngelson

Symptom

A moving immersed boundary with geometry 4, 5, 11 or 12 (airfoil / STL model) hangs forever in s_ibm_setup on any run with more than one rank. Nothing is printed; the job sits at "Transforming model" until the walltime expires. It took two 28-minute hangs and a live backtrace to see what it was, because the mesh size does not change the behaviour.

Cause

s_ibm_setup (src/simulation/m_ibm.fpp:117-120 on master):

do i = 1, num_ibs
    if (patch_ib(i)%moving_ibm /= 0) call s_compute_centroid_offset(i)
    $:GPU_UPDATE(device='[patch_ib(i)]')
end do

s_compute_centroid_offset (m_ibm.fpp:1215) reduces the centre of mass over MPI_COMM_WORLD:

call s_mpi_allreduce_integer_sum(int(num_cells_local, 8), num_cells)
if (num_cells /= 0) then
    call s_mpi_allreduce_sum(center_of_mass_local(1), center_of_mass(1))
    ...

On a multi-rank run num_ibs is not the global patch count. m_start_up.fpp:1306-1323 compacts patch_ib down to the patches whose centroid lies in this rank's neighbourhood, so num_ibs differs from rank to rank and is zero on every rank far from the body. Those ranks never enter the loop, never reach the allreduce, and the body's ranks block in it forever.

Geometries that return before any collective (cuboid, sphere, cylinder, …) are unaffected, which is why this only shows up for STL and airfoil bodies.

Why CI does not see it

The only multi-rank IB cases in the suite use analytic geometries. A moving STL on more than one rank appears to be untested.

Fix

Loop the global patch list and have every rank call the routine for every global patch that needs an offset, in the same order — ranks that do not track the patch contribute zero to each reduction but still join it. That is what we run on our branch (a collective-safe variant taking the global id, with the local index negative on ranks that do not track the patch); the change is confined to the loop bound and the routine's signature.

Related, from the same debugging session: #1863 and the four issues filed alongside this one, all in the moving-IB-across-ranks path.

Found with Claude Code while running a flapping STL wing on OLCF Frontier.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions