Skip to content
This repository was archived by the owner on Nov 18, 2026. It is now read-only.

Add cmake HIDE_MPI flag to esmf_time_f90 build to prevent double mpi_finalize - #13

Merged
jjguerrette merged 1 commit into
release-stablefrom
bugfix/cmake_esmf_HIDE_MPI
Mar 9, 2021
Merged

jjguerrette merged 1 commit into
release-stablefrom
bugfix/cmake_esmf_HIDE_MPI

Conversation

@jamiebresch

Copy link
Copy Markdown

-DHIDE_MPI flag is required in building esmf_time_f90 to prevent calling mpi_finalize twice.
The flag is in src/external/esmf_time_f90/Makefile and should be added in src/external/esmf_time_f90/CMakeLists.txt for cmake.

This fixes the error message:
*** The MPI_Finalize() function was called after MPI_FINALIZE was invoked. reported in https://github.com/JCSDA-internal/mpas-bundle/issues/90

@jjguerrette

jjguerrette commented Feb 16, 2021 •

Copy link
Copy Markdown

@jamiebresch, do you mind pointing me to where the HIDE_MPI flag is normally added? I could not find it in src/external/esmf_time_f90/Makefile as I am not very familiar with the MPAS make procedure. I also wonder whether this is a good solution for mpas-bundle builds.

We need to make mpas_finalize work in mpas-jedi, which requires that MPI_Finalize not be called from within internal MPAS subroutines. MPI initialization and destruction needs to be controlled at the top level in OOPS. I wonder if some ESMF subroutine is called during mpas_finalize, which in turn calls MPI_Finalize. That would be problematic.

@jamiebresch

Copy link
Copy Markdown
Author

@jamiebresch, do you mind pointing me to where the HIDE_MPI flag is normally added? I could not find it in src/external/esmf_time_f90/Makefile as I am not very familiar with the MPAS make procedure. I also wonder whether this is a good solution for mpas-bundle builds.

( cd esmf_time_f90; $(MAKE) FC="$(FC)" FFLAGS="$(FFLAGS)" CPP="$(CPP)" CPPFLAGS="$(CPPFLAGS) -DHIDE_MPI" GEN_F90=$(GEN_F90) )

@jjguerrette

Copy link
Copy Markdown

We need to make mpas_finalize work in mpas-jedi, which requires that MPI_Finalize not be called from within internal MPAS subroutines. MPI initialization and destruction needs to be controlled at the top level in OOPS. I wonder if some ESMF subroutine is called during mpas_finalize, which in turn calls MPI_Finalize. That would be problematic.

Just to clarify, so far we have been unable to use mpas_finalize in mpas-jedi because an error message tells us that MPI_Finalize is being called. There is logic in mpas_dmpar_finalize to avoid such a call here for mpas-jedi simulations:

if (dminfo % initialized_mpi) then

So if ESMF is also calling MPI_Finalize, that is another place to look.

@liujake
liujake requested review from jjguerrette and liujake March 9, 2021 18:25
@jjguerrette

Copy link
Copy Markdown

Sorry @jamiebresch. I totally did not understand what the HIDE_MPI flag is doing. It actually forces the ESMF subroutine to NOT call MPI_Finalize, which is what we want.

Then how do we add the -DHIDE_MPI flag to `src/external/esmf_time_f90/CMakeLists.txt? Is it something like this?:

target_compile_definitions(esmf PRIVATE HIDE_MPI=1)

There are similar lines in src/core_atmosphere/CMakeLists.txt.

@liujake liujake left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems to be an obvious bug in build. Thanks @jamiebresch to catch this. Not sure if this will solve our current MPAS_Finalize issue, but think it should be merged with no harm at least.

@jjguerrette

Copy link
Copy Markdown

Oh I thought this was an issue, not a PR. Looks great!

@jjguerrette
jjguerrette merged commit 84adccf into release-stable Mar 9, 2021
@jjguerrette
jjguerrette deleted the bugfix/cmake_esmf_HIDE_MPI branch March 9, 2021 18:30
@jjguerrette
jjguerrette restored the bugfix/cmake_esmf_HIDE_MPI branch March 9, 2021 18:30
@jjguerrette
jjguerrette deleted the bugfix/cmake_esmf_HIDE_MPI branch August 25, 2022 19:57
climbfuji pushed a commit that referenced this pull request Aug 26, 2022
-DHIDE_MPI flag is required in building esmf_time_f90 to prevent calling mpi_finalize twice.
The flag is in src/external/esmf_time_f90/Makefile and should be added in src/external/esmf_time_f90/CMakeLists.txt for cmake.

This fixes the error message:
`*** The MPI_Finalize() function was called after MPI_FINALIZE was invoked.` reported in JCSDA-internal/mpas-bundle#90
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants