Skip to content
This repository was archived by the owner on Oct 23, 2020. It is now read-only.
This repository was archived by the owner on Oct 23, 2020. It is now read-only.

Memory leak in RK4 #1374

Closed
Closed

Description

@mark-petersen

We don't run RK4 very often. Running EC60to30 on 576 cores on grizzly with gnu optimized, it runs for 7 days and then dies with memory errors.

Activity

  1. mark-petersen commented on Jul 27, 2017

    @mark-petersen
    ContributorAuthor

    This is what I see. I am running from commit 2763249 on mark-petersen/ocean/smoothBLD_Luke_Phil_Todd. Identical runs with split explicit do not have this problem. I expect to see the same thing on MPAS-Dev/ocean/develop, but have not checked.

    gr1112.localdomain> mpirun -n 576 ocean_model_smoothBLD_Luke_Phil_Todd
    App launch reported: 16 (out of 16) daemons - 540 (out of 576) procs
    Operating system error: Operating system error: Cannot allocate memory
    Allocation would exceed memory limit
    Operating system error: Cannot allocate memory
    Allocation would exceed memory limit
    Operating system error: Cannot allocate memory
    Allocation would exceed memory limit
    Operating system error: Cannot allocate memory
    Allocation would exceed memory limit
    Operating system error: Cannot allocate memory
    Allocation would exceed memory limit
    Operating system error: Cannot allocate memory
    Allocation would exceed memory limit
    Cannot allocate memory
    Allocation would exceed memory limit
    Operating system error: Operating system error: Cannot allocate memory
    Allocation would exceed memory limit
    Cannot allocate memory
    Allocation would exceed memory limit
    -------------------------------------------------------
    Primary job  terminated normally, but 1 process returned
    a non-zero exit code.. Per user-direction, the job has been aborted.
    -------------------------------------------------------
    --------------------------------------------------------------------------
    mpirun detected that one or more processes exited with non-zero status, thus causing
    the job to be terminated. The first process to do so was:
    
      Process name: [[46293,1],286]
      Exit code:    1
    --------------------------------------------------------------------------
    gr1112.localdomain> pwd
    /lustre/scratch2/turquoise/mpeterse/runs/c67n/forward
    
  2. mark-petersen commented on Jul 28, 2017

    @mark-petersen
    ContributorAuthor

    Note, this dies within 4 days on wolf, 256 cores. See

    /lustre/scratch2/turquoise/mpeterse/runs/c67q/forward
    
  3. xylar commented on Jan 22, 2019

    @xylar
    Contributor

    As I just discussed with @mark-petersen, this issue is holding up my work on wetting-and-drying under ice shelves. @pwolfram, something you should be aware of, too.

  4. pwolfram commented on Jan 22, 2019

    @pwolfram
    Contributor

    Thanks @xylar

  5. mark-petersen commented on Jan 22, 2019

    @mark-petersen
    ContributorAuthor

    Closing this issue to move conversation to the new MPAS-Model repo. See
    MPAS-Dev/MPAS-Model#137

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions