Skip to content

MPAS QU 60-km ocean experiment reporting NaN/Infinity values at first time step #3

Closed

Description

@hrwinterbottom

To whom it may concern:

I am experimenting using the MPAS ocean core (compiled using PGI) for the 60-km global ocean test case found here: http://mpas-dev.github.io/ocean/release_1.0/release_1.0.html .

When compiled in DEBUG mode, the following is reported in some of the log files:

Caught error: Floating point exception (signal 8)

which suggests either a NaN or INF value may be present.

Has anyone set up a similar experiment, using PGI and the 60-km QU ocean and either had success or similar runtime issues?

Thanks in advance.

Henry Winterbottom

Activity

  1. douglasjacobsen commented on Jul 25, 2013

    @douglasjacobsen
    Member

    What version of code are you using? I assume it's from the 1.x series, but the exact version (and/or commit if you used git to get the code) would be helpful.

  2. hrwinterbottom commented on Jul 25, 2013

    @hrwinterbottom
    Author

    I am using MPAS-Release-1.0. The version released 14 June 2013. I grabbed from here: http://mpas-dev.github.io/

    -H

  3. douglasjacobsen commented on Jul 25, 2013

    @douglasjacobsen
    Member

    Could you please fill our the bug report template linked in the "Bug Tracker" section of http://mpas-dev.github.io/.

    (https://gist.github.com/douglasjacobsen/5868321)

    That will get us started at least.

    Thanks!

  4. hrwinterbottom commented on Jul 25, 2013

    @hrwinterbottom
    Author

    Version of MPAS (in README.md, core's Registry.xml file, and output file):

    MPAS core_ocean version 1.0.

    Description of issue:

    When compiled in DEBUG mode, the following is reported in some of the log files:

    Caught error: Floating point exception (signal 8)

    The experiment subsequently fails before the first timestep.

    Platform used:

    Linux (NOAA hJet machine)

    Compiler used:

    PGI

    Number of processors used:

    Both 64 and 256 cores result in same error.

    Test case used (if applicable):

    namelist.input is as follows, using worldOcean_QU_60km.tar.gz found here: http://mpas-dev.github.io/ocean/release_1.0/release_1.0.html

    &time_management
    config_do_restart = .false.
    config_start_time = "0010-01-01_00:00:00"
    config_run_duration = "02:00:00"
    config_calendar_type = "360day"
    /
    &io
    config_input_name = "grid.nc"
    config_output_name = "output.nc"
    config_restart_name = "restart.nc"
    config_restart_interval = "0001_00:00:00"
    config_output_interval = "01:00:00"
    config_stats_interval = "00:01:00"
    config_write_stats_on_startup = .true.
    config_write_output_on_startup = .true.
    config_frames_per_outfile = 1
    config_pio_num_iotasks = 0
    config_pio_stride = 1
    /
    &time_integration
    config_dt = 10.0
    config_time_integrator = "split_explicit"
    /
    &grid
    config_num_halos = 3
    config_vert_coord_movement = "uniform_stretching"
    config_alter_ICs_for_pbcs = "zlevel_pbcs_off"
    config_min_pbc_fraction = 0.10
    config_check_ssh_consistency = .true.
    /
    &decomposition
    config_block_decomp_file_prefix = "graph.info.part."
    config_number_of_blocks = 0
    config_explicit_proc_decomp = .true.
    config_proc_decomp_file_prefix = "graph.info.part."
    /
    &hmix
    config_hmix_ScaleWithMesh = .false.
    config_maxMeshDensity = -1.0
    config_apvm_scale_factor = 0.0
    /
    &hmix_del2
    config_use_mom_del2 = .true.
    config_use_tracer_del2 = .false.
    config_mom_del2 = 10.0
    config_tracer_del2 = 10.0
    /
    &hmix_del4
    config_use_mom_del4 = .false.
    config_use_tracer_del4 = .false.
    config_mom_del4 = 5.0e13
    config_tracer_del4 = 0.0
    /
    &hmix_Leith
    config_use_Leith_del2 = .false.
    config_Leith_parameter = 1.0
    config_Leith_dx = 60000.0
    config_Leith_visc2_max = 2.5e3
    /
    &standard_GM
    config_h_kappa = 0.0
    config_h_kappa_q = 0.0
    /
    &Rayleigh_damping
    config_Rayleigh_friction = .false.
    config_Rayleigh_damping_coeff = 0.0
    /
    &vmix
    config_convective_visc = 1.0
    config_convective_diff = 1.0
    /
    &vmix_const
    config_use_const_visc = .true.
    config_use_const_diff = .true.
    config_vert_visc = 1.0e-4
    config_vert_diff = 1.0e-4
    /
    &vmix_rich
    config_use_rich_visc = .true.
    config_use_rich_diff = .true.
    config_bkrd_vert_visc = 1.0e-4
    config_bkrd_vert_diff = 1.0e-5
    config_rich_mix = 0.005
    /
    &vmix_tanh
    config_use_tanh_visc = .false.
    config_use_tanh_diff = .false.
    config_max_visc_tanh = 2.5e-1
    config_min_visc_tanh = 1.0e-4
    config_max_diff_tanh = 2.5e-2
    config_min_diff_tanh = 1.0e-5
    config_zMid_tanh = -100
    config_zWidth_tanh = 100
    /
    &forcing
    config_use_monthly_forcing = .false.
    config_restoreTS = .false.
    config_restoreT_timescale = 90.0
    config_restoreS_timescale = 90.0
    /
    &advection
    config_vert_tracer_adv = "stencil"
    config_vert_tracer_adv_order = 3
    config_horiz_tracer_adv_order = 3
    config_coef_3rd_order = 0.25
    config_monotonic = .true.
    /
    &bottom_drag
    config_bottom_drag_coeff = 1.0e-2
    /
    &pressure_gradient
    config_pressure_gradient_type = "pressure_and_zmid"
    config_density0 = 1014.65
    /
    &eos
    config_eos_type = "linear"
    /
    &eos_linear
    config_eos_linear_alpha = 2.55e-1
    config_eos_linear_beta = 7.64e-1
    config_eos_linear_Tref = 19.0
    config_eos_linear_Sref = 35.0
    config_eos_linear_densityref = 1025.022
    /
    &split_explicit_ts
    config_n_ts_iter = 2
    config_n_bcl_iter_beg = 1
    config_n_bcl_iter_mid = 2
    config_n_bcl_iter_end = 2
    config_n_btr_subcycles = 20
    config_n_btr_cor_iter = 2
    config_vel_correction = .true.
    config_btr_subcycle_loop_factor = 2
    config_btr_gam1_velWt1 = 0.5
    config_btr_gam2_SSHWt1 = 1.0
    config_btr_gam3_velWt2 = 1.0
    config_btr_solve_SSH2 = .false.
    /
    &debug
    config_check_zlevel_consistency = .false.
    config_filter_btr_mode = .false.
    config_prescribe_velocity = .false.
    config_prescribe_thickness = .false.
    config_include_KE_vertex = .false.
    config_check_tracer_monotonicity = .false.
    config_disable_thick_all_tend = .false.
    config_disable_thick_hadv = .false.
    config_disable_thick_vadv = .false.
    config_disable_vel_all_tend = .false.
    config_disable_vel_coriolis = .false.
    config_disable_vel_pgrad = .false.
    config_disable_vel_hmix = .false.
    config_disable_vel_windstress = .false.
    config_disable_vel_vmix = .false.
    config_disable_vel_vadv = .false.
    config_disable_tr_all_tend = .false.
    config_disable_tr_adv = .false.
    config_disable_tr_hmix = .false.
    config_disable_tr_vmix = .false.
    /

    Is this issue reproducible in debug mode (make ... DEBUG=TRUE):

    Yes.

  5. douglasjacobsen commented on Jul 26, 2013

    @douglasjacobsen
    Member

    Can you try running with:

    config_explicit_proc_decomp = .false.

    instead of:

    config_explicit_proc_decomp = .true.

    We don't provide any explicit processor decomposition files, so you can't really run with that option on right now. I'm going to continue investigating today, but let me know if this fixes it for you.

  6. hrwinterbottom commented on Jul 26, 2013

    @hrwinterbottom
    Author

    Hi, Doug:

    I ran again, with 256 cores, with the processor decomposition set to false and I am still receiving the following:

    Caught error: Floating point exception (signal 8)

    Thanks.

    -H

  7. douglasjacobsen commented on Jul 26, 2013

    @douglasjacobsen
    Member

    The namelist you posted appears different from the default namelist provided in the tar file for the specific test case you're trying to run.

    Could you run with the default namelist?

    Some other questions:
    What versions of the libraries are you using?
    What versions of the compilers are you using? (PGI 10.9? and OpenMPI or MVAPICH (version number)?)
    Have you tried any of the other test cases, and do the produce similar results? (Specifically the 10km Baroclinic channel one linked here).

  8. hrwinterbottom commented on Jul 26, 2013

    @hrwinterbottom
    Author

    Hi, Doug:

    With the 10-km baroclinic wave experiment, grabbed from the link above, I am still getting the following error: Floating point exception (signal 8)

    Re: the libraries, here is my info:

    (1) PNETCDF: v 1.3.1
    (2) PGI: 12.5
    (3) MVAPICH2: 1.8-r5609-pgi

    -H

  9. hrwinterbottom commented on Jul 26, 2013

    @hrwinterbottom
    Author

    Doug:

    Here is some more info, with DEBUG=true, from a concatenation of all the log.*.err files produced using 4 cores for the baroclinic wave experiment.

    Reading namelist.input

    WARNING: Variable vertCoordMovementWeights not in input file.
    WARNING: Variable seaSurfacePressure not in input file.
    MPAS IO Error: Bad return value from PIO
    Warning: Attribute History not found in grid.nc
    Setting History to ''
    Vertical coordinate movement is: uniform_stretching
    Pressure type is: pressure_and_zmid
    WARNING: Variable vertCoordMovementWeights not in input file.
    WARNING: Variable seaSurfacePressure not in input file.
    MPAS IO Error: Bad return value from PIO
    Warning: Attribute History not found in grid.nc
    Setting History to ''
    Vertical coordinate movement is: uniform_stretching
    Pressure type is: pressure_and_zmid
    [h175:mpi_rank_1][error_sighandler] Caught error: Floating point exception (sign
    al 8)
    WARNING: Variable vertCoordMovementWeights not in input file.
    WARNING: Variable seaSurfacePressure not in input file.
    MPAS IO Error: Bad return value from PIO
    Warning: Attribute History not found in grid.nc
    Setting History to ''
    Vertical coordinate movement is: uniform_stretching
    Pressure type is: pressure_and_zmid
    [h175:mpi_rank_2][error_sighandler] Caught error: Floating point exception (sign
    al 8)
    WARNING: Variable vertCoordMovementWeights not in input file.
    WARNING: Variable seaSurfacePressure not in input file.
    MPAS IO Error: Bad return value from PIO
    Warning: Attribute History not found in grid.nc
    Setting History to ''
    Vertical coordinate movement is: uniform_stretching
    Pressure type is: pressure_and_zmid

    PIO seems to be unhappy.

    -H

  10. douglasjacobsen commented on Jul 26, 2013

    @douglasjacobsen
    Member

    What version of PIO are you using?

    Could you try making two small changes in src/core_ocean/Registry.xml?

    Change the line:
    <var name="vertCoordMovementWeights" type="real" dimensions="nVertLevels" streams="iro" units="unitless"
    to
    <var name="vertCoordMovementWeights" type="real" dimensions="nVertLevels" streams="ro" units="unitless"

    and the line:
    <var name="seaSurfacePressure" type="real" dimensions="nCells Time" streams="ir" units="Pa"
    to
    <var name="seaSurfacePressure" type="real" dimensions="nCells Time" streams="r" units="Pa"

    After you make these changes you'll have to issue a make clean CORE=ocean prior to rebuilding.

  11. hrwinterbottom commented on Jul 26, 2013

    @hrwinterbottom
    Author

    Hi, Doug:

    I am using the version of PIO which was listed in the atmosphere model user's guide by Mike Duda.

    Here is the output using the new executable (defined by the changes you listed above):

    Reading namelist.input

    MPAS IO Error: Bad return value from PIO
    Warning: Attribute History not found in grid.nc
    Setting History to ''
    Vertical coordinate movement is: uniform_stretching
    Pressure type is: pressure_and_zmid
    MPAS IO Error: Bad return value from PIO
    Warning: Attribute History not found in grid.nc
    Setting History to ''
    Vertical coordinate movement is: uniform_stretching
    Pressure type is: pressure_and_zmid
    [h175:mpi_rank_1][error_sighandler] Caught error: Floating point exception (signal 8)
    MPAS IO Error: Bad return value from PIO
    Warning: Attribute History not found in grid.nc
    Setting History to ''
    Vertical coordinate movement is: uniform_stretching
    Pressure type is: pressure_and_zmid
    MPAS IO Error: Bad return value from PIO
    Warning: Attribute History not found in grid.nc
    Setting History to ''
    Vertical coordinate movement is: uniform_stretching
    Pressure type is: pressure_and_zmid

    So, it appears PIO is unhappy. Which version are you using?

    -H

  12. douglasjacobsen commented on Jul 26, 2013

    @douglasjacobsen
    Member

    I don't think PIO is the problem, that error is because it's trying to read the history attribute from the input file and it doesn't exist.

    But, I am currently using version 1.7.2.

    A few more questions:
    Which build target are you using? (i.e. what's the full make command you give to build the code)
    Does the error you're seeing produce any backtrace?

  13. hrwinterbottom commented on Jul 26, 2013

    @hrwinterbottom
    Author

    The full make command I am using is:

    setenv CORE ocean
    make pgi DEBUG=true AUTOCLEAN=true

    And no, no backtrace information even though I provide $(FC) -traceback $(FFLAGS) -c $*.f90 $(FCINCLUDES) -I../framework -I../operators -I../external/esmf_time_f90 to the Makefile in core_ocean

    -H

  14. douglasjacobsen commented on Jul 27, 2013

    @douglasjacobsen
    Member

    I was able to reproduce this issue on yellowstone, as that's the only computer that has access to 12.5 of the pgi compilers. I spent a while trying to figure out what the actual issue is, and the problem is a compiler bug.

    It was fixed in version 12.6 of the pgi compilers, and is specific to the cases where a non-square matrix is transposed using the transpose() command, and then subsequently passed to matmul().

    You can see the version history here: http://www.pgroup.com/support/release_tprs_2012.htm
    with the 12.6 comment of:
    "Wrong answers when "transpose(derc)" is passed to matmul
    Corrected the handling of matmul when the first argument is a transpose of a non-square matrix"

    For now I would say we don't support 12.5, since this bug breaks some necessary parts of MPAS-O.

    Thanks for reporting this, if you have any comments about the support please let me know as we would like to "streamline" it as best as possible.

    I'm going to mark this as closed for now.

  15. 39 remaining items

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions