You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
MPAS QU 60-km ocean experiment reporting NaN/Infinity values at first time step #3
What version of code are you using? I assume it's from the 1.x series, but the exact version (and/or commit if you used git to get the code) would be helpful.
We don't provide any explicit processor decomposition files, so you can't really run with that option on right now. I'm going to continue investigating today, but let me know if this fixes it for you.
The namelist you posted appears different from the default namelist provided in the tar file for the specific test case you're trying to run.
Could you run with the default namelist?
Some other questions:
What versions of the libraries are you using?
What versions of the compilers are you using? (PGI 10.9? and OpenMPI or MVAPICH (version number)?)
Have you tried any of the other test cases, and do the produce similar results? (Specifically the 10km Baroclinic channel one linked here).
Here is some more info, with DEBUG=true, from a concatenation of all the log.*.err files produced using 4 cores for the baroclinic wave experiment.
Reading namelist.input
WARNING: Variable vertCoordMovementWeights not in input file.
WARNING: Variable seaSurfacePressure not in input file.
MPAS IO Error: Bad return value from PIO
Warning: Attribute History not found in grid.nc
Setting History to ''
Vertical coordinate movement is: uniform_stretching
Pressure type is: pressure_and_zmid
WARNING: Variable vertCoordMovementWeights not in input file.
WARNING: Variable seaSurfacePressure not in input file.
MPAS IO Error: Bad return value from PIO
Warning: Attribute History not found in grid.nc
Setting History to ''
Vertical coordinate movement is: uniform_stretching
Pressure type is: pressure_and_zmid
[h175:mpi_rank_1][error_sighandler] Caught error: Floating point exception (sign
al 8)
WARNING: Variable vertCoordMovementWeights not in input file.
WARNING: Variable seaSurfacePressure not in input file.
MPAS IO Error: Bad return value from PIO
Warning: Attribute History not found in grid.nc
Setting History to ''
Vertical coordinate movement is: uniform_stretching
Pressure type is: pressure_and_zmid
[h175:mpi_rank_2][error_sighandler] Caught error: Floating point exception (sign
al 8)
WARNING: Variable vertCoordMovementWeights not in input file.
WARNING: Variable seaSurfacePressure not in input file.
MPAS IO Error: Bad return value from PIO
Warning: Attribute History not found in grid.nc
Setting History to ''
Vertical coordinate movement is: uniform_stretching
Pressure type is: pressure_and_zmid
I am using the version of PIO which was listed in the atmosphere model user's guide by Mike Duda.
Here is the output using the new executable (defined by the changes you listed above):
Reading namelist.input
MPAS IO Error: Bad return value from PIO
Warning: Attribute History not found in grid.nc
Setting History to ''
Vertical coordinate movement is: uniform_stretching
Pressure type is: pressure_and_zmid
MPAS IO Error: Bad return value from PIO
Warning: Attribute History not found in grid.nc
Setting History to ''
Vertical coordinate movement is: uniform_stretching
Pressure type is: pressure_and_zmid
[h175:mpi_rank_1][error_sighandler] Caught error: Floating point exception (signal 8)
MPAS IO Error: Bad return value from PIO
Warning: Attribute History not found in grid.nc
Setting History to ''
Vertical coordinate movement is: uniform_stretching
Pressure type is: pressure_and_zmid
MPAS IO Error: Bad return value from PIO
Warning: Attribute History not found in grid.nc
Setting History to ''
Vertical coordinate movement is: uniform_stretching
Pressure type is: pressure_and_zmid
So, it appears PIO is unhappy. Which version are you using?
I don't think PIO is the problem, that error is because it's trying to read the history attribute from the input file and it doesn't exist.
But, I am currently using version 1.7.2.
A few more questions:
Which build target are you using? (i.e. what's the full make command you give to build the code)
Does the error you're seeing produce any backtrace?
setenv CORE ocean
make pgi DEBUG=true AUTOCLEAN=true
And no, no backtrace information even though I provide $(FC) -traceback $(FFLAGS) -c $*.f90 $(FCINCLUDES) -I../framework -I../operators -I../external/esmf_time_f90 to the Makefile in core_ocean
I was able to reproduce this issue on yellowstone, as that's the only computer that has access to 12.5 of the pgi compilers. I spent a while trying to figure out what the actual issue is, and the problem is a compiler bug.
It was fixed in version 12.6 of the pgi compilers, and is specific to the cases where a non-square matrix is transposed using the transpose() command, and then subsequently passed to matmul().
You can see the version history here: http://www.pgroup.com/support/release_tprs_2012.htm
with the 12.6 comment of:
"Wrong answers when "transpose(derc)" is passed to matmul
Corrected the handling of matmul when the first argument is a transpose of a non-square matrix"
For now I would say we don't support 12.5, since this bug breaks some necessary parts of MPAS-O.
Thanks for reporting this, if you have any comments about the support please let me know as we would like to "streamline" it as best as possible.
Activity
douglasjacobsen commentedon Jul 25, 2013
What version of code are you using? I assume it's from the 1.x series, but the exact version (and/or commit if you used git to get the code) would be helpful.
hrwinterbottom commentedon Jul 25, 2013
I am using MPAS-Release-1.0. The version released 14 June 2013. I grabbed from here: http://mpas-dev.github.io/
-H
douglasjacobsen commentedon Jul 25, 2013
Could you please fill our the bug report template linked in the "Bug Tracker" section of http://mpas-dev.github.io/.
(https://gist.github.com/douglasjacobsen/5868321)
That will get us started at least.
Thanks!
hrwinterbottom commentedon Jul 25, 2013
Version of MPAS (in README.md, core's Registry.xml file, and output file):
MPAS core_ocean version 1.0.
Description of issue:
When compiled in DEBUG mode, the following is reported in some of the log files:
Caught error: Floating point exception (signal 8)
The experiment subsequently fails before the first timestep.
Platform used:
Linux (NOAA hJet machine)
Compiler used:
PGI
Number of processors used:
Both 64 and 256 cores result in same error.
Test case used (if applicable):
namelist.input is as follows, using worldOcean_QU_60km.tar.gz found here: http://mpas-dev.github.io/ocean/release_1.0/release_1.0.html
&time_management
config_do_restart = .false.
config_start_time = "0010-01-01_00:00:00"
config_run_duration = "02:00:00"
config_calendar_type = "360day"
/
&io
config_input_name = "grid.nc"
config_output_name = "output.nc"
config_restart_name = "restart.nc"
config_restart_interval = "0001_00:00:00"
config_output_interval = "01:00:00"
config_stats_interval = "00:01:00"
config_write_stats_on_startup = .true.
config_write_output_on_startup = .true.
config_frames_per_outfile = 1
config_pio_num_iotasks = 0
config_pio_stride = 1
/
&time_integration
config_dt = 10.0
config_time_integrator = "split_explicit"
/
&grid
config_num_halos = 3
config_vert_coord_movement = "uniform_stretching"
config_alter_ICs_for_pbcs = "zlevel_pbcs_off"
config_min_pbc_fraction = 0.10
config_check_ssh_consistency = .true.
/
&decomposition
config_block_decomp_file_prefix = "graph.info.part."
config_number_of_blocks = 0
config_explicit_proc_decomp = .true.
config_proc_decomp_file_prefix = "graph.info.part."
/
&hmix
config_hmix_ScaleWithMesh = .false.
config_maxMeshDensity = -1.0
config_apvm_scale_factor = 0.0
/
&hmix_del2
config_use_mom_del2 = .true.
config_use_tracer_del2 = .false.
config_mom_del2 = 10.0
config_tracer_del2 = 10.0
/
&hmix_del4
config_use_mom_del4 = .false.
config_use_tracer_del4 = .false.
config_mom_del4 = 5.0e13
config_tracer_del4 = 0.0
/
&hmix_Leith
config_use_Leith_del2 = .false.
config_Leith_parameter = 1.0
config_Leith_dx = 60000.0
config_Leith_visc2_max = 2.5e3
/
&standard_GM
config_h_kappa = 0.0
config_h_kappa_q = 0.0
/
&Rayleigh_damping
config_Rayleigh_friction = .false.
config_Rayleigh_damping_coeff = 0.0
/
&vmix
config_convective_visc = 1.0
config_convective_diff = 1.0
/
&vmix_const
config_use_const_visc = .true.
config_use_const_diff = .true.
config_vert_visc = 1.0e-4
config_vert_diff = 1.0e-4
/
&vmix_rich
config_use_rich_visc = .true.
config_use_rich_diff = .true.
config_bkrd_vert_visc = 1.0e-4
config_bkrd_vert_diff = 1.0e-5
config_rich_mix = 0.005
/
&vmix_tanh
config_use_tanh_visc = .false.
config_use_tanh_diff = .false.
config_max_visc_tanh = 2.5e-1
config_min_visc_tanh = 1.0e-4
config_max_diff_tanh = 2.5e-2
config_min_diff_tanh = 1.0e-5
config_zMid_tanh = -100
config_zWidth_tanh = 100
/
&forcing
config_use_monthly_forcing = .false.
config_restoreTS = .false.
config_restoreT_timescale = 90.0
config_restoreS_timescale = 90.0
/
&advection
config_vert_tracer_adv = "stencil"
config_vert_tracer_adv_order = 3
config_horiz_tracer_adv_order = 3
config_coef_3rd_order = 0.25
config_monotonic = .true.
/
&bottom_drag
config_bottom_drag_coeff = 1.0e-2
/
&pressure_gradient
config_pressure_gradient_type = "pressure_and_zmid"
config_density0 = 1014.65
/
&eos
config_eos_type = "linear"
/
&eos_linear
config_eos_linear_alpha = 2.55e-1
config_eos_linear_beta = 7.64e-1
config_eos_linear_Tref = 19.0
config_eos_linear_Sref = 35.0
config_eos_linear_densityref = 1025.022
/
&split_explicit_ts
config_n_ts_iter = 2
config_n_bcl_iter_beg = 1
config_n_bcl_iter_mid = 2
config_n_bcl_iter_end = 2
config_n_btr_subcycles = 20
config_n_btr_cor_iter = 2
config_vel_correction = .true.
config_btr_subcycle_loop_factor = 2
config_btr_gam1_velWt1 = 0.5
config_btr_gam2_SSHWt1 = 1.0
config_btr_gam3_velWt2 = 1.0
config_btr_solve_SSH2 = .false.
/
&debug
config_check_zlevel_consistency = .false.
config_filter_btr_mode = .false.
config_prescribe_velocity = .false.
config_prescribe_thickness = .false.
config_include_KE_vertex = .false.
config_check_tracer_monotonicity = .false.
config_disable_thick_all_tend = .false.
config_disable_thick_hadv = .false.
config_disable_thick_vadv = .false.
config_disable_vel_all_tend = .false.
config_disable_vel_coriolis = .false.
config_disable_vel_pgrad = .false.
config_disable_vel_hmix = .false.
config_disable_vel_windstress = .false.
config_disable_vel_vmix = .false.
config_disable_vel_vadv = .false.
config_disable_tr_all_tend = .false.
config_disable_tr_adv = .false.
config_disable_tr_hmix = .false.
config_disable_tr_vmix = .false.
/
Is this issue reproducible in debug mode (make ... DEBUG=TRUE):
Yes.
douglasjacobsen commentedon Jul 26, 2013
Can you try running with:
config_explicit_proc_decomp = .false.instead of:
config_explicit_proc_decomp = .true.We don't provide any explicit processor decomposition files, so you can't really run with that option on right now. I'm going to continue investigating today, but let me know if this fixes it for you.
hrwinterbottom commentedon Jul 26, 2013
Hi, Doug:
I ran again, with 256 cores, with the processor decomposition set to false and I am still receiving the following:
Caught error: Floating point exception (signal 8)
Thanks.
-H
douglasjacobsen commentedon Jul 26, 2013
The namelist you posted appears different from the default namelist provided in the tar file for the specific test case you're trying to run.
Could you run with the default namelist?
Some other questions:
What versions of the libraries are you using?
What versions of the compilers are you using? (PGI 10.9? and OpenMPI or MVAPICH (version number)?)
Have you tried any of the other test cases, and do the produce similar results? (Specifically the 10km Baroclinic channel one linked here).
hrwinterbottom commentedon Jul 26, 2013
Hi, Doug:
With the 10-km baroclinic wave experiment, grabbed from the link above, I am still getting the following error: Floating point exception (signal 8)
Re: the libraries, here is my info:
(1) PNETCDF: v 1.3.1
(2) PGI: 12.5
(3) MVAPICH2: 1.8-r5609-pgi
-H
hrwinterbottom commentedon Jul 26, 2013
Doug:
Here is some more info, with DEBUG=true, from a concatenation of all the log.*.err files produced using 4 cores for the baroclinic wave experiment.
Reading namelist.input
WARNING: Variable vertCoordMovementWeights not in input file.
WARNING: Variable seaSurfacePressure not in input file.
MPAS IO Error: Bad return value from PIO
Warning: Attribute History not found in grid.nc
Setting History to ''
Vertical coordinate movement is: uniform_stretching
Pressure type is: pressure_and_zmid
WARNING: Variable vertCoordMovementWeights not in input file.
WARNING: Variable seaSurfacePressure not in input file.
MPAS IO Error: Bad return value from PIO
Warning: Attribute History not found in grid.nc
Setting History to ''
Vertical coordinate movement is: uniform_stretching
Pressure type is: pressure_and_zmid
[h175:mpi_rank_1][error_sighandler] Caught error: Floating point exception (sign
al 8)
WARNING: Variable vertCoordMovementWeights not in input file.
WARNING: Variable seaSurfacePressure not in input file.
MPAS IO Error: Bad return value from PIO
Warning: Attribute History not found in grid.nc
Setting History to ''
Vertical coordinate movement is: uniform_stretching
Pressure type is: pressure_and_zmid
[h175:mpi_rank_2][error_sighandler] Caught error: Floating point exception (sign
al 8)
WARNING: Variable vertCoordMovementWeights not in input file.
WARNING: Variable seaSurfacePressure not in input file.
MPAS IO Error: Bad return value from PIO
Warning: Attribute History not found in grid.nc
Setting History to ''
Vertical coordinate movement is: uniform_stretching
Pressure type is: pressure_and_zmid
PIO seems to be unhappy.
-H
douglasjacobsen commentedon Jul 26, 2013
What version of PIO are you using?
Could you try making two small changes in src/core_ocean/Registry.xml?
Change the line:
<var name="vertCoordMovementWeights" type="real" dimensions="nVertLevels" streams="iro" units="unitless"to
<var name="vertCoordMovementWeights" type="real" dimensions="nVertLevels" streams="ro" units="unitless"and the line:
<var name="seaSurfacePressure" type="real" dimensions="nCells Time" streams="ir" units="Pa"to
<var name="seaSurfacePressure" type="real" dimensions="nCells Time" streams="r" units="Pa"After you make these changes you'll have to issue a
make clean CORE=oceanprior to rebuilding.hrwinterbottom commentedon Jul 26, 2013
Hi, Doug:
I am using the version of PIO which was listed in the atmosphere model user's guide by Mike Duda.
Here is the output using the new executable (defined by the changes you listed above):
Reading namelist.input
MPAS IO Error: Bad return value from PIO
Warning: Attribute History not found in grid.nc
Setting History to ''
Vertical coordinate movement is: uniform_stretching
Pressure type is: pressure_and_zmid
MPAS IO Error: Bad return value from PIO
Warning: Attribute History not found in grid.nc
Setting History to ''
Vertical coordinate movement is: uniform_stretching
Pressure type is: pressure_and_zmid
[h175:mpi_rank_1][error_sighandler] Caught error: Floating point exception (signal 8)
MPAS IO Error: Bad return value from PIO
Warning: Attribute History not found in grid.nc
Setting History to ''
Vertical coordinate movement is: uniform_stretching
Pressure type is: pressure_and_zmid
MPAS IO Error: Bad return value from PIO
Warning: Attribute History not found in grid.nc
Setting History to ''
Vertical coordinate movement is: uniform_stretching
Pressure type is: pressure_and_zmid
So, it appears PIO is unhappy. Which version are you using?
-H
douglasjacobsen commentedon Jul 26, 2013
I don't think PIO is the problem, that error is because it's trying to read the history attribute from the input file and it doesn't exist.
But, I am currently using version 1.7.2.
A few more questions:
Which build target are you using? (i.e. what's the full make command you give to build the code)
Does the error you're seeing produce any backtrace?
hrwinterbottom commentedon Jul 26, 2013
The full make command I am using is:
setenv CORE ocean
make pgi DEBUG=true AUTOCLEAN=true
And no, no backtrace information even though I provide$(FC) -traceback $ (FFLAGS) -c $*.f90 $ (FCINCLUDES) -I../framework -I../operators -I../external/esmf_time_f90 to the Makefile in core_ocean
-H
douglasjacobsen commentedon Jul 27, 2013
I was able to reproduce this issue on yellowstone, as that's the only computer that has access to 12.5 of the pgi compilers. I spent a while trying to figure out what the actual issue is, and the problem is a compiler bug.
It was fixed in version 12.6 of the pgi compilers, and is specific to the cases where a non-square matrix is transposed using the transpose() command, and then subsequently passed to matmul().
You can see the version history here: http://www.pgroup.com/support/release_tprs_2012.htm
with the 12.6 comment of:
"Wrong answers when "transpose(derc)" is passed to matmul
Corrected the handling of matmul when the first argument is a transpose of a non-square matrix"
For now I would say we don't support 12.5, since this bug breaks some necessary parts of MPAS-O.
Thanks for reporting this, if you have any comments about the support please let me know as we would like to "streamline" it as best as possible.
I'm going to mark this as closed for now.
39 remaining items
Fix memory leak in mpas_stream_manager.F
Fix memory leak in stream_inquiry.c
Fix memory leak in mpas_io.F
Fix memory leak in mpas_pool_routines.F
Fix memory leak in mpas_framework.F and mpas_domain_routines.F
Fix memory leak in mpas_domain_routines.F
Fix memory leak in xml_stream_parser
Fix memory leak in mpas_io.F
Fix memory leak in mpas_domain_routines.F
Fix memory leak in mpas_pool_routines.F
Fix memory leak in mpas_block_creator.F
Fix memory leak in block_dimension_routines.inc
Fix memory leak in mpas_io.F
Provide default initialization for members of the met_data derived type
Merge branch 'init_atmosphere/met_data_default_init' into develop (PR M…