You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
New COMPASS EC60to30, NAEC60to30cr8 and QU60 test cases don't spin up #235
As part of #229, I tried to run all global_ocean (but not global_ocean_V1) test cases. Most test cases ran to completion or could be fixed with small tweaks to the configuration that I will include in #229. The cases that are broken beyond seemingly easy repair are:
The following is a list of tests that failed (but for reasons I believe are unrelated to this PR):
-o ocean -c global_ocean -r EC60to30 -t spin_up during simulation because of CFL violation:
-o ocean -c global_ocean -r NAEC60to30cr8 -t spin_up: the final (test_final_settings) phase seems to be misconfigured in various ways. Links to restart and Restart_timestamp are missing. timeSeriesStatsMonthly is enabled but there isn't a corresponding stream. It attempts to change to monthly restarts but the last restart is at 0001-01-07, which isn't an allowable combination in MPAS.
-o ocean -c global_ocean -r QU60 -t spin_up during simulation for unknown reasons during the first time step
I don't believe any of these issues are related to #229 but haven't had time to run the tests with ocean/develop to be completely sure.
@mark-petersen, I think you are likely to be the best person to handle these but @sbrus89 was also involved in creating some of these cases so I've assigned you both.
@xylar I ran the EC60to30 and NAEC60to30cr8 a couple of months ago on another branch. At that point, test_final_settings wasn't configured. I was copying files and manually changing the timing in the restart files. I also ended up using the NAEC60to30 spin up for my JIGSAW EC60to30 run. I don't remember what the issue was with the EC60to30 setup, but it was enough of a problem that I decided not to use it for my simulations. Unless something has changed since I ran those simulations, I imagine the issues are still there.
@kristin-hoch, do you have a way to look up what settings you used for your spin-up? It's a pretty big problem in my view if the test cases in the repo don't actually work. It makes it particularly hard to test when things get updated like #229. So it would be really great if we could fix these test cases.
Thanks @kristin-hoch. While I could imagine that working, I think it's probably way overkill since the 8 km resolution for the NAEC60to30cr8 setup requires really small time steps and a lot more stages of spin-up.
I'm working on a 3-stage spin-up of the EC60to30 setup right now. I'll work on the QU60 once that's done. I'll see if the NAEC60to30cr8 spin-up is still broken after recent merges and see if I can fix it (maybe with help from @sbrus89) if so.
@kristin-hoch, I tried your idea (copying the same spin-up procedure as in NAEC60to30cr8 and using it for EC60to30). Perhaps it has changed but I get a blow-up during spin_up6
I also tried running my own tests where I used dt = 10 min and dt_br = 20 seconds. I tried ramping down from Rayleigh damping of 1e-4 to zero without increasing the time step but also got a blow-up when I went to 5e-6.
I am very much open to suggestions from anyone who has experience spinning up Jigsaw runs because I'm stuck.
I'm giving up on the EC60to30 for now. I tried dt = 10 min and dt_br = 20:
config_Rayleigh_damping_coeff = 1.0e-4 for 20 days
config_Rayleigh_damping_coeff = 5.0e-5 for 20 days
config_Rayleigh_damping_coeff = 2.0e-5 for 10 days
config_Rayleigh_damping_coeff = 5.0e-6 for 10 days
...
During stage 4, the CFL keeps spiking to dangerous levels, though the simulation never quite blows up. I could try config_Rayleigh_damping_coeff = 1.0e-5 but I don't think that would then help when I halve the coefficient later on. I just think there's something wrong...
________________________________
From: Xylar Asay-Davis <notifications@github.com>
Sent: Monday, May 13, 2019 12:53:57 PM
To: MPAS-Dev/MPAS-Model
Cc: Hoch, Kristin; Mention
Subject: Re: [MPAS-Dev/MPAS-Model] New COMPASS EC60to30, NAEC60to30cr8 and QU60 test cases don't spin up (#235)
I'm giving up on the EC60to30 for now. I tried dt = 10 min and dt_br = 20:
1. config_Rayleigh_damping_coeff = 1.0e-4 for 20 days
2. config_Rayleigh_damping_coeff = 5.0e-5 for 20 days
3. config_Rayleigh_damping_coeff = 2.0e-5 for 10 days
4. config_Rayleigh_damping_coeff = 5.0e-6 for 10 days
5. ...
During stage 4, the CFL keeps spiking to dangerous levels, though the simulation never quite blows up. I could try config_Rayleigh_damping_coeff = 1.0e-5 but I don't think that would then help when I halve the coefficient later on. I just think there's something wrong...
Here's a graph of the CFL:
[cfl]<https://user-images.githubusercontent.com/4179064/57646588-359b0300-75c1-11e9-986a-02ab2bda4e5d.png>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub<#235?email_source=notifications&email_token=AKE6JCPLHJU2WX725SGABT3PVG2MLA5CNFSM4HLJG242YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODVJHJNI#issuecomment-491943093>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AKE6JCM3JEKG37ORNUNFL4TPVG2MLANCNFSM4HLJG24Q>.
I might have found the problem with the EC60to30. Because the layer depth file is in cm, there are two conversion factors in the namelist for the initial state that need to be set. It looks like by default only one is correct. I ran into a similar problem with the Labrador Sea mesh that Andrew is testing.
In the namelist in initial_state, try changing config_global_ocean_tracer_depth_conversion_factor to 0.01
@sbrus89 and @mark-petersen, I looked further into fixing the issues with test_final_settings in NAEC60to30cr8. (I suspect the issues may be the same with the newly added variant of that test case, but I'm not sure.)
I don't have time to fix this fully but the necessary steps I can foresee are:
continue the spin-up until 0001-02-01 so you're at an even month boundary
add the template for timeSeriesStatsMonthly to the stream section of the config file as well so the output stream is defined
set the timeSeriesStatsMonthlyRestart stream name in the namelist to 'none'
there may also be a missing stream for the eddy stats, or a stream may need to be set to 'none' there
I'm not seeing the issue with linking restart files that I mentioned above so that may have been some kind of an error on my part in my earlier testing.
An alternative to all this would be to disable/remove timeSeriesStatsMonthly so you didn't need to do this stage over an even, full month.
@kristin-hoch, I was just thinking about that. I guess that's better than trying to use a restart as as the new initial condition because the velocity would be set back to zero?
I'm troubled by the manual nature of that step so I think we would need a script to do that automatically as part of COMPASS. Until we can just run the spin_up phase of each test out of the box, I consider them to be broken because it's too hard for anyone but the test's creator to know what the manual steps are.
Xylar,
I agree that it would need to be automated. I also think it might be confusing for a user who doesn't realize that restart_0001-01-01 is after restart 0001-01-04.
Activity
xylar commentedon May 7, 2019
@mark-petersen, I think you are likely to be the best person to handle these but @sbrus89 was also involved in creating some of these cases so I've assigned you both.
mark-petersen commentedon May 7, 2019
@kristin-hoch and I spun up the jigsaw EC60to30 a month ago, we will make sure the spin-up sequence matches what we did for a successful run. Thanks.
kristin-hoch commentedon May 7, 2019
@xylar I ran the EC60to30 and NAEC60to30cr8 a couple of months ago on another branch. At that point, test_final_settings wasn't configured. I was copying files and manually changing the timing in the restart files. I also ended up using the NAEC60to30 spin up for my JIGSAW EC60to30 run. I don't remember what the issue was with the EC60to30 setup, but it was enough of a problem that I decided not to use it for my simulations. Unless something has changed since I ran those simulations, I imagine the issues are still there.
xylar commentedon May 9, 2019
@kristin-hoch, do you have a way to look up what settings you used for your spin-up? It's a pretty big problem in my view if the test cases in the repo don't actually work. It makes it particularly hard to test when things get updated like #229. So it would be really great if we could fix these test cases.
kristin-hoch commentedon May 9, 2019
@xylar
I used a NAEC60to30cr8 spin up setup. I don't believe I changed anything there other than linking to the EC60to30 initialization files.
I don't remember why I didn't originally use the EC60to30 spin up. @mark-petersen ran the spin up initially.
xylar commentedon May 13, 2019
Thanks @kristin-hoch. While I could imagine that working, I think it's probably way overkill since the 8 km resolution for the NAEC60to30cr8 setup requires really small time steps and a lot more stages of spin-up.
I'm working on a 3-stage spin-up of the EC60to30 setup right now. I'll work on the QU60 once that's done. I'll see if the NAEC60to30cr8 spin-up is still broken after recent merges and see if I can fix it (maybe with help from @sbrus89) if so.
xylar commentedon May 13, 2019
@kristin-hoch, I tried your idea (copying the same spin-up procedure as in NAEC60to30cr8 and using it for EC60to30). Perhaps it has changed but I get a blow-up during
spin_up6I also tried running my own tests where I used dt = 10 min and dt_br = 20 seconds. I tried ramping down from Rayleigh damping of 1e-4 to zero without increasing the time step but also got a blow-up when I went to 5e-6.
I am very much open to suggestions from anyone who has experience spinning up Jigsaw runs because I'm stuck.
xylar commentedon May 13, 2019
I'm giving up on the EC60to30 for now. I tried dt = 10 min and dt_br = 20:
During stage 4, the CFL keeps spiking to dangerous levels, though the simulation never quite blows up. I could try config_Rayleigh_damping_coeff = 1.0e-5 but I don't think that would then help when I halve the coefficient later on. I just think there's something wrong...
Here's a graph of the CFL:

kristin-hoch commentedon May 13, 2019
kristin-hoch commentedon May 13, 2019
I might have found the problem with the EC60to30. Because the layer depth file is in cm, there are two conversion factors in the namelist for the initial state that need to be set. It looks like by default only one is correct. I ran into a similar problem with the Labrador Sea mesh that Andrew is testing.
In the namelist in initial_state, try changing config_global_ocean_tracer_depth_conversion_factor to 0.01
xylar commentedon May 14, 2019
@kristin-hoch, thank you so, so much for figuring that out!
With this change, things worked great with the default spin-up.
xylar commentedon May 14, 2019
@mark-petersen, the QU240 and QU60 test cases appear to now be using the 80-layer grid. Is that on purpose?
xylar commentedon May 14, 2019
@sbrus89 and @mark-petersen, I looked further into fixing the issues with
test_final_settingsinNAEC60to30cr8. (I suspect the issues may be the same with the newly added variant of that test case, but I'm not sure.)I don't have time to fix this fully but the necessary steps I can foresee are:
streamsection of the config file as well so the output stream is definedtimeSeriesStatsMonthlyRestartstream name in the namelist to 'none'I'm not seeing the issue with linking restart files that I mentioned above so that may have been some kind of an error on my part in my earlier testing.
An alternative to all this would be to disable/remove timeSeriesStatsMonthly so you didn't need to do this stage over an even, full month.
kristin-hoch commentedon May 14, 2019
Xylar,
My solution to this has been to strip the xtime variable out of the last restart file from the spin up and rename it to 0001-01-01.
xylar commentedon May 14, 2019
@kristin-hoch, I was just thinking about that. I guess that's better than trying to use a restart as as the new initial condition because the velocity would be set back to zero?
I'm troubled by the manual nature of that step so I think we would need a script to do that automatically as part of COMPASS. Until we can just run the
spin_upphase of each test out of the box, I consider them to be broken because it's too hard for anyone but the test's creator to know what the manual steps are.kristin-hoch commentedon May 14, 2019
Xylar,
I agree that it would need to be automated. I also think it might be confusing for a user who doesn't realize that restart_0001-01-01 is after restart 0001-01-04.
xylar commentedon May 7, 2020
Closing this because QU60 was removed in #530 and spin-up of the CUSP mesh is the subject of #525