[ecco-support] Restarts not matching control simulation

[Matthew Mazloff]

Hi Max

It may be chaos originating from roundoff differences when using different sets of machines in parallel computations. 
Do they differ on the first day of simulation or does it takes a few weeks/months before you notice it?

Matt



> On Jul 16, 2024, at 8:22 AM, Max Holloway <max at planetarytech.com> wrote:
> 
> Dear ECCO-support team,
> 
> I am looking for some help in debugging my ECCO setup to allow model restarts and conserve physical fields. I am running the ECCO LLC270 iteration-42 state estimate described in Carroll et al. (2020), which spans the years 1992–2017, on a small MPI cluster (13 machines, 59 processes each) on Amazon Web Services (AWS) Cloud Computing Services.
> 
> I have run a control simulation and am restarting the model from pickup files at various timesteps with an added alkalinity forcing. The problem I am having is that the physical fields of the restarted simulation do not match the control simulation (without restarts). I get a very small deviation in physics. So far I have got around this by re-running the baseline simulation with identical restart points, which allows for model conservation, but will result in a deviation in the solution of the ocean state estimate.
> 
> Any help or suggestions on why my restarts differ from my control simulation would be much appreciated.
> 
> In data I have ensured that pChkptFreq matches dumpFreq in the pickup and output files and I have set pickupStrictlyMatch=TRUE and writePickupAtEnd=TRUE.
> For the restarts, in data.ctrl I have set doSinglePrecTapelev=.FALSE., doinitxx = .FALSE.,. and doPackDiag = .FALSE.,
> 
> Many thanks,
> Max
> 
> _______________________________________________
> ecco-support mailing list
> ecco-support at mit.edu
> https://urldefense.com/v3/__https://mailman.mit.edu/mailman/listinfo/ecco-support__;!!Mih3wA!EDTXhoDP8naMoO0rBkGYuZZMWOIphu8ueDdLUhYC9usGLMBV-ngkd9AcG8o8ViHBwr0sBjG3rrYfy5aoyg$