[ecco-support] Restarts not matching control simulation

[Max Holloway]

Hi Matt,

Thank you for your reply. I've just re-run with daily output and the
anomaly starts on the first day after the restart. The anomalies look to be
consistent between restarts, so are not generated randomly. I will try
running on a single processor to see if this also causes a problem.

Best,
Max

On Tue, Jul 16, 2024 at 5:38 PM Matthew Mazloff <mmazloff at ucsd.edu> wrote:

> Hi Max
>
> It may be chaos originating from roundoff differences when using different
> sets of machines in parallel computations.
> Do they differ on the first day of simulation or does it takes a few
> weeks/months before you notice it?
>
> Matt
>
>
>
> > On Jul 16, 2024, at 8:22 AM, Max Holloway <max at planetarytech.com> wrote:
> >
> > Dear ECCO-support team,
> >
> > I am looking for some help in debugging my ECCO setup to allow model
> restarts and conserve physical fields. I am running the ECCO LLC270
> iteration-42 state estimate described in Carroll et al. (2020), which spans
> the years 1992–2017, on a small MPI cluster (13 machines, 59 processes
> each) on Amazon Web Services (AWS) Cloud Computing Services.
> >
> > I have run a control simulation and am restarting the model from pickup
> files at various timesteps with an added alkalinity forcing. The problem I
> am having is that the physical fields of the restarted simulation do not
> match the control simulation (without restarts). I get a very small
> deviation in physics. So far I have got around this by re-running the
> baseline simulation with identical restart points, which allows for model
> conservation, but will result in a deviation in the solution of the ocean
> state estimate.
> >
> > Any help or suggestions on why my restarts differ from my control
> simulation would be much appreciated.
> >
> > In data I have ensured that pChkptFreq matches dumpFreq in the pickup
> and output files and I have set pickupStrictlyMatch=TRUE and
> writePickupAtEnd=TRUE.
> > For the restarts, in data.ctrl I have set doSinglePrecTapelev=.FALSE.,
> doinitxx = .FALSE.,. and doPackDiag = .FALSE.,
> >
> > Many thanks,
> > Max
> >
> > _______________________________________________
> > ecco-support mailing list
> > ecco-support at mit.edu
> >
> https://urldefense.com/v3/__https://mailman.mit.edu/mailman/listinfo/ecco-support__;!!Mih3wA!EDTXhoDP8naMoO0rBkGYuZZMWOIphu8ueDdLUhYC9usGLMBV-ngkd9AcG8o8ViHBwr0sBjG3rrYfy5aoyg$
>
>
> _______________________________________________
> ecco-support mailing list
> ecco-support at mit.edu
> https://mailman.mit.edu/mailman/listinfo/ecco-support
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.mit.edu/pipermail/ecco-support/attachments/20240717/24e24e80/attachment-0001.htm>