[ecco-support] [EXTERNAL] Restarts not matching control simulation

[Zhang, Hong (US 398K)]

> On Jul 16, 2024, at 9:38 AM, Matthew Mazloff <mmazloff at ucsd.edu> wrote:
> 
> Hi Max
> 
> It may be chaos originating from roundoff differences when using different sets of machines in parallel computations. 
> Do they differ on the first day of simulation or does it takes a few weeks/months before you notice it?
> 
> Matt
> 
Hi Max,
Matt's’ comments make sense.
According to our experience,
using different types of node will cause 6-th or 7-th digit difference in output, like
https://github.com/MITgcm-contrib/ecco_darwin/blob/c0767f818c7ebcea9a226d0b8d2ab412dc2b5b61/v04/llc270_JAMES_paper/input/job_ECCO_darwin#L2
vs
https://github.com/MITgcm-contrib/ecco_darwin/blob/c0767f818c7ebcea9a226d0b8d2ab412dc2b5b61/v04/llc270_JAMES_paper/input/job_ECCO_darwin#L3
due to the compilation  w/ “O2” flag
https://github.com/MITgcm/MITgcm/blob/acacc28f7fa33925e71abf11d29f2fedab031e4e/tools/build_options/linux_amd64_ifort%2Bmpi_ice_nas#L46

Curious to know which ECCO-Darwin version you used
https://github.com/MITgcm-contrib/ecco_darwin/tree/master/v04/llc270_JAMES_paper
or 
https://github.com/MITgcm-contrib/ecco_darwin/tree/master/v05/llc270
Maybe we can take a further look.

Hong