[mesa-users] update on mesa platform dependency bugs: natural log function different on Mac 10.6.8 vs 10.8.5
alexander.heger at monash.edu
Mon Nov 25 15:01:35 EST 2013
Maybe I should add some of my experience with that.
For A long time I was able to able to run the some binary executable
(of KEPLER) on the first Opteron and the following few generations of
Opterons. Across many updates of the OS. probably the maths library
was statically linked. Then came a day when it failed with the next
At the same time, I saw the the same binary executable would already
give different results on different *steppings* of the same Intel CPU
generation. These days things are even more trick with microcode
updates on CPUs.
Recently I could get the same results on all my Ivy Bridge CPUs (using
same compiler, flags (O3, fast, gcc 4.8) and code, however
So, you don't even have to look at compiler flags, just the CPU makes
a difference for the same binary executable.
For many math operations the CPUs have internal tables to estimate a
first results to a maybe 7 digits accuracy. The the result is
"sharpened" by some algorithms. Either may change, and this is where
some of the noise originates. All of this may not be perfect.
Accuracy on the last few digits is sacrificed for the sake of speed.
The internal 80 bit and pc64 flag mentioned before of course also make
difference (I doubt the latter is a 100% fix in all cases).
So, the only way to make sure to get the same results is not just
about compiler flags, in reality you need to use the same CPU and
don't do kernel updates which could install microcode updates to your
On Tue, Nov 26, 2013 at 4:04 AM, Stan Woosley <woosley at ucolick.org> wrote:
> Since this seems to have become a forum for discussing issues related to machine
> hardware, compilers, and variable results, I would like to mention my experience
> (and that of my colleagues and students) with a quite different code used for stellar
> evolution - KEPLER.
> KEPLER is about 35 years old now and has been run on many dozens of platforms
> and a comparable number of compilers and for many applications. We have seen the "noise"
> reported in this thread time and again, and have often tried, without great success to make
> it go away. gfortran gives different results from the intel compiler and linux platforms
> give different results than Apples. Sometimes we have spent months trying to make
> this go away. We have tried compiling with O1, O2, O0. We have tried ieee flags, fp-precise.
> We have gone through and made sure every floating point variable has a d0 at the end.
> Still we get different results. And I am sorry to say those differences are not always microscopic.
> Real differences in presupernova structure can result from running the same code with the
> same compiler and flags on different machines.
> And even if we could make these machine based differences go away, the answer would
> still be chaotic, because these same sorts of differences appear if you change the zoning
> or the time steps or if you change the mass a tiny bit.
> The pre supernova structure of a massive star around 20 Msun can have a significantly
> different core structure if you change its mass by 0.1 Msun or even 0.01 Msun. The
> main reason, probably is the complex interplay of convection during the many burning episodes
> a star goes through, but there may be others. I have come to believe that stellar
> evolution in nature herself is chaotic. Tug and I will post a paper this week
> that discusses a bit of this among other things (the paper is on preSN compactness).
> The only way to get the same answer is to run exactly the same initial model with
> the same code on the same machine with the same compiler and flags.
> Stated another way, if Nomoto, Arnett, Limongi, Hirshi, and Heger (among many others) all ran
> exactly the same massive preSN star with as close a set of physics as we could muster, the
> outcomes will be different. While I do not rule out significant bugs still being
> in KEPLER, or some critical math lib or mixing routine coded in a bad way, the practical
> lessons we have learned are:
> 1) For any published survey always run all the stars on the same machine with the
> same compiler and flags (and version of the code)
> 2) Even then, the answer only has statistical sense (as Tomek just said), so you need to run a lot
> of models. Don't run just 3 and think you have adequately sampled the outcome.
> 3) Avoid, as much as possible, stopping and restarting jobs - as Dave already noted.
> Run all the way through in one calculation if you can. Starting from restart
> dumps changes the answer. (this is especially true of KEPLER because it
> is an implicit hydro code and uses derivatives that are not stored
> in the dump but generated on the fly. This could be improved by storing the derivatives).
> 4) And of course use conservative compiler flags, stable platforms and compilers, etc.
> We will be following these discussions intently to see if some optimal set of flags
> and compilers emerges. Alex, in particular thinks the INTEL compiler may produce
> more variable results than gfortran, but our testing has been sparse.
More information about the Mesa-users