[mesa-users] variability in runs

[Bill Paxton]

Hi,

MESA-USERS: Please report variability in runs!   

As Mike points out in his email below, in some parallel jobs variation is expected.
But it turns out that for mesa, I'm not using any construct that would legitimately
produce variation in runs.  Any variation is sign of a bug somewhere!!!

Just to be clear, I'm talking about running with the exact same everything
(starting model, inlist, version of code, hardware, etc) and getting different numerical
results on different runs.

If (when!) you notice such variability, don't just ignore it please.
Instead, send me inlist, starting model, and a description of what
you have found so I can try to reproduce the problem.  

Thanks,
Bill




On Apr 26, 2012, at 5:42 AM, mzingale at mail.astro.sunysb.edu wrote:

> one comment on parallel jobs.  Sometimes, when you split the work across processors you will lose the constancy of results across processors because in parallel reductions, the numbers may be added up in different orders from one run to the next -- i.e. there is no guarantee that you always add things in the order of processor 1 + 2 + 3 + ...  If there is an extra load on processor 1 (for example), then it may take a little longer and you add things like 2 + 3 + 1 + ...  this gives a slight difference due to roundoff which can be amplified in highly nonlinear problems.  I am not sure if the OpenMP standard details what should be done in these situations.  I think with MPI, there is no guarantee and there, network traffic also comes into play.
> 
> Mike
> 
> 
> On Wed, 25 Apr 2012, andres suarez wrote:
> 
>> Thanks for the advice Bill.
>> For the time being I took the shortest solution: since my runs were not
>> really long, I used only one thread, as Richard suggested, and get the same
>> behaviour in different runs.
>> I'd like to help in the debugging process; as you may have noticed already,
>> there is slight variability since the very first model, using the inlist I
>> previously attached, making me believe that the variability comes from a
>> propagation of this small differences. After your comments, I guess the
>> problem is then in the linear algebra solver, but I don't really know where
>> to follow up.
>> Regards,
>> a.
>> 
>>      On Wed, Apr 25, 2012 at 12:07 PM, Bill Paxton
>>      <paxton at kitp.ucsb.edu> wrote:
>> 
>>            On Apr 23, 2012, at 12:11 PM, andres suarez wrote:
>> 
>>            > I am running mesa version 3851 on a linux platform
>>            on an x86_64 multi-processor computer, using ifort.
>>            >
>> Hi Andres,
>> I've worked on your problem some -- no solution as yet, but a
>> couple of options to get around it.
>> I don't see the variability with gfortran, so the easiest
>> work-around is to switch to it (consider using the mesa SDK).
>> If you are a die-hard ifort fan, you can avoid the bug by using
>> the slower banded lapack option for doing the linear algebra in
>> star instead of the default faster option that uses multicore
>> parallelism.    To do this, add these to &controls in your
>> inlist:
>>         small_mtx_decsol = 'lapack'
>>         large_mtx_decsol = 'lapack'
>> We'll of course let you know if we can find a better solution.
>>  Hopefully it is a bug in my code rather than a compiler bug in
>> ifort, but so far I haven't been able to find it.  Help in
>> debugging is welcome!!!!
>> -Bill
>>