[mesa-users] kap module and thread-safe conflict

[Bill Paxton]

Hi Rob,

On Jul 13, 2015, at 8:52 AM, Robert FARMER wrote:

>> 2) i believe your are wrong when you mention internal OMP constructs in the kap module -- I don't think there are any. 
> Actually, there are two !$OMP critical constructs in op_load.f and kap_def.f90, but I do not think they are the cause of the issues we observe.

Those are for initialization and for on-demand loading of tables.  The actual evaluation of opacities for given arguments doesn't use any OpenMP constructs.  That is intentional to allow the parallel loop to be done by the caller without introducing nested OpenMP loops.
> 
> Critical sections imply that area is being done serially regardless of thread count

Right.  And those are the only uses of OpenMP constructs in kap.

Perhaps Ehsan can supply a stripped down minimal OpenMP loop calling kap evaluation routines that shows his problem in a way that can be reproduced on mac or linux with gfortran and the standard runtime support for openmp.

b


> 
> 
> Rob
> 
> On Mon, Jul 13, 2015 at 8:46 AM, Ehsan Moravveji <e.moravveji at gmail.com> wrote:
> Thanks Bill for your email.
> 
>> On 13 Jul 2015, at 17:06, Bill Paxton <paxton at kitp.ucsb.edu> wrote:
>> 
>> 1) mesa/star routinely calls kap from OpenMP loops without any problems, so i think you are wrong to suggest that kap fails when called from OpenMP loops.  please elaborate on your claim.
> 
> kap does not fail when OMP_NUM_THREADS>1; instead, the result of making a call to kap for fixed logT and logR values depends on OMP_NUM_THREADS when called from an external program: If I set OMP_NUM_THREADS=1, the result of calling kap routines is just excellent, and I am super-happy with that. However, when I request more number of threads, then the kap output changes. In the latter case, it also occasionally returns NaNs.
> 
> With Geert Jan Bex (from central supercomputer centre of our university, in CC), we spent few hours looking into my parallel block, and we did not find a way to resolve this. 
> Then, our guess was that we either need to compile MESA without -fopenmp, and link to that (as Michael proposed), or we better ask the community for their suggestions.
> 
>> 2) i believe your are wrong when you mention internal OMP constructs in the kap module -- I don't think there are any. 
> Actually, there are two !$OMP critical constructs in op_load.f and kap_def.f90, but I do not think they are the cause of the issues we observe.
> 
>> 3) it is of course possible that the problem is in kap rather than in your code; if so, please let us know so we can fix it.
> Before putting any blame on MESA (that I always avoid to), we tried to inspect my block of code carefully, and we failed to find any flaw. In such a case, the mesa-user community can come to rescue ;-)
> The whole purpose of my email is to consult with experts.
> 
> By the way, is it possible to install a “minimal” version of MESA, in a sense of having the kap module and its dependencies (I think const, chem, crlibm, interp_1d, interp_2d, …)?
> 
> Thanks for your email again, and best regards
> Ehsan.
> 
> 
>> 
>> bill
>> 
>> 
>> 
>>  
>> 
>> On Jul 13, 2015, at 4:33 AM, Ehsan Moravveji wrote:
>> 
>>> Dear MESA users,
>>> I am using and calling the kap module in an external code to get opacities for different temperatures and densities, using v.7678. Therefore, I have a nested do-loop that I can parallelise with OpenMP for (hopefully) faster fetching of opacities for a range of temperatures and densities. 
>>> With only one thread, the script runs OK, and the results are what I expect. The pseudo-code looks like
>>> 
>>>    !$OMP parallel do default(none) private(…) shared(…)
>>>    do logT in a range of temperatures
>>>         do logR in a range of densities
>>> 
>>>              kappa = get_kappa(logT, logR)
>>>              
>>>          enddo
>>>   enddo
>>>   !$OMP end parallel do
>>> 
>>> However, setting OMP_NUM_THREADS > 1, also invokes the internal OMP constructs in the kap module; the result of this is a significant speed up, but very spaghetti results. 
>>> 
>>> The problem is, by default MESA is installed with -fopenmp and I would like to keep it so, because of my other applications of MESA. However, my call to get_kappa() needs to be restricted to use only one thread. 
>>> Geert Jan Bex (in CC) and I also tried calling OMP_set_num_threads(1) before calling get_kappa(), but that did not work either.
>>> 
>>> So, my question is: Is it possible to use MESA modules in external programs that also include OpenMP directives, making sure that the whole computations are thread safe?
>>> I would be grateful if you enlighten me with your comments and ideas.
>>> 
>>> Best regards
>>> Ehsan.
>>> 
>>> ------------------------------------------------------------------------------
>>> Don't Limit Your Business. Reach for the Cloud.
>>> GigeNET's Cloud Solutions provide you with the tools and support that
>>> you need to offload your IT needs and focus on growing your business.
>>> Configured For All Businesses. Start Your Cloud Today.
>>> https://www.gigenetcloud.com/_______________________________________________
>>> mesa-users mailing list
>>> mesa-users at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/mesa-users
>> 
>> ------------------------------------------------------------------------------
>> Don't Limit Your Business. Reach for the Cloud.
>> GigeNET's Cloud Solutions provide you with the tools and support that
>> you need to offload your IT needs and focus on growing your business.
>> Configured For All Businesses. Start Your Cloud Today.
>> https://www.gigenetcloud.com/_______________________________________________
>> mesa-users mailing list
>> mesa-users at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/mesa-users
> 
> 
> ------------------------------------------------------------------------------
> Don't Limit Your Business. Reach for the Cloud.
> GigeNET's Cloud Solutions provide you with the tools and support that
> you need to offload your IT needs and focus on growing your business.
> Configured For All Businesses. Start Your Cloud Today.
> https://www.gigenetcloud.com/
> _______________________________________________
> mesa-users mailing list
> mesa-users at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/mesa-users
> 
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.mesastar.org/pipermail/mesa-users/attachments/20150713/481eb142/attachment.html>