[mesa-users] kap module and thread-safe conflict

Mon Jul 13 20:08:19 EDT 2015

Hi folks,

There is a program called sample_kap in kap/test.  The purpose of this code
was (and still is) to demonstrate how to use the kap module in your code.

It reads in an AGB star model and then evaluates both Type1 and Type2
opacities in a single, parallel loop.

I attach an updated version, consistent with MESA r7632.  The version in
MESA was out of date in a few places.

It produces identical results when compiled with the MESA SDK and then run
with 1, 2, 4, 8, and 16 threads.

If you encounter any problems with it, please let me know.

Aaron

On Tue, Jul 14, 2015 at 2:17 AM, Bill Paxton <paxton at kitp.ucsb.edu> wrote:

> Hi Rob,
>
> On Jul 13, 2015, at 8:52 AM, Robert FARMER wrote:
>
> 2) i believe your are wrong when you mention internal OMP constructs in
> the kap module -- I don't think there are any.
>
> Actually, there are two !$OMP critical constructs in op_load.f and
> kap_def.f90, but I do not think they are the cause of the issues we observe.
>
>
> Those are for initialization and for on-demand loading of tables.  The
> actual evaluation of opacities for given arguments doesn't use any OpenMP
> constructs.  That is intentional to allow the parallel loop to be done by
> the caller without introducing nested OpenMP loops.
>
>
> Critical sections imply that area is being done serially regardless of
> thread count
>
>
> Right.  And those are the only uses of OpenMP constructs in kap.
>
> Perhaps Ehsan can supply a stripped down minimal OpenMP loop calling kap
> evaluation routines that shows his problem in a way that can be reproduced
> on mac or linux with gfortran and the standard runtime support for openmp.
>
> b
>
>
>
>
> Rob
>
> On Mon, Jul 13, 2015 at 8:46 AM, Ehsan Moravveji <e.moravveji at gmail.com>
> wrote:
>
>> Thanks Bill for your email.
>>
>> On 13 Jul 2015, at 17:06, Bill Paxton <paxton at kitp.ucsb.edu> wrote:
>>
>> 1) mesa/star routinely calls kap from OpenMP loops without any problems,
>> so i think you are wrong to suggest that kap fails when called from OpenMP
>> loops.  please elaborate on your claim.
>>
>>
>> kap does not fail when OMP_NUM_THREADS>1; instead, the result of making a
>> call to kap for fixed logT and logR values depends on OMP_NUM_THREADS when
>> called from an external program: If I set OMP_NUM_THREADS=1, the result of
>> calling kap routines is just excellent, and I am super-happy with that.
>> However, when I request more number of threads, then the kap output
>> changes. In the latter case, it also occasionally returns NaNs.
>>
>> With Geert Jan Bex (from central supercomputer centre of our university,
>> in CC), we spent few hours looking into my parallel block, and we did not
>> find a way to resolve this.
>> Then, our guess was that we either need to compile MESA without -fopenmp,
>> and link to that (as Michael proposed), or we better ask the community for
>> their suggestions.
>>
>> 2) i believe your are wrong when you mention internal OMP constructs in
>> the kap module -- I don't think there are any.
>>
>> Actually, there are two !$OMP critical constructs in op_load.f and
>> kap_def.f90, but I do not think they are the cause of the issues we observe.
>>
>> 3) it is of course possible that the problem is in kap rather than in
>> your code; if so, please let us know so we can fix it.
>>
>> Before putting any blame on MESA (that I always avoid to), we tried to
>> inspect my block of code carefully, and we failed to find any flaw. In such
>> a case, the mesa-user community can come to rescue ;-)
>> The whole purpose of my email is to consult with experts.
>>
>> By the way, is it possible to install a “minimal” version of MESA, in a
>> sense of having the kap module and its dependencies (I think const, chem,
>> crlibm, interp_1d, interp_2d, …)?
>>
>> Thanks for your email again, and best regards
>> Ehsan.
>>
>>
>>
>> bill
>>
>>
>>
>>
>>
>> On Jul 13, 2015, at 4:33 AM, Ehsan Moravveji wrote:
>>
>>  Dear MESA users,
>> I am using and calling the kap module in an external code to get
>> opacities for different temperatures and densities, using v.7678.
>> Therefore, I have a nested do-loop that I can parallelise with OpenMP for
>> (hopefully) faster fetching of opacities for a range of temperatures and
>> densities.
>> With only one thread, the script runs OK, and the results are what I
>> expect. The pseudo-code looks like
>>
>>     !$OMP parallel do default(none) private(…) shared(…)
>>    do logT in a range of temperatures
>>         do logR in a range of densities
>>
>>               kappa = get_kappa(logT, logR)
>>
>>          enddo
>>   enddo
>>   !$OMP end parallel do
>>
>>  However, setting OMP_NUM_THREADS > 1, also invokes the internal OMP
>> constructs in the kap module; the result of this is a significant speed up,
>> but very spaghetti results.
>>
>>  The problem is, by default MESA is installed with -fopenmp and I would
>> like to keep it so, because of my other applications of MESA. However, my
>> call to get_kappa() needs to be restricted to use only one thread.
>> Geert Jan Bex (in CC) and I also tried calling OMP_set_num_threads(1)
>> before calling get_kappa(), but that did not work either.
>>
>>  So, my question is: Is it possible to use MESA modules in external
>> programs that also include OpenMP directives, making sure that the whole
>> computations are thread safe?
>> I would be grateful if you enlighten me with your comments and ideas.
>>
>>  Best regards
>> Ehsan.
>>
>>
>> ------------------------------------------------------------------------------
>> Don't Limit Your Business. Reach for the Cloud.
>> GigeNET's Cloud Solutions provide you with the tools and support that
>> you need to offload your IT needs and focus on growing your business.
>> Configured For All Businesses. Start Your Cloud Today.
>>
>> https://www.gigenetcloud.com/_______________________________________________
>> mesa-users mailing list
>> mesa-users at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/mesa-users
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Don't Limit Your Business. Reach for the Cloud.
>> GigeNET's Cloud Solutions provide you with the tools and support that
>> you need to offload your IT needs and focus on growing your business.
>> Configured For All Businesses. Start Your Cloud Today.
>>
>> https://www.gigenetcloud.com/_______________________________________________
>> mesa-users mailing list
>> mesa-users at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/mesa-users
>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Don't Limit Your Business. Reach for the Cloud.
>> GigeNET's Cloud Solutions provide you with the tools and support that
>> you need to offload your IT needs and focus on growing your business.
>> Configured For All Businesses. Start Your Cloud Today.
>> https://www.gigenetcloud.com/
>> _______________________________________________
>> mesa-users mailing list
>> mesa-users at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/mesa-users
>>
>>
>
>
>
> ------------------------------------------------------------------------------
> Don't Limit Your Business. Reach for the Cloud.
> GigeNET's Cloud Solutions provide you with the tools and support that
> you need to offload your IT needs and focus on growing your business.
> Configured For All Businesses. Start Your Cloud Today.
> https://www.gigenetcloud.com/
> _______________________________________________
> mesa-users mailing list
> mesa-users at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/mesa-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.mesastar.org/pipermail/mesa-users/attachments/20150714/b6305932/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: sample_kap.f
Type: text/x-fortran
Size: 8976 bytes
Desc: not available
URL: <https://lists.mesastar.org/pipermail/mesa-users/attachments/20150714/b6305932/attachment.bin>