[mesa-users] pgstar slowdown
Robert Farmer
rjfarmer at asu.edu
Thu Dec 8 13:26:38 EST 2016
thanks for the plots, it confirms theres nothing really odd going on on
your machine, but something is still definitely making xorg need much more
% of the cpu time than others, though the absolute values for each step are
similar to what i get.
Some total timings from a different model i ran to a fixed model number:
no pgstar: 6m30s,
with "normal" pgstar and only history based plots: 8m 20s
plotting every 100th history point (but with pgstar redrawing the window
every step): 8m
plotting no points but we redraw the plot windows every step: 8m 5s
So from my numbers you see that most of the cost is from redrawing the
windows, with plotting data on top being smaller chunk. So i'd suggest:
Update any software/drivers available
Make sure you either save plots to file or display on screen (if you do
both you'll double the redrawing cost)
use pgstar_interval to redraw the image fewer times
Rob
On Wed, Dec 7, 2016 at 11:14 AM, Matthew Clayton <
matthew.clayton at exeter.ox.ac.uk> wrote:
> Hi Rob,
>
> I see the same gradual increase as you: the CPU usage of Xorg increases
> slowly over the course of a run, except in my case it ends up using pretty
> much an entire core to itself. The sudden jumps I was talking about earlier
> are what you see if you turn the history plot window on and off whilst a
> simulation is running (e.g. by commenting out the line
> Summary_History_win_flag = .true. in your pgstar inlist during the run).
>
>
> I've run my test case twice with your run_star_extras.f, and attach two
> history files:
>
> > history_noplot.data is from a simulation with no plotting windows turned
> on (although pgstar_flag = .true. is still set)
>
> > history_plot.data is from an identical simulation but with the summary
> history window turned on (with pgstar_cnt=1).
>
> NOTE: these files are inside a .7z archive because I need to squeeze this
> email past my university's max size filter.
>
> I've taken the liberty of adding another output column which measures the
> wall time between steps in milliseconds (wall_time), and I've attached the
> version of run_star_extras.f with this added in case anyone wants to try
> for themselves.
>
> Also attached is a figure comparing the CPU and wall times between the two
> runs. Since the time between steps tends to be very noisy, I've also
> plotted the 100-point moving averages of all values to smooth things out.
>
> Cheers,
>
> Matthew
> ------------------------------
> *From:* Robert Farmer [rjfarmer at asu.edu]
> *Sent:* 05 December 2016 21:03
> *To:* Matthew Clayton
> *Cc:* mesa-users at lists.sourceforge.net
>
> *Subject:* Re: [mesa-users] pgstar slowdown
>
> Hi
> If you watch top with MESA running, pgstar on and the history summary on
> (from the start), do you see a gradual increase in the cpu usage of xorg?
> On my machine i'm seeing a gradual rise to ~40% rather than a sudden jump
> (Exact % may depend on cpu/graphics card/software versions).
>
> Secondly can you try this run_star_extras.f which adds a new history
> column "clock_time" which measures the difference in the cpu clock between
> each step and then send the history.data file back to me?
>
> Rob
>
> On Mon, Dec 5, 2016 at 9:04 AM, Matthew Clayton <
> matthew.clayton at exeter.ox.ac.uk> wrote:
>
>> Hi Frank,
>>
>> I don't think memory can be my problem. I've run my test case again, and
>> played around with turning the summary history plot window on and off at a
>> model number of about 16,000 (using pgstar_cnt=1 and displaying the entire
>> history of the model). I've reproduced below the output of "top" when I
>> have the summary history window off (upper set of output), and when it's on
>> (lower set of output). Hopefully the column formatting will survive its way
>> to the mailing list. As you can see, the change leads to a strong reduction
>> in the amount of CPU used by star, whilst plenty of free memory is
>> available.
>>
>> For background, I'm running this on a machine that has 4 physical cores
>> with hyperthreading, so 8 logical cores, and I've told MESA to use 6 of
>> them.
>>
>> Thanks for your help,
>>
>> Matthew
>>
>>
>> top - 15:30:49 up 13 days, 21:40, 7 users, load average: 1.62, 1.77,
>> 2.88
>> Tasks: 362 total, 2 running, 359 sleeping, 0 stopped, 1 zombie
>> %Cpu(s): 68.5 us, 0.5 sy, 0.0 ni, 30.9 id, 0.1 wa, 0.0 hi, 0.0 si,
>> 0.0 st
>> KiB Mem: 16320000 total, 11716296 used, 4603704 free, 611792 buffers
>> KiB Swap: 11034620 total, 114860 used, 10919760 free. 5952480 cached
>> Mem
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>> COMMAND
>> 23234 clayton 20 0 1876684 1.389g 9788 R 527.7 8.9 118:18.18
>> star
>> 2152 root 20 0 1030204 260196 194116 S 13.6 1.6 268:50.10
>> Xorg
>> 3887 clayton 20 0 1393996 100340 36952 S 4.0 0.6 71:46.48
>> compiz
>> 28870 clayton 20 0 708264 55476 41196 S 3.3 0.3 0:25.45
>> gnome-syst+
>> 9875 clayton 20 0 667364 31548 17324 S 1.0 0.2 11:58.54
>> gnome-term+
>> 23432 clayton 20 0 3468824 287172 64432 S 1.0 1.8 105:28.03
>> mendeleyde+
>> 29447 root 20 0 30284 7580 5732 S 1.0 0.0 0:00.35
>> cfagent
>> 7 root 20 0 0 0 0 S 0.3 0.0 12:03.37
>> rcu_sched
>> 1747 couchdb 20 0 602348 41280 4680 S 0.3 0.3 29:41.28
>> beam.smp
>> 3555 clayton 20 0 377044 27212 4664 S 0.3 0.2 7:17.17
>> ibus-daemon
>> 10639 clayton 20 0 1788440 240992 34656 S 0.3 1.5 32:14.86
>> sublime_te+
>> 28813 clayton 20 0 31172 3372 2704 R 0.3 0.0 0:01.74
>> top
>> 1 root 20 0 30812 4364 2492 S 0.0 0.0 0:04.46
>> init
>> 2 root 20 0 0 0 0 S 0.0 0.0 0:00.10
>> kthreadd
>> 3 root 20 0 0 0 0 S 0.0 0.0 0:09.01
>> ksoftirqd/0
>> 8 root 20 0 0 0 0 S 0.0 0.0 0:00.00
>> rcu_bh
>> 9 root rt 0 0 0 0 S 0.0 0.0 0:01.72
>> migration/0
>>
>>
>>
>> top - 15:30:07 up 13 days, 21:39, 7 users, load average: 1.16, 1.75,
>> 2.93
>> Tasks: 362 total, 2 running, 359 sleeping, 0 stopped, 1 zombie
>> %Cpu(s): 20.7 us, 1.0 sy, 0.0 ni, 78.1 id, 0.1 wa, 0.0 hi, 0.1 si,
>> 0.0 st
>> KiB Mem: 16320000 total, 11693180 used, 4626820 free, 611728 buffers
>> KiB Swap: 11034620 total, 114860 used, 10919760 free. 5930800 cached
>> Mem
>>
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
>> COMMAND
>> 2152 root 20 0 1031544 261496 195416 R 93.2 1.6 268:23.31
>> Xorg
>> 23234 clayton 20 0 1876684 1.389g 9788 S 73.6 8.9 116:55.06
>> star
>> 28870 clayton 20 0 708264 55476 41196 S 3.0 0.3 0:23.95
>> gnome-syst+
>> 3887 clayton 20 0 1393872 100292 36904 S 1.7 0.6 71:44.15
>> compiz
>> 9875 clayton 20 0 667364 31548 17324 S 1.0 0.2 11:58.30
>> gnome-term+
>> 23432 clayton 20 0 3468824 287172 64432 S 1.0 1.8 105:27.59
>> mendeleyde+
>> 3645 clayton 20 0 125020 4704 4400 S 0.7 0.0 0:11.48
>> at-spi2-re+
>> 1747 couchdb 20 0 602348 41280 4680 S 0.3 0.3 29:41.22
>> beam.smp
>> 10031 clayton 20 0 4975824 176892 49608 S 0.3 1.1 1:59.19
>> dropbox
>> 10639 clayton 20 0 1788440 240992 34656 S 0.3 1.5 32:14.69
>> sublime_te+
>> 21931 clayton 20 0 669812 24848 20424 S 0.3 0.2 0:09.79
>> gnome-sett+
>> 22270 clayton 20 0 517116 14448 10136 S 0.3 0.1 0:05.87
>> zeitgeist-+
>> 25039 clayton 20 0 1400448 73648 56484 S 0.3 0.5 0:01.63
>> nautilus
>> 28813 clayton 20 0 31172 3372 2704 R 0.3 0.0 0:01.66
>> top
>> 1 root 20 0 30812 4364 2492 S 0.0 0.0 0:04.46
>> init
>> 2 root 20 0 0 0 0 S 0.0 0.0 0:00.10
>> kthreadd
>> 3 root 20 0 0 0 0 S 0.0 0.0 0:09.01
>> ksoftirqd/0
>>
>> ________________________________________
>> From: Frank Timmes [fxt44 at mac.com]
>> Sent: 05 December 2016 14:50
>> To: Matthew Clayton
>> Cc: Frank Timmes; mesa-users at lists.sourceforge.net
>> Subject: Re: [mesa-users] pgstar slowdown
>>
>> hi matthew,
>>
>> we looked fairly deeply into the cost of pgstar on a variety of runs,
>> including your model, with rob doing most of the heavy lifting.
>>
>> typically we find the cost of pgstar is 20-30%, with about half of that
>> due to
>> system the kernal/graphics driver and about half due to mesa (dominated by
>> pgstar frame refreshes). there is a very slight trend of increasing the
>> cost of pgstar as an evolution proceeds due to the additional history
>> data.
>>
>> in no case, including yours, can we replicate order of magnitude
>> slowdowns.
>> is it possible you are running out of real memory on your machine?
>> please run the job and watch your system's free/app/wired/compressed
>> memory usage.
>>
>> modulo running out of memory, the best solution for those who care
>> is to set pgstar_interval to a larger number to reduce the pgstar frame
>> drawing costs.
>>
>> fxt
>>
>>
>>
>>
>> > On Nov 28, 2016, at 11:39 AM, Matthew Clayton <
>> matthew.clayton at exeter.ox.ac.uk> wrote:
>> >
>> > Hi all,
>> >
>> > I use pgstar a lot to see live plots of simulations I run, it's great.
>> =]
>> >
>> > However, when running reasonably long (10k models +) simulations using
>> pgstar, I've noticed that it starts to use a lot of processor time* and
>> cause the simulation to slow down considerably (in a "factor of 10" kind of
>> sense). By turning different plots on and off I've been able to work out
>> that this is due to drawing plots like the summary history plot, which show
>> the past behaviour of the model (as opposed to, say a plot that shows a
>> profile through the current model). I think what's going on is that every
>> time the plot is updated, pgstar is drawing a point (or line segment) for
>> every model in the star's history, which eventually gets quite slow and can
>> lead runs to grind to a halt.
>> >
>> > I've been able to circumvent this problem by having the plots updated
>> less often (pgstar_interval = large number) or (in the case of the summary
>> history) manually truncating the plot to only show recent models
>> (Summary_History_xmin = slightly less than current model number), neither
>> of which is very satisfactory.
>> >
>> > I'm wondering whether anyone else is having this problem, or whether it
>> suggests that my X Window environment is set up in some inefficient way or
>> some such. If this is a general problem, is there a general workaround?
>> Perhaps pgstar could be made to only draw a certain maximum number of
>> points on any plot, and start missing out intermediate values if it's asked
>> to draw more than this number. Or maybe it does that already and my problem
>> is coming from something else. Either way, I'd welcome some input from
>> anyone who's experienced this issue or has ideas about how to fix it.
>> >
>> > Cheers,
>> >
>> > Matthew
>> >
>> > (obligatory data, in case it's relevant: I'm using version 7624 on an
>> Ubuntu machine)
>> >
>> > *Well, specifically it's Xorg using processor time
>> > ------------------------------------------------------------
>> ------------------
>>
>>
>> ------------------------------------------------------------
>> ------------------
>>
>> _______________________________________________
>> mesa-users mailing list
>> mesa-users at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/mesa-users
>>
>>
>
> ------------------------------------------------------------
> ------------------
> Developer Access Program for Intel Xeon Phi Processors
> Access to Intel Xeon Phi processor-based developer platforms.
> With one year of Intel Parallel Studio XE.
> Training and support from Colfax.
> Order your platform today.http://sdm.link/xeonphi
> _______________________________________________
> mesa-users mailing list
> mesa-users at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/mesa-users
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.mesastar.org/pipermail/mesa-users/attachments/20161208/4d8b3567/attachment.html>
More information about the Mesa-users
mailing list