[mesa-users] how to extract columns from existing log file

Richard Townsend townsend at astro.wisc.edu
Mon Oct 20 15:00:15 EDT 2014 

Note that on OS X systems, it should be /usr/bin/env on the first line.

> On Oct 20, 2014, at 1:45 PM, Michael Zingale <michael.zingale at stonybrook.edu> wrote:
> 
> I'd go the python route too.  Here's a stripped down script that doesn't know about MESA or do any sort of error checking:
> 
> #!/bin/env python
> 
> import numpy as np
> import sys
> 
> def doit(file, col):
> 
>     data = np.genfromtxt(file, names=True)
> 
>     od = data[col]
> 
>     for n in range(len(od)):
>         print od[n]
> 
> 
> if __name__ == "__main__":
>     doit(sys.argv[1], sys.argv[2])
> 
> 
> 
> put it in an executable like select_col.py, then
> 
> ./select_col.py filename colname
> 
> will print out the column whose header (assumed to be the first line) is colname
> 
> 
> 
> On Mon, Oct 20, 2014 at 2:31 PM, Robert FARMER <rjfarmer at asu.edu <mailto:rjfarmer at asu.edu>> wrote:
> Here's a python script based on a python library i made:
> 
> You'll need python and numpy
> Assuming your in a mesa work dir (so that history.data is in LOGS/history.data)
> Run:
> python /path/to/extractMesa.py col_names.txt
> 
> and assuming in col_names.txt each column name is on separate line (also the file name for the columns isn't fixed just needs to be first parameter to the script)
> 
> Output goes to stdout
> You'll get the header plus the columns numbers as well. So it should look like a normal history file except now smaller
> 
> It should also remove the backups and retries
> 
> Never tried this though on 100mb file before.
> 
> Rob
> 
> 
> 
> 
> On Mon, Oct 20, 2014 at 10:49 AM, Bill Paxton <paxton at kitp.ucsb.edu <mailto:paxton at kitp.ucsb.edu>> wrote:
> Dear Santa's Helpers,
> 
> Thanks for the quick replies.
> 
>> ideally, i'd give this marvelous tool a file with a list of the column names that i want and it would do the rest.
> 
> 
> Now I just need a wrapper script that reads a file with column names, uses the header of the history.data file to map names to column numbers, and then calls gawk to do the work.   Please Santa, can I have that too?  ;D
> 
> -Bill
> 
> 
> 
> 
> 
> On Oct 20, 2014, at 10:42 AM, Aaron Dotter wrote:
> 
>> Hi Bill,
>> 
>> I was just about to recommend gawk but Rob sorta beat me to it.  It's like awk but traditional awk might choke on 100s of MBs of data.  gawk should not.
>> 
>> Otherwise it works just like awk in Rob's email.
>> 
>> Aaron
>> 
>> On Mon, Oct 20, 2014 at 1:39 PM, Robert FARMER <rjfarmer at asu.edu <mailto:rjfarmer at asu.edu>> wrote:
>> If you know the column numbers you could do:
>> 
>> awk '{print $NUM1,$NUM2...}' file
>> 
>> where $NUM1 is column numbers, 
>> $0 is the whole line
>> $1 first column etc
>> 
>> If you want to skip the header then you can do this:
>> awk '{if(NR>2) print $1,$2...}' file
>> 
>> Where NR tracks the number of rows
>> 
>> Rob
>> 
>> 
>> 
>> On Mon, Oct 20, 2014 at 10:35 AM, Bill Paxton <paxton at kitp.ucsb.edu <mailto:paxton at kitp.ucsb.edu>> wrote:
>> Some unix wizard out there will have a solution for this one I'm sure.
>> 
>> I have a monster history.data file (100s of MB) with lots and lots of columns, most of which are unneeded for my current task --
>> it would be lovely to have a simple way to extract a small subset of the columns to create a much smaller history.data file.
>> 
>> and of course i don't want to have to do any work to create this tool for myself.  i want Santa Claus to give it to me.  now.  ;D
>> 
>> Thanks,
>> Bill
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Comprehensive Server Monitoring with Site24x7.
>> Monitor 10 servers for $9/Month.
>> Get alerted through email, SMS, voice calls or mobile push notifications.
>> Take corrective actions from your mobile device.
>> http://p.sf.net/sfu/Zoho <http://p.sf.net/sfu/Zoho>
>> _______________________________________________
>> mesa-users mailing list
>> mesa-users at lists.sourceforge.net <mailto:mesa-users at lists.sourceforge.net>
>> https://lists.sourceforge.net/lists/listinfo/mesa-users <https://lists.sourceforge.net/lists/listinfo/mesa-users>
>> 
>> 
>> ------------------------------------------------------------------------------
>> Comprehensive Server Monitoring with Site24x7.
>> Monitor 10 servers for $9/Month.
>> Get alerted through email, SMS, voice calls or mobile push notifications.
>> Take corrective actions from your mobile device.
>> http://p.sf.net/sfu/Zoho <http://p.sf.net/sfu/Zoho>
>> _______________________________________________
>> mesa-users mailing list
>> mesa-users at lists.sourceforge.net <mailto:mesa-users at lists.sourceforge.net>
>> https://lists.sourceforge.net/lists/listinfo/mesa-users <https://lists.sourceforge.net/lists/listinfo/mesa-users>
>> 
>> 
> 
> 
> 
> ------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
> http://p.sf.net/sfu/Zoho <http://p.sf.net/sfu/Zoho>
> _______________________________________________
> mesa-users mailing list
> mesa-users at lists.sourceforge.net <mailto:mesa-users at lists.sourceforge.net>
> https://lists.sourceforge.net/lists/listinfo/mesa-users <https://lists.sourceforge.net/lists/listinfo/mesa-users>
> 
> 
> 
> 
> -- 
> Michael Zingale
> Associate Professor
> 
> Dept. of Physics & Astronomy • Stony Brook University • Stony Brook, NY 11794-3800
> phone:  631-632-8225
> e-mail: Michael.Zingale at stonybrook.edu <mailto:Michael.Zingale at stonybrook.edu>
> web: http://www.astro.sunysb.edu/mzingale <http://www.astro.sunysb.edu/mzingale>------------------------------------------------------------------------------
> Comprehensive Server Monitoring with Site24x7.
> Monitor 10 servers for $9/Month.
> Get alerted through email, SMS, voice calls or mobile push notifications.
> Take corrective actions from your mobile device.
> http://p.sf.net/sfu/Zoho_______________________________________________
> mesa-users mailing list
> mesa-users at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/mesa-users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.mesastar.org/pipermail/mesa-users/attachments/20141020/24b2b32f/attachment.html>

More information about the Mesa-users mailing list