[mesa-users] how to extract columns from existing log file

[Bill Paxton]

Fantastic!

Now, dear Santa, so I can run Rob's script, please help me set up python on an old Mac running OSX 10.6.8 --- last time I tried I got lost in a maze of twisty turning contradictory instructions on the web that left me convinced that before I could run python I'd have to update my mac.  ;D    and if I update my mac, I'll need to take a year off to rewrite my Tioga GUI that runs fine on 10.6.8, but not on any newer OSX because of "improvements".  ah the joys of software tools that make life so much easier.  

-B


On Oct 20, 2014, at 11:31 AM, Robert FARMER wrote:

> Here's a python script based on a python library i made:
> 
> You'll need python and numpy
> Assuming your in a mesa work dir (so that history.data is in LOGS/history.data)
> Run:
> python /path/to/extractMesa.py col_names.txt
> 
> and assuming in col_names.txt each column name is on separate line (also the file name for the columns isn't fixed just needs to be first parameter to the script)
> 
> Output goes to stdout
> You'll get the header plus the columns numbers as well. So it should look like a normal history file except now smaller
> 
> It should also remove the backups and retries
> 
> Never tried this though on 100mb file before.
> 
> Rob
> 
> 
> 
> 
> On Mon, Oct 20, 2014 at 10:49 AM, Bill Paxton <paxton at kitp.ucsb.edu> wrote:
> Dear Santa's Helpers,
> 
> Thanks for the quick replies.
> 
>> ideally, i'd give this marvelous tool a file with a list of the column names that i want and it would do the rest.
> 
> 
> Now I just need a wrapper script that reads a file with column names, uses the header of the history.data file to map names to column numbers, and then calls gawk to do the work.   Please Santa, can I have that too?  ;D
> 
> -Bill
> 
> 
> 
> 
> 
> On Oct 20, 2014, at 10:42 AM, Aaron Dotter wrote:
> 
>> Hi Bill,
>> 
>> I was just about to recommend gawk but Rob sorta beat me to it.  It's like awk but traditional awk might choke on 100s of MBs of data.  gawk should not.
>> 
>> Otherwise it works just like awk in Rob's email.
>> 
>> Aaron
>> 
>> On Mon, Oct 20, 2014 at 1:39 PM, Robert FARMER <rjfarmer at asu.edu> wrote:
>> If you know the column numbers you could do:
>> 
>> awk '{print $NUM1,$NUM2...}' file
>> 
>> where $NUM1 is column numbers, 
>> $0 is the whole line
>> $1 first column etc
>> 
>> If you want to skip the header then you can do this:
>> awk '{if(NR>2) print $1,$2...}' file
>> 
>> Where NR tracks the number of rows
>> 
>> Rob
>> 
>> 
>> 
>> On Mon, Oct 20, 2014 at 10:35 AM, Bill Paxton <paxton at kitp.ucsb.edu> wrote:
>> Some unix wizard out there will have a solution for this one I'm sure.
>> 
>> I have a monster history.data file (100s of MB) with lots and lots of columns, most of which are unneeded for my current task --
>> it would be lovely to have a simple way to extract a small subset of the columns to create a much smaller history.data file.
>> 
>> and of course i don't want to have to do any work to create this tool for myself.  i want Santa Claus to give it to me.  now.  ;D
>> 
>> Thanks,
>> Bill
>> 
>> 
>> 
>> ------------------------------------------------------------------------------
>> Comprehensive Server Monitoring with Site24x7.
>> Monitor 10 servers for $9/Month.
>> Get alerted through email, SMS, voice calls or mobile push notifications.
>> Take corrective actions from your mobile device.
>> http://p.sf.net/sfu/Zoho
>> _______________________________________________
>> mesa-users mailing list
>> mesa-users at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/mesa-users
>> 
>> 
>> ------------------------------------------------------------------------------
>> Comprehensive Server Monitoring with Site24x7.
>> Monitor 10 servers for $9/Month.
>> Get alerted through email, SMS, voice calls or mobile push notifications.
>> Take corrective actions from your mobile device.
>> http://p.sf.net/sfu/Zoho
>> _______________________________________________
>> mesa-users mailing list
>> mesa-users at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/mesa-users
>> 
>> 
> 
> 
> <mesaExtract.py>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.mesastar.org/pipermail/mesa-users/attachments/20141020/5f4f27e3/attachment.html>