[mesa-users] how to extract columns from existing log file

[Robert FARMER]

Here's a python script based on a python library i made:

You'll need python and numpy
Assuming your in a mesa work dir (so that history.data is in
LOGS/history.data)
Run:
python /path/to/extractMesa.py col_names.txt

and assuming in col_names.txt each column name is on separate line (also
the file name for the columns isn't fixed just needs to be first parameter
to the script)

Output goes to stdout
You'll get the header plus the columns numbers as well. So it should look
like a normal history file except now smaller

It should also remove the backups and retries

Never tried this though on 100mb file before.

Rob




On Mon, Oct 20, 2014 at 10:49 AM, Bill Paxton <paxton at kitp.ucsb.edu> wrote:

> Dear Santa's Helpers,
>
> Thanks for the quick replies.
>
> ideally, i'd give this marvelous tool a file with a list of the column
>>> names that i want and it would do the rest.
>>>
>>
> Now I just need a wrapper script that reads a file with column names, uses
> the header of the history.data file to map names to column numbers, and
> then calls gawk to do the work.   Please Santa, can I have that too?  ;D
>
> -Bill
>
>
>
>
>
> On Oct 20, 2014, at 10:42 AM, Aaron Dotter wrote:
>
> Hi Bill,
>
> I was just about to recommend gawk but Rob sorta beat me to it.  It's like
> awk but traditional awk might choke on 100s of MBs of data.  gawk should
> not.
>
> Otherwise it works just like awk in Rob's email.
>
> Aaron
>
> On Mon, Oct 20, 2014 at 1:39 PM, Robert FARMER <rjfarmer at asu.edu> wrote:
>
>> If you know the column numbers you could do:
>>
>> awk '{print $NUM1,$NUM2...}' file
>>
>> where $NUM1 is column numbers,
>> $0 is the whole line
>> $1 first column etc
>>
>> If you want to skip the header then you can do this:
>> awk '{if(NR>2) print $1,$2...}' file
>>
>> Where NR tracks the number of rows
>>
>> Rob
>>
>>
>>
>> On Mon, Oct 20, 2014 at 10:35 AM, Bill Paxton <paxton at kitp.ucsb.edu>
>> wrote:
>>
>>> Some unix wizard out there will have a solution for this one I'm sure.
>>>
>>> I have a monster history.data file (100s of MB) with lots and lots of
>>> columns, most of which are unneeded for my current task --
>>> it would be lovely to have a simple way to extract a small subset of the
>>> columns to create a much smaller history.data file.
>>>
>>> and of course i don't want to have to do any work to create this tool
>>> for myself.  i want Santa Claus to give it to me.  now.  ;D
>>>
>>> Thanks,
>>> Bill
>>>
>>>
>>>
>>>
>>> ------------------------------------------------------------------------------
>>> Comprehensive Server Monitoring with Site24x7.
>>> Monitor 10 servers for $9/Month.
>>> Get alerted through email, SMS, voice calls or mobile push notifications.
>>> Take corrective actions from your mobile device.
>>> http://p.sf.net/sfu/Zoho
>>> _______________________________________________
>>> mesa-users mailing list
>>> mesa-users at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/mesa-users
>>>
>>
>>
>>
>> ------------------------------------------------------------------------------
>> Comprehensive Server Monitoring with Site24x7.
>> Monitor 10 servers for $9/Month.
>> Get alerted through email, SMS, voice calls or mobile push notifications.
>> Take corrective actions from your mobile device.
>> http://p.sf.net/sfu/Zoho
>> _______________________________________________
>> mesa-users mailing list
>> mesa-users at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/mesa-users
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.mesastar.org/pipermail/mesa-users/attachments/20141020/d7379edd/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: mesaExtract.py
Type: text/x-python
Size: 1784 bytes
Desc: not available
URL: <https://lists.mesastar.org/pipermail/mesa-users/attachments/20141020/d7379edd/attachment.py>