Re: [MacPerl] efficiency & memory

[Alex Satrapa <packrat@blitzen.canberra.edu.au>]

On Thu, 20 Aug 1998, Nathaniel Irons wrote:
> I'm writing a suite of tools for custom log analysis.  Some of the
> datafiles are Rather Large (2-20MB) and macperl is crashing when I
> attempt to process them.  

> Tue Jul 14 19:59:00 1998 [27132] PORT
> Tue Jul 14 19:59:00 1998 [27132] RETR mrfile.hqx
> Tue Jul 14 19:59:40 1998 [26717] NOOP

I would be tempted to use one of the built-in database tools. Rather than
copying the logs into memory on the fly, I'd do something like...

while(<>) {
    SplitToFields( $_, @record );
    DumpToDatabase( @record );
}


If I remember correctly, you can sort by the primary key (in this case,
the 5 digit number you're using) and pull the data back out, row by row,
into a summary file.

Sorry if I seem a little out of touch with Perl reality - my work's had me
programming in Visual Basic using Microsoft Access for the last three
months. I haven't touched a line of Perl code in that time. I'm just
hoping to get you thinking of a different way to handle the large amounts
of data the log files probably contain.

Note, too, that when you're processing a stream of data like that, you can
collect the information you need for statistical analysis on the fly. For
example, if you keep count of the number of records, the sum of a
particular field, and the sum of the squares of that field, you can derive
the mean and standard deviation.

I hope there's enough here to get you thinking.

Regards,
Alex


***** Want to unsubscribe from this list?
***** Send mail with body "unsubscribe" to mac-perl-request@iis.ee.ethz.ch