Re: [MacPerl-AnyPerl] using open() on zipped files

[Chris Nandor <pudge@pobox.com>]

At 23.22 -0400 2000.06.18, Paul Corr wrote:
>I have a number of large zipped log files. I ran our Web log reporting
>program (WebTrends) on them but the higher-ups would like other sorts of
>reports. They are proxy logs and the program gives 'top 10' (or 20, 50,
>etc.) but the desire is to track a particular resource over time, for
>example. I've done this on these files by unzipping and running on a
>group of them. There has to be a better way and I'm open to suggestions.
>
>This is more of a novice question, but I was wondering about the actual
>operation of the 'open()' function. I consulted Programming Perl and
>couple of other books and it seems like this function opens a connection
>to the a file and then the data is read a record (line) at a time. Do I
>have this right? On the Mac, I'm concerned about exceeding available
>memory. Manually opening the files exceeds memory quickly and I'm
>wondering if using perl and open on the file is most efficient.
>
>I also did find a Zip module on CPAN which will zip and unzip, etc. but I'd
>like tips on best practice, if this is the way.

I am not sure what you are asking here.  If you want to access data 
in zipped files, then yes, use the Archive::Zip module.  I don't know 
if it has memory problems or not.  It uses Compress::Zlib, which can 
work in low-memory situations, so perhaps it can.

I am not sure what you want to know about open() though.  open() just 
opens a file.  How you get data from that file differs depending on 
what function you use.  readline() and <> read one line at a time. 
read() reads in bytes.  But in any case, reading data in does not 
cause memory problems unless you keep it in memory.

   open LARGE_FILE, "< some_2GB_file.txt" or die $!;
   while (<LARGE_FILE>) {
     $x{foo}++ if /bar/;
   }
   close LARGE_FILE;

That can run without any memory problems (assuming the lines 
themselves are not several MBs each), because you are not saving the 
file data, you are looking at it and then dumping it.

-- 
Chris Nandor       |     pudge@pobox.com      |     http://pudge.net/
Andover.Net        | chris.nandor@andover.net | http://slashcode.com/

==== Want to unsubscribe from this list?
==== Send mail with body "unsubscribe" to macperl-anyperl-request@macperl.org