Re: [MacPerl] Memory problems in OO context

[Matthew Langford <langfml [at] eng [dot] auburn [dot] edu>]

On Fri, 19 Nov 1999, Bart Lateur wrote:

> On Thu, 18 Nov 1999 01:34:09 -0600 (CST), Matthew Langford wrote:
> 
> >To get the whole deal I had to use more than 55 MB of memory.  (I think
> >this is what I came back to; for a while I had to crank out some virtual
> >memory and set it higher.)  The hash, after I finally got it, takes up 4.6
> >MB. If I read in the file, I don't have too many memory problems.  
> >Clearly, there are serious memory leaks here in the "getting" stage.
> 
> That doesn't prove a thing. I tried HTML::TreeBuilder on a 15k HTML
> file, on a PC (DOS Perl), and I got an out of memory error, with many
> megabytes available for Perl.
> 
> This HTML parsing seems to be very memory hungry. My guess is that this
> is because every HTML tag is a not so small object in memory. A lot of
> those objects, and you're quickly out of memory.

Thanks for the reply.  Maybe this is a common problem.  Parsing HTML may
be hungry, but I'm trying to free up the memory after each file (they run
about 24k).

The data structure I'm saving out with Storable is small (maybe 8mb?
in memory).  I know this, because I can read it into a different Perl
program with a much smaller memory partition (and the Finder memory window
still shows memory free).


> Just give an acceptable proof that there IS indeed a memory leak, e.g.
> by parsing the same file over and over again, clearing memory (=
> releasing objects) every time, and watching it get out of memory after
> many loops.

I'm _trying_ to release the objects.  That's my question:  how do I get
MacPerl to let go of the parse object for each file?

This line should clear my URI object, LWP::UserAgent
object, HTTP::Request object, and the result object:

         # try to clear up some memory
        undef $theFile; undef $ua; undef $req; undef $res;



I'm not saying MacPerl has a memory leak, I'm saying my program has a
memory leak.  I say this because I parse a file, return the data I need
(which given the above observations the data I need is much smaller than
the data allocated for parsing), and loop--over some two hundred files.  I
have to keep the data I need, but I want to free the data allocated for
parsing.  This is not happening:  I see the memory usage creep up for each
file.

In the past, I hate to admit, I have just given MacPerl more memory.
Sloppy.  I no longer have more memory to give.

Another option I have used is to bypass memory problems with an XS module
(I switched from a regex to CSV_XS), which has a dramatically smaller
memory footprint.  I find it ironic that I'm switching to a
non-garbage-collected language to solve memory allocation problems in a
garbage-collected language.  That may be the wrong way to look at it,
though.

I notice there is a HTML::Parser_XS available.  Chris, if you are still
reading this, how do I request/enter this module from pudge.net/mmp?

I may build it myself, but I'll have to re-download the MacPerl source
again.  I need a new hard drive...

I've built Text::CSV_XS 0.20, but I didn't submit it because it was my
first XS module, I didn't know how, and I wasn't confident it was working.
Since then, I am convinced that my build works exactly the same as the
0.18 build.  <babble off>



--
MattLangford 




# ===== Want to unsubscribe from this list?
# ===== Send mail with body "unsubscribe" to macperl-request@macperl.org