Re: [MacPerl] Memory problems in OO context

[Matthew Langford <langfml [at] eng [dot] auburn [dot] edu>]

On Sun, 5 Dec 1999, Ken Williams wrote:

> Nicholas.G.Thornton@directory.reed.edu (Nicholas G. Thornton) wrote:
> >I have been noticing memory leaks laetely. Not so bad as some of my
> >other programs (which are in beta afterall), but still there.
> >Noticeable after a few days of running. Arguably could be the scripts
> >I'm running. Arguably.
> 
> Well, the rules for yelling "memory leak" in a crowded theatre remain
> the same: whittle your code down to the smallest possible program that
> demonstrates the leak, then submit a bug report.  More likely what will
> happen is that you'll discover where your code isn't cleaning up
> properly.  Sure, Perl usually cleans up after you, but sometimes it
> can't.

Good advice.  I've spent the day, partly whittling code, but mostly
watching my machine reboot.  All hail protected memory!  

The first, and probably easiest to fix, problem I've noticed:  why does
AutoLoader come up when I call exit(1) in my program?

I apologize to those who are tired of seeing me post.  I'm tired of
posting, so I know others are bound to be sick of my posts.  I hope to
find some resolution soon, and return to a state of blissful awe at the
creativity and productivity of Alan Fry and Chris Nandor and many others.

Here's what I've found in my memory travels/travails.

This is the main body of code:

  for ($elem = 0; $elem < 250; $elem++)
  {
        # print every third country name
      print "$elem...Afghanistan again...\n";
    
      $info{ "$elem" }{'data'} = GetOneCountry( "af.html" );
      
      # Give MacPerl time off to respond to cancels and mouse clicks
      sleep 1;
  }

  Storable::nstore \%info, 'slurp.store';

  undef %info;

  die "Finishing storage and shutting down.\n";


One interesting thing I've learned is that Storable takes an amount of
memory roughly equal to the size of the file it writes.  So I've got to
have at least twice as much memory as this data structure or file size.
For this test, 'slurp.store' ended up as 5,258,651 bytes.

Another interesting thing is that 'undef %info;' doesn't free up a
boatload of memory like I'd hoped.  Before I inserted the die statement, I
tried retrieving the data structure into an entirely different variable:

  $iref = Storable::retrieve( 'slurp.store' );

This, in combination with the amount of memory used for the first store,
together with the memory for all the iterations, was too much and MacPerl
crashed, taking the OS with it.  For those who care, MacPerl crashed in
_pool_find_ptr_bucket (actually several times during the day), and MacsBug
StdLog got no further than the header.  Where the ROM version normally
shows, MacsBug sez "Unable to access that address".  Blammo.

Anyway, back to the code.  Clearly all the non-trivial work is done in the
GetOneCountry sub, so here it is:

	sub GetOneCountry
	{
	    my( $uri ) = @_;
	    
	    my $p;
	    $p = OneCountryGrabber->new->parse_file("$uri");
	    
	    if ( defined($p))
	    {
	          # try to clear up some memory
	        $theInfo = \%{$p->{metaData}};
	        $p->{metaData} = undef;
	        undef $p->{metaData};
	          # this is the big memory hog
	        undef $p;
	        return $theInfo;
	    } 
	    else 
	    {
	        print "Bad luck this time\n";
	        die   "Parse_file failed."; 
	    }	
	}

Notice all the feeble efforts to slash, undef, clear up, or otherwise
scavenge some memory.  Futile.

If you are wondering about OneCountryGrabber, it is a subclass of
HTML::Filter.  I haven't included it because it isn't so small as these
two functions, and because I think the problem is outside it.  If there's
demand, I'll post it too.

Now, for the stats on memory usage, which I gathered using Metrowerks
ZoneRanger.  There was some variance between runs, but within reason. 
 After opening the source in Perl and running a syntax check on it,
MacPerl's memory looked like this:

      Blocks          Used          Free          Ptrs         Handles
         324           291            33           159             132
  33,952,928     2,228,832    31,724,096     2,109,328         119,504

after running the 250-loop, but before the Storable (I scribbled this down
once, when I had a sleep statement in the code).  I didn't write down the
totals, but you can figure them from Used+Free:

      Blocks          Used          Free          Ptrs         Handles
           ?          1430            14          1272             158
           ?    24,428,432     9,524,496    24,300,016         128,416

After the 250-loop and the Storable, this is how things looked:

      Blocks          Used          Free          Ptrs         Handles
        1700          1659            41          1508             151
  35,317,344    29,973,472     5,343,872    29,836,864         136,608


Clearly memory fragmentation is not my main problem.

It seems to me $p is not freed in sub GetOneCountry, despite all my
efforts.  Nor does it seem to get freed even after deleting %info in the
main function, despite %info being the last possible reference to it.
(Though the data structure is several levels nested, ultimately all its
values are strings.  It does not refer to itself.)

Why doesn't _every_ module returning a reference to something leak memory?

Originally I didn't use $theInfo in GetOneCountry at all, I just returned
\%{$p->{metaData}}.  I thought maybe $theInfo would allow me to clear the
metaData hashref, which in turn would allow $p to be freed.  It didn't
happen that way.

I guess I need to copy the metaData element-by-element:  a deep copy,
rather than just copying the reference.  Hopefully by not using the
reference, when I delete it, it will die as it should.  I'm going to try
Ref->copyref next.

Any thoughts?


--
MattLangford 




# ===== Want to unsubscribe from this list?
# ===== Send mail with body "unsubscribe" to macperl-request@macperl.org