Re: [MacPerl] Parsing Script

[mark@cheers.jsc.nasa.gov (Mark Manning/Muniz Eng.)]

Ok, before I get into this, remember that Chris has alread
posted that File::Sort will do what you want and I believe
that Matthias has also already answered you about this.
:-)


With that said:

According to Strider:
> 
> Mainly because of the memory problems.
<snip>

*nod* But what I mean is that you are trying too hard to
make this all work.  It should be a lot simpler than what
you posted originally.  :-)

> Although I'd do that, I need the WHOLE thing sorted. Eventually, the whole
> database would have to be in memory to do that, unless I misunderstand you.
<snip>

Actually - yes.  You did misunderstand what I said.  But -
I'm not always the best when I'm explaining things.  :-)

So let me try this again.

Since all of the database can not fit into memory at one
time, you have to break it up into smaller pieces.

It's that simple.

The size of these pieces depends upon how much memory you
have on your computer.  If you have a megabyte of memory
over and above what MacPerl needs in order to work, then
the size of these pieces must be 1/3 the size of that
memory or about 300k.  The reason is: You are going to be
pulling in two of these files at a time, sorting them, and
then writing back out the last half to the second file.
Like so:

Let's say you have 300mb of data to sort.
Then let's say you have so much memory that you can divide
the database up into 60mb chunks.
This gives you five files with 60mb in each of them.
Then the simple bubble file sort method would mean that you
have to sort these five files in this manner:

Sort Files 1 & 2 giving file #2 back.
Sort Files 1 & 3 giving file #3 back.
Sort Files 1 & 4 giving file #4 back.
Sort Files 1 & 5 giving file #5 back.
Save File #1.
Load File #2.
Sort Files 2 & 3 giving file #3 back.
Sort Files 2 & 4 giving file #4 back.
Sort Files 2 & 5 giving file #5 back.
Save File #2.
Load File #3.
Sort Files 3 & 4 giving file #4 back.
Sort Files 3 & 5 giving file #5 back.
Save File #3.
Load File #4.
Sort Files 4 & 5 giving file #5 back.
Save File #4.

This would sort all five files properly.  But this is only
one method of doing this.  There are many others.
Including (but not limited to) bucket sorts, B-tree sorts,
etc....  It's just that this is the easiest to set up.

> I don't believe it is, although I'm not sure what you're referring to. All
> I planned to do was sort the file using a low amount of memory- it seems
> that the sorting (and duplication combination or replacement) would be well
> fitted here. The only problem is that of the output array, which I will
> work on once this works. I won't be working on 300mb files for a little
> while, so this one will work for now, if I can get it to work at all.
<snip>

I was referring to an earlier question which I thought you
had posed about keeping track of unique userid's while
reading a huge file.  :-)

Anyway, if the File::Sort works fine - heck - then go for it!  :-)

***** Want to unsubscribe from this list?
***** Send mail with body "unsubscribe" to mac-perl-request@iis.ee.ethz.ch