Java out of memory error

Alright i have reserved 1.25GB of ram in the JVM, will report back

still not working. I am not writing it to memory (i think), i am basically doing this:

buffered reader read in file,

while (buffered_reader_file readline() != null)
{
string newstring = buffered_reader_file.readLine()
clean/split and add to random_access_file
}

help plox?

try this one....

string newstring = "";
int totalTerms = 0;

while ((newstring = buffered_reader_file.readLine()) != null)
{
// do whatever you want here
totalTerms++;
if (totalTerms % 10000 == 0)
{
System.out.println("Processing count: " + totalTerms + "...");
}
}

trying that now.

Thanks a lot Art.

You might not want to read the whole line. If the term has more than 50000 pages you just want to skip it. So instead of saving those gigantic lines in memory you can skip them.

So first read till you find the document frequency and if that is lower than 50000 save the line in a String.

Also if you find one with more than 50000 pages use
buffered_reader_file.skip(20*df) (The 20 is something like the minimum number of bytes per docId=termfrecuency pair)
so you skip a lot of bytes and then read the rest of the line. Saves tons of time on those long lines!

hmm i thought we are only skipping in the cosine scoring part.

so basically if the term appears more than 50k times, i shouldnt even include it in the posting.data?

you can exclude those either in the process of generating posting.data or in the cosine scoring part.

Great!

i think its finally running correctly now.

did you guys get any null pointer errors?

thanks for the help guys!