Yasser's dashboard

Yasser replied on March 14, 2009 17:13 to the question "Firefox Extension Question" in LUCI:

To be more accurate, FF extension monitors user browsing and whenever a new page is browsed it extracts some keywords from it and sends them to a web server through Ajax. Then the results that are returned by the web server are shown in the side bar.

Yasser replied on March 14, 2009 01:42 to the question "Missing keyword in the Wikipedia-Top500000.txt file" in LUCI:

Just ignore that line.

Yasser replied on March 13, 2009 23:24 to the question "Making the Binary File" in LUCI:

Always try to make your programs responsive by printing a status message once a while. For example, in a loop that should handle 1 million items, it is a good idea to print the number of processed items after processing each 10K items.

Yasser replied on March 13, 2009 19:58 to the question "50,000 occurance and/or 10% of documents +code submission question" in LUCI:

Use one of them in your code.

Just submit your codes and not the generated data.

Yasser replied on March 13, 2009 05:55 to the question "Demos on Monday" in LUCI:

Sorry! You should answer my questions.

Yasser replied on March 13, 2009 04:34 to the question "calculating magnitude" in LUCI:

Yes, you don't need to use the document magnitude and dot product is enough. See slide 5 in Discussion07.pdf

Yasser replied on March 13, 2009 04:27 to the question "calculating magnitude" in LUCI:

As explained in class and discussion session, you should ignore the normalization part. Check my slides!

Yasser replied on March 11, 2009 21:58 to the question "Loading the Terms.Offset" in LUCI:

Yes sir. That is part of the "boot" procedure and is not counted in the limit.

Yasser replied on March 11, 2009 03:54 to the question "url file" in LUCI:

I have a copy of it here: http://www.ics.uci.edu/~yganjisa/TA/A...

Yasser replied on March 10, 2009 05:36 to the question "RandomAccessFile Question" in LUCI:

No, you should use the same RandomAccessFile for writing in the file. Check my slides. I have examples for both reading from and writing to RandomAccessFiles.

Yasser replied on March 10, 2009 05:22 to the question "RandomAccessFile Question" in LUCI:

Jordan, when you get an unexpected integer, it means that you have written the data incorrectly. You can also put your write snippet here, so that we can check.

What do you mean by 1 line? These files are binary files and there is no notion of line in them.

Yasser replied on March 04, 2009 07:49 to the question "Tail: Did you guys get crap too?" in LUCI:

Both are OK to submit. I only need to see a portion of your posting list to know more about its structure.

Yasser replied on March 03, 2009 02:03 to the question "URLs/Doc ids" in LUCI:

Yes, docids are enough.

Yasser replied on March 02, 2009 20:52 to the question "job tracker down?" in LUCI:

Yes, be patient for a few minutes and it would probably become stable again.

Yasser replied on March 02, 2009 20:39 to the question "Why does it get to 100% MAP and 100% REDUCE and then fail?" in LUCI:

Probably, your reducer is taking a lot of time and you are not reporting any status to hadoop master in the mean time.

Yasser shared an idea in LUCI on March 02, 2009 20:34:

Check for page being null

If your program is running fine on small inputs and it cannot handle larger inputs, with a high probability you are not checking the "page" variable being null. In case of any error in fetching page, the downloader would return null. Your code should be like this:

Downloader downloader = new Downloader();
Page page = downloader.download(URL, docid);
if (page != null) {
// Map code here
}

Yasser replied on March 02, 2009 20:27 to the question "job tracker down?" in LUCI:

No, it's up: http://palantir.ics.uci.edu:1755/jobt...

Yasser replied on March 02, 2009 15:48 to the question "Not working on a larger scale?" in LUCI:

You should make sure that "page" is not null. If there is any problem in downloading the page this variable would be null. In that case you should ignore that page.

Yasser replied on March 02, 2009 08:14 to the question "Few problems" in LUCI:

This is not the right way of doing this. Your job is being executed on the distributed file system and you cannot dump your outputs to the local file system. Use this process instead:

1) Crawl the pages.
2) Create the posting lists.
3) In the generated posting lists find the line which is associated with term which is your name and report the list of the docids in that line.

Yasser replied on March 02, 2009 08:12 to the question "Question for URLs" in LUCI:

I am not sure if I explained it clearly! So again this is the normal process:
1) Crawl the pages.
2) Create the posting lists.
3) In the generated posting lists find the line which is associated with term which is your name and report the list of the docids in that line.