infection0's dashboard

HMM I am also unsure now of what they meant:

"List of (at most 1000) URLs containing first name of member 1 of your team"

Do they mean "/wiki/Barack_Obama" if my name was Barack, or do they mean the text...

Oh and also are we supposed to use nohup for this, or is it OK to just use

bin/hadoop jar hadooptest.jar ir.assignment05.PostingList /assignment06/input10 output

And close the terminal?

P.S. I have a static class keeping track of the URLs found that contain our name, and also the most common 10 terms. however, when I run the thing on the DFS, it outputs to my /extra/ugrad_space folder with nothing registered. How do we keep track of this stuff?

Oh. Stupid me. It saves into my /extra/ugrad_space/ folder.

See, I did this like,

if(URL.toLowerCase().contains(FIRSTNAME.toLowerCase()))
...blahblah,

But, I can't save the file on the dfs :(
So I have no idea how to do it.

Thanks! That job.setCombinerClass() was the thing that was doing me in. It was in the code example so I used it not thinking about what it did. Now to see if I can start this thing...

We're also having trouble getting ours running...

Map class - Downloads the text, parses it into tokens,
output.collect(token, new Text(docID + ""));

I think I have Map down and I am just messing with Reduce.

I used to filter out duplicate words in the map function so that only the first occurrence of a term in a document is added to the output collector.

We are considering removing that and counting the number of duplicate docIDs in the Reducefunction... but for some reason I am having an ungodly amount of trouble with that. You want us to count the number of occurrences of a term for each docID right? I am trying to tack on that number to the end of the docID before collection, but I think that is screwing everything up when it calls reduce again.

I have no idea why but my output looks like this:

Term (0) (0) [ ]

when without the changes (before I tried to add this stuff) it looks like:
Term [24, 48, ...]

This is how I'm trying to format the final output:
Term (3 occurrences) [24 (2), 48(3), ...]

I am doing this by adding the CF to the Term text, and adding the DF into each docID collected. I think this is a weird way of doing it, but I don't know what else to do.

Here's a small code snippet from Reduce that illustrates the current state of my program:

ArrayList<string> docIDOutputs = new ArrayList<string>();
String docIDPlusCount = docID + " (" + documentTermCount + ")";
docIDOutputs.add(docIDPlusCount);

String outputString = (iterating through the list of docIDs and adding them to the string one by one)

String termPlusCorpusCount = term.toString() + corpusTermCount;
output.collect(new Text(termPlusCorpusCount), new Text(outputString));

I know I'm doing something wrong, I just don't know how to do it right...</string></string>

Hmm, my problem is that I don't have a good idea of where or how I should collect the data. My current methods are really "hacky" and cause my program to fail.

Strange, now my crawler fails to fetch half of all the pages... did Wikipedia just go down? I can't load any of its pages from my computer either.

Holy crud, it DID JUST GO DOWN. My phone, my computer, my friends cant load it either! What's next? Google?

Wikimedia Foundation
Error
العربية Bahasa Indonesia Česky Dansk Deutsch Eesti Ελληνικά English Español فارسی Français עברית Italiano 日本語 한국어
Nederlands Norsk (Bokmål) Polski Português Română Русский Српски Suomi Svenska ไทย Tiếng Việt Türkçe 繁體中文 简体中文
English

Our servers are currently experiencing a technical problem. This is probably temporary and should be fixed soon. Please try again in a few minutes.

You may be able to get further information in the #wikipedia channel on the Freenode IRC network.

Hmm can someone give me a tip on where to count the total number of unique terms? I have no idea where to put it. I originally put it in REDUCE, but that seems impossible now due to the fact that reduce can be run so many times...

Anyone solve this one? I'm getting it also.

EDIT:
lol stupid error, I just had the JAR file name wrong. For me, it was:

bin/hadoop jar hadooptest.jar PostingList input output

INSTEAD OF

bin/hadoop jar hadoop-0.19.0-hadooptest.jar PostingList input output

My error was that I thought the hadoop-0.19.0- was part of the directory structure (which it sort of is, that's the name of my folder where hadoop is in).

I have just emailed helpdesk@ics.uci.edu to wipe my account. Hopefully they can return everything to normal before the assignment deadline. I edited the files per your instructions and it did not help. Thanks anyway, at least I know what the problem is.

Ok so they're probably in my /home directory, but nedit doesn't work and neither does nano... WinSCP can't see these files either.

Pardon my noobness, but where are my .bashrc and .cshrc?

This happens across the board:

$ module load nano
ModuleCmd_Load.c(199):ERROR:105: Unable to locate a modulefile for 'nano'

$ nedit changes.txt
NEdit: Can't open display

I suspect there's a problem with my ICS account in general... I'll ask them on Tuesday if they can "reset" my account or anything like that. Right now it's like typing to a vegetable... there's so many things that don't work.

Now I can do module and echo. But I lost nano... it says command not found. What the heck is going on?