Loading Profile...
Why does it get to 100% MAP and 100% REDUCE and then fail?
My attempts to map/reduce get to 100% map and 100% reduce, but then it shows up as "FAILED" and only some of the parts are present. Is there any specific problem that I should address or is it just the quirky bugs of Hadoop/DFS? Should I just brute-force this until I get a complete copy of the my posting list?
I’m FAILED
1
person has this question
I have this question, too!
Tell me when someone answers.
The more people who ask this question, the more it gets noticed.
The more people who ask this question, the more it gets noticed.
Create a customer community for your own organization
Plans starting at $19/month
-
Inappropriate?You may notice that it is 4:30AM and yes I am working on my posting list...
I’m really tired
-
Inappropriate?Probably, your reducer is taking a lot of time and you are not reporting any status to hadoop master in the mean time.
-
Inappropriate?I've got all the updating in the reduce method:
reporter.incrCounter(Counters.REDUCER,1);
reporter.setStatus("REDUCE: " + term.toString());
All of the jobs I have run have gotten to 100% and then showed up as "Failed".
-
Inappropriate?You might want to put the reporter into your for loop instead of outside the loop since some of the terms have around 500,000 docid, so if the collection which you choose to hold the list in the reduce class is not efficient enough, it could take sometimes to process.
1 person says
this answers the question
-
Inappropriate?Hmm. That is a good idea. Thanks. I will try that out. Do I need to increment the counter each loop or is setStatus sufficient?
-
Inappropriate?I actually only add the setStatus into the loop since the counter can still help me trace how many terms are found during the process. I think as long as you report somethings back to the hadoop, you should be fine.
1 person says
this answers the question
-
1 person says
this answers the question
EMPLOYEE
