Hadoop Speed
When using hadoop on the ics server, it takes about 10 minutes to run a test. We packaged it using the FatJar extension to eclipse. Are there any recommendations to make this run any faster? The example takes like 1 minute. Even when we use the example code in our project that we make it still takes 10 minutes. It takes like 3 minutes to say "map 0%..." and then another 3 minutes to do anything and show up as a job. Once the map starts it's extremely fast. But when it finishes it takes another 2 minutes to close the job.
Thanks for your help!
Thanks for your help!
2
people have this question
I have this question, too!
Tell me when someone answers.
The more people who ask this question, the more it gets noticed.
The more people who ask this question, the more it gets noticed.
Create a customer community for your own organization
Plans starting at $19/month
-
Inappropriate?have you tried another machine? I took me only 14 seconds on one of the family-guy machines. Although, from your description, it sounds like you might not even be using the ICS computers.
on another note, it took 45 minutes to untar the hadoop file... -
Inappropriate?It takes a while to launch the processes but once they are launched things go faster. That seems to be what you experienced although even that was a bit on the slow side.
-
Inappropriate?Also if you have a lot of stuff happening automatically for you when you log in, that could be slowing down the start up process of your system. (Like an elaborate .cshrc file or something)
I’m empathetically inefficient
Loading Profile...



EMPLOYEE