Loading Profile...
NullPointerException running Hadoop Job on Fedora
I tried running the Hadoop example at:
http://berkeley-mltea.pbwiki.com/f/ha...
(file comes with a readme) I followed the readme, and attempted to run the example as ryan (myself), hadoop as well as root. I am running Fedora 9 with Sun Java 1.6, and get the following errors:
[ryan@localhost hadoop_example]$ hadoop jar simpleLogReg.jar hadoopml.SimpleLogReg data.txt 10
Starting iteration 1
Running evalObjGrad job...
09/03/17 21:26:45 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
09/03/17 21:26:45 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
09/03/17 21:26:45 INFO mapred.FileInputFormat: Total input paths to process : 1
09/03/17 21:26:45 INFO mapred.FileInputFormat: Total input paths to process : 1
09/03/17 21:26:46 INFO mapred.FileInputFormat: Total input paths to process : 1
09/03/17 21:26:46 INFO mapred.FileInputFormat: Total input paths to process : 1
09/03/17 21:26:46 INFO mapred.JobClient: Running job: job_local_0001
09/03/17 21:26:46 INFO mapred.MapTask: numReduceTasks: 1
09/03/17 21:26:46 INFO mapred.MapTask: io.sort.mb = 100
09/03/17 21:26:46 INFO mapred.MapTask: data buffer = 79691776/99614720
09/03/17 21:26:46 INFO mapred.MapTask: record buffer = 262144/327680
09/03/17 21:26:46 WARN mapred.LocalJobRunner: job_local_0001
java.lang.NullPointerException
at hadoopml.SimpleLogReg.readWFromParamsFile(SimpleLogReg.java:230)
at hadoopml.SimpleLogReg$EvalObjGradMapper.configure(SimpleLogReg.java:256)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:33)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:223)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:157)
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1113)
at hadoopml.SimpleLogReg.runJob(SimpleLogReg.java:217)
at hadoopml.SimpleLogReg.evalObjGrad(SimpleLogReg.java:138)
at hadoopml.SimpleLogReg.train(SimpleLogReg.java:116)
at hadoopml.SimpleLogReg.run(SimpleLogReg.java:80)
at hadoopml.SimpleLogReg.main(SimpleLogReg.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
with the same configuration on Mac OS 10.6, it works fine.
http://berkeley-mltea.pbwiki.com/f/ha...
(file comes with a readme) I followed the readme, and attempted to run the example as ryan (myself), hadoop as well as root. I am running Fedora 9 with Sun Java 1.6, and get the following errors:
[ryan@localhost hadoop_example]$ hadoop jar simpleLogReg.jar hadoopml.SimpleLogReg data.txt 10
Starting iteration 1
Running evalObjGrad job...
09/03/17 21:26:45 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
09/03/17 21:26:45 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
09/03/17 21:26:45 INFO mapred.FileInputFormat: Total input paths to process : 1
09/03/17 21:26:45 INFO mapred.FileInputFormat: Total input paths to process : 1
09/03/17 21:26:46 INFO mapred.FileInputFormat: Total input paths to process : 1
09/03/17 21:26:46 INFO mapred.FileInputFormat: Total input paths to process : 1
09/03/17 21:26:46 INFO mapred.JobClient: Running job: job_local_0001
09/03/17 21:26:46 INFO mapred.MapTask: numReduceTasks: 1
09/03/17 21:26:46 INFO mapred.MapTask: io.sort.mb = 100
09/03/17 21:26:46 INFO mapred.MapTask: data buffer = 79691776/99614720
09/03/17 21:26:46 INFO mapred.MapTask: record buffer = 262144/327680
09/03/17 21:26:46 WARN mapred.LocalJobRunner: job_local_0001
java.lang.NullPointerException
at hadoopml.SimpleLogReg.readWFromParamsFile(SimpleLogReg.java:230)
at hadoopml.SimpleLogReg$EvalObjGradMapper.configure(SimpleLogReg.java:256)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82)
at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:33)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:58)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:82)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:223)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:157)
java.io.IOException: Job failed!
at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1113)
at hadoopml.SimpleLogReg.runJob(SimpleLogReg.java:217)
at hadoopml.SimpleLogReg.evalObjGrad(SimpleLogReg.java:138)
at hadoopml.SimpleLogReg.train(SimpleLogReg.java:116)
at hadoopml.SimpleLogReg.run(SimpleLogReg.java:80)
at hadoopml.SimpleLogReg.main(SimpleLogReg.java:92)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
with the same configuration on Mac OS 10.6, it works fine.
I’m sad it doesn't work on Fedora, but happy it works on Mac!
1
person has this problem
I have this problem, too!
Tell me when someone solves it.
The more people who report this problem, the more it gets noticed.
The more people who report this problem, the more it gets noticed.
-
Inappropriate?Hey Ryan, we're going to take a close look at this in the morning.
At first glance, this looks like a bug in the user code. I'm wondering if subtle differences in how i/o works in OSX vs Fedora is the root cause.
Out of curiosity, have you tried the same code on stock hadoop running on fedora? -
Inappropriate?Hi Ryan,
Looking at the stack trace, it says that it's running your job through the LocalJobRunner. The DistributedCache doesn't work in local mode, so the example code you're running NPE's when it tries to access a file through the DistributedCache.
I've noticed that your example program ships with its own copy of the Hadoop 19 jar, which has its own (default, local-only) configuration inside. Hadoop might be using that jar (especially if the current directory is on the classpath) and reading the config from there. If you move hadoop-0.19.1-core.jar to some far-away directory, does that improve the situation?
The other possibility is that Hadoop can't find a valid distributed configuration to run with on your Fedora box. Can you confirm that the Hadoop daemons (NameNode, JobTracker, TaskTracker, etc) are all running, and that /etc/hadoop/conf points somewhere sane?
Cheers,
- Aaron
EMPLOYEE
EMPLOYEE
