NullPointerException when starting Hadoop service deamons

I am trying to restart my hadoop cluster but the service deamons all generate a NullPointerException.

Here is the excerpt from the logs of the namenode

************************************************************/
2009-11-05 10:24:13,148 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: STARTUP_MSG:
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG: host = athena.stillwater.org/10.1.1.1
STARTUP_MSG: args = []
STARTUP_MSG: version = 0.20.1+152
STARTUP_MSG: build = -r c15291d10caa19c2355f437936c7678d537adf94; compiled by 'root' on Mon Nov 2 00:44:35 EST 2009
************************************************************/
2009-11-05 10:24:13,265 ERROR org.apache.hadoop.hdfs.server.namenode.NameNode: java.lang.NullPointerException
at org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:134)
at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:161)
at org.apache.hadoop.hdfs.server.namenode.NameNode.getAddress(NameNode.java:165)
at org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:180)
at org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:288)
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:968)
at org.apache.hadoop.hdfs.server.namenode.NameNode.main(NameNode.java:977)

2009-11-05 10:24:13,266 INFO org.apache.hadoop.hdfs.server.namenode.NameNode: SHUTDOWN_MSG:
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at athena.stillwater.org/10.1.1.1
************************************************************/
</init>

I’m confused

Inappropriate?

Reply to this problem

1 person has this problem

I have this problem, too! Tell me when someone solves it.
The more people who report this problem, the more it gets noticed.

EMPLOYEE

Remove Inappropriate?
Delete or Edit

Carl Steinbach, Employee, replied on November 05, 2009 19:22

Hi Theo,

This is probably happening because you haven't defined the fs.default.name property in your hadoop-site.xml configuration file, or possibly Hadoop is not able to find your hadoop-site.xml conf file.

this solves the problem

Add a comment This solves the problem Mark as a company solution
Remove Inappropriate?
Delete or Edit

theo replied on November 05, 2009 19:39

Carl:

Thanks for taking a look at this problem. I searched for the hadoop-site.xml file and find it in the distribution under src. However, the content of the file states emphatically:

DO NOT PUT ANY PROPERTY IN THIS FILE. INSTEAD USE core-site.xml

That file, core-site.xml, does exist in the /etc/hadoop-0.20/conf.pseudo configuration area, and in it, the property fs.default.name is set to hdfs://localhost:8020. I have no idea what that means, or what goes wrong. Is there more detailed information that describes the boot process?

I’m confused

this solves the problem

Add a comment This solves the problem
Remove Inappropriate?
Delete or Edit

theo replied on November 05, 2009 20:15

Maybe the better question to ask is how do you start the hadoop after an installation and you have restarted the server?

I installed Hadoop yesterday and was going through the documentation and all was fine. I then shutdown the server for the night, and then tried to continue where I left off from yesterday. I went back to the point were the documentation states to start all the Hadoop services with the script:

for service in /etc/init.d/hadoop-0.20-*
do
sudo $service start
done

And that is where I get stuck right from the start since none of the deamons fire up, all dying with a message from org.apache.hadoop.net.NetUtils.createSocketAddr(NetUtils.java:134) delivering a null pointer to the requesting function.

So, what is the right procedure to restart a Hadoop instance?

I’m frustrated

this solves the problem

Add a comment This solves the problem
EMPLOYEE

Remove Inappropriate?
Delete or Edit

Carl Steinbach, Employee, replied on November 05, 2009 21:03

Hi Theo,

The set of files required for configuration changed between Hadoop 0.19 and 0.20. My initial response referred to the pre 0.20 configuration files.

Given that you're working with 0.20, the configuration files you mentioned and the steps you outlined all sound good.

Two things:

- /etc/hadoop-0.20/conf should point to /etc/hadoop-0.20/conf.pseudo. Can you please verify that this is the case.

- Can you post the file contents of /etc/hadoop-0.20/conf.pseudo to pastebin and post the link to this ticket? I'll take a look and let you know if I find anything strange.

Thanks.

Carl

1 person says this solves the problem

Add a comment This solves the problem Mark as a company solution
Remove Inappropriate?
Delete or Edit

theo replied on November 05, 2009 22:54

[root@athena ~]# ll /etc/hadoop-0.20/conf
lrwxrwxrwx 1 root root 34 Nov 4 17:11 /etc/hadoop-0.20/conf -> /etc/alternatives/hadoop-0.20-conf
[root@athena ~]# ll /etc/alternatives/hadoop-0.20-conf/
total 60
-rwxr-xr-x 1 root root 3936 Nov 4 18:18 capacity-scheduler.xml
-rwxr-xr-x 1 root root 535 Nov 4 18:18 configuration.xsl
-rwxr-xr-x 1 root root 178 Nov 4 18:18 core-site.xml
-rwxr-xr-x 1 root root 3032 Nov 4 18:18 fair-scheduler.xml
-rwxr-xr-x 1 root root 2237 Nov 4 18:18 hadoop-env.sh
-rwxr-xr-x 1 root root 1489 Nov 4 18:18 hadoop-metrics.properties
-rwxr-xr-x 1 root root 4190 Nov 4 18:18 hadoop-policy.xml
-rwxr-xr-x 1 root root 178 Nov 4 18:18 hdfs-site.xml
-rwxr-xr-x 1 root root 3425 Nov 4 18:18 log4j.properties
-rwxr-xr-x 1 root root 178 Nov 4 18:18 mapred-site.xml
-rwxr-xr-x 1 root root 10 Nov 4 18:18 masters
-rwxr-xr-x 1 root root 10 Nov 4 18:18 slaves
-rwxr-xr-x 1 root root 1243 Nov 4 18:18 ssl-client.xml.example
-rwxr-xr-x 1 root root 1195 Nov 4 18:18 ssl-server.xml.example

I’m undecided

this solves the problem

Add a comment This solves the problem
Remove Inappropriate?
Delete or Edit

theo replied on November 05, 2009 22:59

so /etc/hadoop-0.20/conf does not point to what you indicated. Here is the content of /etc/hadoop-0.20/conf.pseudo:
[root@athena hadoop-0.20]# ll conf.pseudo/
total 64
-rwxr-xr-x 1 root root 3936 Nov 2 00:45 capacity-scheduler.xml
-rwxr-xr-x 1 root root 535 Nov 2 00:45 configuration.xsl
lrwxrwxrwx 1 root root 28 Nov 4 18:01 conf.pseudo -> /etc/hadoop-0.20/conf.pseudo
-rwxr-xr-x 1 root root 338 Nov 1 22:22 core-site.xml
-rwxr-xr-x 1 root root 3032 Nov 2 00:45 fair-scheduler.xml
-rwxr-xr-x 1 root root 2248 Nov 5 15:06 hadoop-env.sh
-rwxr-xr-x 1 root root 2108 Nov 1 22:22 hadoop-metrics.properties
-rwxr-xr-x 1 root root 4190 Nov 2 00:45 hadoop-policy.xml
-rwxr-xr-x 1 root root 496 Nov 1 22:22 hdfs-site.xml
-rwxr-xr-x 1 root root 3425 Nov 2 00:45 log4j.properties
-rwxr-xr-x 1 root root 213 Nov 1 22:22 mapred-site.xml
-rwxr-xr-x 1 root root 10 Nov 2 00:45 masters
-rwxr-xr-x 1 root root 321 Nov 1 22:22 README
-rwxr-xr-x 1 root root 10 Nov 2 00:45 slaves
-rwxr-xr-x 1 root root 1243 Nov 2 00:45 ssl-client.xml.example
-rwxr-xr-x 1 root root 1195 Nov 2 00:45 ssl-server.xml.example

So did I mess something up with this 'alternatives' environment? What is the state it needs to get back into? How do I recognize that it is proper again?

this solves the problem

Add a comment This solves the problem
Remove Inappropriate?
Delete or Edit

theo replied on November 06, 2009 00:24

Ok, I found the problem. I read up on this 'alternatives' and I realized that I had issued a set of inconsistent updates. I had three alternatives, regular, pseudo, and cluster and had upped the priority. But alternatives was stating that it was in manual mode and was selecting the cluster version. Once I set the 'alternatives' stuff so that it was pointing to conf.pseudo it fired back up.

Then I ran into the problem that the config files (hadoop-env.sh) states that you should set the JAVA_HOME environment. And that lead me down another bad path. Java is managed through alternatives as well so that was a mess. After I completely ignored the information in the config files and unset JAVA_HOME all is well.

My hadoop pseudo instance is back up..... just in time to shut down the server again for the evening. Let's hope I remember all this tomorrow morning.

I’m confident

this solves the problem

Add a comment This solves the problem

Reply to theo's problem

(Some HTML allowed)

Company participation

The mood in here

Products & Services

This topic is tagged

Related topics in Cloudera

NullPointerException when starting Hadoop service deamons