Answering Cloudera configurator questions with NFS directories

Also, the following properties can specify multiple directories (on multiple drives):

dfs.name.dir
dfs.data.dir
mapred.local.dir

The namenode will write multiple copies of its filesystem metadata: one in each directory specified in dfs.name.dir. This is for reasons of resilience: if one drive is lost then the metadata is still safe. You should make sure that one of these drives is NFS so you have a remote backup. I wouldn't recommend using an NFS drive for any of the other settings.

Specifying multiple drives for dfs.data.dir and mapred.local.dir has a different meaning. Rather than replicating data between them, the datanode/tasktracker just uses the multiple drives in a round-robin fashion for storage. You shouldn't use NFS for these since you want to take advantage of local disk speeds.

Tom

Thanks for your replies. But I have a specific problem. If the NFS directory has the same on all the client machines. Eg. /user/rja/hadoop_dirs

Then every client will be writing to the same directory . In other words Client A will be overwriting Client B's data. I see a lot of NFS Stale handle in the hadoop log files, and I suspect that the scenario described above must have taken place.

Hi,

First off, what type of data are you storing in a NFS mount? Typically NFS is only useful for duplicating the name node metadata on a different machine. It's most likely a bad idea for a NFS mount be shared by all task trackers and data nodes, as the NFS server will probably be a huge bottleneck for any MapReduce job. If you can avoid using NFS for data nodes and task trackers, then you should. It doesn't fit well with the concept of a distributed filesystem.

Lastly, if you have several machines all mounting the same NFS server, you'll definitely need top-level directories for each node in the NFS root directory to ensure nodes don't step on each others' feet.

Alex

.

Why did I use NFS ? The lab setup I inherited had a large amount of space on nfs directories. That is one reason to use it. I did keep the anythings starting with "dfs" as local. Example dfs.data.dir. But I kept names starting with hadoop as Nfs e.g "hadoop,tmp.dir"/

Secondly, I was lazy in copying the same code to all the nodes and keeping to same hadoop version in all. So i installed hadoop source in a NFS directory

i added hadoop.tmp.dir to the NFS. From the name "tmp.dir" looked harmless and very little IO , but it gives the above problem.

From your reply, it seems correct use to duplicate namenode meta data via a common NFS directory.

Note: When i mean NFS, i mean the pathname on any of the nodes will lead to the same file.

I don't recommend setting anything other than the dfs.name.dir to a NFS mount. Note that dfs.name.dir is a comma-delimited list of directories. At least one of these directories should be on the name node itself, and the other should be the NFS mount. The configuration I just mentioned will store name node metadata on two machines: the NFS server and the name node itself. This is ideal for recovering the name node if it catches fire.

Configuring hadoop.tmp.dir to an NFS mount is going to slow down your MapReduce jobs significantly. Each task tracker will write its temporary data here, which means you'll be getting tons of simultaneous writes to the same NFS server. I recommend configuring hadoop.tmp.dir to be local.

It sounds like you want to use NFS because you have a lot of disk space there. I understand that you want to utilize the available disk space there, but doing so will only harm your Hadoop cluster. However, using NFS to distribute configuration, etc., isn't that crazy of an idea. You might also mount each user's home directory on NFS. This will allow your home directory on each machine to be the same.

Hope this helps.

Alex

Good answers. Thanks I have removed NFS from all of the above uses. In fact, I was quite surprised that I could simply do a cp -r <nfs> <local> on the name dfs.name.dir and restart hadoop.

Two More questions.

Now only the source code is on NFS.

I find hadoop.log by default goes to the "logs" directory. The hadoop wiki says use log4j configuration. That seems to be the only write that is happening TO Nfs.

#1 How can i optimize this. Should I give local disk space for logging or NFS.

#2 If the logs are written to local disk on each of the nodes, will the logs be collated together for viewing by the hadoop framework ?
</local></nfs>

Hi,

I wouldn't recommend writing log files to the NFS for the same reasons I mentioned above. The NFS server will get slammed when all of your nodes write to it. Hadoop by default does not collect logs to a central place for you, but we've written some software to allow for this.

Facebook released a tool called Scribe that is meant for collecting logs. It uses a Thrift interface. I recommend installing a central Scribe server on your NFS box and using our log4j Scribe appender to collect Hadoop logs. Take a look here:

http://www.cloudera.com/blog/2008/11/...

Alex

Besides dfs.name.dir, does it make sense to direct fs.checkpoint.dir to the NFS?

I run a secondary namenode on a separate machine. Both the 2NN and NN have two local partitions as well as the NFS. My NFS partition isn't particularly fast, and I'm not completely clear how write-intensive the checkpoint operation is. On the other hand, the NFS server is fault-tolerant and is regularly backed up, and I like the idea of having the fs metadata be recoverable.

Right now, I direct dfs.name.dir to both a local drive and the NFS partition, and fs.checkpoint.dir to the NFS only. Do you have any advice on whether this is likely to be choking performance or unnecessarily redundant?

Hi Philip,

What you're doing with the NameNode (NN) and NFS makes perfect sense. However, I'm a little confused why you're using NFS for the SecondaryNameNode (SNN). The SNN will fetch the FS image file and edits log from the NN over HTTP every so often (by default, each hour). It will then do a HTTP post back to the NN with the new FS image. fs.checkpoint.dir specifies where on the SNN machine the temporary FS image and edits log are stored to.

So anyway, it doesn't make sense to have the SNN write to a NFS. If you want to do periodic backups of the NN metadata, then you can fetch the FS image at http://hostname.of.namenode:50070/get... and the edits log at http://hostname.of.namenode:50070/get... via a cron job.

Learn more here: http://hadoop.apache.org/common/docs/...+NameNode

Let me know what you think, Philip.

Alex