SyRenity's dashboard

SyRenity asked a question in Cloudera on September 29, 2009 20:46:

Monitor Hadoop processes with watchdog

Hi.

What is the best way to monitor Hadoop processes with a watchdog - say monit for example?

Hadoop doesn't make a pid file, which makes it difficult to work with monit.

Thanks in advance!

SyRenity replied on September 24, 2009 16:20 to the question "Should we release/host RPMs for all releases?" in Cloudera:

Hi.

Just wanted to follow-up on the question - how much 0.20 stable, and what are the upgrade instructions?

Thanks!

SyRenity replied on September 01, 2009 18:55 to the question "How much RAM Datanode should take?" in Cloudera:

Hi.

Will check and report.

Any idea if these 2 tickets may have any effect on my issue?

https://issues.apache.org/jira/browse...

https://issues.apache.org/jira/browse...

Thanks.

SyRenity replied on August 31, 2009 22:12 to the question "How much RAM Datanode should take?" in Cloudera:

It's disabled and stands on 2000:

# The maximum amount of heap to use, in MB. Default is 1000.
# export HADOOP_HEAPSIZE=2000

Should I enable it and lower to say 256, if the only thing I use there is Datanode?

Or it will cause OOM's?

SyRenity replied on August 31, 2009 19:52 to the question "How much RAM Datanode should take?" in Cloudera:

Hi.

1) It was an idle test cluster with 2 DN - nothing was written or read from it during the check.

2) I'm actually not sure what the -Xmx value would be, as I launched it using Cloudera wizard build config.
By the way, I do remember specifying the total RAM of the node in Cloudera wizard as1GB, I think, (it's a VM with limited memory), could it be that Cloudera set this as Xmx and let Java to use it all?

3) In my hadoop-site.conf, there is the following value:
<property>
<name>mapred.child.java.opts</name>
<value>-Xmx250m</value>
</property>

But it seems to be related only to map reduce, and not to DFS - correct?

SyRenity asked a question in Cloudera on August 31, 2009 17:11:

How much RAM Datanode should take?

I measured the Datanode memory usage, and noticed they take up to 700 MB of RAM.

As their main job is to store files to disk, any idea why they take so much RAM?

SyRenity replied on August 26, 2009 11:54 to the question "Should we release/host RPMs for all releases?" in Cloudera:

Hi.

Two questions:

1) How stable the 0.20 release? I see it's in testing stage, any idea when it will be usable for production?

2) What are the upgrade procedures to go from 0.18.3 to 0.20?

Thanks in advance!

SyRenity asked a question in Cloudera on August 12, 2009 10:59:

Secondary NameNode is not working?

Hi.

I've moved the Secondary Name Node to another node, but it doesn't seems to work at all.

There is no any check-point data in SecNameNode hadoop data directory.

When I checked the logs, I seen a lot of "connection refused" exceptions.

Any idea what causing this?

SyRenity replied on August 08, 2009 12:53 to the question "Compiling x86_64 libraries" in Cloudera:

Hi Matt.

Thanks for answering, that what I looked for!

Stas.

SyRenity marked one of matt's replies in Cloudera as useful. matt replied to the question "Compiling x86_64 libraries".

SyRenity replied on August 02, 2009 11:56 to the question "Compiling x86_64 libraries" in Cloudera:

Hi.

Can someone comment regarding the above?

Thanks!

SyRenity replied on July 28, 2009 09:16 to the question "Compiling x86_64 libraries" in Cloudera:

Hi.

I finally got some time to install the latest Cloudera "0.18.3-14.cloudera.CH0_3".

I noticed that there are hadoop-native.x86_64 packages and have installed them.

Two questions:

1) Any idea if this basically replaces the need to compile the 64-bit libs myself, and provides full native Hadoop 64-bit integration?

2) Do I need to do anything specific to use them - or they are used automatically?

Thanks!

SyRenity replied on June 29, 2009 18:17 to the question "Is hadoop-4346 fix included in Cloudera?" in Cloudera:

Great, thanks for the info.

SyRenity replied on June 22, 2009 20:57 to the question "Is hadoop-4346 fix included in Cloudera?" in Cloudera:

Cool - Cloudera saves the day! :)

Just a small question - it was fixed in the initial version of Cloudera (released on Match?) or in the latest one, released a ~month ago.

Thanks Christophe.

SyRenity asked a question in Cloudera on June 22, 2009 18:54:

Is hadoop-4346 fix included in Cloudera?

Hi.

I wanted to clarify, is HADOOP-4346 fix included in latest Cloudera distribution (the latest 0.18.3 version)?

The issue seems quite severe, and I have a reason to believe I have encountered it myself.

Thanks for information!

SyRenity replied on June 02, 2009 23:22 to the question "Hadoop update procedure" in Cloudera:

Thanks.

SyRenity marked one of Tom's replies in Cloudera as useful. Tom replied to the question "Hadoop update procedure".

SyRenity replied on June 01, 2009 18:12 to the question "Fixes in latest Cloudera update" in Cloudera:

Great, please do.

This patch main benefit that application under any context can see how much free space and how much space is left.

Without it the only way to find out these details is to run the application under root context, which is much more worse from security POV.

Regards.

SyRenity asked a question in Cloudera on June 01, 2009 10:42:

Fixes in latest Cloudera update

Hi.

Any idea if the latest update includes the following fix:

Allows non-administrative user to get the overall disk space (aka HADOOP-4368)?

Regards.

SyRenity asked a question in Cloudera on June 01, 2009 10:37:

Hadoop update procedure

Hi.

As the new 0.18.3 version of Cloudera Hadoop came out, I wanted to ask about the upgrade process.

Is there any specific way to upgrade it, or I can simply run the yum update on all the nodes and it would be enough?

Regards.