Michael Driscoll's dashboard

Michael Driscoll replied on November 05, 2009 01:44 to the question "is it possible to add more nodes to a running cluster?" in Cloudera:

Dayakar (and anyone who may be reading this) - You'll also want to make sure you read this other response,

Balancer Won't Start http://getsatisfaction.com/cloudera/t...

This issue didn't rear it's head until I attempted to run the rebalancer after shutting down some nodes, then adding some nodes.

Michael Driscoll replied on November 05, 2009 00:13 to the question "Balancer wont start" in Cloudera:

Thanks Alex - I had this exact same issue trying to do remove nodes, then add nodes on a cluster running on EC2. It look like adding that :8020 to hadoop-site.xml cleared it up.

Michael Driscoll started following the question "is it possible to add more nodes to a running cluster?" in Cloudera.

A comment on the question "cloudera-desktop missing from debian repo" in Cloudera:

Agree that the question is a little vague, but of course it works for you :) -- you work at Cloudera! – Michael Driscoll, on November 01, 2009 22:40

Michael Driscoll replied on November 01, 2009 22:36 to the question "cloudera-desktop missing from debian repo" in Cloudera:

Zenkalia - I had a similar issue, just double-check that your /etc/apt/sources.list has the right lines. Running Ubuntu Intrepid, I added the following lines:

deb http://archive.cloudera.com/debian intrepid-testing contrib

deb-src http://archive.cloudera.com/debian intrepid-testing contrib

deb http://archive.cloudera.com/debian intrepid-stable contrib

deb-src http://archive.cloudera.com/debian intrepid-stable contrib

Then added the Cloudera key with:

curl -s http://archive.cloudera.com/debian/ar... | sudo apt-key add -

This is documented at the following page at Cloudera:

http://archive.cloudera.com/docs/_apt...

Michael Driscoll asked a question in Cloudera on November 01, 2009 13:49:

Hadoop Namenode issues with Cloudera EBS scripts

I've been using the Cloudera's EBS scripts to launch and persist clusters over time. But the last few times I've been getting the following error when doing any basic HDFS operations (such as 'hadoop fs -ls').

"Bad connection to FS. command aborted."

I restarted my cluster and began from scratch. But the error reappeared.

I took a look at my Hadoop logs and found the following:

ERROR dfs.NameNode: java.net.UnknownHostException: Invalid hostname for server: ec2-174-129-103-21.compute-1.amazonaws.com

I guess I could manually change these to the internal IP, but I'm wondering if anyone could shed insight on why that's not happening automatically?

Thanks in advance!

Michael Driscoll marked one of Tom's replies in Cloudera as useful. Tom replied to the question "Persisting data on EBS across varying cluster sizes".

Michael Driscoll marked one of Tom's replies in Cloudera as useful. Tom replied to the question "Possible to include R in the cloudera-hadoop(32/64)-ami?".

Michael Driscoll started following the problem "Cloudera's Distribution for Hadoop EC2 with EBS" in Cloudera.

Michael Driscoll started following the question "Update configuration files cluster-wide using hadoop-ec2" in Cloudera.

Michael Driscoll marked one of Tom's replies in Cloudera as useful. Tom replied to the question "Update configuration files cluster-wide using hadoop-ec2".

Michael Driscoll replied on September 20, 2009 02:26 to the question "Attaching additional nodes and volumes to EBS" in Cloudera:

Abhijit - I just (unknowingly) asked a similar question, I believe. My question is titled Persisting data on EBS across varying cluster sizes -- hopefully if it's answered, that may help you out too.

Michael Driscoll started following the question "Attaching additional nodes and volumes to EBS" in Cloudera.

Michael Driscoll asked a question in Cloudera on September 20, 2009 01:12:

Persisting data on EBS across varying cluster sizes

I've been using Cloudera launch scripts that support HDFS storage on EBS, using the fine Cloudera EBS tutorial:

I'm wondering whether it's possible to vary the number of slaves used between cluster launches.

For example, I'd like to launch a cluster with 2 slaves, run some analysis and store results in HDFS.

% hadoop-ec2 launch-cluster my-ebs-cluster 2
% ... stuff ...
% hadoop fs -put mystuff output


Then, later on, I'd like to have a cluster with 8 slaves, and have the same HDFS data available.

% hadoop-ec2 launch-cluster my-ebs-cluster 8
% hadoop fs -ls output


I'm not clear on how -- or if -- the new cluster can rebalance data across these additional nodes.