zjffdu's dashboard

zjffdu replied on November 02, 2009 10:28 to the question "How to read hdfs outside ec2 ?" in Cloudera:

I changed the conf file in ec2, set the dfs.default.name and mapred.job.tracker using public DNS, then I can connect to hdfs, and can execute command hadoop fs -ls dir
. But I can not execute command hadoop fs -cat file.txt and can not submit job to jobtracker ( put pig.jar to hdfs), You suggestion of using proxy will work in command.

So how can I tell the Java API that I want to use the proxy ?

zjffdu replied on November 01, 2009 06:09 to the question "How to read hdfs outside ec2 ?" in Cloudera:

Hi Carl, I know the Java API, but does it work outside ec2 ?

I have my own machine as pig client and have my hdfs cluster in ec2, but when I run pig script, It seems that I can not read from data node.

zjffdu replied on October 31, 2009 02:01 to the question "How to read hdfs outside ec2 ?" in Cloudera:

Carl, thank you for your answer. Now I want to read files using Java API, it there any way to do that ?

zjffdu asked a question in Cloudera on October 30, 2009 12:33:

How to read hdfs outside ec2 ?

as a client outside ec2, I can run command hadoop fs -ls /

but I can not run hadoop fs -cat /output.txt

It looks like I should open some port on slaves, anyone know which should I open ?

Thank you.

zjffdu replied on October 30, 2009 10:54 to the question "How can I run pig script when my client is outside of ec2 ?" in Cloudera:

I found the problem.
Because fs.default.name use private DNS rather than public DNS.
I think using public DNS is better so that outside users can also use the cluster in EC2.

zjffdu asked a question in Cloudera on October 30, 2009 10:47:

The web UI of hdfs is not accessible on ec2 ?

The web UI of hdfs is not accessible on ec2?

I install hadoop using cloudera's distribution on ec2. But I can not access the web ui of hdfs on port 50070, but I can access the job tracker ui on port 50030.

And I have already grant permission to the group on these ports

Anyone know the problem?

Thank you

Jeff zhang

zjffdu asked a question in Cloudera on October 30, 2009 10:42:

How can I run pig script when my client is outside of ec2 ?

Anyone have ever used the no-ec2 machine as client of pig ?

Today I'd like to have my own machine as client for pig which is installed on ec2.

and when I submit pig script, the Path always contain the private DNS rather than the public DNS, of course I can not connect to the private DNS:

The returned Path from RPC: hdfs://ip-10-242-205-155.ec2.internal:9000/hadoop/system/mapred
But I'd like it to be: hdfs://ec2-174-129-75-73.compute-1.amazonaws.com:9000/hadoop/system/mapred so I can connect it

Anyone has encountered this problem before ?

Thank you.

Jeff zhang

zjffdu asked a question in Cloudera on June 06, 2009 11:49:

Thanks

about pig, how can I get the exception message in hadoop mode ?
my script works ok in local mode,but failed in mapreduce mode, maybe the load func or udf cause this, Is there any way I can get the exception iformation on the slave node.

Thank you

zjffdu replied on May 08, 2009 09:41 to the question "error when use the command : hadoop fs -put /etc/hadoop/conf/*.xml" in Cloudera:

I think my problem is the same with this one.

http://getsatisfaction.com/cloudera/t...

zjffdu replied on May 08, 2009 09:09 to the question "error when use the command : hadoop fs -put /etc/hadoop/conf/*.xml" in Cloudera:

Seems I can only do the ls operation, if I want to some file tranfer operation there will be errors, this is the error message I invoke the command: copyFromLocal.

copyFromLocal: java.io.IOException: File /user/root/input/ec2-ami-tools-1.3-19974.noarch.rpm could only be replicated to 0 nodes, instead of 1
Exception closing file /user/root/input/ec2-ami-tools-1.3-19974.noarch.rpm
java.io.IOException: Filesystem closed
at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:198)
at org.apache.hadoop.hdfs.DFSClient.access$600(DFSClient.java:65)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.closeInternal(DFSClient.java:3084)
at org.apache.hadoop.hdfs.DFSClient$DFSOutputStream.close(DFSClient.java:3053)
at org.apache.hadoop.hdfs.DFSClient$LeaseChecker.close(DFSClient.java:942)
at org.apache.hadoop.hdfs.DFSClient.close(DFSClient.java:210)
at org.apache.hadoop.hdfs.DistributedFileSystem.close(DistributedFileSystem.java:243)
at org.apache.hadoop.fs.FsShell.close(FsShell.java:1842)
at org.apache.hadoop.fs.FsShell.main(FsShell.java:1856)

zjffdu replied on May 08, 2009 09:05 to the question "error when use the command : hadoop fs -put /etc/hadoop/conf/*.xml" in Cloudera:

Yes, I run it on EC2

zjffdu replied on May 08, 2009 04:35 to the question "error when use the command : hadoop fs -put /etc/hadoop/conf/*.xml" in Cloudera:

I remove that folder, but the error still exist.

zjffdu replied on May 08, 2009 01:21 to the question "error when use the command : hadoop fs -put /etc/hadoop/conf/*.xml" in Cloudera:

sorry for mistyping,
the whole command : hadoop fs -put /etc/hadoop/conf/*.xml input

I follow the way on this page: http://www.cloudera.com/hadoop-ec2

zjffdu asked a question in Cloudera on May 07, 2009 16:02:

error when use the command : hadoop fs -put /etc/hadoop/conf/*.xml

This is the error message:

09/05/07 11:56:41 WARN dfs.DFSClient: Error Recovery for block null bad datanode[0]
put: Could not get block locations. Aborting...
Could not get block locations. Aborting...
Target input/hadoop-site.xml already exists

Exception closing file /user/root/input/fairscheduler.xml
java.io.IOException: Could not get block locations. Aborting...
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2153)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1745)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1899)
Exception closing file /user/root/input/hadoop-default.xml
java.io.IOException: Could not get block locations. Aborting...
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:2153)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1400(DFSClient.java:1745)
at org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1899)

Does anyone know the reasons ?

Thank you very much.