Aaron's dashboard

Aaron replied on December 18, 2009 04:36 to the problem "sqoop - Null Pointer Exception problem" in Cloudera:

Thanks for describing this. I've submitted a patch to Apache that fixes the issue (see https://issues.apache.org/jira/browse.... This'll probably go out in the next CDH release.

Cheers,
- Aaron

Aaron replied on December 18, 2009 03:38 to the problem "Generated jar with sqoop hiveimport lacks the generated class file" in Cloudera:

Chris, I think that this patch will fix this issue: https://issues.apache.org/jira/browse...

This is available on Hadoop trunk but not yet available in a public CDH release. This will probably be out after the holidays.

Cheers,
- Aaron

Aaron replied on December 18, 2009 01:35 to the question "sqoop hive import giving null query values" in Cloudera:

I think I've fixed this bug now. Test case and patch available at https://issues.apache.org/jira/browse...

Aaron replied on December 17, 2009 23:42 to the question "Decomissioning nodes" in Cloudera:

If you modify hdfs-site.xml, you'll have to restart the namenode before those changes take effect. The -refreshNodes argument only re-reads an already-configured dfs.hosts.exclude file.

Aaron replied on December 17, 2009 19:48 to the question "Wrong FS Error when trying to use Hive with S3" in Cloudera:

What is the exact DDL command you were running? Are you on Hadoop 0.20?

What happens if you try starting the hive shell with 'hive -D fs.default.name=s3n://blablabla' instead of just 'hive' ?

Aaron replied on December 16, 2009 22:27 to the problem "Generated jar with sqoop hiveimport lacks the generated class file" in Cloudera:

Good to know. Also added to the list.

Aaron replied on December 16, 2009 22:13 to the problem "Generated jar with sqoop hiveimport lacks the generated class file" in Cloudera:

Actually, you can manually specify the entire class name.

Try using --class-name com.mycompany.hadoop.friends

Aaron replied on December 16, 2009 22:02 to the problem "Generated jar with sqoop hiveimport lacks the generated class file" in Cloudera:

Ooh. There's a '.' in the table name. I don't think this is a case that I considered. Between that and the package you've chosen, it's getting confused about the filename and what package the class belongs in.

Note the discrepancy here:

09/12/16 13:43:55 DEBUG orm.ClassWriter: sourceFilename is com/mycompany/hadoop/mycomp/friends.java

09/12/16 13:43:57 DEBUG orm.CompilationManager: Considering entry: /tmp/sqoop/compile/com/mycompany/hadoop/friends.class

If you can workaround this by not putting a '.' in your tablename, I think that would be your best bet for today. In the meantime, I'll add this to the bug list.

Thanks for helping out!

Aaron replied on December 16, 2009 21:31 to the problem "Generated jar with sqoop hiveimport lacks the generated class file" in Cloudera:

That's strange. What was the exact set of command-line arguments you used?

Please enable DEBUG logging (see http://archive.cloudera.com/docs/sqoo... for instructions), run again, and paste in the trace to this thread.

Also, what JDK version are you using? (Run 'java version' and print the results here) What OS?

Thanks,
- Aaron

Aaron replied on December 15, 2009 04:45 to the question "where can I find hive in vm image?" in Cloudera:

You can start hive just by typing 'hive' at the command prompt. There's a directory on the desktop called "instructions" that contains all the documentation you should need to get started working with examples. This documentation is more up-to-date than any documentation on the web.

Aaron replied on December 11, 2009 19:17 to the problem "sqoop - Null Pointer Exception problem" in Cloudera:

Hm. That's unfortunate. Can you add some more information?

* What command line did you use to invoke sqoop?
* What is the schema of the table you were trying to import? (e.g., what does "DESCRIBE time_by_day" say?)

Aaron replied on December 09, 2009 19:24 to the question "Sqoop can't find the OracleDriver class" in Cloudera:

Karthik,

Is your Hadoop cluster multi-node? Is ojdbc6.jar in the $HADOOP_HOME/lib dir on all slave nodes? You may need to restart the tasktrackers after installing this library.

Aaron replied on December 07, 2009 21:07 to the question "New Tutorials using non-deprecated Classes?" in Cloudera:

We're in the process of updating things. New-API examples will be made available soon. For now, though, the existing API is still the preferred means of interacting with Hadoop. There are also a few "gotchas" associated with the new API that won't be corrected until 0.21, and introducing workarounds for these in example code is also similarly confusing.

Stay tuned...

Aaron replied on November 28, 2009 02:23 to the question "Getting Eclipse to Work in the Training Virtual Machine" in Cloudera:

If you do 'jar tf RowNumberTwoPass.jar', what directory hierarchy does it show inside? It might think that 'bin' is part of the package name inside the jar.

Try maybe instead switching to the bin dir before creating the jar:
cd bin && jar cvf RowNumberTwoPass.jar *

or equivalently:

jar cvf RowNumberTwoPass.jar -C bin .

Aaron replied on November 27, 2009 19:19 to the question "Getting Eclipse to Work in the Training Virtual Machine" in Cloudera:

The Yahoo VM isn't particularly well-maintained. It's running a much older version of Hadoop. Our VM stays current with our Hadoop distribution. Of course, given that the Hadoop project itself has more-or-less moved on from maintaining the Eclipse plugin over the past 18 months, take that as you will :)

If you're going to do serious Hadoop development, sooner or later you'll need to learn a bit about how the javac, jar, and ant tools work. You can probably start by taking one of the ant build.xml files from the sample projects and copy it into a new directory, and modify that to suit a new project.

Aaron replied on November 27, 2009 19:07 to the question "Getting Eclipse to Work in the Training Virtual Machine" in Cloudera:

The latest VM had a bug where it referenced jars that don't exist. You should remove libfb303 and libthrift from all projects' build paths. That will fix that problem. This will be corrected in the next VM release.

Also, the training VM does not come with the MapReduce perspective; the Hadoop eclipse plugin does not work in recent editions of Eclipse. You can run in local-mode in Eclipse or use JUnit, but for submitting to the cluster you will need to use the command line. See instructions in the various exercises (in the instructions/exercises/ folder on the desktop) for how to run your jobs.

- Aaron

Aaron replied on November 25, 2009 22:40 to the question "sqoop hive import giving null query values" in Cloudera:

Hi Jonathan,

Sorry to let this drop for a while. This is perplexing. I suspect that there might be some trailing characters or something else in the file that's making Hive misinterpret the columns. Could you maybe send me the part-m-00000 file that it creates? (email: aaron at cloudera dot com).

Also, which version of Hive are you using? Are you using the debian package from CDH?

Thanks,
- Aaron

Aaron replied on November 20, 2009 03:13 to the question "sqoop hive import giving null query values" in Cloudera:

Oops, the missing class error is due to commons-io-1.4.jar not being packaged with that release of CDH2. This is also logged as a bug to fix before the next release. If you want, you can fix this manually by downloading commons-io-1.4.jar from Apache Commons (commons.apache.org) and putting it in $HADOOP_HOME/lib/.

As for the non-direct import, this looks like a new bug. I'll have to look into writing a fix for it. Can you please change your log level to DEBUG, run the import again, and post the output here? To change your log level, follow these instructions: http://archive.cloudera.com/docs/sqoo...

Sorry for all the frustration -- thanks for helping out
- Aaron

Aaron replied on November 19, 2009 04:46 to the question "sqoop hive import giving null query values" in Cloudera:

Jonathan,

I've actually recently independently confirmed this bug as an open issue; it's on my TODO list to fix. The bug is specific to using mysql direct mode without manually specifying the delimiters.

For now, there is a workaround: explicitly set the delimiter characters.

e.g.: sqoop --connect <foo> --direct --table <bar> --fields-terminated-by '\0001' --lines-terminated-by '\n'
</bar></foo>

Aaron replied on November 10, 2009 19:51 to the question "How to run only one Reduce Task and it begins its execution only when all Map Tasks have been completed?" in Cloudera:

The task JVM will spawn early in order to start copying data from its input mappers, but your actual Reducer.reduce() method will never begin executing until all the mappers are finished; that is a contract already enforced by Hadoop for you.