Loading Profile...
sqoop getting java.lang.NoClassDefFoundError when trying to import a table
Everything is working well when I use sqoop to list databases or tables in my MySQL server but not when I want to import data into hdfs
sqoop --connect jdbc:mysql://127.0.0.1/sakila --username root --password root --table staff
09/07/21 12:31:01 INFO sqoop.Sqoop: Beginning code generation
09/07/21 12:31:02 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM staff AS t WHERE 1 = 1
09/07/21 12:31:02 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM staff AS t WHERE 1 = 1
09/07/21 12:31:02 ERROR orm.ClassWriter: Cannot resolve SQL type -4
09/07/21 12:31:02 ERROR orm.ClassWriter: No Java type for SQL type -4
09/07/21 12:31:02 ERROR orm.ClassWriter: No Java type for SQL type -4
09/07/21 12:31:02 ERROR orm.ClassWriter: No Java type for SQL type -4
09/07/21 12:31:02 ERROR orm.ClassWriter: No Java type for SQL type -4
09/07/21 12:31:02 ERROR orm.ClassWriter: No Java type for SQL type -4
09/07/21 12:31:02 DEBUG orm.ClassWriter: Writing source file: ./staff.java
09/07/21 12:31:02 DEBUG orm.ClassWriter: Table name: staff
09/07/21 12:31:02 DEBUG orm.ClassWriter: Columns: staff_id:-6, first_name:12, last_name:12, address_id:5, picture:-4, email:12, store_id:-6, active:-7, usern
ame:12, password:12, last_update:93,
09/07/21 12:31:02 DEBUG orm.ClassWriter: Could not create directory tree for .
09/07/21 12:31:02 DEBUG orm.CompilationManager: Warning: Could not make directories for /tmp/sqoop/compile/
09/07/21 12:31:02 INFO orm.CompilationManager: HADOOP_HOME is /usr/lib/hadoop
09/07/21 12:31:02 INFO orm.CompilationManager: Found hadoop core jar at: /usr/lib/hadoop/hadoop-0.18.3-6cloudera0.3.0-core.jar
09/07/21 12:31:02 INFO orm.CompilationManager: Invoking javac with args: -sourcepath ./ -d /tmp/sqoop/compile/ -classpath /etc/hadoop/conf:/usr/lib/jvm/java-
6-sun//lib/tools.jar:/usr/lib/hadoop:/usr/lib/hadoop/hadoop-0.18.3-6cloudera0.3.0-core.jar:/usr/lib/hadoop/lib/commons-cli-2.0-SNAPSHOT.jar:/usr/lib/hadoop/l
ib/commons-codec-1.3.jar:/usr/lib/hadoop/lib/commons-httpclient-3.0.1.jar:/usr/lib/hadoop/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop/lib/commons-logging-a
pi-1.0.4.jar:/usr/lib/hadoop/lib/commons-net-1.4.1.jar:/usr/lib/hadoop/lib/hadoop-0.18.3-6cloudera0.3.0-fairscheduler.jar:/usr/lib/hadoop/lib/hadoop-0.18.3-6
cloudera0.3.0-scribe-log4j.jar:/usr/lib/hadoop/lib/hsqldb.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jetty-5.1.4.jar:/usr/lib/hadoop/lib/ju
nit-4.5.jar:/usr/lib/hadoop/lib/kfs-0.1.3.jar:/usr/lib/hadoop/lib/libfb303.jar:/usr/lib/hadoop/lib/libthrift.jar:/usr/lib/hadoop/lib/log4j-1.2.15.jar:/usr/li
b/hadoop/lib/mysql-connector-java-5.0.8-bin.jar:/usr/lib/hadoop/lib/oro-2.0.8.jar:/usr/lib/hadoop/lib/servlet-api.jar:/usr/lib/hadoop/lib/slf4j-api-1.4.3.jar
:/usr/lib/hadoop/lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/jetty-ext/commons-el.jar:/usr/lib/hadoop/lib/jetty-ext/j
asper-compiler.jar:/usr/lib/hadoop/lib/jetty-ext/jasper-runtime.jar:/usr/lib/hadoop/lib/jetty-ext/jsp-api.jar:/usr/lib/hadoop/hadoop-0.18.3-6cloudera0.3.0-co
re.jar:/usr/lib/hadoop/contrib/sqoop/hadoop-0.18.3-6cloudera0.3.0-sqoop.jar ./staff.java
java.lang.NoClassDefFoundError: com/sun/tools/javac/Main
at org.apache.hadoop.sqoop.orm.CompilationManager.compile(CompilationManager.java:166)
at org.apache.hadoop.sqoop.Sqoop.generateORM(Sqoop.java:65)
at org.apache.hadoop.sqoop.Sqoop.importTable(Sqoop.java:76)
at org.apache.hadoop.sqoop.Sqoop.run(Sqoop.java:160)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.sqoop.Sqoop.main(Sqoop.java:178)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
Caused by: java.lang.ClassNotFoundException: com.sun.tools.javac.Main
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 16 more
FYI, I've enabled sqoop debug in /etc/hadoop/conf/log4j.properties
sqoop --connect jdbc:mysql://127.0.0.1/sakila --username root --password root --table staff
09/07/21 12:31:01 INFO sqoop.Sqoop: Beginning code generation
09/07/21 12:31:02 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM staff AS t WHERE 1 = 1
09/07/21 12:31:02 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM staff AS t WHERE 1 = 1
09/07/21 12:31:02 ERROR orm.ClassWriter: Cannot resolve SQL type -4
09/07/21 12:31:02 ERROR orm.ClassWriter: No Java type for SQL type -4
09/07/21 12:31:02 ERROR orm.ClassWriter: No Java type for SQL type -4
09/07/21 12:31:02 ERROR orm.ClassWriter: No Java type for SQL type -4
09/07/21 12:31:02 ERROR orm.ClassWriter: No Java type for SQL type -4
09/07/21 12:31:02 ERROR orm.ClassWriter: No Java type for SQL type -4
09/07/21 12:31:02 DEBUG orm.ClassWriter: Writing source file: ./staff.java
09/07/21 12:31:02 DEBUG orm.ClassWriter: Table name: staff
09/07/21 12:31:02 DEBUG orm.ClassWriter: Columns: staff_id:-6, first_name:12, last_name:12, address_id:5, picture:-4, email:12, store_id:-6, active:-7, usern
ame:12, password:12, last_update:93,
09/07/21 12:31:02 DEBUG orm.ClassWriter: Could not create directory tree for .
09/07/21 12:31:02 DEBUG orm.CompilationManager: Warning: Could not make directories for /tmp/sqoop/compile/
09/07/21 12:31:02 INFO orm.CompilationManager: HADOOP_HOME is /usr/lib/hadoop
09/07/21 12:31:02 INFO orm.CompilationManager: Found hadoop core jar at: /usr/lib/hadoop/hadoop-0.18.3-6cloudera0.3.0-core.jar
09/07/21 12:31:02 INFO orm.CompilationManager: Invoking javac with args: -sourcepath ./ -d /tmp/sqoop/compile/ -classpath /etc/hadoop/conf:/usr/lib/jvm/java-
6-sun//lib/tools.jar:/usr/lib/hadoop:/usr/lib/hadoop/hadoop-0.18.3-6cloudera0.3.0-core.jar:/usr/lib/hadoop/lib/commons-cli-2.0-SNAPSHOT.jar:/usr/lib/hadoop/l
ib/commons-codec-1.3.jar:/usr/lib/hadoop/lib/commons-httpclient-3.0.1.jar:/usr/lib/hadoop/lib/commons-logging-1.0.4.jar:/usr/lib/hadoop/lib/commons-logging-a
pi-1.0.4.jar:/usr/lib/hadoop/lib/commons-net-1.4.1.jar:/usr/lib/hadoop/lib/hadoop-0.18.3-6cloudera0.3.0-fairscheduler.jar:/usr/lib/hadoop/lib/hadoop-0.18.3-6
cloudera0.3.0-scribe-log4j.jar:/usr/lib/hadoop/lib/hsqldb.jar:/usr/lib/hadoop/lib/jets3t-0.6.1.jar:/usr/lib/hadoop/lib/jetty-5.1.4.jar:/usr/lib/hadoop/lib/ju
nit-4.5.jar:/usr/lib/hadoop/lib/kfs-0.1.3.jar:/usr/lib/hadoop/lib/libfb303.jar:/usr/lib/hadoop/lib/libthrift.jar:/usr/lib/hadoop/lib/log4j-1.2.15.jar:/usr/li
b/hadoop/lib/mysql-connector-java-5.0.8-bin.jar:/usr/lib/hadoop/lib/oro-2.0.8.jar:/usr/lib/hadoop/lib/servlet-api.jar:/usr/lib/hadoop/lib/slf4j-api-1.4.3.jar
:/usr/lib/hadoop/lib/slf4j-log4j12-1.4.3.jar:/usr/lib/hadoop/lib/xmlenc-0.52.jar:/usr/lib/hadoop/lib/jetty-ext/commons-el.jar:/usr/lib/hadoop/lib/jetty-ext/j
asper-compiler.jar:/usr/lib/hadoop/lib/jetty-ext/jasper-runtime.jar:/usr/lib/hadoop/lib/jetty-ext/jsp-api.jar:/usr/lib/hadoop/hadoop-0.18.3-6cloudera0.3.0-co
re.jar:/usr/lib/hadoop/contrib/sqoop/hadoop-0.18.3-6cloudera0.3.0-sqoop.jar ./staff.java
java.lang.NoClassDefFoundError: com/sun/tools/javac/Main
at org.apache.hadoop.sqoop.orm.CompilationManager.compile(CompilationManager.java:166)
at org.apache.hadoop.sqoop.Sqoop.generateORM(Sqoop.java:65)
at org.apache.hadoop.sqoop.Sqoop.importTable(Sqoop.java:76)
at org.apache.hadoop.sqoop.Sqoop.run(Sqoop.java:160)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.sqoop.Sqoop.main(Sqoop.java:178)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)
Caused by: java.lang.ClassNotFoundException: com.sun.tools.javac.Main
at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
... 16 more
FYI, I've enabled sqoop debug in /etc/hadoop/conf/log4j.properties
I’m sad
1
person has this question
I have this question, too!
Tell me when someone answers.
The more people who ask this question, the more it gets noticed.
The more people who ask this question, the more it gets noticed.
-
Inappropriate?Hi srand,
Sorry for the trouble :( It seems as though sqoop can't find your Java compiler. (It currently auto-generates a class for your table definition, and then compiles and loads that class.) Where does your JAVA_HOME variable point? If it's at a JRE, then this won't work. If it's at a JDK, then it should. In your $JAVA_HOME/lib/ directory there should be a file named tools.jar -- that's what Sqoop currently can't find.
Thanks for the full debug output by the way --- it's very helpful.
Looking at the classpath argument, it seems to be trying /usr/lib/jvm/java-6-sun//lib/tools.jar. Can you let me know if this file exists? If it does, what does the output of:
jar tf /usr/lib/jvm/java-6-sun/lib/tools.jar | grep com/sun/tools/javac/Main
report?
Thanks,
- Aaron -
Inappropriate?Hi Aaron,
thanks for your suggestion. I hadn't set JAVA_HOME as hadoop was working well without specifying it, I was thinking that each binaries-oop was finding the JDK by itself.
Now it 's working better but .......... fails !
09/07/21 22:57:23 WARN mapred.JobClient: Error reading task outputConnection refused
09/07/21 22:57:23 WARN mapred.JobClient: Error reading task outputConnection refused
09/07/21 22:57:24 INFO mapred.JobClient: Task Id : attempt_200907211208_0006_m_000001_2, Status : FAILED
Failed to rename output with the exception: java.io.IOException: Can not get the relative path: base = hdfs://hadoopM:54310/user/hadoop/store/_temporary/_attempt_200907211208_0006_m_000001_2 child = hdfs://HadoopM:54310/user/hadoop/store/_temporary/_attempt_200907211208_0006_m_000001_2/part-00001
at org.apache.hadoop.mapred.Task.getFinalPath(Task.java:586)
at org.apache.hadoop.mapred.Task.moveTaskOutputs(Task.java:599)
at org.apache.hadoop.mapred.Task.moveTaskOutputs(Task.java:617)
at org.apache.hadoop.mapred.Task.saveTaskOutput(Task.java:561)
at org.apache.hadoop.mapred.JobTracker$TaskCommitQueue.run(JobTracker.java:2061)
09/07/21 22:57:24 WARN mapred.JobClient: Error reading task outputConnection refused
09/07/21 22:57:24 WARN mapred.JobClient: Error reading task outputConnection refused
09/07/21 22:57:26 ERROR sqoop.Sqoop: Encountered IOException running import job: java.io.IOException: Job failed!
I have to investiguate.
thanks
-
Inappropriate?Hi srand,
I think this is a subtle Hadoop configuration issue. Hadoop is extremely particular about the NameNode's canonical address, but it breaks in non-obvious ways. Somewhere you're referring to hdfs://hadoopM:54310 and elsewhere you use hdfs://HadoopM:54310 (with a capital "H"). DNS may resolve both of these to the same IP, but the IP necessarily can't reverse-dns to both of these at once. You should check that your various options in the hadoop config xml files and and any parameters you use all refer to the NameNode via the exact same DNS name. -
Inappropriate?You were absolutely right ! I changed the hostname of the namenode from HadoopM to hadoopM and now it's working !
It's a really strange behaviour and I would'nt have find it without your help
thanks אהרון ! (Aaron better written :)
I’m happy
EMPLOYEE
