Get your own customer support community

About me

Recent activity

Subscribe to this feed
  • problem

    Simonluca Landi replied on June 18, 2009 19:35 to the problem "sqoop OutOfMemoryError while importing table" in Cloudera:

    Simonluca Landi
    Aaron,
    I think this post could help: http://benjchristensen.wordpress.com/...

    Thanks for your work
    -=[SLL]=-
    Do you think this solves the problem? Mark it as solved
  • problem

    Simonluca Landi reported a problem in Cloudera on June 18, 2009 14:37:

    Simonluca Landi
    sqoop OutOfMemoryError while importing table
    Hi all.
    I'm trying to import a few GBs table from MySql to Hadoop using sqoop, but the job fails. The job is running with 1GB ram.
    The stack trace is:
    09/06/18 16:19:31 INFO sqoop.Sqoop: Beginning code generation
    09/06/18 16:19:31 INFO manager.SqlManager: Executing SQL statement: SELECT t.* FROM outbound_messages AS t WHERE 1 = 1
    java.lang.OutOfMemoryError: Java heap space
    at com.mysql.jdbc.Buffer.getBytes(Buffer.java:198)
    at com.mysql.jdbc.Buffer.readLenByteArray(Buffer.java:318)
    at com.mysql.jdbc.MysqlIO.nextRow(MysqlIO.java:1375)
    at com.mysql.jdbc.MysqlIO.readSingleRowSet(MysqlIO.java:2369)
    at com.mysql.jdbc.MysqlIO.getResultSet(MysqlIO.java:451)
    at com.mysql.jdbc.MysqlIO.readResultsForQueryOrUpdate(MysqlIO.java:2076)
    at com.mysql.jdbc.MysqlIO.readAllResults(MysqlIO.java:1451)
    at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:1787)
    at com.mysql.jdbc.Connection.execSQL(Connection.java:3283)
    at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1332)
    at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1467)
    at org.apache.hadoop.sqoop.manager.SqlManager.execute(SqlManager.java:254)
    at org.apache.hadoop.sqoop.manager.SqlManager.getColumnTypes(SqlManager.java:97)
    at org.apache.hadoop.sqoop.orm.ClassWriter.generate(ClassWriter.java:445)
    at org.apache.hadoop.sqoop.Sqoop.generateORM(Sqoop.java:64)
    at org.apache.hadoop.sqoop.Sqoop.importTable(Sqoop.java:76)
    at org.apache.hadoop.sqoop.Sqoop.run(Sqoop.java:160)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
    at org.apache.hadoop.sqoop.Sqoop.main(Sqoop.java:178)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
    at org.apache.hadoop.mapred.JobShell.run(JobShell.java:54)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
    at org.apache.hadoop.mapred.JobShell.main(JobShell.java:68)

    Any help will be appreciated

    Thanks