hadoop集群外运行sqoop报错

Hadoop版本:Apache 版本Hadoop2.6.0
Sqoop版本:Apache版本Sqoop1.4.6
在集群外的一台机器上安装了sqoop,并在这台机器上配置了要连接的hadoop集群的
fs.defaultFS和yarn.resourcemanager.hostname,并指定定了HADOOP_HOME环境变量
尝试通过sqoop将mysql中的一个表中的数据导入到HDFS中,出现如下错误,
[hadoopStudy@hadoopStudy1 ~]$ sqoop import --connect jdbc:mysql://localhost:3306/sqoopTest --username root --password hadoopStudy2016.5c --query 'select * from student where $CONDITIONS' --split-by id --target-dir /sqoopDir -m2 --null-string '' --null-non-string ''
Warning: /data/application/sqoop/bin/../../hbase does not exist! HBase imports will fail.
Please set $HBASE_HOME to the root of your HBase installation.
Warning: /data/application/sqoop/bin/../../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /data/application/sqoop/bin/../../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.
Warning: /data/application/sqoop/bin/../../zookeeper does not exist! Accumulo imports will fail.
Please set $ZOOKEEPER_HOME to the root of your Zookeeper installation.
16/11/08 14:35:25 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6
16/11/08 14:35:25 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
16/11/08 14:35:26 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset.
16/11/08 14:35:26 INFO tool.CodeGenTool: Beginning code generation
16/11/08 14:35:27 INFO manager.SqlManager: Executing SQL statement: select * from student where  (1 = 0)
16/11/08 14:35:27 INFO manager.SqlManager: Executing SQL statement: select * from student where  (1 = 0)
16/11/08 14:35:27 INFO manager.SqlManager: Executing SQL statement: select * from student where  (1 = 0)
16/11/08 14:35:27 INFO orm.CompilationManager: HADOOP_MAPRED_HOME is /data/application/hadoop-2.6.0
Note: /tmp/sqoop-hadoopStudy/compile/0831a58db3ebb7ec80076e10c1540cc0/QueryResult.java uses or overrides a deprecated API.
Note: Recompile with -Xlint:deprecation for details.
16/11/08 14:35:30 INFO orm.CompilationManager: Writing jar file: /tmp/sqoop-hadoopStudy/compile/0831a58db3ebb7ec80076e10c1540cc0/QueryResult.jar
16/11/08 14:35:30 INFO mapreduce.ImportJobBase: Beginning query import.
16/11/08 14:35:31 INFO Configuration.deprecation: mapred.jar is deprecated. Instead, use mapreduce.job.jar
16/11/08 14:35:33 INFO Configuration.deprecation: mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
16/11/08 14:35:33 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
16/11/08 14:35:33 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
16/11/08 14:35:34 INFO mapreduce.JobSubmitter: Cleaning up the staging area file:/tmp/hadoop-hadoopStudy/mapred/staging/hadoopStudy1846784522/.staging/job_local1846784522_0001
16/11/08 14:35:34 ERROR tool.ImportTool: Encountered IOException running import job: java.io.FileNotFoundException: File does not exist: hdfs://hadoopStudy2:8020/data/application/sqoop/lib/parquet-common-1.4.1.jar
        at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1122)
        at org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:1114)
        at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
        at org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1114)
        at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
        at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224)
        at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:93)
        at org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57)
        at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:269)
        at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:390)
        at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:483)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
        at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:415)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
        at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
        at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
        at org.apache.sqoop.mapreduce.ImportJobBase.doSubmitJob(ImportJobBase.java:196)
        at org.apache.sqoop.mapreduce.ImportJobBase.runJob(ImportJobBase.java:169)
        at org.apache.sqoop.mapreduce.ImportJobBase.runImport(ImportJobBase.java:266)
        at org.apache.sqoop.manager.SqlManager.importQuery(SqlManager.java:729)
        at org.apache.sqoop.tool.ImportTool.importTable(ImportTool.java:499)
        at org.apache.sqoop.tool.ImportTool.run(ImportTool.java:605)
        at org.apache.sqoop.Sqoop.run(Sqoop.java:143)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.sqoop.Sqoop.runSqoop(Sqoop.java:179)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:218)
        at org.apache.sqoop.Sqoop.runTool(Sqoop.java:227)
        at org.apache.sqoop.Sqoop.main(Sqoop.java:236)
异常中报的“/data/application/sqoop”是我在本机安装sqoop的路径,跑任务时怎么去hdfs上去找这个路径了???????????我哪个地方配错了???
另外我在Apache官方下载的sqoop页面如下

1478588556610.png

我现在选的是sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz
下面的那个sqoop-1.4.6.tar.gz是什么??最初还以为是这个,结果根本没用起来,大小才2M多,是源码包么???

wangxiaolei

赞同来自:

基于不同的hadoop版本编译发布的sqoop, 没有带hadoop的不基于hadoop编译发布。 提示parquet-common-1.4.1.jar包不存在。 你换成2.0的sqoop

wangxiaolei

赞同来自:

我下载的sqoop-1.4.6.bin__hadoop-2.0.4-alpha.tar.gz里面是有这个jar包的
1478598629680.png
 

hadoopStudy

赞同来自:

刚才试了一下如果把sqoop装到集群内的机器上,数据是可以导入成功的……

要回复问题请先登录注册