mapreduce执行程序HDFS路径找不到
因为mapreduce执行程序的数据输入路径是在HDFS路径下,而这个路径下的数据是我shell远程执行hive查询语句获得,但是mapreduce程序读取不到,可是我java端的DFS location就是连接HDFS里面有这个文件夹和数据,请问怎么办?
mapreduce中文件路径是这样设置的:
FileInputFormat.addInputPath(job, new Path("hdfs://192.168.100.223:9000/home/hadoop/test/000000_0"));
运行报错信息如下:
17/07/27 09:49:05 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
17/07/27 09:49:05 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
17/07/27 09:49:06 WARN mapreduce.JobResourceUploader: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
17/07/27 09:49:06 INFO mapreduce.JobSubmitter: Cleaning up the staging area file:/home/hadoop/temp/mapred/staging/hadoop1319276731/.staging/job_local1319276731_0001
Exception in thread "main" org.apache.hadoop.mapreduce.lib.input.InvalidInputException: Input path does not exist: hdfs://192.168.100.223:9000/home/hadoop/test/000000_0
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.singleThreadedListStatus(FileInputFormat.java:323)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:265)
at org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:387)
at org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:301)
at org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:318)
at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:196)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1287)
at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1308)
at com.easeye.adtarget.cloud.trafficStatis.hadoop.Test.main(Test.java:48)
可是我用hadoop用户查看HDFS目录文件,有这个test文件,但是却不能获取或者打开;我直接用root用户在hive里面输入导出命令导出的数据也是root用户的权限,却可以在hadoop中找到并查看!很奇怪,唯一的不同只是我远程使用shell脚本执行了hive查询和导出数据而已!不明白为什么一个可以找到,一个不能找到,查询和导出命令是一样的。
shell脚本代码为:
7 个回复
fish - Hadooper
赞同来自:
fish - Hadooper
赞同来自:
fish - Hadooper
赞同来自:
fish - Hadooper
赞同来自:
jane3von
赞同来自:
fish - Hadooper
赞同来自:
fish - Hadooper
赞同来自: