代码中自定义的参数显示找不到

老师好,我在修改Combiner调度机制的作业中,已经明确了怎么改,但是定义了一个参数,在eclipse中冒红叉,说找不到,我怎么能消除这个问题呢?还是说等到命令行 hadoop jar XXX.jar -D mapreduce.combiner.run.only.once=true 就好了?
下图是我在eclipse中的截图:
1IUGH08(N321L9OV@D1JLZI.png

fish - Hadooper

赞同来自: zhudejun_1985

在编译时,用mvn package -DskipTests跳过test阶段。Hadoop的JUnit回归测试需要搭建测试环境,现在只是为了编译,先跳过它吧。

wangxiaolei

赞同来自: zhudejun_1985

就是将编译好jar包hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/target/目录下的hadoop-mapreduce-client-core-xxx.jar包拷贝到当前集群的安装目录/root/hadoop-2.6.2/share/hadoop/mapreduce/下面就可以了。 后面的操作和以前的操作一样,执行hadoop jar命令 注意你编译的源代码的jar包要和当前的hadoop集群版本一致,否则会因为新jar包的源代码和集群的源码代码不一样,在运行程序的时候可能导致一些问题。

wangxiaolei

赞同来自:

不是在这里写的,需要修改源代码。 在MapTask.java中修改,代码是: Boolean.valueOf(job.get("mapreduce.combiner.run.only.once"));

zhudejun_1985 - 天下大事,必作于细

赞同来自:

在主机上/root/hadoop-cdh/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core# mvn package 报错,这个是什么问题?测试阶段错误?不懂了。 OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapreduce.lib.db.TestDbClasses Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 2.385 sec <<< FAILURE! - in org.apache.hadoop.mapreduce.lib.db.TestDbClasses testDataDrivenDBInputFormat(org.apache.hadoop.mapreduce.lib.db.TestDbClasses)  Time elapsed: 1.054 sec  <<< ERROR! java.lang.Exception: test timed out after 1000 milliseconds     at org.mockito.cglib.proxy.Enhancer.rename(Enhancer.java:398)     at org.mockito.cglib.proxy.Enhancer.access$400(Enhancer.java:61)     at org.mockito.cglib.proxy.Enhancer$6.getImplSignature(Enhancer.java:922)     at org.mockito.cglib.proxy.MethodInterceptorGenerator.generate(MethodInterceptorGenerator.java:89)     at org.mockito.cglib.proxy.Enhancer.emitMethods(Enhancer.java:948)     at org.mockito.cglib.proxy.Enhancer.generateClass(Enhancer.java:499)     at org.mockito.cglib.core.DefaultGeneratorStrategy.generate(DefaultGeneratorStrategy.java:25)     at org.mockito.cglib.core.AbstractClassGenerator.create(AbstractClassGenerator.java:217)     at org.mockito.cglib.proxy.Enhancer.createHelper(Enhancer.java:378)     at org.mockito.cglib.proxy.Enhancer.createClass(Enhancer.java:318)     at org.mockito.internal.creation.jmock.ClassImposterizer.createProxyClass(ClassImposterizer.java:93)     at org.mockito.internal.creation.jmock.ClassImposterizer.imposterise(ClassImposterizer.java:50)     at org.mockito.internal.util.MockUtil.createMock(MockUtil.java:54)     at org.mockito.internal.MockitoCore.mock(MockitoCore.java:45)     at org.mockito.Mockito.mock(Mockito.java:921)     at org.mockito.Mockito.mock(Mockito.java:816)     at org.apache.hadoop.mapreduce.lib.db.TestDbClasses.testDataDrivenDBInputFormat(TestDbClasses.java:73) OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapreduce.lib.output.TestFileOutputFormat Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.601 sec - in org.apache.hadoop.mapreduce.lib.output.TestFileOutputFormat OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapreduce.lib.output.TestPreemptableFileOutputCommitter Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.332 sec - in org.apache.hadoop.mapreduce.lib.output.TestPreemptableFileOutputCommitter OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapreduce.lib.output.TestFileOutputCommitter Tests run: 14, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.999 sec - in org.apache.hadoop.mapreduce.lib.output.TestFileOutputCommitter OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapreduce.lib.input.TestCombineFileRecordReader Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.847 sec - in org.apache.hadoop.mapreduce.lib.input.TestCombineFileRecordReader OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapreduce.lib.input.TestLineRecordReader Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 11.667 sec - in org.apache.hadoop.mapreduce.lib.input.TestLineRecordReader OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapreduce.lib.input.TestFileInputFormat Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.171 sec - in org.apache.hadoop.mapreduce.lib.input.TestFileInputFormat OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapreduce.split.TestJobSplitWriter Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.476 sec - in org.apache.hadoop.mapreduce.split.TestJobSplitWriter OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapreduce.TestJobMonitorAndPrint Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.809 sec - in org.apache.hadoop.mapreduce.TestJobMonitorAndPrint OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapreduce.security.TestTokenCache Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.479 sec - in org.apache.hadoop.mapreduce.security.TestTokenCache OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapreduce.TestShufflePlugin Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.704 sec - in org.apache.hadoop.mapreduce.TestShufflePlugin OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapreduce.TestJob Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.681 sec - in org.apache.hadoop.mapreduce.TestJob OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapreduce.TestContextFactory Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.334 sec - in org.apache.hadoop.mapreduce.TestContextFactory OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapreduce.tools.TestCLI Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.35 sec - in org.apache.hadoop.mapreduce.tools.TestCLI OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapred.lib.db.TestDBInputFormat Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.184 sec - in org.apache.hadoop.mapred.lib.db.TestDBInputFormat OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapred.lib.TestCombineFileRecordReader Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.713 sec - in org.apache.hadoop.mapred.lib.TestCombineFileRecordReader OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapred.TestSkipBadRecords Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.45 sec - in org.apache.hadoop.mapred.TestSkipBadRecords OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapred.TestJobConf Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.425 sec - in org.apache.hadoop.mapred.TestJobConf OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapred.TestTaskLogAppender Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.615 sec - in org.apache.hadoop.mapred.TestTaskLogAppender OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapred.TestOldMethodsJobID Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.737 sec - in org.apache.hadoop.mapred.TestOldMethodsJobID OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapred.TestTaskLog Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.051 sec - in org.apache.hadoop.mapred.TestTaskLog OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapred.TestLineRecordReader Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.485 sec - in org.apache.hadoop.mapred.TestLineRecordReader OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapred.TestJobEndNotifier Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.912 sec - in org.apache.hadoop.mapred.TestJobEndNotifier OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapred.TestJobAclsManager Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.51 sec - in org.apache.hadoop.mapred.TestJobAclsManager OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapred.TestClusterStatus Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.244 sec - in org.apache.hadoop.mapred.TestClusterStatus OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapred.TestCounters Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.034 sec - in org.apache.hadoop.mapred.TestCounters OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapred.TestJobInfo Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.289 sec - in org.apache.hadoop.mapred.TestJobInfo OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapred.TestClock Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.222 sec - in org.apache.hadoop.mapred.TestClock OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapred.TestMaster Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.297 sec - in org.apache.hadoop.mapred.TestMaster OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapred.TestJobQueueClient Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.247 sec - in org.apache.hadoop.mapred.TestJobQueueClient OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapred.TestFileInputFormat Tests run: 12, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.522 sec - in org.apache.hadoop.mapred.TestFileInputFormat OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapred.TestJvmContext Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.543 sec - in org.apache.hadoop.mapred.TestJvmContext OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapred.TestIndexCache Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.435 sec - in org.apache.hadoop.mapred.TestIndexCache OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapred.TestQueue Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.777 sec - in org.apache.hadoop.mapred.TestQueue OpenJDK 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.mapred.TestFileOutputCommitter Tests run: 13, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.855 sec - in org.apache.hadoop.mapred.TestFileOutputCommitter Results : Tests in error:    TestDbClasses.testDataDrivenDBInputFormat:73 »  test timed out after 1000 mill... Tests run: 183, Failures: 0, Errors: 1, Skipped: 0 [INFO] ------------------------------------------------------------------------ [INFO] BUILD FAILURE [INFO] ------------------------------------------------------------------------ [INFO] Total time: 03:16 min [INFO] Finished at: 2016-11-23T12:58:21+08:00 [INFO] Final Memory: 29M/91M [INFO] ------------------------------------------------------------------------ [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.17:test (default-test) on project hadoop-mapreduce-client-core: There are test failures. [ERROR]  [ERROR] Please refer to /root/hadoop-cdh/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/target/surefire-reports for the individual test results. [ERROR] -> [Help 1] [ERROR]  [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR]  [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/conflu ... ption [root@zhumac1 hadoop-mapreduce-client-core]# ll

zhudejun_1985 - 天下大事,必作于细

赞同来自:

[INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 01:03 min [INFO] Finished at: 2016-11-23T23:32:04+08:00 [INFO] Final Memory: 43M/114M [INFO] ------------------------------------------------------------------------ [root@zhumac1 hadoop-mapreduce-client-core]# pwd /root/hadoop-cdh/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core [root@zhumac1 hadoop-mapreduce-client-core]# ll total 12 -rw-rw-r--  1 root root 3510 Oct 16  2015 pom.xml drwxrwxr-x  5 root root 4096 Oct 16  2015 src drwxr-xr-x 10 root root 4096 Nov 23 23:31 target [root@zhumac1 hadoop-mapreduce-client-core]#    老师好,这个是我编译好的包,但我现在想知道的是,这个目录是在/root/hadoop-cdh/下的源代码包里完成的,并不是我集群中平时跑程序用的hadoop jar的那个真实的配置在/etc/profile文件里的HADOOP_HOME,我怎么能把这个修改应用在咱们的/root/bootcamp/practise/testcombiner/target/practise-0.0.1-SNAPSHOT.jar测试程序上面呢?

fish - Hadooper

赞同来自:

编译好hadoop包,将其拷贝到你执行(使用)hadoop时所要读取Hadoop包的地方。

要回复问题请先登录注册