Hadoop 运行MapReduce 自带的例子wordcount报错

请老师帮忙看看这导致这个错误的原因是什么?错误详细信息如下:
=============================================================================================2016-08-24 17:43:20,733 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: IOException in offerService
java.net.ConnectException: Call From slave-202/10.161.75.241 to slave-202:8020 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
    at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)
    at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)
    at org.apache.hadoop.ipc.Client.call(Client.java:1473)
    at org.apache.hadoop.ipc.Client.call(Client.java:1400)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)
    at com.sun.proxy.$Proxy14.sendHeartbeat(Unknown Source)
    at org.apache.hadoop.hdfs.protocolPB.DatanodeProtocolClientSideTranslatorPB.sendHeartbeat(DatanodeProtocolClientSideTranslatorPB.java:140)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:617)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:715)
    at org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:889)
    at java.lang.Thread.run(Thread.java:744)
Caused by: java.net.ConnectException: Connection refused
    at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
    at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)
    at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)
    at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)
    at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:608)
    at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:706)
    at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:369)
    at org.apache.hadoop.ipc.Client.getConnection(Client.java:1522)
    at org.apache.hadoop.ipc.Client.call(Client.java:1439)
    ... 8 more

IT_Angel

赞同来自:

slave-202 机器上 telnet slave-202 8020 是通的

fish - Hadooper

赞同来自:

问答私信给我发送一下你机器的IP跟密码,以及你在什么路径下面执行了什么命令吧。

wangxiaolei

赞同来自:

我看你发给我的命令本身就是错的bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.2.jar WordCount /home/input/wordcount.txt /home/output WordCount是小写。 我修改成小写,执行该命令后并没有报错,job出于等待状态。 请参考下 http://wenda.chinahadoop.cn/question/3958  

IT_Angel

赞同来自:

有错啊,这是我在slave-202/yarn-root-nodemanager-slave-202.log 中截取的错误日志!   ===============================================================================================   2016-08-25 10:52:59,387 INFO org.apache.hadoop.yarn.server.nodemanager.NMAuditLogger: USER=root    IP=10.161.79.123    OPERATION=Start Container Request    TARGET=ContainerManageImpl    RESULT=SUCCESS    APPID=application_1472032054895_0002    CONTAINERID=container_1472032054895_0002_01_000001 2016-08-25 10:52:59,675 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread java.lang.IllegalArgumentException: java.net.UnknownHostException: hadoop-cluster     at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374)     at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:310)     at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:176)     at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:668)     at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:604)     at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:148)     at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2596)     at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)     at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)     at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)     at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:169)     at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.getFileSystem(LogAggregationService.java:179)     at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.verifyAndCreateRemoteLogDir(LogAggregationService.java:186)     at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.initApp(LogAggregationService.java:319)     at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:443)     at org.apache.hadoop.yarn.server.nodemanager.containermanager.logaggregation.LogAggregationService.handle(LogAggregationService.java:67)     at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:180)     at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:107)     at java.lang.Thread.run(Thread.java:744) Caused by: java.net.UnknownHostException: hadoop-cluster     ... 20 more 2016-08-25 10:52:59,685 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.application.Application: Adding container_1472032054895_0002_01_000001 to application application_1472032054895_0002  

IT_Angel

赞同来自:

而且在主Namenode的yarn-root-resourcemanager-master.log 日志中有也有错误啊!   ===============================================================================================2016-08-25 11:04:24,990 INFO org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher: Error cleaning master  java.net.ConnectException: Call From master/10.161.79.123 to slave-202:58393 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefused     at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)     at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)     at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)     at java.lang.reflect.Constructor.newInstance(Constructor.java:526)     at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:791)     at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:731)     at org.apache.hadoop.ipc.Client.call(Client.java:1473)     at org.apache.hadoop.ipc.Client.call(Client.java:1400)     at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:232)     at com.sun.proxy.$Proxy32.stopContainers(Unknown Source)     at org.apache.hadoop.yarn.api.impl.pb.client.ContainerManagementProtocolPBClientImpl.stopContainers(ContainerManagementProtocolPBClientImpl.java:110)     at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.cleanup(AMLauncher.java:139)     at org.apache.hadoop.yarn.server.resourcemanager.amlauncher.AMLauncher.run(AMLauncher.java:268)     at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)     at java.lang.Thread.run(Thread.java:744) Caused by: java.net.ConnectException: Connection refused     at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)     at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:735)     at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)     at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:530)     at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:494)     at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:608)     at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:706)     at org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:369)     at org.apache.hadoop.ipc.Client.getConnection(Client.java:1522)     at org.apache.hadoop.ipc.Client.call(Client.java:1439)     ... 9 more  

IT_Angel

赞同来自:

我觉得执行JOB时等待是因为执行过程中已经出错了,所以等待!

IT_Angel

赞同来自:

老师我还是希望您到我的主机环境中,帮我把环境调试好,能够正常运行MR的JOB,现在很多日志中都有报错的!

wangxiaolei

赞同来自:

java.net.UnknownHostException: hadoop-cluster 报这个错误,之前你碰到过这个问题。需要配置下,参考下面 http://wenda.chinahadoop.cn/question/3948 是因为配置完后,还没有正确执行吗?

wangxiaolei

赞同来自:

你先把所有的机器上的hdfs-site.xml文件中都配置上
<property>
 <name>dfs.client.failover.proxy.provider.hadoop-cluster</name>
 <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
修改好后再给我回个消息

wangxiaolei

赞同来自:

现在跑wordcount有没有问题

wangxiaolei

赞同来自:

你在hdfs-site.xml中,配置的
<property>
  <name>dfs.namenode.http-address.hadoop-cluster.nn1</name>
  <value>0.0.0.0:50070</value>
 </property>
 <property>
  <name>dfs.namenode.rpc-address.hadoop-cluster.nn1</name>
  <value>master:8020</value>
 </property>
0.0.0.0是一开始就这么配置的还是后来修改的?  

wangxiaolei

赞同来自:

现在我修改了下配置文件,然后把集群重新启动下,在执行bin/hdfs haadmin -transitionToActive nn1命令时报下面的错误
2016-08-25 16:17:52,335 INFO org.apache.hadoop.hdfs.StateChange: STATE* UnderReplicatedBlocks has 0 blocks
2016-08-25 16:18:20,896 WARN org.apache.hadoop.hdfs.server.namenode.FSEditLog: Unable to determine input streams from QJM to [10.161.75.241:8485, 10.161.71.41:8485, 10.251.35.163:8485]. Skipping.
org.apache.hadoop.hdfs.qjournal.client.QuorumException: Got too many exceptions to achieve quorum size 2/3. 3 exceptions thrown:
10.251.35.163:8485: Journal Storage Directory /data/hadoop/dfs/journal/hadoop-cluster not formatted
        at org.apache.hadoop.hdfs.qjournal.server.Journal.checkFormatted(Journal.java:457)
没有格式化,需要重新格式化。 你删除/data/hadoop/dfs/journal目录下的数据了?还是做了什么其他操作?

wangxiaolei

赞同来自:

重新格式化了,现在可以了。 发现一些问题: 1、journalnode就启动你配置的三台机器就可以了,不要4台机器都启动。 2、不要配置0.0.0.0。机器总共4台,都搭建了集群服务,叫做服务端。      当这台机器跑mapreduce的时候就是client端了。 3、每台机器的配置文件内容要保持一致。 4、重点看下我在mapred-site.xml和yarn-site.xml文件中新增的内容,      每个属性的含义在第三周第一次直播课有讲。

IT_Angel

赞同来自:

好的,谢谢老师!

要回复问题请先登录注册