日志开始生成,启动Flume后出错

大作业在启动sh log-generator后,连通的Flume也被启动起来,但是一直在报下面的错误,最后只能强行终止。请问老师哪儿出问题了吗?

2016-11-22 17:25:45,042 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.register(MonitoredCounterGroup.java:120)] Monitored counter group for type: SOURCE, name: r1: Successfully registered new MBean.
2016-11-22 17:25:45,043 (lifecycleSupervisor-1-1) [INFO - org.apache.flume.instrumentation.MonitoredCounterGroup.start(MonitoredCounterGroup.java:96)] Component type: SOURCE, name: r1 started
2016-11-22 18:26:54,954 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 250 milliseconds
2016-11-22 18:26:55,205 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
2016-11-22 18:26:58,209 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 500 milliseconds
2016-11-22 18:26:58,709 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
2016-11-22 18:27:01,714 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 1000 milliseconds
2016-11-22 18:27:02,714 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
2016-11-22 18:27:05,716 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 2000 milliseconds
2016-11-22 18:27:07,716 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
2016-11-22 18:27:10,718 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
2016-11-22 18:27:14,718 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
2016-11-22 18:27:17,720 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
2016-11-22 18:27:21,720 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
2016-11-22 18:27:24,723 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
2016-11-22 18:27:28,724 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
2016-11-22 18:27:31,725 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
2016-11-22 18:27:35,726 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
2016-11-22 18:27:38,728 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
2016-11-22 18:27:42,728 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
2016-11-22 18:27:45,730 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
2016-11-22 18:27:49,730 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
2016-11-22 18:27:52,735 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
2016-11-22 18:27:56,736 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
2016-11-22 18:27:59,738 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
2016-11-22 18:28:03,738 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
2016-11-22 18:28:06,742 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
2016-11-22 18:28:10,743 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
2016-11-22 18:28:13,744 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
2016-11-22 18:28:17,745 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
2016-11-22 18:28:20,748 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
2016-11-22 18:28:24,749 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
2016-11-22 18:28:27,752 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
2016-11-22 18:28:31,753 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
2016-11-22 18:28:34,755 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
2016-11-22 18:28:38,755 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
2016-11-22 18:28:41,757 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
2016-11-22 18:28:45,758 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
2016-11-22 18:28:48,759 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
2016-11-22 18:28:52,760 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
2016-11-22 18:28:55,761 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
2016-11-22 18:28:59,762 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
2016-11-22 18:29:02,763 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
2016-11-22 18:29:06,764 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
2016-11-22 18:29:09,765 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
2016-11-22 18:29:13,766 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
2016-11-22 18:29:16,768 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
2016-11-22 18:29:20,769 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.
2016-11-22 18:29:23,773 (pool-3-thread-1) [WARN - org.apache.flume.source.SpoolDirectorySource$SpoolDirectoryRunnable.run(SpoolDirectorySource.java:239)] The channel is full, and cannot write data now. The source will try again after 4000 milliseconds
^C2016-11-22 18:29:25,325 (agent-shutdown-hook) [INFO - org.apache.flume.lifecycle.LifecycleSupervisor.stop(LifecycleSupervisor.java:79)] Stopping lifecycle supervisor 10
2016-11-22 18:29:25,332 (agent-shutdown-hook) [INFO - org.apache.flume.node.PollingPropertiesFileConfigurationProvider.stop(PollingPropertiesFileConfigurationProvider.java:83)] Configuration provider stopping
2016-11-22 18:29:27,773 (pool-3-thread-1) [INFO - org.apache.flume.client.avro.ReliableSpoolingFileEventReader.readEvents(ReliableSpoolingFileEventReader.java:238)] Last read was never committed - resetting mark position.

wangxiaolei

赞同来自: zhudejun_1985

代码写完后,使用mvn package命令打成jar包。 然后执行命令:hadoop jar /root/bootcamp_sogou/kafka-to-hdfs/sogoukafka/target/sogoukafka-1.0-SNAPSHOT.jar consumer就启动了

wangxiaolei

赞同来自:

没有看到ERROR,贴的不是报错信息 代码写在那个目录下

wangxiaolei

赞同来自:

新建了个test.sh执行后可以在控制台输出信息,如下:
erSink.java:94)] Event: { headers:{key=/root/bootcamp/practise/sogouquery/data/generated/SogouQ.reduced.20150601}  3A 31 35 3A 35 35 09 37 34 38 33 32 36 36 00:15:55.7483266 }
2016-11-22 20:03:56,723 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.perSink.java:94)] Event: { headers:{key=/root/bootcamp/practise/sogouquery/data/generated/SogouQ.reduced.20150601}  3A 31 35 3A 35 35 09 30 35 39 37 38 34 31 00:15:55.0597841 }
2016-11-22 20:03:56,723 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.perSink.java:94)] Event: { headers:{key=/root/bootcamp/practise/sogouquery/data/generated/SogouQ.reduced.20150601}  3A 31 35 3A 35 35 09 38 38 34 32 32 34 35 00:15:55.8842245 }
2016-11-22 20:03:56,724 (SinkRunner-PollingRunner-DefaultSinkProcessor) [INFO - org.apache.flume.sink.LoggerSink.perSink.java:94)] Event: { headers:{key=/root/bootcamp/practise/sogouquery/data/generated/SogouQ.reduced.20150601}  3A 31 35 3A 35 35 09 32 39 34 31 36 37 33 00:15:55.2941673 }
说明你的kafka那边有问题。 每个步骤都做一个测试,会交容易找出在那里出了问题的。 你检查下Kafka的consumer,是否正常消费数据。

zhudejun_1985 - 天下大事,必作于细

赞同来自:

我的start_consumer.sh脚本内容是这样的:#!/bin/sh cd /root/kafka-0.8.2 bin/kafka-console-consumer.sh --zookeeper zhumac1:2181 --topic sogouTopic --from-beginning   这里还有一个疑问,就是我在脚本中怎么能应用consumer.properties这个文件配置呢?虽然已经配置好了,但是如果我执行bin/kafka-console-consumer.sh  consumer.properties会报错。上网也没搜索出来,求教.

wangxiaolei

赞同来自:

consumer.properties这个文件和你的代码是对应TestConsumer.java中的值。 我看你采用代码写死的方式,并没有采用配置文件的方法,所以不用配置consumer.properties
图片.png
 

zhudejun_1985 - 天下大事,必作于细

赞同来自:

意思就是,对我来说所谓的启动kafka consumer这个操作,应该是直接运行java -cp JAR包的这个动作了? 我已经mvn编译好了,就是前几天问你的那些参数配置问题,都已经解决了。是这样子吗?不再用脚本来启动了?

wangxiaolei

赞同来自:

kafka的服务启动了吗? learder为什么是-1
图片.png
 

wangxiaolei

赞同来自:

kafka启动成功后,你再试试。

要回复问题请先登录注册