各位老师,我遇到一个奇怪现象。spark-submit 执行jar,只有第一次有结果,往后的每次都没有结果。

 详细情况是这样的:
   有一个java程序,功能是使用KafkaUtils.createDirectStream... 方式五分钟获取一次数据并计算,得到结果。
   但是现在第一次执行结果是正常的。第一次后的每次执行结果都不正常,日志信息中没有也没有报错。
   kafka在正常接收新数据,hadoop也正常。
 
   为什么会这样的呢。
 
 
spark-submit执行正常日志(第一次执行)
---------------------------------------------------------------------------------------------
    17/04/13 15:30:00 INFO VerifiableProperties: Verifying properties
17/04/13 15:30:00 INFO VerifiableProperties: Property group.id is overridden to 
17/04/13 15:30:00 INFO VerifiableProperties: Property zookeeper.connect is overridden to 
17/04/13 15:30:00 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 230.4 KB, free 366.1 MB)
17/04/13 15:30:00 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 20.6 KB, free 366.1 MB)
17/04/13 15:30:00 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 192.168.2.10:42542 (size: 20.6 KB, free: 366.3 MB)
17/04/13 15:30:00 INFO SparkContext: Created broadcast 0 from textFile at UserCountGroupByMCC.java:58
17/04/13 15:30:00 INFO FileInputFormat: Total input paths to process : 1
17/04/13 15:30:00 INFO SparkContext: Starting job: collectAsMap at UserCountGroupByMCC.java:61
17/04/13 15:30:00 INFO DAGScheduler: Got job 0 (collectAsMap at UserCountGroupByMCC.java:61) with 2 output partitions
17/04/13 15:30:00 INFO DAGScheduler: Final stage: ResultStage 0 (collectAsMap at UserCountGroupByMCC.java:61)
17/04/13 15:30:00 INFO DAGScheduler: Parents of final stage: List()
17/04/13 15:30:00 INFO DAGScheduler: Missing parents: List()
17/04/13 15:30:00 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[5] at filter at UserCountGroupByMCC.java:61), which has no missing parents
17/04/13 15:30:00 INFO MemoryStore: Block broadcast_1 stored as values in memory (estimated size 3.8 KB, free 366.1 MB)
17/04/13 15:30:00 INFO MemoryStore: Block broadcast_1_piece0 stored as bytes in memory (estimated size 2.2 KB, free 366.0 MB)
17/04/13 15:30:00 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on 192.168.2.10:42542 (size: 2.2 KB, free: 366.3 MB)
17/04/13 15:30:00 INFO SparkContext: Created broadcast 1 from broadcast at DAGScheduler.scala:1012
17/04/13 15:30:00 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[5] at filter at UserCountGroupByMCC.java:61)
17/04/13 15:30:00 INFO YarnScheduler: Adding task set 0.0 with 2 tasks
17/04/13 15:30:00 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, slave01, partition 0, NODE_LOCAL, 5683 bytes)
17/04/13 15:30:00 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, slave01, partition 1, NODE_LOCAL, 5683 bytes)
17/04/13 15:30:00 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 0 on executor id: 1 hostname: slave01.
17/04/13 15:30:00 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 1 on executor id: 1 hostname: slave01.
17/04/13 15:30:01 INFO BlockManagerInfo: Added broadcast_1_piece0 in memory on slave01:52086 (size: 2.2 KB, free: 912.3 MB)
17/04/13 15:30:02 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on slave01:52086 (size: 20.6 KB, free: 912.3 MB)
17/04/13 15:30:03 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 3446 ms on slave01 (1/2)
17/04/13 15:30:03 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 3507 ms on slave01 (2/2)
17/04/13 15:30:03 INFO YarnScheduler: Removed TaskSet 0.0, whose tasks have all completed, from pool 
17/04/13 15:30:04 INFO DAGScheduler: ResultStage 0 (collectAsMap at UserCountGroupByMCC.java:61) finished in 3.515 s
17/04/13 15:30:04 INFO DAGScheduler: Job 0 finished: collectAsMap at UserCountGroupByMCC.java:61, took 3.606687 s
17/04/13 15:30:04 INFO MemoryStore: Block broadcast_2 stored as values in memory (estimated size 4.8 MB, free 361.2 MB)
17/04/13 15:30:04 INFO MemoryStore: Block broadcast_2_piece0 stored as bytes in memory (estimated size 378.9 KB, free 360.9 MB)
17/04/13 15:30:04 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on 192.168.2.10:42542 (size: 378.9 KB, free: 365.9 MB)
17/04/13 15:30:04 INFO SparkContext: Created broadcast 2 from broadcast at UserCountGroupByMCC.java:78
17/04/13 15:30:04 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 230.5 KB, free 360.6 MB)
17/04/13 15:30:04 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 20.6 KB, free 360.6 MB)
17/04/13 15:30:04 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on 192.168.2.10:42542 (size: 20.6 KB, free: 365.9 MB)
17/04/13 15:30:04 INFO SparkContext: Created broadcast 3 from textFile at UserCountGroupByMCC.java:159
17/04/13 15:30:04 INFO FileInputFormat: Total input paths to process : 1
17/04/13 15:30:04 INFO JobScheduler: Added jobs for time 1492068600000 ms
17/04/13 15:30:04 INFO JobGenerator: Checkpointing graph for time 1492068600000 ms
17/04/13 15:30:04 INFO DStreamGraph: Updating checkpoint data for time 1492068600000 ms
17/04/13 15:30:04 INFO DStreamGraph: Updated checkpoint data for time 1492068600000 ms
17/04/13 15:30:04 INFO JobScheduler: Starting job streaming job 1492068600000 ms.0 from job set of time 1492068600000 ms
17/04/13 15:30:04 INFO CheckpointWriter: Submitted checkpoint of time 1492068600000 ms to writer queue
17/04/13 15:30:04 INFO CheckpointWriter: Saving checkpoint for time 1492068600000 ms to file 'hdfs://192.168.2.9:9000/streaming_checkpoint/checkpoint-1492068600000'
17/04/13 15:30:04 INFO SparkContext: Starting job: foreach at UserCountGroupByMCC.java:103
17/04/13 15:30:04 INFO DAGScheduler: Registering RDD 6 (mapToPair at UserCountGroupByMCC.java:81)
17/04/13 15:30:04 INFO DAGScheduler: Got job 1 (foreach at UserCountGroupByMCC.java:103) with 16 output partitions
17/04/13 15:30:04 INFO DAGScheduler: Final stage: ResultStage 2 (foreach at UserCountGroupByMCC.java:103)
17/04/13 15:30:04 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 1)
17/04/13 15:30:04 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 1)
17/04/13 15:30:04 INFO DAGScheduler: Submitting ShuffleMapStage 1 (MapPartitionsRDD[6] at mapToPair at UserCountGroupByMCC.java:81), which has no missing parents
17/04/13 15:30:04 INFO MemoryStore: Block broadcast_4 stored as values in memory (estimated size 5.5 KB, free 360.6 MB)
17/04/13 15:30:04 INFO MemoryStore: Block broadcast_4_piece0 stored as bytes in memory (estimated size 2.9 KB, free 360.6 MB)
17/04/13 15:30:04 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on 192.168.2.10:42542 (size: 2.9 KB, free: 365.9 MB)
17/04/13 15:30:04 INFO SparkContext: Created broadcast 4 from broadcast at DAGScheduler.scala:1012
17/04/13 15:30:04 INFO DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 1 (MapPartitionsRDD[6] at mapToPair at UserCountGroupByMCC.java:81)
17/04/13 15:30:04 INFO YarnScheduler: Adding task set 1.0 with 4 tasks
17/04/13 15:30:04 INFO TaskSetManager: Starting task 0.0 in stage 1.0 (TID 2, slave03, partition 0, RACK_LOCAL, 5703 bytes)
17/04/13 15:30:04 INFO TaskSetManager: Starting task 1.0 in stage 1.0 (TID 3, slave01, partition 1, RACK_LOCAL, 5703 bytes)
17/04/13 15:30:04 INFO TaskSetManager: Starting task 2.0 in stage 1.0 (TID 4, slave03, partition 2, RACK_LOCAL, 5703 bytes)
17/04/13 15:30:04 INFO TaskSetManager: Starting task 3.0 in stage 1.0 (TID 5, slave01, partition 3, RACK_LOCAL, 5703 bytes)
17/04/13 15:30:04 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 2 on executor id: 2 hostname: slave03.
17/04/13 15:30:04 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 4 on executor id: 2 hostname: slave03.
17/04/13 15:30:04 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 3 on executor id: 1 hostname: slave01.
17/04/13 15:30:04 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 5 on executor id: 1 hostname: slave01.
17/04/13 15:30:04 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on slave01:52086 (size: 2.9 KB, free: 912.3 MB)
17/04/13 15:30:04 INFO BlockManagerInfo: Added rdd_1_3 in memory on slave01:52086 (size: 4.0 B, free: 912.3 MB)
17/04/13 15:30:04 INFO BlockManagerInfo: Added rdd_1_1 in memory on slave01:52086 (size: 4.0 B, free: 912.3 MB)
17/04/13 15:30:04 INFO TaskSetManager: Finished task 1.0 in stage 1.0 (TID 3) in 143 ms on slave01 (1/4)
17/04/13 15:30:04 INFO TaskSetManager: Finished task 3.0 in stage 1.0 (TID 5) in 142 ms on slave01 (2/4)
17/04/13 15:30:07 INFO BlockManagerInfo: Added broadcast_4_piece0 in memory on slave03:56489 (size: 2.9 KB, free: 912.3 MB)
17/04/13 15:30:08 INFO BlockManagerInfo: Added rdd_1_0 in memory on slave03:56489 (size: 5.1 MB, free: 907.2 MB)
17/04/13 15:30:08 INFO BlockManagerInfo: Added broadcast_2_piece0 in memory on slave03:56489 (size: 378.9 KB, free: 906.8 MB)
17/04/13 15:30:08 INFO BlockManagerInfo: Added rdd_1_2 in memory on slave03:56489 (size: 5.5 MB, free: 901.3 MB)
17/04/13 15:30:11 INFO TaskSetManager: Finished task 2.0 in stage 1.0 (TID 4) in 7276 ms on slave03 (3/4)
17/04/13 15:30:11 INFO TaskSetManager: Finished task 0.0 in stage 1.0 (TID 2) in 7301 ms on slave03 (4/4)
17/04/13 15:30:11 INFO DAGScheduler: ShuffleMapStage 1 (mapToPair at UserCountGroupByMCC.java:81) finished in 7.302 s
17/04/13 15:30:11 INFO YarnScheduler: Removed TaskSet 1.0, whose tasks have all completed, from pool 
17/04/13 15:30:11 INFO DAGScheduler: looking for newly runnable stages
17/04/13 15:30:11 INFO DAGScheduler: running: Set()
17/04/13 15:30:11 INFO DAGScheduler: waiting: Set(ResultStage 2)
17/04/13 15:30:11 INFO DAGScheduler: failed: Set()
17/04/13 15:30:11 INFO DAGScheduler: Submitting ResultStage 2 (MapPartitionsRDD[8] at groupByKey at UserCountGroupByMCC.java:51), which has no missing parents
17/04/13 15:30:11 INFO MemoryStore: Block broadcast_5 stored as values in memory (estimated size 3.8 KB, free 360.6 MB)
17/04/13 15:30:11 INFO MemoryStore: Block broadcast_5_piece0 stored as bytes in memory (estimated size 2.1 KB, free 360.6 MB)
17/04/13 15:30:11 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on 192.168.2.10:42542 (size: 2.1 KB, free: 365.9 MB)
17/04/13 15:30:11 INFO SparkContext: Created broadcast 5 from broadcast at DAGScheduler.scala:1012
17/04/13 15:30:11 INFO DAGScheduler: Submitting 16 missing tasks from ResultStage 2 (MapPartitionsRDD[8] at groupByKey at UserCountGroupByMCC.java:51)
17/04/13 15:30:11 INFO YarnScheduler: Adding task set 2.0 with 16 tasks
17/04/13 15:30:11 INFO TaskSetManager: Starting task 0.0 in stage 2.0 (TID 6, slave03, partition 0, NODE_LOCAL, 5597 bytes)
17/04/13 15:30:11 INFO TaskSetManager: Starting task 1.0 in stage 2.0 (TID 7, slave03, partition 1, NODE_LOCAL, 5597 bytes)
17/04/13 15:30:11 INFO TaskSetManager: Starting task 2.0 in stage 2.0 (TID 8, slave03, partition 2, NODE_LOCAL, 5597 bytes)
17/04/13 15:30:11 INFO TaskSetManager: Starting task 3.0 in stage 2.0 (TID 9, slave03, partition 3, NODE_LOCAL, 5597 bytes)
17/04/13 15:30:11 INFO TaskSetManager: Starting task 4.0 in stage 2.0 (TID 10, slave03, partition 4, NODE_LOCAL, 5597 bytes)
17/04/13 15:30:11 INFO TaskSetManager: Starting task 5.0 in stage 2.0 (TID 11, slave03, partition 5, NODE_LOCAL, 5597 bytes)
17/04/13 15:30:11 INFO TaskSetManager: Starting task 7.0 in stage 2.0 (TID 12, slave03, partition 7, NODE_LOCAL, 5597 bytes)
17/04/13 15:30:11 INFO TaskSetManager: Starting task 8.0 in stage 2.0 (TID 13, slave03, partition 8, NODE_LOCAL, 5597 bytes)
17/04/13 15:30:11 INFO TaskSetManager: Starting task 6.0 in stage 2.0 (TID 14, slave01, partition 6, PROCESS_LOCAL, 5597 bytes)
17/04/13 15:30:11 INFO TaskSetManager: Starting task 9.0 in stage 2.0 (TID 15, slave01, partition 9, PROCESS_LOCAL, 5597 bytes)
17/04/13 15:30:11 INFO TaskSetManager: Starting task 10.0 in stage 2.0 (TID 16, slave01, partition 10, PROCESS_LOCAL, 5597 bytes)
17/04/13 15:30:11 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 6 on executor id: 2 hostname: slave03.
17/04/13 15:30:11 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 7 on executor id: 2 hostname: slave03.
17/04/13 15:30:11 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 8 on executor id: 2 hostname: slave03.
17/04/13 15:30:11 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 9 on executor id: 2 hostname: slave03.
17/04/13 15:30:11 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 10 on executor id: 2 hostname: slave03.
17/04/13 15:30:11 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 11 on executor id: 2 hostname: slave03.
17/04/13 15:30:11 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 12 on executor id: 2 hostname: slave03.
17/04/13 15:30:11 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 13 on executor id: 2 hostname: slave03.
17/04/13 15:30:11 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 14 on executor id: 1 hostname: slave01.
17/04/13 15:30:11 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 15 on executor id: 1 hostname: slave01.
17/04/13 15:30:11 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 16 on executor id: 1 hostname: slave01.
17/04/13 15:30:11 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on slave01:52086 (size: 2.1 KB, free: 912.3 MB)
17/04/13 15:30:11 INFO BlockManagerInfo: Added broadcast_5_piece0 in memory on slave03:56489 (size: 2.1 KB, free: 901.3 MB)
17/04/13 15:30:11 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 192.168.2.10:46424
17/04/13 15:30:11 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 0 to 192.168.2.6:56971
17/04/13 15:30:11 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 0 is 196 bytes
17/04/13 15:30:11 INFO TaskSetManager: Finished task 9.0 in stage 2.0 (TID 15) in 80 ms on slave01 (1/16)
17/04/13 15:30:11 INFO TaskSetManager: Finished task 6.0 in stage 2.0 (TID 14) in 82 ms on slave01 (2/16)
17/04/13 15:30:11 INFO TaskSetManager: Finished task 10.0 in stage 2.0 (TID 16) in 81 ms on slave01 (3/16)
17/04/13 15:30:12 INFO TaskSetManager: Starting task 11.0 in stage 2.0 (TID 17, slave03, partition 11, NODE_LOCAL, 5597 bytes)
17/04/13 15:30:12 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 17 on executor id: 2 hostname: slave03.
17/04/13 15:30:12 INFO TaskSetManager: Finished task 5.0 in stage 2.0 (TID 11) in 757 ms on slave03 (4/16)
17/04/13 15:30:13 INFO TaskSetManager: Starting task 12.0 in stage 2.0 (TID 18, slave03, partition 12, NODE_LOCAL, 5597 bytes)
17/04/13 15:30:13 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 18 on executor id: 2 hostname: slave03.
17/04/13 15:30:13 INFO TaskSetManager: Finished task 4.0 in stage 2.0 (TID 10) in 1210 ms on slave03 (5/16)
17/04/13 15:30:13 INFO TaskSetManager: Starting task 13.0 in stage 2.0 (TID 19, slave03, partition 13, NODE_LOCAL, 5597 bytes)
17/04/13 15:30:13 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 19 on executor id: 2 hostname: slave03.
17/04/13 15:30:13 INFO TaskSetManager: Finished task 7.0 in stage 2.0 (TID 12) in 1309 ms on slave03 (6/16)
17/04/13 15:30:13 INFO TaskSetManager: Starting task 14.0 in stage 2.0 (TID 20, slave03, partition 14, NODE_LOCAL, 5597 bytes)
17/04/13 15:30:13 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 20 on executor id: 2 hostname: slave03.
17/04/13 15:30:13 INFO TaskSetManager: Finished task 13.0 in stage 2.0 (TID 19) in 268 ms on slave03 (7/16)
17/04/13 15:30:13 INFO TaskSetManager: Starting task 15.0 in stage 2.0 (TID 21, slave03, partition 15, NODE_LOCAL, 5597 bytes)
17/04/13 15:30:13 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 21 on executor id: 2 hostname: slave03.
17/04/13 15:30:13 INFO TaskSetManager: Finished task 3.0 in stage 2.0 (TID 9) in 1923 ms on slave03 (8/16)
17/04/13 15:30:16 INFO TaskSetManager: Finished task 11.0 in stage 2.0 (TID 17) in 4363 ms on slave03 (9/16)
17/04/13 15:30:16 INFO TaskSetManager: Finished task 8.0 in stage 2.0 (TID 13) in 5118 ms on slave03 (10/16)
17/04/13 15:30:16 INFO TaskSetManager: Finished task 2.0 in stage 2.0 (TID 8) in 5124 ms on slave03 (11/16)
17/04/13 15:30:17 INFO CheckpointWriter: Deleting hdfs://192.168.2.9:9000/streaming_checkpoint/checkpoint-1492061700000
17/04/13 15:30:17 INFO CheckpointWriter: Checkpoint for time 1492068600000 ms saved to file 'hdfs://192.168.2.9:9000/streaming_checkpoint/checkpoint-1492068600000', took 5685 bytes and 13377 ms
17/04/13 15:30:24 INFO TaskSetManager: Finished task 0.0 in stage 2.0 (TID 6) in 12351 ms on slave03 (12/16)
17/04/13 15:30:29 INFO TaskSetManager: Finished task 14.0 in stage 2.0 (TID 20) in 16008 ms on slave03 (13/16)
17/04/13 15:30:43 INFO TaskSetManager: Finished task 1.0 in stage 2.0 (TID 7) in 32017 ms on slave03 (14/16)
17/04/13 15:31:07 INFO TaskSetManager: Finished task 12.0 in stage 2.0 (TID 18) in 54295 ms on slave03 (15/16)
17/04/13 15:32:00 INFO TaskSetManager: Finished task 15.0 in stage 2.0 (TID 21) in 106672 ms on slave03 (16/16)
17/04/13 15:32:00 INFO YarnScheduler: Removed TaskSet 2.0, whose tasks have all completed, from pool 
17/04/13 15:32:00 INFO DAGScheduler: ResultStage 2 (foreach at UserCountGroupByMCC.java:103) finished in 108.600 s
17/04/13 15:32:00 INFO DAGScheduler: Job 1 finished: foreach at UserCountGroupByMCC.java:103, took 116.036874 s
17/04/13 15:32:00 INFO JobScheduler: Finished job streaming job 1492068600000 ms.0 from job set of time 1492068600000 ms
17/04/13 15:32:00 INFO JobScheduler: Starting job streaming job 1492068600000 ms.1 from job set of time 1492068600000 ms
17/04/13 15:32:00 INFO SparkContext: Starting job: foreach at UserCountGroupByMCC.java:234
17/04/13 15:32:00 INFO DAGScheduler: Registering RDD 12 (mapToPair at UserCountGroupByMCC.java:172)
17/04/13 15:32:00 INFO DAGScheduler: Registering RDD 11 (mapToPair at UserCountGroupByMCC.java:163)
17/04/13 15:32:00 INFO DAGScheduler: Got job 2 (foreach at UserCountGroupByMCC.java:234) with 4 output partitions
17/04/13 15:32:00 INFO DAGScheduler: Final stage: ResultStage 5 (foreach at UserCountGroupByMCC.java:234)
17/04/13 15:32:00 INFO DAGScheduler: Parents of final stage: List(ShuffleMapStage 3, ShuffleMapStage 4)
17/04/13 15:32:00 INFO DAGScheduler: Missing parents: List(ShuffleMapStage 3, ShuffleMapStage 4)
17/04/13 15:32:00 INFO DAGScheduler: Submitting ShuffleMapStage 3 (MapPartitionsRDD[12] at mapToPair at UserCountGroupByMCC.java:172), which has no missing parents
17/04/13 15:32:00 INFO MemoryStore: Block broadcast_6 stored as values in memory (estimated size 4.4 KB, free 360.6 MB)
17/04/13 15:32:00 INFO MemoryStore: Block broadcast_6_piece0 stored as bytes in memory (estimated size 2.4 KB, free 360.6 MB)
17/04/13 15:32:00 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on 192.168.2.10:42542 (size: 2.4 KB, free: 365.9 MB)
17/04/13 15:32:00 INFO SparkContext: Created broadcast 6 from broadcast at DAGScheduler.scala:1012
17/04/13 15:32:00 INFO DAGScheduler: Submitting 4 missing tasks from ShuffleMapStage 3 (MapPartitionsRDD[12] at mapToPair at UserCountGroupByMCC.java:172)
17/04/13 15:32:00 INFO YarnScheduler: Adding task set 3.0 with 4 tasks
17/04/13 15:32:00 INFO DAGScheduler: Submitting ShuffleMapStage 4 (MapPartitionsRDD[11] at mapToPair at UserCountGroupByMCC.java:163), which has no missing parents
17/04/13 15:32:00 INFO TaskSetManager: Starting task 0.0 in stage 3.0 (TID 22, slave03, partition 0, PROCESS_LOCAL, 5703 bytes)
17/04/13 15:32:00 INFO TaskSetManager: Starting task 1.0 in stage 3.0 (TID 23, slave01, partition 1, PROCESS_LOCAL, 5703 bytes)
17/04/13 15:32:00 INFO MemoryStore: Block broadcast_7 stored as values in memory (estimated size 4.0 KB, free 360.6 MB)
17/04/13 15:32:00 INFO TaskSetManager: Starting task 2.0 in stage 3.0 (TID 24, slave03, partition 2, PROCESS_LOCAL, 5703 bytes)
17/04/13 15:32:00 INFO MemoryStore: Block broadcast_7_piece0 stored as bytes in memory (estimated size 2.4 KB, free 360.6 MB)
17/04/13 15:32:00 INFO TaskSetManager: Starting task 3.0 in stage 3.0 (TID 25, slave01, partition 3, PROCESS_LOCAL, 5703 bytes)
17/04/13 15:32:00 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on 192.168.2.10:42542 (size: 2.4 KB, free: 365.9 MB)
17/04/13 15:32:00 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 22 on executor id: 2 hostname: slave03.
17/04/13 15:32:00 INFO SparkContext: Created broadcast 7 from broadcast at DAGScheduler.scala:1012
17/04/13 15:32:00 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 24 on executor id: 2 hostname: slave03.
17/04/13 15:32:00 INFO DAGScheduler: Submitting 2 missing tasks from ShuffleMapStage 4 (MapPartitionsRDD[11] at mapToPair at UserCountGroupByMCC.java:163)
17/04/13 15:32:00 INFO YarnScheduler: Adding task set 4.0 with 2 tasks
17/04/13 15:32:00 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 23 on executor id: 1 hostname: slave01.
17/04/13 15:32:00 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 25 on executor id: 1 hostname: slave01.
17/04/13 15:32:00 INFO TaskSetManager: Starting task 0.0 in stage 4.0 (TID 26, slave01, partition 0, NODE_LOCAL, 5883 bytes)
17/04/13 15:32:00 INFO TaskSetManager: Starting task 1.0 in stage 4.0 (TID 27, slave01, partition 1, NODE_LOCAL, 5883 bytes)
17/04/13 15:32:00 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 26 on executor id: 1 hostname: slave01.
17/04/13 15:32:00 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 27 on executor id: 1 hostname: slave01.
17/04/13 15:32:00 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on slave03:56489 (size: 2.4 KB, free: 901.3 MB)
17/04/13 15:32:00 INFO BlockManagerInfo: Added broadcast_6_piece0 in memory on slave01:52086 (size: 2.4 KB, free: 912.3 MB)
17/04/13 15:32:00 INFO BlockManagerInfo: Added broadcast_7_piece0 in memory on slave01:52086 (size: 2.4 KB, free: 912.3 MB)
17/04/13 15:32:00 INFO TaskSetManager: Finished task 3.0 in stage 3.0 (TID 25) in 36 ms on slave01 (1/4)
17/04/13 15:32:00 INFO TaskSetManager: Finished task 1.0 in stage 3.0 (TID 23) in 48 ms on slave01 (2/4)
17/04/13 15:32:00 INFO BlockManagerInfo: Added broadcast_3_piece0 in memory on slave01:52086 (size: 20.6 KB, free: 912.2 MB)
17/04/13 15:32:00 WARN DFSClient: Slow ReadProcessor read fields took 108623ms (threshold=30000ms); ack: seqno: 11 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 9309745, targets: [192.168.2.10:50010, 192.168.2.13:50010, 192.168.2.6:50010]
17/04/13 15:32:00 INFO TaskSetManager: Finished task 0.0 in stage 3.0 (TID 22) in 307 ms on slave03 (3/4)
17/04/13 15:32:00 INFO TaskSetManager: Finished task 2.0 in stage 3.0 (TID 24) in 306 ms on slave03 (4/4)
17/04/13 15:32:00 INFO YarnScheduler: Removed TaskSet 3.0, whose tasks have all completed, from pool 
17/04/13 15:32:00 INFO DAGScheduler: ShuffleMapStage 3 (mapToPair at UserCountGroupByMCC.java:172) finished in 0.309 s
17/04/13 15:32:00 INFO DAGScheduler: looking for newly runnable stages
17/04/13 15:32:00 INFO DAGScheduler: running: Set(ShuffleMapStage 4)
17/04/13 15:32:00 INFO DAGScheduler: waiting: Set(ResultStage 5)
17/04/13 15:32:00 INFO DAGScheduler: failed: Set()
17/04/13 15:32:16 INFO TaskSetManager: Finished task 1.0 in stage 4.0 (TID 27) in 16125 ms on slave01 (1/2)
17/04/13 15:32:17 INFO TaskSetManager: Finished task 0.0 in stage 4.0 (TID 26) in 16595 ms on slave01 (2/2)
17/04/13 15:32:17 INFO YarnScheduler: Removed TaskSet 4.0, whose tasks have all completed, from pool 
17/04/13 15:32:17 INFO DAGScheduler: ShuffleMapStage 4 (mapToPair at UserCountGroupByMCC.java:163) finished in 16.595 s
17/04/13 15:32:17 INFO DAGScheduler: looking for newly runnable stages
17/04/13 15:32:17 INFO DAGScheduler: running: Set()
17/04/13 15:32:17 INFO DAGScheduler: waiting: Set(ResultStage 5)
17/04/13 15:32:17 INFO DAGScheduler: failed: Set()
17/04/13 15:32:17 INFO DAGScheduler: Submitting ResultStage 5 (MapPartitionsRDD[18] at filter at UserCountGroupByMCC.java:187), which has no missing parents
17/04/13 15:32:17 INFO MemoryStore: Block broadcast_8 stored as values in memory (estimated size 4.6 KB, free 360.6 MB)
17/04/13 15:32:17 INFO MemoryStore: Block broadcast_8_piece0 stored as bytes in memory (estimated size 2.4 KB, free 360.6 MB)
17/04/13 15:32:17 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on 192.168.2.10:42542 (size: 2.4 KB, free: 365.9 MB)
17/04/13 15:32:17 INFO SparkContext: Created broadcast 8 from broadcast at DAGScheduler.scala:1012
17/04/13 15:32:17 INFO DAGScheduler: Submitting 4 missing tasks from ResultStage 5 (MapPartitionsRDD[18] at filter at UserCountGroupByMCC.java:187)
17/04/13 15:32:17 INFO YarnScheduler: Adding task set 5.0 with 4 tasks
17/04/13 15:32:17 INFO TaskSetManager: Starting task 0.0 in stage 5.0 (TID 28, slave01, partition 0, PROCESS_LOCAL, 5660 bytes)
17/04/13 15:32:17 INFO TaskSetManager: Starting task 1.0 in stage 5.0 (TID 29, slave03, partition 1, PROCESS_LOCAL, 5660 bytes)
17/04/13 15:32:17 INFO TaskSetManager: Starting task 2.0 in stage 5.0 (TID 30, slave01, partition 2, PROCESS_LOCAL, 5660 bytes)
17/04/13 15:32:17 INFO TaskSetManager: Starting task 3.0 in stage 5.0 (TID 31, slave03, partition 3, PROCESS_LOCAL, 5660 bytes)
17/04/13 15:32:17 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 28 on executor id: 1 hostname: slave01.
17/04/13 15:32:17 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 30 on executor id: 1 hostname: slave01.
17/04/13 15:32:17 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 29 on executor id: 2 hostname: slave03.
17/04/13 15:32:17 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 31 on executor id: 2 hostname: slave03.
17/04/13 15:32:17 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on slave01:52086 (size: 2.4 KB, free: 912.2 MB)
17/04/13 15:32:17 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 192.168.2.10:46424
17/04/13 15:32:17 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 1 is 171 bytes
17/04/13 15:32:17 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 192.168.2.10:46424
17/04/13 15:32:17 INFO MapOutputTrackerMaster: Size of output statuses for shuffle 2 is 151 bytes
17/04/13 15:32:18 INFO BlockManagerInfo: Added broadcast_8_piece0 in memory on slave03:56489 (size: 2.4 KB, free: 901.3 MB)
17/04/13 15:32:18 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 1 to 192.168.2.6:56971
17/04/13 15:32:18 INFO MapOutputTrackerMasterEndpoint: Asked to send map output locations for shuffle 2 to 192.168.2.6:56971
 
 
 
 
spark-submit执行不正常日志(第一次后的执行)
---------------------------------------------------------------------------------------------
17/04/13 15:25:00 INFO MemoryStore: Block broadcast_17 stored as values in memory (estimated size 230.5 KB, free 349.3 MB)
17/04/13 15:25:00 INFO MemoryStore: Block broadcast_17_piece0 stored as bytes in memory (estimated size 20.6 KB, free 349.3 MB)
17/04/13 15:25:00 INFO BlockManagerInfo: Added broadcast_17_piece0 in memory on 192.168.2.10:48597 (size: 20.6 KB, free: 365.1 MB)
17/04/13 15:25:00 INFO SparkContext: Created broadcast 17 from textFile at UserCountGroupByMCC.java:58
17/04/13 15:25:00 INFO FileInputFormat: Total input paths to process : 1
17/04/13 15:25:00 INFO SparkContext: Starting job: collectAsMap at UserCountGroupByMCC.java:61
17/04/13 15:25:00 INFO DAGScheduler: Got job 5 (collectAsMap at UserCountGroupByMCC.java:61) with 2 output partitions
17/04/13 15:25:00 INFO DAGScheduler: Final stage: ResultStage 8 (collectAsMap at UserCountGroupByMCC.java:61)
17/04/13 15:25:00 INFO DAGScheduler: Parents of final stage: List()
17/04/13 15:25:00 INFO DAGScheduler: Missing parents: List()
17/04/13 15:25:00 INFO DAGScheduler: Submitting ResultStage 8 (MapPartitionsRDD[62] at filter at UserCountGroupByMCC.java:61), which has no missing parents
17/04/13 15:25:00 INFO MemoryStore: Block broadcast_18 stored as values in memory (estimated size 3.8 KB, free 349.3 MB)
17/04/13 15:25:00 INFO MemoryStore: Block broadcast_18_piece0 stored as bytes in memory (estimated size 2.2 KB, free 349.3 MB)
17/04/13 15:25:00 INFO BlockManagerInfo: Added broadcast_18_piece0 in memory on 192.168.2.10:48597 (size: 2.2 KB, free: 365.1 MB)
17/04/13 15:25:00 INFO SparkContext: Created broadcast 18 from broadcast at DAGScheduler.scala:1012
17/04/13 15:25:00 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 8 (MapPartitionsRDD[62] at filter at UserCountGroupByMCC.java:61)
17/04/13 15:25:00 INFO YarnScheduler: Adding task set 8.0 with 2 tasks
17/04/13 15:25:00 WARN DFSClient: Slow ReadProcessor read fields took 148337ms (threshold=30000ms); ack: seqno: 22 status: SUCCESS status: SUCCESS status: SUCCESS downstreamAckTimeNanos: 1022460, targets: [192.168.2.10:50010, 192.168.2.13:50010, 192.168.2.7:50010]
17/04/13 15:25:00 INFO TaskSetManager: Starting task 0.0 in stage 8.0 (TID 28, slave02, partition 0, NODE_LOCAL, 5683 bytes)
17/04/13 15:25:00 INFO TaskSetManager: Starting task 1.0 in stage 8.0 (TID 29, slave04, partition 1, NODE_LOCAL, 5683 bytes)
17/04/13 15:25:00 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 28 on executor id: 2 hostname: slave02.
17/04/13 15:25:00 INFO YarnSchedulerBackend$YarnDriverEndpoint: Launching task 29 on executor id: 1 hostname: slave04.
17/04/13 15:25:00 INFO BlockManagerInfo: Added broadcast_18_piece0 in memory on slave04:50897 (size: 2.2 KB, free: 912.2 MB)
17/04/13 15:25:00 INFO BlockManagerInfo: Added broadcast_17_piece0 in memory on slave04:50897 (size: 20.6 KB, free: 912.2 MB)
17/04/13 15:25:00 INFO TaskSetManager: Finished task 1.0 in stage 8.0 (TID 29) in 332 ms on slave04 (1/2)
17/04/13 15:25:00 INFO BlockManagerInfo: Added broadcast_18_piece0 in memory on slave02:46769 (size: 2.2 KB, free: 902.2 MB)
17/04/13 15:25:00 INFO BlockManagerInfo: Added broadcast_17_piece0 in memory on slave02:46769 (size: 20.6 KB, free: 902.2 MB)
17/04/13 15:25:02 INFO TaskSetManager: Finished task 0.0 in stage 8.0 (TID 28) in 2312 ms on slave02 (2/2)
17/04/13 15:25:02 INFO YarnScheduler: Removed TaskSet 8.0, whose tasks have all completed, from pool 
17/04/13 15:25:02 INFO DAGScheduler: ResultStage 8 (collectAsMap at UserCountGroupByMCC.java:61) finished in 2.313 s
17/04/13 15:25:02 INFO DAGScheduler: Job 5 finished: collectAsMap at UserCountGroupByMCC.java:61, took 2.321058 s
17/04/13 15:25:02 INFO MemoryStore: Block broadcast_19 stored as values in memory (estimated size 4.8 MB, free 344.5 MB)
17/04/13 15:25:02 INFO MemoryStore: Block broadcast_19_piece0 stored as bytes in memory (estimated size 378.9 KB, free 344.1 MB)
17/04/13 15:25:02 INFO BlockManagerInfo: Added broadcast_19_piece0 in memory on 192.168.2.10:48597 (size: 378.9 KB, free: 364.7 MB)
17/04/13 15:25:02 INFO SparkContext: Created broadcast 19 from broadcast at UserCountGroupByMCC.java:78
17/04/13 15:25:02 INFO MemoryStore: Block broadcast_20 stored as values in memory (estimated size 230.5 KB, free 343.9 MB)
17/04/13 15:25:02 INFO MemoryStore: Block broadcast_20_piece0 stored as bytes in memory (estimated size 20.6 KB, free 343.8 MB)
17/04/13 15:25:02 INFO BlockManagerInfo: Added broadcast_20_piece0 in memory on 192.168.2.10:48597 (size: 20.6 KB, free: 364.7 MB)
17/04/13 15:25:02 INFO SparkContext: Created broadcast 20 from textFile at UserCountGroupByMCC.java:159
17/04/13 15:25:02 INFO FileInputFormat: Total input paths to process : 1
17/04/13 15:25:02 INFO JobScheduler: Added jobs for time 1492068300000 ms
17/04/13 15:25:02 INFO JobGenerator: Checkpointing graph for time 1492068300000 ms
17/04/13 15:25:02 INFO DStreamGraph: Updating checkpoint data for time 1492068300000 ms
17/04/13 15:25:02 INFO DStreamGraph: Updated checkpoint data for time 1492068300000 ms
17/04/13 15:25:02 INFO CheckpointWriter: Submitted checkpoint of time 1492068300000 ms to writer queue
17/04/13 15:25:02 INFO CheckpointWriter: Saving checkpoint for time 1492068300000 ms to file 'hdfs://192.168.2.9:9000/streaming_checkpoint/checkpoint-1492068300000'
17/04/13 15:25:15 INFO CheckpointWriter: Deleting hdfs://192.168.2.9:9000/streaming_checkpoint/checkpoint-1492061400000
17/04/13 15:25:15 INFO CheckpointWriter: Checkpoint for time 1492068300000 ms saved to file 'hdfs://192.168.2.9:9000/streaming_checkpoint/checkpoint-1492068300000', took 5889 bytes and 13327 ms
 
 
 
 

Dong - Hulu

赞同来自: fish

日志显示没有任何错误。   可能是kafka中没有数据了,所以没有输出。   你要持续不断的向kafka中写数据,才能看到结果。不然默认情况下,spark streaming会依次把所有kafka中的数据处理完,除非再写入新数据。

要回复问题请先登录注册