各位老师好,最近在做项目组的一个任务,遇到一个error,不知道如何解决,请帮忙看一下,不胜感激。

下面截取部分日志。
 
任务是通过pyspark 调用sql ,执行一个表的相关操作。
 
04/09 10:01:57 INFO DiskBlockManager: Created local directory at /hadoop/7/scratch/local/usercache/b_xxxxx/appcache/application_1489098474031_186483/blockmgr-00b06f87-9d17-4a8f-8f6d-2d7418e3718e17/04/09 10:01:57 INFO DiskBlockManager: Created local directory at /hadoop/8/scratch/local/usercache/b_xxxxx/appcache/application_1489098474031_186483/blockmgr-70d1011f-157f-457d-bb7d-736152efb39d
17/04/09 10:01:57 INFO DiskBlockManager: Created local directory at /hadoop/9/scratch/local/usercache/b_xxxxx/appcache/application_1489098474031_186483/blockmgr-1c16136d-c2fc-4f02-ae35-b9803d062303
17/04/09 10:01:57 INFO DiskBlockManager: Created local directory at /hadoop/10/scratch/local/usercache/b_xxxxx/appcache/application_1489098474031_186483/blockmgr-53c54650-6ee4-4ccb-ba8b-794fb7aeb855
17/04/09 10:01:57 INFO DiskBlockManager: Created local directory at /hadoop/11/scratch/local/usercache/b_xxxxx/appcache/application_1489098474031_186483/blockmgr-1ccc611e-afa3-4ba4-a965-80caf07efda6
17/04/09 10:01:57 INFO DiskBlockManager: Created local directory at /hadoop/12/scratch/local/usercache/b_xxxxx/appcache/application_1489098474031_186483/blockmgr-88289dd2-2650-4149-b76b-86cd200ad3c9
17/04/09 10:01:57 INFO MemoryStore: MemoryStore started with capacity 366.3 MB
17/04/09 10:01:57 INFO CoarseGrainedExecutorBackend: Connecting to driver: spark://CoarseGrainedScheduler@10.103.83.17:45033
17/04/09 10:01:57 INFO CoarseGrainedExecutorBackend: Successfully registered with driver
17/04/09 10:01:57 INFO Executor: Starting executor ID 67 on host hdc3-lvs01-400-1401-006.stratus.lvs.weizhi.com
17/04/09 10:01:57 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 37963.
17/04/09 10:01:57 INFO NettyBlockTransferService: Server created on hdc3-lvs01-400-1401-006.stratus.lvs.weizhi.com:37963
17/04/09 10:01:57 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
17/04/09 10:01:57 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(67, hdc3-lvs01-400-1401-006.stratus.lvs.weizhi.com, 37963, None)
17/04/09 10:01:57 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(67, hdc3-lvs01-400-1401-006.stratus.lvs.weizhi.com, 37963, None)
17/04/09 10:01:57 INFO BlockManager: external shuffle service port = 7337
17/04/09 10:01:57 INFO BlockManager: Registering executor with local external shuffle service.
17/04/09 10:01:57 INFO TransportClientFactory: Successfully created connection to hdc3-lvs01-400-1401-006.stratus.lvs.weizhi.com/10.142.33.101:7337 after 1 ms (0 ms spent in bootstraps)
17/04/09 10:01:57 INFO BlockManager: Initialized BlockManager: BlockManagerId(67, hdc3-lvs01-400-1401-006.stratus.lvs.weizhi.com, 37963, None)
17/04/09 10:02:05 INFO CoarseGrainedExecutorBackend: Got assigned task 221
17/04/09 10:02:05 INFO Executor: Running task 19.0 in stage 2.0 (TID 221)
17/04/09 10:02:05 INFO MapOutputTrackerWorker: Updating epoch to 2 and clearing cache
17/04/09 10:02:05 INFO TorrentBroadcast: Started reading broadcast variable 3
17/04/09 10:02:05 INFO TransportClientFactory: Successfully created connection to lvsaishdc3dn2413.stratus.lvs.weizhi.com/10.142.44.112:32920 after 2 ms (0 ms spent in bootstraps)
17/04/09 10:02:05 INFO MemoryStore: Block broadcast_3_piece0 stored as bytes in memory (estimated size 44.9 KB, free 366.3 MB)
17/04/09 10:02:05 INFO TorrentBroadcast: Reading broadcast variable 3 took 129 ms
17/04/09 10:02:05 INFO MemoryStore: Block broadcast_3 stored as values in memory (estimated size 117.5 KB, free 366.1 MB)
17/04/09 10:02:06 INFO deprecation: mapred.job.id is deprecated. Instead, use mapreduce.job.id
17/04/09 10:02:06 INFO deprecation: mapred.tip.id is deprecated. Instead, use mapreduce.task.id
17/04/09 10:02:06 INFO deprecation: mapred.task.id is deprecated. Instead, use mapreduce.task.attempt.id
17/04/09 10:02:06 INFO deprecation: mapred.task.is.map is deprecated. Instead, use mapreduce.task.ismap
17/04/09 10:02:06 INFO deprecation: mapred.task.partition is deprecated. Instead, use mapreduce.task.partition
17/04/09 10:02:06 INFO MapOutputTrackerWorker: Don't have map outputs for shuffle 1, fetching them
17/04/09 10:02:06 INFO MapOutputTrackerWorker: Doing the fetch; tracker endpoint = NettyRpcEndpointRef(spark://MapOutputTracker@10.103.83.17:45033)
17/04/09 10:02:06 INFO MapOutputTrackerWorker: Got the output locations
17/04/09 10:02:06 INFO ShuffleBlockFetcherIterator: Getting 0 non-empty blocks out of 200 blocks
17/04/09 10:02:06 INFO ShuffleBlockFetcherIterator: Started 0 remote fetches in 5 ms
17/04/09 10:02:06 INFO CodeGenerator: Code generated in 298.158557 ms
17/04/09 10:02:06 INFO FileOutputCommitter: File Output Committer Algorithm version is 1
17/04/09 10:02:07 INFO CodeGenerator: Code generated in 17.565168 ms
17/04/09 10:02:07 INFO CodeGenerator: Code generated in 14.314253 ms
17/04/09 10:02:08 INFO CodeGenerator: Code generated in 65.800343 ms
17/04/09 10:02:08 INFO CodeGenerator: Code generated in 11.181807 ms
17/04/09 10:02:08 INFO CodeGenerator: Code generated in 25.424587 ms
17/04/09 10:02:08 INFO SparkHiveDynamicPartitionWriterContainer: Sorting complete. Writing out partition files one at a time.
17/04/09 10:02:08 INFO SparkHadoopMapRedUtil: No need to commit output of task because needsTaskCommit=false: attempt_20170409100047_0002_m_000019_0
17/04/09 10:02:08 INFO Executor: Finished task 19.0 in stage 2.0 (TID 221). 2753 bytes result sent to driver
17/04/09 10:03:09 ERROR CoarseGrainedExecutorBackend: RECEIVED SIGNAL TERM
17/04/09 10:03:09 INFO DiskBlockManager: Shutdown hook called
17/04/09 10:03:09 INFO ShutdownHookManager: Shutdown hook called
End of LogType:stderr

那小子真帅

赞同来自:

以前,我用hive 做的,是可以的。但是用pyspark 调用sql,数据量大的时候就不行了。我尝试google 得到一些答案,没有给与我很好的原因。 另外我的spark版本是2.0

要回复问题请先登录注册