老师您好,我在三台机器使用ambari搭建了集群,使用tez执行hive任务时,明明查询结果已经返回,但是程序还在占用资源,该怎么办呢?

摘取日志信息如下:
2018-11-26 10:06:26,611 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: No taskRequests. Container's idle timeout delay expired or is new. Releasing container, containerId=container_e20_1542879091392_0071_01_000006, containerExpiryTime=1543197986558, idleTimeout=10000, taskRequestsCount=0, heldContainers=3, delayedContainers=2, isNew=false
2018-11-26 10:06:26,611 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1542879091392_0071_1][Event:CONTAINER_STOPPED]: containerId=container_e20_1542879091392_0071_01_000006, stoppedTime=1543197986611, exitStatus=0
2018-11-26 10:06:26,611 [INFO] [ContainerLauncher #7] |launcher.ContainerLauncherImpl|: Stopping container_e20_1542879091392_0071_01_000006
2018-11-26 10:06:26,611 [INFO] [ContainerLauncher #7] |impl.ContainerManagementProtocolProxy|: Opening proxy : genecast-slave01.hadoop:45454
2018-11-26 10:06:26,789 [INFO] [Dispatcher thread {Central}] |container.AMContainerImpl|: Container container_e20_1542879091392_0071_01_000006 exited with diagnostics set to Container failed, exitCode=-105. Container killed by the ApplicationMaster.
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143

2018-11-26 10:06:27,018 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: No taskRequests. Container's idle timeout delay expired or is new. Releasing container, containerId=container_e20_1542879091392_0071_01_000008, containerExpiryTime=1543197987014, idleTimeout=10000, taskRequestsCount=0, heldContainers=2, delayedContainers=1, isNew=false
2018-11-26 10:06:27,018 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1542879091392_0071_1][Event:CONTAINER_STOPPED]: containerId=container_e20_1542879091392_0071_01_000008, stoppedTime=1543197987018, exitStatus=0
2018-11-26 10:06:27,019 [INFO] [ContainerLauncher #8] |launcher.ContainerLauncherImpl|: Stopping container_e20_1542879091392_0071_01_000008
2018-11-26 10:06:27,019 [INFO] [ContainerLauncher #8] |impl.ContainerManagementProtocolProxy|: Opening proxy : genecast-master.hadoop:45454
2018-11-26 10:06:27,040 [INFO] [Dispatcher thread {Central}] |container.AMContainerImpl|: Container container_e20_1542879091392_0071_01_000008 exited with diagnostics set to Container failed, exitCode=-100. Container released by application
2018-11-26 10:06:30,792 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: No taskRequests. Container's idle timeout delay expired or is new. Releasing container, containerId=container_e20_1542879091392_0071_01_000011, containerExpiryTime=1543197990702, idleTimeout=10000, taskRequestsCount=0, heldContainers=1, delayedContainers=0, isNew=false
2018-11-26 10:06:30,792 [INFO] [Dispatcher thread {Central}] |history.HistoryEventHandler|: [HISTORY][DAG:dag_1542879091392_0071_1][Event:CONTAINER_STOPPED]: containerId=container_e20_1542879091392_0071_01_000011, stoppedTime=1543197990792, exitStatus=0
2018-11-26 10:06:30,792 [INFO] [ContainerLauncher #9] |launcher.ContainerLauncherImpl|: Stopping container_e20_1542879091392_0071_01_000011
2018-11-26 10:06:30,792 [INFO] [ContainerLauncher #9] |impl.ContainerManagementProtocolProxy|: Opening proxy : genecast-master.hadoop:45454
2018-11-26 10:06:30,801 [INFO] [Dispatcher thread {Central}] |container.AMContainerImpl|: Container container_e20_1542879091392_0071_01_000011 exited with diagnostics set to Container failed, exitCode=-100. Container released by application
2018-11-26 10:06:37,571 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 301 lastPreemptionHeartbeat: 300
2018-11-26 10:06:50,106 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 351 lastPreemptionHeartbeat: 350
2018-11-26 10:07:02,639 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 401 lastPreemptionHeartbeat: 400
2018-11-26 10:07:15,173 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 451 lastPreemptionHeartbeat: 450
2018-11-26 10:07:27,708 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 501 lastPreemptionHeartbeat: 500
2018-11-26 10:07:40,239 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 551 lastPreemptionHeartbeat: 550
2018-11-26 10:07:52,771 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 601 lastPreemptionHeartbeat: 600
2018-11-26 10:08:05,304 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 651 lastPreemptionHeartbeat: 650
2018-11-26 10:08:17,834 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 701 lastPreemptionHeartbeat: 700
2018-11-26 10:08:30,363 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 751 lastPreemptionHeartbeat: 750
2018-11-26 10:08:42,893 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 801 lastPreemptionHeartbeat: 800
2018-11-26 10:08:55,424 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 851 lastPreemptionHeartbeat: 850
2018-11-26 10:09:07,958 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 901 lastPreemptionHeartbeat: 900
2018-11-26 10:09:20,489 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 951 lastPreemptionHeartbeat: 950
2018-11-26 10:09:33,019 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 1001 lastPreemptionHeartbeat: 1000
2018-11-26 10:09:45,548 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 1051 lastPreemptionHeartbeat: 1050
2018-11-26 10:09:58,076 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 1101 lastPreemptionHeartbeat: 1100
2018-11-26 10:10:10,602 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 1151 lastPreemptionHeartbeat: 1150
2018-11-26 10:10:23,129 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 1201 lastPreemptionHeartbeat: 1200
2018-11-26 10:10:35,656 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 1251 lastPreemptionHeartbeat: 1250
2018-11-26 10:10:48,185 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 1301 lastPreemptionHeartbeat: 1300
2018-11-26 10:11:00,714 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 1351 lastPreemptionHeartbeat: 1350
2018-11-26 10:11:13,240 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 1401 lastPreemptionHeartbeat: 1400
2018-11-26 10:11:25,772 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 1451 lastPreemptionHeartbeat: 1450
2018-11-26 10:11:38,300 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 1501 lastPreemptionHeartbeat: 1500
2018-11-26 10:11:50,828 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 1551 lastPreemptionHeartbeat: 1550
2018-11-26 10:12:03,354 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 1601 lastPreemptionHeartbeat: 1600
2018-11-26 10:12:15,886 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 1651 lastPreemptionHeartbeat: 1650
2018-11-26 10:12:28,416 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 1701 lastPreemptionHeartbeat: 1700
2018-11-26 10:12:40,947 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 1751 lastPreemptionHeartbeat: 1750
2018-11-26 10:12:53,475 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 1801 lastPreemptionHeartbeat: 1800
2018-11-26 10:13:06,001 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 1851 lastPreemptionHeartbeat: 1850
2018-11-26 10:13:18,531 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 1901 lastPreemptionHeartbeat: 1900
2018-11-26 10:13:31,063 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 1951 lastPreemptionHeartbeat: 1950
2018-11-26 10:13:43,592 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 2001 lastPreemptionHeartbeat: 2000
2018-11-26 10:13:56,122 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 2051 lastPreemptionHeartbeat: 2050
2018-11-26 10:14:08,651 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 2101 lastPreemptionHeartbeat: 2100
2018-11-26 10:14:21,179 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 2151 lastPreemptionHeartbeat: 2150
2018-11-26 10:14:33,705 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 2201 lastPreemptionHeartbeat: 2200
2018-11-26 10:14:46,235 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 2251 lastPreemptionHeartbeat: 2250
2018-11-26 10:14:58,763 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 2301 lastPreemptionHeartbeat: 2300
2018-11-26 10:15:11,290 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 2351 lastPreemptionHeartbeat: 2350
2018-11-26 10:15:23,820 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 2401 lastPreemptionHeartbeat: 2400
2018-11-26 10:15:36,346 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 2451 lastPreemptionHeartbeat: 2450
2018-11-26 10:15:48,874 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 2501 lastPreemptionHeartbeat: 2500
2018-11-26 10:16:01,403 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 2551 lastPreemptionHeartbeat: 2550
2018-11-26 10:16:13,934 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 2601 lastPreemptionHeartbeat: 2600
2018-11-26 10:16:21,885 [INFO] [Timer-1] |app.DAGAppMaster|: Session timed out, lastDAGCompletionTime=1543197973310 ms, sessionTimeoutInterval=600000 ms
2018-11-26 10:16:21,885 [INFO] [Timer-1] |rm.TaskSchedulerEventHandler|: TaskScheduler notified that it should unregister from RM
2018-11-26 10:16:21,885 [INFO] [Timer-1] |app.DAGAppMaster|: No current running DAG, shutting down the AM
2018-11-26 10:16:21,885 [INFO] [Timer-1] |app.DAGAppMaster|: DAGAppMasterShutdownHandler invoked
2018-11-26 10:16:21,885 [INFO] [Timer-1] |app.DAGAppMaster|: Handling DAGAppMaster shutdown
2018-11-26 10:16:21,886 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Sleeping for 5 seconds before shutting down
2018-11-26 10:16:26,476 [INFO] [AMRM Callback Handler Thread] |rm.YarnTaskSchedulerService|: Allocated: <memory:0, vCores:0> Free: <memory:20480, vCores:8> pendingRequests: 0 delayedContainers: 0 heartbeats: 2651 lastPreemptionHeartbeat: 2650
2018-11-26 10:16:26,886 [INFO] [AMShutdownThread] |app.DAGAppMaster|: Calling stop for all the services
2018-11-26 10:16:26,886 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Initiating stop of YarnTaskScheduler
2018-11-26 10:16:26,886 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Releasing held containers
2018-11-26 10:16:26,886 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Removing all pending taskRequests
2018-11-26 10:16:26,887 [INFO] [AMShutdownThread] |history.HistoryEventHandler|: Stopping HistoryEventHandler
2018-11-26 10:16:26,887 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Stopping RecoveryService
2018-11-26 10:16:26,887 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Handle the remaining events in queue, queue size=0
2018-11-26 10:16:26,887 [INFO] [RecoveryEventHandlingThread] |recovery.RecoveryService|: EventQueue take interrupted. Returning
2018-11-26 10:16:26,887 [INFO] [AMShutdownThread] |recovery.RecoveryService|: Closing Summary Stream
2018-11-26 10:16:27,293 [INFO] [AMShutdownThread] |ats.ATSV15HistoryLoggingService|: Stopping ATSService, eventQueueBacklog=0
2018-11-26 10:16:27,294 [INFO] [DelayedContainerManager] |rm.YarnTaskSchedulerService|: AllocatedContainerManager Thread interrupted
2018-11-26 10:16:27,296 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Unregistering application from RM, exitStatus=SUCCEEDED, exitMessage=Session stats:submittedDAGs=0, successfulDAGs=1, failedDAGs=0, killedDAGs=0
, trackingURL=http://genecast-ambari.hadoop: ... _0071
2018-11-26 10:16:27,302 [INFO] [AMShutdownThread] |impl.AMRMClientImpl|: Waiting for application to be successfully unregistered.
2018-11-26 10:16:27,402 [INFO] [AMShutdownThread] |impl.AMRMClientImpl|: Waiting for application to be successfully unregistered.
2018-11-26 10:16:27,503 [INFO] [AMShutdownThread] |rm.YarnTaskSchedulerService|: Successfully unregistered application from RM
2018-11-26 10:16:27,504 [INFO] [AMRM Callback Handler Thread] |impl.AMRMClientAsyncImpl|: Interrupted while waiting for queue
java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2014)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2048)
	at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
	at org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$CallbackHandlerThread.run(AMRMClientAsyncImpl.java:287)

car2008 - 90后IT

赞同来自:

就是上一次am一直处于running状态,我是不是改一下yarn的分配资源的策略模式?

car2008 - 90后IT

赞同来自:

现在使用的是容量调度器

car2008 - 90后IT

赞同来自:

果然是的 我改为公平模式就好了

要回复问题请先登录注册