spark-submit 提交失败,Stack trace: ExitCodeException exitCode=10:

代码是在spark-shell中已经成功实现,然后在intellj idea运行成功并打包JAR的(运行在本地local),现在在spark-submit失败。spark2.2,intellj idea是用的sbt,(intellj idea成功读取MySQL并处理保存)
这里也进行资源调度--num-executors 2 --executor-memory 3G --executor-cores 4 --driver-memory 3G 
但是同样报错,同时在spark-shell和intellj idea中实验2000页的mysql数据成功实现,但4w多页时,无论如何调整资源,都只有一个节点active,其他节点都dead,什么原因?
[hdfs@master4 bin]$ spark-submit --class wordcount --master yarn --deploy-mode cluster /home/wordcount_test.jar
18/08/06 00:34:32 INFO client.RMProxy: Connecting to ResourceManager at master4/10.6.3.40:8032
18/08/06 00:34:33 INFO yarn.Client: Requesting a new application from cluster with 2 NodeManagers
18/08/06 00:34:33 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (3779 MB per container)
18/08/06 00:34:33 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
18/08/06 00:34:33 INFO yarn.Client: Setting up container launch context for our AM
18/08/06 00:34:33 INFO yarn.Client: Setting up the launch environment for our AM container
18/08/06 00:34:33 INFO yarn.Client: Preparing resources for our AM container
18/08/06 00:34:34 INFO yarn.Client: Uploading resource file:/home/wordcount_test.jar -> hdfs://master4:8020/user/hdfs/.sparkStaging/application_1531861476380_0029/wordcount_test.jar
18/08/06 00:34:40 INFO yarn.Client: Uploading resource file:/tmp/spark-082ce11f-d5a5-468d-83e3-e4b38d3b07fd/__spark_conf__731945147152243490.zip -> hdfs://master4:8020/user/hdfs/.sparkStaging/application_1531861476380_0029/__spark_conf__731945147152243490.zip
18/08/06 00:34:40 INFO spark.SecurityManager: Changing view acls to: hdfs
18/08/06 00:34:40 INFO spark.SecurityManager: Changing modify acls to: hdfs
18/08/06 00:34:40 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hdfs); users with modify permissions: Set(hdfs)
18/08/06 00:34:41 INFO yarn.Client: Submitting application 29 to ResourceManager
18/08/06 00:34:41 INFO impl.YarnClientImpl: Submitted application application_1531861476380_0029
18/08/06 00:34:42 INFO yarn.Client: Application report for application_1531861476380_0029 (state: ACCEPTED)
18/08/06 00:34:42 INFO yarn.Client: 
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.users.hdfs
start time: 1533486881042
final status: UNDEFINED
tracking URL: http://master4:8088/proxy/appl ... 0029/
user: hdfs
18/08/06 00:34:43 INFO yarn.Client: Application report for application_1531861476380_0029 (state: ACCEPTED)
18/08/06 00:34:44 INFO yarn.Client: Application report for application_1531861476380_0029 (state: ACCEPTED)
18/08/06 00:34:45 INFO yarn.Client: Application report for application_1531861476380_0029 (state: ACCEPTED)
18/08/06 00:34:46 INFO yarn.Client: Application report for application_1531861476380_0029 (state: ACCEPTED)
18/08/06 00:34:47 INFO yarn.Client: Application report for application_1531861476380_0029 (state: ACCEPTED)
18/08/06 00:34:48 INFO yarn.Client: Application report for application_1531861476380_0029 (state: ACCEPTED)
18/08/06 00:34:49 INFO yarn.Client: Application report for application_1531861476380_0029 (state: ACCEPTED)
18/08/06 00:34:50 INFO yarn.Client: Application report for application_1531861476380_0029 (state: ACCEPTED)
18/08/06 00:34:51 INFO yarn.Client: Application report for application_1531861476380_0029 (state: ACCEPTED)
18/08/06 00:34:52 INFO yarn.Client: Application report for application_1531861476380_0029 (state: FAILED)
18/08/06 00:34:52 INFO yarn.Client: 
client token: N/A
diagnostics: Application application_1531861476380_0029 failed 2 times due to AM Container for appattempt_1531861476380_0029_000002 exited with  exitCode: 10
For more detailed output, check application tracking page:http://master4:8088/proxy/appl ... /Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1531861476380_0029_02_000001
Exit code: 10
Stack trace: ExitCodeException exitCode=10: 
at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
at org.apache.hadoop.util.Shell.run(Shell.java:504)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
 
 
Container exited with a non-zero exit code 10
Failing this attempt. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.users.hdfs
start time: 1533486881042
final status: FAILED
tracking URL: http://master4:8088/cluster/ap ... _0029
user: hdfs
Exception in thread "main" org.apache.spark.SparkException: Application application_1531861476380_0029 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1030)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1077)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
18/08/06 00:34:52 INFO util.ShutdownHookManager: Shutdown hook called
18/08/06 00:34:52 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-082ce11f-d5a5-468d-83e3-e4b38d3b07fd
You have new mail in /var/spool/mail/root
[hdfs@master4 bin]$ spark-submit --class wordcount --master yarn --deploy-mode cluster /home/wordcount_test.jar
18/08/06 00:44:52 INFO client.RMProxy: Connecting to ResourceManager at master4/10.6.3.40:8032
18/08/06 00:44:52 INFO yarn.Client: Requesting a new application from cluster with 1 NodeManagers
18/08/06 00:44:52 INFO yarn.Client: Verifying our application has not requested more than the maximum memory capability of the cluster (3779 MB per container)
18/08/06 00:44:52 INFO yarn.Client: Will allocate AM container, with 1408 MB memory including 384 MB overhead
18/08/06 00:44:52 INFO yarn.Client: Setting up container launch context for our AM
18/08/06 00:44:52 INFO yarn.Client: Setting up the launch environment for our AM container
18/08/06 00:44:52 INFO yarn.Client: Preparing resources for our AM container
18/08/06 00:44:53 INFO yarn.Client: Uploading resource file:/home/wordcount_test.jar -> hdfs://master4:8020/user/hdfs/.sparkStaging/application_1533487371481_0001/wordcount_test.jar
18/08/06 00:45:13 INFO yarn.Client: Uploading resource file:/tmp/spark-a7c6de19-aab0-45a7-9a3c-99af5bafc8d5/__spark_conf__6054076072140717571.zip -> hdfs://master4:8020/user/hdfs/.sparkStaging/application_1533487371481_0001/__spark_conf__6054076072140717571.zip
18/08/06 00:45:13 INFO spark.SecurityManager: Changing view acls to: hdfs
18/08/06 00:45:13 INFO spark.SecurityManager: Changing modify acls to: hdfs
18/08/06 00:45:13 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hdfs); users with modify permissions: Set(hdfs)
18/08/06 00:45:13 INFO yarn.Client: Submitting application 1 to ResourceManager
18/08/06 00:45:13 INFO impl.YarnClientImpl: Submitted application application_1533487371481_0001
18/08/06 00:45:14 INFO yarn.Client: Application report for application_1533487371481_0001 (state: ACCEPTED)
18/08/06 00:45:14 INFO yarn.Client: 
client token: N/A
diagnostics: N/A
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.users.hdfs
start time: 1533487513634
final status: UNDEFINED
tracking URL: http://master4:8088/proxy/appl ... 0001/
user: hdfs
18/08/06 00:45:15 INFO yarn.Client: Application report for application_1533487371481_0001 (state: ACCEPTED)
18/08/06 00:45:16 INFO yarn.Client: Application report for application_1533487371481_0001 (state: ACCEPTED)
18/08/06 00:45:17 INFO yarn.Client: Application report for application_1533487371481_0001 (state: ACCEPTED)
18/08/06 00:45:18 INFO yarn.Client: Application report for application_1533487371481_0001 (state: ACCEPTED)
18/08/06 00:45:19 INFO yarn.Client: Application report for application_1533487371481_0001 (state: ACCEPTED)
18/08/06 00:45:20 INFO yarn.Client: Application report for application_1533487371481_0001 (state: ACCEPTED)
18/08/06 00:45:21 INFO yarn.Client: Application report for application_1533487371481_0001 (state: ACCEPTED)
18/08/06 00:45:22 INFO yarn.Client: Application report for application_1533487371481_0001 (state: ACCEPTED)
18/08/06 00:45:24 INFO yarn.Client: Application report for application_1533487371481_0001 (state: FAILED)
18/08/06 00:45:24 INFO yarn.Client: 
client token: N/A
diagnostics: Application application_1533487371481_0001 failed 2 times due to AM Container for appattempt_1533487371481_0001_000002 exited with  exitCode: 10
For more detailed output, check application tracking page:http://master4:8088/proxy/appl ... /Then, click on links to logs of each attempt.
Diagnostics: Exception from container-launch.
Container id: container_1533487371481_0001_02_000001
Exit code: 10
Stack trace: ExitCodeException exitCode=10: 
at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
at org.apache.hadoop.util.Shell.run(Shell.java:504)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
 
 
Container exited with a non-zero exit code 10
Failing this attempt. Failing the application.
ApplicationMaster host: N/A
ApplicationMaster RPC port: -1
queue: root.users.hdfs
start time: 1533487513634
final status: FAILED
tracking URL: http://master4:8088/cluster/ap ... _0001
user: hdfs
Exception in thread "main" org.apache.spark.SparkException: Application application_1533487371481_0001 finished with failed status
at org.apache.spark.deploy.yarn.Client.run(Client.scala:1030)
at org.apache.spark.deploy.yarn.Client$.main(Client.scala:1077)
at org.apache.spark.deploy.yarn.Client.main(Client.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731)
at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181)
at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
18/08/06 00:45:24 INFO util.ShutdownHookManager: Shutdown hook called
18/08/06 00:45:24 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-a7c6de19-aab0-45a7-9a3c-99af5bafc8d5
E@Q}54{8BHZ5EHLW}P)7A9.png

haley1983

赞同来自:

[hdfs@master4 bin]$ yarn logs -applicationId application_1531861476380_0028 18/08/06 01:04:02 INFO client.RMProxy: Connecting to ResourceManager at master4/10.6.3.40:8032 Unable to get ApplicationState. Attempting to fetch logs directly from the filesystem.     Container: container_1531861476380_0028_02_000001 on slave41_8041 =================================================================== LogType:stderr Log Upload Time:Sun Aug 05 11:20:01 +0800 2018 LogLength:4006 Log Contents: 18/08/05 11:19:57 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT] 18/08/05 11:19:58 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1531861476380_0028_000002 18/08/05 11:19:59 INFO spark.SecurityManager: Changing view acls to: yarn,hdfs 18/08/05 11:19:59 INFO spark.SecurityManager: Changing modify acls to: yarn,hdfs 18/08/05 11:19:59 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, hdfs); users with modify permissions: Set(yarn, hdfs) 18/08/05 11:19:59 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread 18/08/05 11:19:59 ERROR yarn.ApplicationMaster: Uncaught exception:  java.lang.SecurityException: Invalid signature file digest for Manifest main attributes at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:284) at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:238) at java.util.jar.JarVerifier.processEntry(JarVerifier.java:316) at java.util.jar.JarVerifier.update(JarVerifier.java:228) at java.util.jar.JarFile.initializeVerifier(JarFile.java:383) at java.util.jar.JarFile.getInputStream(JarFile.java:450) at sun.misc.JarIndex.getJarIndex(JarIndex.java:137) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:839) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:831) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:830) at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:803) at sun.misc.URLClassPath$3.run(URLClassPath.java:530) at sun.misc.URLClassPath$3.run(URLClassPath.java:520) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath.getLoader(URLClassPath.java:519) at sun.misc.URLClassPath.getLoader(URLClassPath.java:492) at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:457) at sun.misc.URLClassPath.getResource(URLClassPath.java:211) at java.net.URLClassLoader$1.run(URLClassLoader.java:365) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.spark.deploy.yarn.ApplicationMaster.startUserApplication(ApplicationMaster.scala:546) at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:334) at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:197) at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:680) at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:69) at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:68) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796) at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:68) at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:678) at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) 18/08/05 11:19:59 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 10, (reason: Uncaught exception: java.lang.SecurityException: Invalid signature file digest for Manifest main attributes) 18/08/05 11:19:59 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with FAILED (diag message: Uncaught exception: java.lang.SecurityException: Invalid signature file digest for Manifest main attributes) 18/08/05 11:19:59 INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1531861476380_0028 18/08/05 11:20:00 INFO util.ShutdownHookManager: Shutdown hook called   LogType:stdout Log Upload Time:Sun Aug 05 11:20:01 +0800 2018 LogLength:0 Log Contents:       Container: container_1531861476380_0028_01_000001 on slave42_8041 =================================================================== LogType:stderr Log Upload Time:Sun Aug 05 11:20:00 +0800 2018 LogLength:3672 Log Contents: 18/08/05 11:19:56 INFO yarn.ApplicationMaster: Registered signal handlers for [TERM, HUP, INT] 18/08/05 11:19:57 INFO yarn.ApplicationMaster: ApplicationAttemptId: appattempt_1531861476380_0028_000001 18/08/05 11:19:58 INFO spark.SecurityManager: Changing view acls to: yarn,hdfs 18/08/05 11:19:58 INFO spark.SecurityManager: Changing modify acls to: yarn,hdfs 18/08/05 11:19:58 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(yarn, hdfs); users with modify permissions: Set(yarn, hdfs) 18/08/05 11:19:58 INFO yarn.ApplicationMaster: Starting the user application in a separate Thread 18/08/05 11:19:58 ERROR yarn.ApplicationMaster: Uncaught exception:  java.lang.SecurityException: Invalid signature file digest for Manifest main attributes at sun.security.util.SignatureFileVerifier.processImpl(SignatureFileVerifier.java:284) at sun.security.util.SignatureFileVerifier.process(SignatureFileVerifier.java:238) at java.util.jar.JarVerifier.processEntry(JarVerifier.java:316) at java.util.jar.JarVerifier.update(JarVerifier.java:228) at java.util.jar.JarFile.initializeVerifier(JarFile.java:383) at java.util.jar.JarFile.getInputStream(JarFile.java:450) at sun.misc.JarIndex.getJarIndex(JarIndex.java:137) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:839) at sun.misc.URLClassPath$JarLoader$1.run(URLClassPath.java:831) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath$JarLoader.ensureOpen(URLClassPath.java:830) at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:803) at sun.misc.URLClassPath$3.run(URLClassPath.java:530) at sun.misc.URLClassPath$3.run(URLClassPath.java:520) at java.security.AccessController.doPrivileged(Native Method) at sun.misc.URLClassPath.getLoader(URLClassPath.java:519) at sun.misc.URLClassPath.getLoader(URLClassPath.java:492) at sun.misc.URLClassPath.getNextLoader(URLClassPath.java:457) at sun.misc.URLClassPath.getResource(URLClassPath.java:211) at java.net.URLClassLoader$1.run(URLClassLoader.java:365) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at org.apache.spark.deploy.yarn.ApplicationMaster.startUserApplication(ApplicationMaster.scala:546) at org.apache.spark.deploy.yarn.ApplicationMaster.runDriver(ApplicationMaster.scala:334) at org.apache.spark.deploy.yarn.ApplicationMaster.run(ApplicationMaster.scala:197) at org.apache.spark.deploy.yarn.ApplicationMaster$$anonfun$main$1.apply$mcV$sp(ApplicationMaster.scala:680) at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:69) at org.apache.spark.deploy.SparkHadoopUtil$$anon$1.run(SparkHadoopUtil.scala:68) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1796) at org.apache.spark.deploy.SparkHadoopUtil.runAsSparkUser(SparkHadoopUtil.scala:68) at org.apache.spark.deploy.yarn.ApplicationMaster$.main(ApplicationMaster.scala:678) at org.apache.spark.deploy.yarn.ApplicationMaster.main(ApplicationMaster.scala) 18/08/05 11:19:58 INFO yarn.ApplicationMaster: Final app status: FAILED, exitCode: 10, (reason: Uncaught exception: java.lang.SecurityException: Invalid signature file digest for Manifest main attributes) 18/08/05 11:19:58 INFO util.ShutdownHookManager: Shutdown hook called   LogType:stdout Log Upload Time:Sun Aug 05 11:20:00 +0800 2018 LogLength:0 Log Contents:

要回复问题请先登录注册