运行spark-pi出现错误,下面是运行log,配置文件spark-env.sh spark-default.conf 自己没有找到配置文件哪里出错,希望老师帮忙下

## error: ``` [root@HTY-1 spark]# spark-submit --master spark://10.118.100.4:7077 --class org.apache.spark.examples.SparkPi --name Spark-Pi /root/spark/lib/spark-examples-1.6.1-hadoop2.6.0.jar Warning: Ignoring non-spark config property: saprk.eventLog.compress=true 16/06/07 00:13:23 INFO spark.SparkContext: Running Spark version 1.6.1 16/06/07 00:13:24 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 16/06/07 00:13:25 INFO spark.SecurityManager: Changing view acls to: root 16/06/07 00:13:25 INFO spark.SecurityManager: Changing modify acls to: root 16/06/07 00:13:25 INFO spark.SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); users with modify permissions: Set(root) 16/06/07 00:13:27 INFO util.Utils: Successfully started service 'sparkDriver' on port 43742. 16/06/07 00:13:27 INFO slf4j.Slf4jLogger: Slf4jLogger started 16/06/07 00:13:27 INFO Remoting: Starting remoting 16/06/07 00:13:28 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriverActorSystem@10.118.100.4:58524] 16/06/07 00:13:28 INFO util.Utils: Successfully started service 'sparkDriverActorSystem' on port 58524. 16/06/07 00:13:28 INFO spark.SparkEnv: Registering MapOutputTracker 16/06/07 00:13:28 INFO spark.SparkEnv: Registering BlockManagerMaster 16/06/07 00:13:28 INFO storage.DiskBlockManager: Created local directory at /tmp/blockmgr-9d26a340-a4f6-463b-b56f-1db6c0d8ee44 16/06/07 00:13:28 INFO storage.MemoryStore: MemoryStore started with capacity 517.4 MB 16/06/07 00:13:29 INFO spark.SparkEnv: Registering OutputCommitCoordinator 16/06/07 00:13:29 INFO server.Server: jetty-8.y.z-SNAPSHOT 16/06/07 00:13:30 INFO server.AbstractConnector: Started SelectChannelConnector@0.0.0.0:4040 16/06/07 00:13:30 INFO util.Utils: Successfully started service 'SparkUI' on port 4040. 16/06/07 00:13:30 INFO ui.SparkUI: Started SparkUI at http://10.118.100.4:4040 16/06/07 00:13:30 INFO spark.HttpFileServer: HTTP File server directory is /tmp/spark-91c90b64-bf5d-4625-ad7f-1102be8e7324/httpd-f00a659e-6c4e-4707-9ba7-b28d24cb8782 16/06/07 00:13:30 INFO spark.HttpServer: Starting HTTP Server 16/06/07 00:13:30 INFO server.Server: jetty-8.y.z-SNAPSHOT 16/06/07 00:13:30 INFO server.AbstractConnector: Started SocketConnector@0.0.0.0:55767 16/06/07 00:13:30 INFO util.Utils: Successfully started service 'HTTP file server' on port 55767. 16/06/07 00:13:31 INFO spark.SparkContext: Added JAR file:/root/spark/lib/spark-examples-1.6.1-hadoop2.6.0.jar at http://10.118.100.4:55767/jars/spark-examples-1.6.1-hadoop2.6.0.jar with timestamp 1465229611103 16/06/07 00:13:31 INFO client.AppClient$ClientEndpoint: Connecting to master spark://10.118.100.4:7077... 16/06/07 00:13:32 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20160607001332-0002 16/06/07 00:13:32 INFO scheduler.TaskSchedulerImpl: Starting speculative execution thread 16/06/07 00:13:32 INFO client.AppClient$ClientEndpoint: Executor added: app-20160607001332-0002/0 on worker-20160606230256-10.118.100.6-47675 (10.118.100.6:47675) with 1 cores 16/06/07 00:13:32 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160607001332-0002/0 on hostPort 10.118.100.6:47675 with 1 cores, 512.0 MB RAM 16/06/07 00:13:32 INFO util.Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 57387. 16/06/07 00:13:32 INFO netty.NettyBlockTransferService: Server created on 57387 16/06/07 00:13:32 INFO storage.BlockManagerMaster: Trying to register BlockManager 16/06/07 00:13:32 INFO client.AppClient$ClientEndpoint: Executor added: app-20160607001332-0002/1 on worker-20160606230300-10.118.100.4-47420 (10.118.100.4:47420) with 1 cores 16/06/07 00:13:32 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160607001332-0002/1 on hostPort 10.118.100.4:47420 with 1 cores, 512.0 MB RAM 16/06/07 00:13:32 INFO storage.BlockManagerMasterEndpoint: Registering block manager 10.118.100.4:57387 with 517.4 MB RAM, BlockManagerId(driver, 10.118.100.4, 57387) 16/06/07 00:13:32 INFO storage.BlockManagerMaster: Registered BlockManager 16/06/07 00:13:32 INFO client.AppClient$ClientEndpoint: Executor added: app-20160607001332-0002/2 on worker-20160606230253-10.118.100.2-50505 (10.118.100.2:50505) with 1 cores 16/06/07 00:13:32 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160607001332-0002/2 on hostPort 10.118.100.2:50505 with 1 cores, 512.0 MB RAM 16/06/07 00:13:32 INFO client.AppClient$ClientEndpoint: Executor added: app-20160607001332-0002/3 on worker-20160606230255-10.118.100.5-40538 (10.118.100.5:40538) with 1 cores 16/06/07 00:13:32 INFO cluster.SparkDeploySchedulerBackend: Granted executor ID app-20160607001332-0002/3 on hostPort 10.118.100.5:40538 with 1 cores, 512.0 MB RAM 16/06/07 00:13:32 INFO client.AppClient$ClientEndpoint: Executor updated: app-20160607001332-0002/0 is now RUNNING 16/06/07 00:13:32 INFO client.AppClient$ClientEndpoint: Executor updated: app-20160607001332-0002/2 is now RUNNING 16/06/07 00:13:32 INFO client.AppClient$ClientEndpoint: Executor updated: app-20160607001332-0002/1 is now RUNNING 16/06/07 00:13:32 INFO client.AppClient$ClientEndpoint: Executor updated: app-20160607001332-0002/3 is now RUNNING ###16/06/07 00:13:35 ERROR spark.SparkContext: Error initializing SparkContext. java.lang.IllegalArgumentException: java.net.UnknownHostException: systex at org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374) at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:312) at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:178) at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:665) at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:601) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:148) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2596) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1650) at org.apache.spark.scheduler.EventLoggingListener.(EventLoggingListener.scala:66) at org.apache.spark.SparkContext.(SparkContext.scala:547) at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:29) at org.apache.spark.examples.SparkPi.main(SparkPi.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ### Caused by: java.net.UnknownHostException: systex ... 25 more 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/metrics/json,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/kill,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/api,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/static,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump/json,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/threadDump,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors/json,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/executors,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment/json,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/environment,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd/json,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/rdd,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage/json,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/storage,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool/json,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/pool,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage/json,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/stage,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages/json,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/stages,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job/json,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/job,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs/json,null} 16/06/07 00:13:35 INFO handler.ContextHandler: stopped o.s.j.s.ServletContextHandler{/jobs,null} 16/06/07 00:13:35 INFO ui.SparkUI: Stopped Spark web UI at http://10.118.100.4:4040 16/06/07 00:13:35 INFO cluster.SparkDeploySchedulerBackend: Shutting down all executors 16/06/07 00:13:35 INFO cluster.SparkDeploySchedulerBackend: Asking each executor to shut down 16/06/07 00:13:35 INFO spark.MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped! 16/06/07 00:13:35 INFO storage.MemoryStore: MemoryStore cleared 16/06/07 00:13:35 INFO storage.BlockManager: BlockManager stopped 16/06/07 00:13:35 INFO storage.BlockManagerMaster: BlockManagerMaster stopped 16/06/07 00:13:35 INFO scheduler.OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped! 16/06/07 00:13:35 INFO spark.SparkContext: Successfully stopped SparkContext 16/06/07 00:13:35 INFO remote.RemoteActorRefProvider$RemotingTerminator: Shutting down remote daemon. ###Exception in thread "main" java.lang.IllegalArgumentException: java.net.UnknownHostException: systex org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374) at org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:312) at org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:178) at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:665) at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:601) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:148) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2596) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370) at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1650) at org.apache.spark.scheduler.EventLoggingListener.(EventLoggingListener.scala:66) at org.apache.spark.SparkContext.(SparkContext.scala:547) at org.apache.spark.examples.SparkPi$.main(SparkPi.scala:29) at org.apache.spark.examples.SparkPi.main(SparkPi.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) ###Caused by: java.net.UnknownHostException: systex 25 more 16/06/07 00:13:36 INFO remote.RemoteActorRefProvider$RemotingTerminator: Remote daemon shut down; proceeding with flushing remote transports. 16/06/07 00:13:36 INFO util.ShutdownHookManager: Shutdown hook called 16/06/07 00:13:36 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-91c90b64-bf5d-4625-ad7f-1102be8e7324 16/06/07 00:13:36 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-91c90b64-bf5d-4625-ad7f-1102be8e7324/httpd-f00a659e-6c4e-4707-9ba7-b28d24cb8782 [root@HTY-1 spark]# ``` ### spark-env.sh ``` #!/usr/bin/env bash export JAVA_HOME=/root/hadoop/jdk1.7.0_67 export SCALA_HOME=/root/scala export HADOOP_HOME=/root/hadoop/hadoop-2.6.2 export HADOOP_CONF_DIR=/root/hadoop/hadoop-2.6.2/etc/hadoop SAPRK_LOCAL_IP=10.118.100.4 SAPRK_LOCAL_DIRS=/root/data/spark/local SPARK_MASTER_IP=10.118.100.4 SPARK_MASTER_WEBUI_PORT=8080 SPARK_WORKER_CORES=2 SPARK_WORKER_MEMORY=8g SPARK_WORKER_DIR=/root/data/spark/work export SPARK_MASTER_OPTS="-Dspark.deploy.defaultCores=4" export SPARK_WORKER_OPTS="-Dspark.worker.cleanup.enabled=true -Dspark.worker.cleanup.appDataTtl=604800" export SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 -Dspark.history.retainedApplication=3 -Dspark.history.fs.logDirectory=hdfs://systex/user/spark/applicationHistory" SPARK_LOG_DIR=/root/data/spark/log "spark-env.sh" 89L, 4965C ``` ### spark-default.conf ``` spark.eventLog.enabled true spark.eventLog.dir hdfs://systex/user/spark/applicationHistory saprk.eventLog.compress true spark.broadcast.blockSize 8m spark.executor.cores 1 spark.executor.memory 512m spark.executor.heartbeatInterval 20s spark.files.fetch Timeout 120s spark.task.maxFailures 6 spark.serializer org.apache.spark.serializer.KryoSerializer spark.kryoserializer.buffer.max 256m spark.akka.frameSize 128 spark.default.parallelism 20 spark.network.timeout 300s spark.speculation true ```

yanglei

赞同来自: apostate4

"Caused by: java.net.UnknownHostException: systex"   log中不是已经提示错误了吗?  在spark集群中每个节点机器上的 /etc/hosts中加入该host和Ip的配对试试。

apostate4 - step by step

赞同来自:

确保你的hdfs用 systex 这个hostname是可以访问的。

trueyao

赞同来自:

如楼上所说,是host设置不对。看具体错误信息,发现是hadoop路径出了问题。在 spark-env.sh 中的第20行
-Dspark.history.fs.logDirectory=hdfs://systex/user/spark/applicationHistor
以及 spark-default.conf 中的第2行
spark.eventLog.dir hdfs://systex/user/spark/applicationHistory
hdfs的路径设置应包含主机:端口,类似于hdfs://10.118.100.4:8020/systex/user/spark/applicationHistory  这个端口号应该要和你hadoop中core-site.xmlfs.defaultFS 设置的一样。由于使用了hdfs,所以在运行Spark-Pi之前,需要先启动hdfs。 顺便说一句,以上设计到"history","eventLog"的配置是为了开启spark的history服务器,用于程序运行完成后查看应用历史。所以如果你暂时不需要查看应用历史,那么可以先把相关配置注释掉,再看运行有没有问题吧。

@CrazyChao - 人生不止眼前的苟且,还有诗和远方的田野!^.^

赞同来自:

路径错误,相对路径与绝对路径没有分清!我应该使用/user/root/spark....这个路径

要回复问题请先登录注册