Hbase 启动时 HRegionserver 进程启动失败

[root@slave1 logs]# cat hbase-root-regionserver-slave1.log
Wed Jul 11 17:50:16 CST 2018 Starting regionserver on slave1
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7190
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 7190
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited
2018-07-11 17:50:39,644 INFO  [main] regionserver.HRegionServer: STARTING executorService HRegionServer
2018-07-11 17:50:39,647 INFO  [main] util.VersionInfo: HBase 2.0.0
2018-07-11 17:50:39,647 INFO  [main] util.VersionInfo: Source code repository git://kalashnikov.att.net/Users/stack/checkouts/hbase.git revision=7483b111e4da77adbfc8062b3b22cbe7c2cb91c1
2018-07-11 17:50:39,647 INFO  [main] util.VersionInfo: Compiled by stack on Sun Apr 22 20:26:55 PDT 2018
2018-07-11 17:50:39,647 INFO  [main] util.VersionInfo: From source with checksum a59e806496ef216732e730c746bbe5ac
2018-07-11 17:50:41,728 INFO  [main] util.ServerCommandLine: hbase.tmp.dir: /tmp/hbase-root
2018-07-11 17:50:41,728 INFO  [main] util.ServerCommandLine: hbase.rootdir: hdfs://192.168.1.129:9000/hbase
2018-07-11 17:50:41,728 INFO  [main] util.ServerCommandLine: hbase.cluster.distributed: true
2018-07-11 17:50:41,728 INFO  [main] util.ServerCommandLine: hbase.zookeeper.quorum: slave1,slave2
2018-07-11 17:50:41,731 INFO  [main] util.ServerCommandLine: env:HBASE_LOGFILE=hbase-root-regionserver-slave1.log
2018-07-11 17:50:41,731 INFO  [main] util.ServerCommandLine: env:PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
2018-07-11 17:50:41,731 INFO  [main] util.ServerCommandLine: env:JAVA_HOME=/usr/java/jdk1.8.0_171
2018-07-11 17:50:41,731 INFO  [main] util.ServerCommandLine: env:LANG=en_US.UTF-8
2018-07-11 17:50:41,731 INFO  [main] util.ServerCommandLine: env:XDG_SESSION_ID=4
2018-07-11 17:50:41,732 INFO  [main] util.ServerCommandLine: env:JRE_HOME=/usr/java/jdk1.8.0_171/jre
2018-07-11 17:50:41,732 INFO  [main] util.ServerCommandLine: env:MAIL=/var/mail/root
2018-07-11 17:50:41,732 INFO  [main] util.ServerCommandLine: env:LD_LIBRARY_PATH=:/usr/hadoop-2.8.4/lib/native
2018-07-11 17:50:41,732 INFO  [main] util.ServerCommandLine: env:LOGNAME=root
2018-07-11 17:50:41,732 INFO  [main] util.ServerCommandLine: env:HBASE_REST_OPTS=
2018-07-11 17:50:41,732 INFO  [main] util.ServerCommandLine: env:PWD=/root
2018-07-11 17:50:41,732 INFO  [main] util.ServerCommandLine: env:HBASE_CLASSPATH=/usr/hbase/conf
2018-07-11 17:50:41,732 INFO  [main] util.ServerCommandLine: env:HBASE_ROOT_LOGGER=INFO,RFA
2018-07-11 17:50:41,732 INFO  [main] util.ServerCommandLine: env:LESSOPEN=||/usr/bin/lesspipe.sh %s
2018-07-11 17:50:41,733 INFO  [main] util.ServerCommandLine: env:SHELL=/bin/bash
2018-07-11 17:50:41,733 INFO  [main] util.ServerCommandLine: env:HBASE_HEAPSIZE=4G
2018-07-11 17:50:41,733 INFO  [main] util.ServerCommandLine: env:HBASE_ENV_INIT=true
2018-07-11 17:50:41,733 INFO  [main] util.ServerCommandLine: env:HBASE_IDENT_STRING=root
2018-07-11 17:50:41,733 INFO  [main] util.ServerCommandLine: env:HBASE_ZNODE_FILE=/tmp/hbase-root-regionserver.znode
2018-07-11 17:50:41,733 INFO  [main] util.ServerCommandLine: env:SSH_CLIENT=192.168.1.129 59070 22
2018-07-11 17:50:41,733 INFO  [main] util.ServerCommandLine: env:HBASE_LOG_PREFIX=hbase-root-regionserver-slave1
2018-07-11 17:50:41,733 INFO  [main] util.ServerCommandLine: env:HBASE_LOG_DIR=/usr/hbase/logs
2018-07-11 17:50:41,733 INFO  [main] util.ServerCommandLine: env:USER=root
2018-07-11 17:50:41,734 INFO  [main] util.ServerCommandLine: lib/metrics-core-3.2.1.jar:/usr/hbase/lib/netty-all-4.0.23.Final.jar:/usr/hbase/lib/org.eclipse.jdt.core-3.8.2.v20130121.jar:/usr/hbase/lib/osgi-resource-locator-1.0.1.jar:/usr/hbase/lib/paranamer-2.3.jar:/usr/hbase/lib/protobuf-java-2.5.0.jar:/usr/hbase/lib/slf4j-api-1.7.25.jar:/usr/hbase/lib/slf4j-log4j12-1.7.25.jar:/usr/hbase/lib/snappy-java-1.0.5.jar:/usr/hbase/lib/spymemcached-2.12.2.jar:/usr/hbase/lib/validation-api-1.1.0.Final.jar:/usr/hbase/lib/xmlenc-0.52.jar:/usr/hbase/lib/xz-1.0.jar:/usr/hbase/lib/zookeeper-3.4.10.jar:/usr/hadoop-2.8.4/etc/hadoop:/usr/hadoop-2.8.4/share/hadoop/common/lib/*:/usr/hadoop-2.8.4/share/hadoop/common/*:/usr/hadoop-2.8.4/share/hadoop/hdfs:/usr/hadoop-2.8.4/share/hadoop/hdfs/lib/*:/usr/hadoop-2.8.4/share/hadoop/hdfs/*:/usr/hadoop-2.8.4/share/hadoop/yarn/lib/*:/usr/hadoop-2.8.4/share/hadoop/yarn/*:/usr/hadoop-2.8.4/share/hadoop/mapreduce/lib/*:/usr/hadoop-2.8.4/share/hadoop/mapreduce/*:/usr/hadoop-2.8.4/contrib/capacity-scheduler/*.jar:/usr/hbase/conf
2018-07-11 17:50:41,734 INFO  [main] util.ServerCommandLine: env:HBASE_MANAGES_ZK=false
2018-07-11 17:50:41,734 INFO  [main] util.ServerCommandLine: env:SSH_CONNECTION=192.168.1.129 59070 192.168.1.128 22
2018-07-11 17:50:41,734 INFO  [main] util.ServerCommandLine: env:HBASE_AUTOSTART_FILE=/tmp/hbase-root-regionserver.autostart
2018-07-11 17:50:41,734 INFO  [main] util.ServerCommandLine: env:HADOOP_HOME=/usr/hadoop-2.8.4
2018-07-11 17:50:41,734 INFO  [main] util.ServerCommandLine: env:HBASE_NICENESS=0
2018-07-11 17:50:41,734 INFO  [main] util.ServerCommandLine: env:HBASE_OPTS= -XX:+UseConcMarkSweepGC   -Dhbase.log.dir=/usr/hbase/logs -Dhbase.log.file=hbase-root-regionserver-slave1.log -Dhbase.home.dir=/usr/hbase -Dhbase.id.str=root -Dhbase.root.logger=INFO,RFA -Djava.library.path=/usr/hadoop-2.8.4/lib/native -Dhbase.security.logger=INFO,RFAS
2018-07-11 17:50:41,734 INFO  [main] util.ServerCommandLine: env:HBASE_SECURITY_LOGGER=INFO,RFAS
2018-07-11 17:50:41,734 INFO  [main] util.ServerCommandLine: env:XDG_RUNTIME_DIR=/run/user/0
2018-07-11 17:50:41,734 INFO  [main] util.ServerCommandLine: env:HBASE_THRIFT_OPTS=
2018-07-11 17:50:41,734 INFO  [main] util.ServerCommandLine: env:HBASE_HOME=/usr/hbase
2018-07-11 17:50:41,735 INFO  [main] util.ServerCommandLine: env:SHLVL=3
2018-07-11 17:50:41,735 INFO  [main] util.ServerCommandLine: env:HOME=/root
2018-07-11 17:50:41,735 INFO  [main] util.ServerCommandLine: env:MALLOC_ARENA_MAX=4
2018-07-11 17:50:41,760 INFO  [main] util.ServerCommandLine: vmName=Java HotSpot(TM) 64-Bit Server VM, vmVendor=Oracle Corporation, vmVersion=25.171-b11
2018-07-11 17:50:41,760 INFO  [main] util.ServerCommandLine: vmInputArguments=[-Dproc_regionserver, -XX:OnOutOfMemoryError=kill -9 %p, -Xmx4G, -XX:+UseConcMarkSweepGC, -Dhbase.log.dir=/usr/hbase/logs, -Dhbase.log.file=hbase-root-regionserver-slave1.log, -Dhbase.home.dir=/usr/hbase, -Dhbase.id.str=root, -Dhbase.root.logger=INFO,RFA, -Djava.library.path=/usr/hadoop-2.8.4/lib/native, -Dhbase.security.logger=INFO,RFAS]
2018-07-11 17:50:44,871 INFO  [main] metrics.MetricRegistries: Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl
2018-07-11 17:50:46,008 INFO  [main] regionserver.RSRpcServices: regionserver/slave1:16020 server-side Connection retries=45
2018-07-11 17:50:46,034 INFO  [main] ipc.RpcExecutor: Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=3, maxQueueLength=300, handlerCount=30
2018-07-11 17:50:46,036 INFO  [main] ipc.RpcExecutor: Instantiated priority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=300, handlerCount=20
2018-07-11 17:50:46,037 INFO  [main] ipc.RpcExecutor: Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=300, handlerCount=3
2018-07-11 17:50:46,266 INFO  [main] ipc.RpcServerFactory: Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService
2018-07-11 17:50:46,789 INFO  [main] io.ByteBufferPool: Created with bufferSize=64 KB and maxPoolSize=1.88 KB
2018-07-11 17:50:47,115 INFO  [main] ipc.NettyRpcServer: Bind to /192.168.1.128:16020
2018-07-11 17:50:47,411 INFO  [main] hfile.CacheConfig: Allocating onheap LruBlockCache size=1.60 GB, blockSize=64 KB
2018-07-11 17:50:47,456 INFO  [main] hfile.CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1.20 MB, freeSize=1.60 GB, maxSize=1.60 GB, heapSize=1.20 MB, minSize=1.52 GB, minFactor=0.95, multiSize=776.66 MB, multiFactor=0.5, singleSize=388.33 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2018-07-11 17:50:47,458 INFO  [main] hfile.CacheConfig: Created cacheConfig: blockCache=LruBlockCache{blockCount=0, currentSize=1.20 MB, freeSize=1.60 GB, maxSize=1.60 GB, heapSize=1.20 MB, minSize=1.52 GB, minFactor=0.95, multiSize=776.66 MB, multiFactor=0.5, singleSize=388.33 MB, singleFactor=0.25}, cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false
2018-07-11 17:50:51,138 INFO  [main] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
2018-07-11 17:50:51,145 INFO  [main] fs.HFileSystem: Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks
2018-07-11 17:50:51,483 INFO  [main] zookeeper.RecoverableZooKeeper: Process identifier=regionserver:16020 connecting to ZooKeeper ensemble=slave1:2181,slave2:2181
2018-07-11 17:50:51,516 INFO  [main] zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.10-39d3a4f269333c922ed3db283be479f9deacaa0f, built on 03/23/2017 10:13 GMT
2018-07-11 17:50:51,516 INFO  [main] zookeeper.ZooKeeper: Client environment:host.name=slave1
2018-07-11 17:50:51,517 INFO  [main] zookeeper.ZooKeeper: Client environment:java.version=1.8.0_171
2018-07-11 17:50:51,517 INFO  [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
2018-07-11 17:50:51,517 INFO  [main] zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.8.0_171/jre
2018-07-11 17:50:51,517 INFO  [main] zookeeper.ZooKeeper: :/usr/hadoop-2.8.4/share/hadoop/mapreduce/lib/commons-io-2.4.jar:/usr/hadoop-2.8.4/share/hadoop/mapreduce/lib/guice-3.0.jar:/usr/hadoop-2.8.4/share/hadoop/mapreduce/lib/snappy-java-1.0.4.1.jar:/usr/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-plugins-2.8.4.jar:/usr/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-app-2.8.4.jar:/usr/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.8.4.jar:/usr/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-common-2.8.4.jar:/usr/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-hs-2.8.4.jar:/usr/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-shuffle-2.8.4.jar:/usr/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-core-2.8.4.jar:/usr/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.4-tests.jar:/usr/hadoop-2.8.4/share/hadoop/mapreduce/hadoop-mapreduce-client-jobclient-2.8.4.jar:/usr/hadoop-2.8.4/contrib/capacity-scheduler/*.jar:/usr/hbase/conf
2018-07-11 17:50:51,517 INFO  [main] zookeeper.ZooKeeper: Client environment:java.library.path=/usr/hadoop-2.8.4/lib/native
2018-07-11 17:50:51,517 INFO  [main] zookeeper.ZooKeeper: Client environment:java.io.tmpdir=/tmp
2018-07-11 17:50:51,517 INFO  [main] zookeeper.ZooKeeper: Client environment:java.compiler=<NA>
2018-07-11 17:50:51,517 INFO  [main] zookeeper.ZooKeeper: Client environment:os.name=Linux
2018-07-11 17:50:51,517 INFO  [main] zookeeper.ZooKeeper: Client environment:os.arch=amd64
2018-07-11 17:50:51,517 INFO  [main] zookeeper.ZooKeeper: Client environment:os.version=3.10.0-862.el7.x86_64
2018-07-11 17:50:51,517 INFO  [main] zookeeper.ZooKeeper: Client environment:user.name=root
2018-07-11 17:50:51,517 INFO  [main] zookeeper.ZooKeeper: Client environment:user.home=/root
2018-07-11 17:50:51,517 INFO  [main] zookeeper.ZooKeeper: Client environment:user.dir=/root
2018-07-11 17:50:51,519 INFO  [main] zookeeper.ZooKeeper: Initiating client connection, connectString=slave1:2181,slave2:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.PendingWatcher@35088e87
2018-07-11 17:50:51,663 INFO  [main-SendThread(slave2:2181)] zookeeper.ClientCnxn: Opening socket connection to server slave2/192.168.1.130:2181. Will not attempt to authenticate using SASL (unknown error)
2018-07-11 17:50:51,686 INFO  [main-SendThread(slave2:2181)] zookeeper.ClientCnxn: Socket connection established to slave2/192.168.1.130:2181, initiating session
2018-07-11 17:50:51,708 INFO  [main-SendThread(slave2:2181)] zookeeper.ClientCnxn: Session establishment complete on server slave2/192.168.1.130:2181, sessionid = 0x36488ab4beb0002, negotiated timeout = 40000
2018-07-11 17:50:51,838 INFO  [main] util.log: Logging initialized @23218ms
2018-07-11 17:50:51,990 INFO  [main] http.HttpRequestLog: Http request log for http.requests.regionserver is not defined
2018-07-11 17:50:52,012 INFO  [main] http.HttpServer: Added global filter 'safety' (class=org.apache.hadoop.hbase.http.HttpServer$QuotingInputFilter)
2018-07-11 17:50:52,014 INFO  [main] http.HttpServer: Added global filter 'clickjackingprevention' (class=org.apache.hadoop.hbase.http.ClickjackingPreventionFilter)
2018-07-11 17:50:52,017 INFO  [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context regionserver
2018-07-11 17:50:52,017 INFO  [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context logs
2018-07-11 17:50:52,018 INFO  [main] http.HttpServer: Added filter static_user_filter (class=org.apache.hadoop.hbase.http.lib.StaticUserWebFilter$StaticUserFilter) to context static
2018-07-11 17:50:52,081 INFO  [main] http.HttpServer: Jetty bound to port 16030
2018-07-11 17:50:52,084 INFO  [main] server.Server: jetty-9.3.19.v20170502
2018-07-11 17:50:52,182 INFO  [main] handler.ContextHandler: Started o.e.j.s.ServletContextHandler@1de9d54{/logs,file:///usr/hbase/logs/,AVAILABLE}
2018-07-11 17:50:52,183 INFO  [main] handler.ContextHandler: Started o.e.j.s.ServletContextHandler@47ad69f7{/static,file:///usr/hbase/hbase-webapps/static/,AVAILABLE}
2018-07-11 17:50:52,498 INFO  [main] handler.ContextHandler: Started o.e.j.w.WebAppContext@46731692{/,file:///usr/hbase/hbase-webapps/regionserver/,AVAILABLE}{file:/usr/hbase/hbase-webapps/regionserver}
2018-07-11 17:50:52,509 INFO  [main] server.AbstractConnector: Started ServerConnector@3b9632d1{HTTP/1.1,[http/1.1]}{0.0.0.0:16030}
2018-07-11 17:50:52,509 INFO  [main] server.Server: Started @23890ms
2018-07-11 17:50:52,553 INFO  [regionserver/slave1:16020] regionserver.HRegionServer: ClusterId : 4475c45c-36d4-4c3c-a3d8-63579c1a8352
2018-07-11 17:50:52,710 INFO  [regionserver/slave1:16020] zookeeper.ReadOnlyZKClient: Connect 0x1b92d5dc to slave1:2181,slave2:2181 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms
2018-07-11 17:50:52,730 INFO  [ReadOnlyZKClient-slave1:2181,slave2:2181@0x1b92d5dc] zookeeper.ZooKeeper: Initiating client connection, connectString=slave1:2181,slave2:2181 sessionTimeout=90000 watcher=org.apache.hadoop.hbase.zookeeper.ReadOnlyZKClient$$Lambda$37/244688886@4b972840
2018-07-11 17:50:52,732 INFO  [ReadOnlyZKClient-slave1:2181,slave2:2181@0x1b92d5dc-SendThread(slave1:2181)] zookeeper.ClientCnxn: Opening socket connection to server slave1/192.168.1.128:2181. Will not attempt to authenticate using SASL (unknown error)
2018-07-11 17:50:52,733 INFO  [ReadOnlyZKClient-slave1:2181,slave2:2181@0x1b92d5dc-SendThread(slave1:2181)] zookeeper.ClientCnxn: Socket connection established to slave1/192.168.1.128:2181, initiating session
2018-07-11 17:50:52,743 INFO  [ReadOnlyZKClient-slave1:2181,slave2:2181@0x1b92d5dc-SendThread(slave1:2181)] zookeeper.ClientCnxn: Session establishment complete on server slave1/192.168.1.128:2181, sessionid = 0x26488a678f60003, negotiated timeout = 40000
2018-07-11 17:50:52,832 INFO  [regionserver/slave1:16020] regionserver.RegionServerCoprocessorHost: System coprocessor loading is enabled
2018-07-11 17:50:52,833 INFO  [regionserver/slave1:16020] regionserver.RegionServerCoprocessorHost: Table coprocessor loading is enabled
2018-07-11 17:50:52,850 INFO  [regionserver/slave1:16020] regionserver.HRegionServer: reportForDuty to master=master,16000,1531302618708 with port=16020, startcode=1531302642058
2018-07-11 17:50:53,298 INFO  [regionserver/slave1:16020] wal.WALFactory: Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider
2018-07-11 17:50:53,594 INFO  [regionserver/slave1:16020] regionserver.MetricsRegionServerWrapperImpl: Computing regionserver metrics every 5000 milliseconds
2018-07-11 17:50:53,660 INFO  [regionserver/slave1:16020] regionserver.MemStoreFlusher: globalMemStoreLimit=1.6 G, globalMemStoreLimitLowMark=1.5 G, Offheap=false
2018-07-11 17:50:53,694 INFO  [regionserver/slave1:16020] throttle.PressureAwareCompactionThroughputController: Compaction throughput configurations, higher bound: 20.00 MB/second, lower bound 10.00 MB/second, off peak: unlimited, tuning period: 60000 ms
2018-07-11 17:50:53,706 INFO  [regionserver/slave1:16020] regionserver.HRegionServer: CompactionChecker runs every PT10S
2018-07-11 17:50:54,100 INFO  [regionserver/slave1:16020] regionserver.HeapMemoryManager: Starting, tuneOn=false
2018-07-11 17:50:54,107 INFO  [regionserver/slave1:16020] regionserver.ChunkCreator: Allocating data MemStoreChunkPool with chunk size 2 MB, max count 735, initial count 0
2018-07-11 17:50:54,109 INFO  [regionserver/slave1:16020] regionserver.ChunkCreator: Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 817, initial count 0
2018-07-11 17:50:54,216 INFO  [SplitLogWorker-slave1:16020] regionserver.SplitLogWorker: SplitLogWorker slave1,16020,1531302642058 starting
2018-07-11 17:50:54,251 INFO  [ReplicationExecutor-0] regionserver.ReplicationSourceManager: Current list of replicators: [slave1,16020,1531215877492, slave2,16020,1531282693583, slave2,16020,1531209762102, slave1,16020,1531294545774, slave1,16020,1531209761968, slave2,16020,1531292513897, slave1,16020,1531289321699, slave2,16020,1531292885825, slave1,16020,1531282693671, slave1,16020,1531278747603, slave1,16020,1531297827564, slave2,16020,1531297827578, master,16020,1531300588994, slave2,16020,1531302641448, slave1,16020,1531302642058, slave2,16020,1531215878406, slave1,16020,1531292885786, slave1,16020,1531292513901, slave2,16020,1531289321717, slave2,16020,1531294545098, slave1,16020,1531214678920, slave1,16020,1531214630161, slave2,16020,1531278747573] other RSs: [slave2,16020,1531302641448, slave1,16020,1531302642058]
2018-07-11 17:50:54,424 INFO  [regionserver/slave1:16020] regionserver.HRegionServer: Serving as slave1,16020,1531302642058, RpcServer on slave1/192.168.1.128:16020, sessionid=0x36488ab4beb0002
2018-07-11 17:50:54,463 INFO  [regionserver/slave1:16020] quotas.RegionServerRpcQuotaManager: Quota support disabled
2018-07-11 17:50:54,463 INFO  [regionserver/slave1:16020] quotas.RegionServerSpaceQuotaManager: Quota support disabled, not starting space quota manager.
2018-07-11 17:50:56,784 INFO  [main-EventThread] replication.ReplicationTrackerZKImpl: /hbase/rs/slave2,16020,1531302641448 znode expired, triggering replicatorRemoved event
2018-07-11 17:50:57,548 INFO  [regionserver/slave1:16020] wal.AbstractFSWAL: WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=slave1%2C16020%2C1531302642058, suffix=, logDir=hdfs://192.168.1.129:9000/hbase/WALs/slave1,16020,1531302642058, archiveDir=hdfs://192.168.1.129:9000/hbase/oldWALs
2018-07-11 17:50:57,623 ERROR [regionserver/slave1:16020] regionserver.HRegionServer: pache/hadoop/fs/ContentSummary; @98: invokestatic
  Reason:
    Type 'org/apache/hadoop/fs/ContentSummary$Builder' (current frame, stack[1]) is not assignable to 'org/apache/hadoop/fs/QuotaUsage$Builder'
  Current Frame:
    bci: @98
    flags: { }
    locals: { 'org/apache/hadoop/hdfs/protocol/proto/HdfsProtos$ContentSummaryProto', 'org/apache/hadoop/fs/ContentSummary$Builder' }
    stack: { 'org/apache/hadoop/hdfs/protocol/proto/HdfsProtos$StorageTypeQuotaInfosProto', 'org/apache/hadoop/fs/ContentSummary$Builder' }
  Bytecode:
    0x0000000: 2ac7 0005 01b0 bb03 4159 b703 424c 2b2a
    0x0000010: b603 43b6 0344 2ab6 0345 b603 462a b603
    0x0000020: 47b6 0348 2ab6 0349 b603 4a2a b603 4bb6
    0x0000030: 034c 2ab6 034d b603 4e2a b603 4fb6 0350
    0x0000040: 2ab6 0351 b603 522a b603 53b6 0354 2ab6
    0x0000050: 0355 b603 5657 2ab6 0357 9900 0b2a b603
    0x0000060: 582b b803 592b b603 5ab0
  Stackmap Table:
    same_frame(@6)
    append_frame(@101,Object[#2149])
 *****
java.lang.VerifyError: Bad type on operand stack
Exception Details:
  Location:
    org/apache/hadoop/hdfs/protocolPB/PBHelperClient.convert(Lorg/apache/hadoop/hdfs/protocol/proto/HdfsProtos$ContentSummaryProto;)Lorg/apache/hadoop/fs/ContentSummary; @98: invokestatic
  Reason:
    Type 'org/apache/hadoop/fs/ContentSummary$Builder' (current frame, stack[1]) is not assignable to 'org/apache/hadoop/fs/QuotaUsage$Builder'
  Current Frame:
    bci: @98
    flags: { }
    locals: { 'org/apache/hadoop/hdfs/protocol/proto/HdfsProtos$ContentSummaryProto', 'org/apache/hadoop/fs/ContentSummary$Builder' }
    stack: { 'org/apache/hadoop/hdfs/protocol/proto/HdfsProtos$StorageTypeQuotaInfosProto', 'org/apache/hadoop/fs/ContentSummary$Builder' }
  Bytecode:
    0x0000000: 2ac7 0005 01b0 bb03 4159 b703 424c 2b2a
    0x0000010: b603 43b6 0344 2ab6 0345 b603 462a b603
    0x0000020: 47b6 0348 2ab6 0349 b603 4a2a b603 4bb6
    0x0000030: 034c 2ab6 034d b603 4e2a b603 4fb6 0350
    0x0000040: 2ab6 0351 b603 522a b603 53b6 0354 2ab6
    0x0000050: 0355 b603 5657 2ab6 0357 9900 0b2a b603
    0x0000060: 582b b803 592b b603 5ab0
  Stackmap Table:
    same_frame(@6)
    append_frame(@101,Object[#2149])

        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.createPBHelper(FanOutOneBlockAsyncDFSOutputHelper.java:418)
        at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.<clinit>(FanOutOneBlockAsyncDFSOutputHelper.java:555)
        at org.apache.hadoop.hbase.io.asyncfs.AsyncFSOutputHelper.createOutput(AsyncFSOutputHelper.java:51)
        at org.apache.hadoop.hbase.regionserver.wal.AsyncProtobufLogWriter.initOutput(AsyncProtobufLogWriter.java:168)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractProtobufLogWriter.init(AbstractProtobufLogWriter.java:167)
        at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createAsyncWriter(AsyncFSWALProvider.java:99)
        at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:612)
        at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.createWriterInstance(AsyncFSWAL.java:124)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:759)
        at org.apache.hadoop.hbase.regionserver.wal.AbstractFSWAL.rollWriter(AbstractFSWAL.java:489)
        at org.apache.hadoop.hbase.regionserver.wal.AsyncFSWAL.<init>(AsyncFSWAL.java:251)
        at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:69)
        at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.createWAL(AsyncFSWALProvider.java:44)
        at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:138)
        at org.apache.hadoop.hbase.wal.AbstractFSWALProvider.getWAL(AbstractFSWALProvider.java:57)
        at org.apache.hadoop.hbase.wal.WALFactory.getWAL(WALFactory.java:252)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.getWAL(HRegionServer.java:2065)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.buildServerLoad(HRegionServer.java:1291)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.tryRegionServerReport(HRegionServer.java:1172)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:989)
        at java.lang.Thread.run(Thread.java:748)
2018-07-11 17:50:57,625 ERROR [regionserver/slave1:16020] regionserver.HRegionServer: RegionServer abort: loaded coprocessors are: []
2018-07-11 17:50:57,661 INFO  [regionserver/slave1:16020] regionserver.HRegionServer: 0,
    "RequestSize_99th_percentile" : 0,
    "RequestSize_99.9th_percentile" : 0,
    "sentBytes" : 0,
    "QueueCallTime_num_ops" : 0,
    "QueueCallTime_min" : 0,
    "QueueCallTime_max" : 0,
    "QueueCallTime_mean" : 0,
    "QueueCallTime_25th_percentile" : 0,
    "QueueCallTime_median" : 0,
    "QueueCallTime_75th_percentile" : 0,
    "QueueCallTime_90th_percentile" : 0,
    "QueueCallTime_95th_percentile" : 0,
    "QueueCallTime_98th_percentile" : 0,
    "QueueCallTime_99th_percentile" : 0,
    "QueueCallTime_99.9th_percentile" : 0,
    "authenticationFailures" : 0
  } ],
  "beans" : [ {
    "name" : "Hadoop:service=HBase,name=RegionServer,sub=Replication",
    "modelerType" : "RegionServer,sub=Replication",
    "tag.Context" : "regionserver",
    "tag.Hostname" : "slave1"
  } ],
  "beans" : [ {
    "name" : "Hadoop:service=HBase,name=RegionServer,sub=Server",
    "modelerType" : "RegionServer,sub=Server",
    "tag.Context" : "regionserver",
    "tag.Hostname" : "slave1"
  } ]
}
2018-07-11 17:50:57,689 INFO  [regionserver/slave1:16020] regionserver.HRegionServer: ***** STOPPING region server 'slave1,16020,1531302642058' *****
2018-07-11 17:50:57,689 INFO  [regionserver/slave1:16020] regionserver.HRegionServer: Lorg/apache/hadoop/fs/ContentSummary; @98: invokestatic
  Reason:
    Type 'org/apache/hadoop/fs/ContentSummary$Builder' (current frame, stack[1]) is not assignable to 'org/apache/hadoop/fs/QuotaUsage$Builder'
  Current Frame:
    bci: @98
    flags: { }
    locals: { 'org/apache/hadoop/hdfs/protocol/proto/HdfsProtos$ContentSummaryProto', 'org/apache/hadoop/fs/ContentSummary$Builder' }
    stack: { 'org/apache/hadoop/hdfs/protocol/proto/HdfsProtos$StorageTypeQuotaInfosProto', 'org/apache/hadoop/fs/ContentSummary$Builder' }
  Bytecode:
    0x0000000: 2ac7 0005 01b0 bb03 4159 b703 424c 2b2a
    0x0000010: b603 43b6 0344 2ab6 0345 b603 462a b603
    0x0000020: 47b6 0348 2ab6 0349 b603 4a2a b603 4bb6
    0x0000030: 034c 2ab6 034d b603 4e2a b603 4fb6 0350
    0x0000040: 2ab6 0351 b603 522a b603 53b6 0354 2ab6
    0x0000050: 0355 b603 5657 2ab6 0357 9900 0b2a b603
    0x0000060: 582b b803 592b b603 5ab0
  Stackmap Table:
    same_frame(@6)
    append_frame(@101,Object[#2149])

2018-07-11 17:50:57,689 INFO  [regionserver/slave1:16020] regionserver.SplitLogWorker: Sending interrupt to stop the worker thread
2018-07-11 17:50:57,689 INFO  [regionserver/slave1:16020] regionserver.HRegionServer: Stopping infoServer
2018-07-11 17:50:57,690 INFO  [SplitLogWorker-slave1:16020] regionserver.SplitLogWorker: SplitLogWorker interrupted. Exiting.
2018-07-11 17:50:57,692 INFO  [SplitLogWorker-slave1:16020] regionserver.SplitLogWorker: SplitLogWorker slave1,16020,1531302642058 exiting
2018-07-11 17:50:57,703 INFO  [regionserver/slave1:16020] handler.ContextHandler: Stopped o.e.j.w.WebAppContext@46731692{/,null,UNAVAILABLE}{file:/usr/hbase/hbase-webapps/regionserver}
2018-07-11 17:50:57,708 INFO  [regionserver/slave1:16020] server.AbstractConnector: Stopped ServerConnector@3b9632d1{HTTP/1.1,[http/1.1]}{0.0.0.0:16030}
2018-07-11 17:50:57,709 INFO  [regionserver/slave1:16020] handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@47ad69f7{/static,file:///usr/hbase/hbase-webapps/static/,UNAVAILABLE}
2018-07-11 17:50:57,709 INFO  [regionserver/slave1:16020] handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@1de9d54{/logs,file:///usr/hbase/logs/,UNAVAILABLE}
2018-07-11 17:50:57,711 INFO  [regionserver/slave1:16020] regionserver.HeapMemoryManager: Stopping
2018-07-11 17:50:57,711 INFO  [regionserver/slave1:16020] flush.RegionServerFlushTableProcedureManager: Stopping region server flush procedure manager abruptly.
2018-07-11 17:50:57,711 INFO  [regionserver/slave1:16020] snapshot.RegionServerSnapshotManager: Stopping RegionServerSnapshotManager abruptly.
2018-07-11 17:50:57,711 INFO  [regionserver/slave1:16020] regionserver.HRegionServer: aborting server slave1,16020,1531302642058
2018-07-11 17:50:57,711 INFO  [regionserver/slave1:16020] zookeeper.ReadOnlyZKClient: Close zookeeper connection 0x1b92d5dc to slave1:2181,slave2:2181
2018-07-11 17:50:57,716 INFO  [regionserver/slave1:16020] regionserver.HRegionServer: stopping server slave1,16020,1531302642058; all regions closed.
2018-07-11 17:50:57,716 INFO  [regionserver/slave1:16020] regionserver.Leases: Closed leases
2018-07-11 17:50:57,717 INFO  [MemStoreFlusher.0] regionserver.MemStoreFlusher: MemStoreFlusher.0 exiting
2018-07-11 17:50:57,722 INFO  [MemStoreFlusher.1] regionserver.MemStoreFlusher: MemStoreFlusher.1 exiting
2018-07-11 17:50:57,724 INFO  [ReadOnlyZKClient-slave1:2181,slave2:2181@0x1b92d5dc] zookeeper.ZooKeeper: Session: 0x26488a678f60003 closed
2018-07-11 17:50:57,725 INFO  [ReadOnlyZKClient-slave1:2181,slave2:2181@0x1b92d5dc-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x26488a678f60003
2018-07-11 17:50:57,728 INFO  [regionserver/slave1:16020] hbase.ChoreService: Chore service for: regionserver/slave1:16020 had [[ScheduledChore: Name: CompactedHFilesCleaner Period: 120000 Unit: MILLISECONDS], [ScheduledChore: Name: MemstoreFlusherChore Period: 10000 Unit: MILLISECONDS], [ScheduledChore: Name: CompactionThroughputTuner Period: 60000 Unit: MILLISECONDS], [ScheduledChore: Name: MovedRegionsCleaner for region slave1,16020,1531302642058 Period: 120000 Unit: MILLISECONDS]] on shutdown
2018-07-11 17:50:57,729 INFO  [regionserver/slave1:16020] regionserver.CompactSplit: Waiting for Split Thread to finish...
2018-07-11 17:50:57,729 INFO  [regionserver/slave1:16020] regionserver.CompactSplit: Waiting for Large Compaction Thread to finish...
2018-07-11 17:50:57,729 INFO  [regionserver/slave1:16020] regionserver.CompactSplit: Waiting for Small Compaction Thread to finish...
2018-07-11 17:50:57,729 INFO  [regionserver/slave1:16020.logRoller] regionserver.LogRoller: LogRoller exiting.
2018-07-11 17:50:57,731 INFO  [regionserver/slave1:16020] ipc.NettyRpcServer: Stopping server on /192.168.1.128:16020
2018-07-11 17:50:57,765 INFO  [regionserver/slave1:16020] zookeeper.ZooKeeper: Session: 0x36488ab4beb0002 closed
2018-07-11 17:50:57,765 INFO  [regionserver/slave1:16020] regionserver.HRegionServer: Exiting; stopping=slave1,16020,1531302642058; zookeeper connection closed.
2018-07-11 17:50:57,765 INFO  [main-EventThread] zookeeper.ClientCnxn: EventThread shut down for session: 0x36488ab4beb0002
2018-07-11 17:50:57,765 ERROR [main] regionserver.HRegionServerCommandLine: Region server exiting
java.lang.RuntimeException: HRegionServer Aborted

        at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.start(HRegionServerCommandLine.java:67)
        at org.apache.hadoop.hbase.regionserver.HRegionServerCommandLine.run(HRegionServerCommandLine.java:87)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
        at org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
        at org.apache.hadoop.hbase.regionserver.HRegionServer.main(HRegionServer.java:2971)
2018-07-11 17:50:57,788 INFO  [Thread-4] regionserver.ShutdownHook: Shutdown hook starting; hbase.shutdown.hook=true; fsShutdownHook=org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer@1ba05e38
2018-07-11 17:50:57,788 INFO  [Thread-4] regionserver.ShutdownHook: Starting fs shutdown hook thread.
2018-07-11 17:50:57,792 INFO  [Thread-4] regionserver.ShutdownHook: Shutdown hook finished.
[root@slave1 logs]#

KKKK6dn

赞同来自:

解决了吗  

要回复问题请先登录注册