Hbase regionserver 报错然后造成 region下线,最终所有region都下线后,这个regionserver就挂掉了

``` org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.server.namenode.LeaseExpiredException): No lease on /hbase/WALs/xhn12,60020,1483801123833-splitting/xhn12%2C60020% 2C1483801123833.1483850465810 (inode 53042): File is not open for writing. [Lease. Holder: DFSClient_hb_rs_xhn12,60020,1483801123833_-808411320_33, pendingcreates: 1] 2017-01-08 13:06:23,534 WARN [regionserver60020-WAL.AsyncSyncer0] hdfs.DFSClient: Error while syncing at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:496) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:945) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1194) at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.addDatanode2ExistingPipeline(DFSOutputStream.java:1040) at com.sun.proxy.$Proxy16.getAdditionalDatanode(Unknown Source) at org.apache.hadoop.hbase.fs.HFileSystem$1.invoke(HFileSystem.java:294) at java.lang.reflect.Method.invoke(Method.java:606) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at com.sun.proxy.$Proxy15.getAdditionalDatanode(Unknown Source) at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187) at java.lang.reflect.Method.invoke(Method.java:606) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolTranslatorPB.java:416) at com.sun.proxy.$Proxy14.getAdditionalDatanode(Unknown Source) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206) at org.apache.hadoop.ipc.Client.call(Client.java:1364) at org.apache.hadoop.ipc.Client.call(Client.java:1411) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2007) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614) at javax.security.auth.Subject.doAs(Subject.java:415) at java.security.AccessController.doPrivileged(Native Method) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2009) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2013) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:587) at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java) at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getAdditionalDatanode(ClientNamenodeProtocolServerSideTranslatorPB.java:499) at org.apache.hadoop.hdfs.server.namenode.AuthorizationProviderProxyClientProtocol.getAdditionalDatanode(AuthorizationProviderProxyClientProtocol.java:204) at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getAdditionalDatanode(NameNodeRpcServer.java:647) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalDatanode(FSNamesystem.java:3237) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:3334) ```

wangxiaolei

赞同来自:

提供下更多的regionserver的日志信息?

Hagrid

赞同来自:

2017-01-10 13:10:55,983 DEBUG [MemStoreFlusher.0] regionserver.CompactSplitThread: Small Compaction requested: system; Because: MemStoreFlusher.0; compaction_queue=(0:1), split_queue=0, merge_queue=0 2017-01-10 13:10:55,983 DEBUG [regionserver60020-smallCompactions-1483945539168] compactions.RatioBasedCompactionPolicy: Selecting compaction from 3 store files, 0 compacting, 3 eligible, 10 blocking 2017-01-10 13:11:55,715 WARN  [JvmPauseMonitor] util.JvmPauseMonitor: Detected pause in JVM or host machine (eg GC): pause of approximately 58923ms GC pool 'ParNew' had collection(s): count=1 time=57632ms GC pool 'ConcurrentMarkSweep' had collection(s): count=1 time=1379ms java.io.EOFException: Premature EOF: no length prefix available         at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:2103)         at org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)         at org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:810) 2017-01-10 13:11:55,715 WARN  [regionserver60020] util.Sleeper: We slept 60077ms instead of 3000ms, this is likely due t