Hadoop2.6.0|Hbase0.98.13的Mapreduce开发环境搭建

在[button href=http://www.bcmeng.com/hadoop-zookeeper-hbase/ color=red]阿里ECSHadoop2.6.0|Zookeeper3.4.6|Hbase0.98.13集群搭建[/button]一文中我们已经搭建了Hadoop和Hbase集群,但是还是无法运行Hbase Mapreduce 任务。我们运行Mapreduce 任务时会遇到许多java.lang.ClassNotFoundException。这个问题是因为Hadoop classpath中没有hbase的相关jar包,所以我们可以将hbase的相关jar包导入Hadoop的classpath中。同时为了避免以后一些不确定错误的发生,我们应该将hbase的lib的Hadoop的2.2.0版本的jar包替换为Hadoop的2.6.0的jar包。所以,为了搭建好完整健全的hbase的Mapreduce 开发环境,我们需要以下3步:

1:用Hadoop2.6.0的jar包替换掉Hbase的lib下Hadoop相关j ar包

cd /home/hbase/lib
vim f.sh
find -name "hadoop*jar" | sed 's/2.2.0/2.6.0/g' | sed 's/\.\///g' > f.log
rm ./hadoop*jar
cat ./f.log | while read Line
do
find /home/hadoop/share/hadoop -name "$Line" | xargs -i cp {} ./
done
rm ./f.log
chmod u+x f.sh
./f.sh
rm -f f.sh

cp /home/hadoop/share/hadoop/common/lib/htrace-core-3.0.4.jar /home/hbase/lib
替换完成后,Hbase的lib目录如下:

activation-1.1.jar
aopalliance-1.0.jar
asm-3.1.jar
avro-1.7.4.jar
commons-beanutils-1.7.0.jar
commons-beanutils-core-1.8.0.jar
commons-cli-1.2.jar
commons-codec-1.7.jar
commons-collections-3.2.1.jar
commons-compress-1.4.1.jar
commons-configuration-1.6.jar
commons-daemon-1.0.13.jar
commons-digester-1.8.jar
commons-el-1.0.jar
commons-httpclient-3.1.jar
commons-io-2.4.jar
commons-lang-2.6.jar
commons-logging-1.1.1.jar
commons-math-2.1.jar
commons-net-3.1.jar
findbugs-annotations-1.3.9-1.jar
gmbal-api-only-3.0.0-b023.jar
grizzly-framework-2.1.2.jar
grizzly-http-2.1.2.jar
grizzly-http-server-2.1.2.jar
grizzly-http-servlet-2.1.2.jar
grizzly-rcm-2.1.2.jar
guava-12.0.1.jar
guice-3.0.jar
guice-servlet-3.0.jar
hadoop-annotations-2.6.0.jar
hadoop-auth-2.6.0.jar
hadoop-common-2.6.0.jar
hadoop-hdfs-2.6.0.jar
hadoop-mapreduce-client-app-2.6.0.jar
hadoop-mapreduce-client-common-2.6.0.jar
hadoop-mapreduce-client-core-2.6.0.jar
hadoop-mapreduce-client-jobclient-2.6.0.jar
hadoop-mapreduce-client-shuffle-2.6.0.jar
hadoop-yarn-api-2.6.0.jar
hadoop-yarn-client-2.6.0.jar
hadoop-yarn-common-2.6.0.jar
hadoop-yarn-server-common-2.6.0.jar
hadoop-yarn-server-nodemanager-2.6.0.jar
hamcrest-core-1.3.jar
hbase-annotations-0.98.13-hadoop2.jar
hbase-checkstyle-0.98.13-hadoop2.jar
hbase-client-0.98.13-hadoop2.jar
hbase-common-0.98.13-hadoop2.jar
hbase-common-0.98.13-hadoop2-tests.jar
hbase-examples-0.98.13-hadoop2.jar
hbase-hadoop2-compat-0.98.13-hadoop2.jar
hbase-hadoop-compat-0.98.13-hadoop2.jar
hbase-it-0.98.13-hadoop2.jar
hbase-it-0.98.13-hadoop2-tests.jar
hbase-prefix-tree-0.98.13-hadoop2.jar
hbase-protocol-0.98.13-hadoop2.jar
hbase-rest-0.98.13-hadoop2.jar
hbase-server-0.98.13-hadoop2.jar
hbase-server-0.98.13-hadoop2-tests.jar
hbase-shell-0.98.13-hadoop2.jar
hbase-testing-util-0.98.13-hadoop2.jar
hbase-thrift-0.98.13-hadoop2.jar
high-scale-lib-1.1.1.jar
htrace-core-2.04.jar
htrace-core-3.0.4.jar
httpclient-4.1.3.jar
httpcore-4.1.3.jar
jackson-core-asl-1.8.8.jar
jackson-jaxrs-1.8.8.jar
jackson-mapper-asl-1.8.8.jar
jackson-xc-1.8.8.jar
jamon-runtime-2.3.1.jar
jasper-compiler-5.5.23.jar
jasper-runtime-5.5.23.jar
javax.inject-1.jar
javax.servlet-3.1.jar
javax.servlet-api-3.0.1.jar
jaxb-api-2.2.2.jar
jaxb-impl-2.2.3-1.jar
jcodings-1.0.8.jar
jersey-client-1.8.jar
jersey-core-1.8.jar
jersey-grizzly2-1.9.jar
jersey-guice-1.9.jar
jersey-json-1.8.jar
jersey-server-1.8.jar
jersey-test-framework-core-1.9.jar
jersey-test-framework-grizzly2-1.9.jar
jets3t-0.6.1.jar
jettison-1.3.1.jar
jetty-6.1.26.jar
jetty-sslengine-6.1.26.jar
jetty-util-6.1.26.jar
joni-2.1.2.jar
jruby-complete-1.6.8.jar
jsch-0.1.42.jar
jsp-2.1-6.1.14.jar
jsp-api-2.1-6.1.14.jar
jsr305-1.3.9.jar
junit-4.11.jar
libthrift-0.9.0.jar
log4j-1.2.17.jar
management-api-3.0.0-b012.jar
metrics-core-2.2.0.jar
netty-3.6.6.Final.jar
paranamer-2.3.jar
protobuf-java-2.5.0.jar
ruby
servlet-api-2.5-6.1.14.jar
slf4j-api-1.6.4.jar
snappy-java-1.0.4.1.jar
xmlenc-0.52.jar
xz-1.0.jar
zookeeper-3.4.6.jar
2:将Hbase的相关jar包导入Hadoop的classpath:

cp /home/hbase/lib/hbase-*.jar /home/hadoop/share/hadoop/common

cp /home/hbase/lib/htrace-*.jar /home/hadoop/share/hadoop/common

cp /home/hbase/lib/high-scale-lib-1.1.1.jar /home/hadoop/share/hadoop/common

cp /home/hbase/lib/metrics-core-2.2.0.jar /home/hadoop/share/hadoop/common

cp /home/hbase/conf/hbase-site.xml /home/hadoop/etc/hadoop
注:以上包均为必须jar包。如果没有high-scale-lib-1.1.1.jar和metrics-core-2.2.0.jarHadoop只可以将mapreduce导入hbase,但是无法使用TableMapReduceUtil将hbase导入mapreduce.

完成后/home/hadoop/share/hadoop/common如下:

hadoop-common-2.6.0.jar hbase-rest-0.98.13-hadoop2.jar
hadoop-common-2.6.0-tests.jar hbase-server-0.98.13-hadoop2.jar
hadoop-nfs-2.6.0.jar hbase-server-0.98.13-hadoop2-tests.jar
hbase-annotations-0.98.13-hadoop2.jar hbase-shell-0.98.13-hadoop2.jar
hbase-checkstyle-0.98.13-hadoop2.jar hbase-testing-util-0.98.13-hadoop2.jar
hbase-client-0.98.13-hadoop2.jar hbase-thrift-0.98.13-hadoop2.jar
hbase-common-0.98.13-hadoop2.jar high-scale-lib-1.1.1.jar
hbase-common-0.98.13-hadoop2-tests.jar htrace-core-2.04.jar
hbase-examples-0.98.13-hadoop2.jar htrace-core-3.0.4.jar
hbase-hadoop2-compat-0.98.13-hadoop2.jar jdiff
hbase-hadoop-compat-0.98.13-hadoop2.jar lib
hbase-it-0.98.13-hadoop2.jar metrics-core-2.2.0.jar
hbase-it-0.98.13-hadoop2-tests.jar sources
hbase-prefix-tree-0.98.13-hadoop2.jar templates
hbase-protocol-0.98.13-hadoop2.jar
然后重启启动hadoop 和hbase集群即可。

3:配置eclipse的Hbase的mapreduce的 maven开发环境:

图形界面安装:
安装
yum groupinstall "Desktop"
yum install xrdp
yum install tigervnc-server
service xrdp start
service xrdp restart

eclipse安装:

wget http://www.eclipse.org/downloa ... D1071

桌面上右键创建启动器
浏览eclipse路径
点击图标 进入到eclipse根目录下
即可将eclipse添加到桌面
远程登录kks1,在eclipse建立maven项目。配置pom.xml如下:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/ma ... gt%3B
<modelVersion>4.0.0</modelVersion>
<groupId>com.kks</groupId>
<artifactId>hbase</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>hbase</name>
<description>hbase test</description>
<dependencies>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-client</artifactId>
<version>0.98.13-hadoop2</version>
</dependency>
<dependency>
<groupId>org.apache.hbase</groupId>
<artifactId>hbase-server</artifactId>
<version>0.98.13-hadoop2</version>
</dependency>
</dependencies>
</project>
就可以正常开发mapreduce程序。编写完成后将到处jar包。按如下方式运行即可:

hadoop jar job.jar hbase.HbaseScanRow 后面可以添加参数
具体如何将利用mapreduce将hbase导入导入,请看下文:

[button href=http://www.bcmeng.com/hbasemrdemo color=red]Mapreduce导入导出Hbase0.98代码示例[/button]

0 个评论

要回复文章请先登录注册