sqoop 1.99的安装配置

摘要:
1. sqoop 1.99的安装配置
2. client使用
3. 从HBase,Hive导数据到mysql
版本
sqoop-1.99.2-bin-hadoop100
Sqoop是Hadoop系统数据和RDBMS互导数据的工具。1.99新版的一个包里包含两个部分:客户端和服务器端。必须在集群中安装好一个节点的服务器端。所有客户端和该服务器端相连。服务器端作为mapreduce的client,所以Hadoop必须和Sqoop服务器端装在一起。客户端则不限。这种设计让导数据更灵活。原1.44版则没有区分服务器端和客户端。
服务器端安装
1.确认本机有hadoop
hadoop fs -ls
由于hadoop的主版本1.xx和2.xx不兼容,所以sqoop的二进制版本也是分100和200的。如我在hadoop 1.1.2中使用sqoop-1.99.2-bin-hadoop100来进行配套。
解压
tar zxvf sqoop-1.99.2-bin-hadoop100.tar.gz
cd sqoop-1.99.2-bin-hadoop100
2. 安装依赖库和组件
./bin/addtowar.sh -hadoop-auto
[hadoop@hs11 sqoop-1.99.2-bin-hadoop100]$ ./bin/addtowar.sh -hadoop-auto
Non of expected directories with Hadoop exists
Usage : addtowar.sh 
Options: -hadoop-auto Try to guess hadoop version and path
-hadoop-version HADOOP_VERSION Specify used version
-hadoop-path HADOOP_PATHS Where to find hadoop jars (multiple paths with Hadoop jars separated by ‘:’)
-jars JARS_PATH Special jars that should be added (multiple JAR paths separated by ‘:’)
-war SQOOP_WAR Target Sqoop war file where all jars should be ingested
由于我的hadoop没有安装到系统路径,所以需修改配置文件。
[hadoop@hs11 sqoop-1.99.2-bin-hadoop100]$ vi ./bin/addtowar.sh
hadoopPossiblePaths=”/home/hadoop/hadoop-1.1.2 /usr/lib/hadoop /usr/lib/hadoop-mapreduce/ /usr/lib/hadoop-yarn/ /usr/lib/hadoop-hdfs”
[hadoop@hs11 sqoop-1.99.2-bin-hadoop100]$ ./bin/addtowar.sh -hadoop-auto
./bin/addtowar.sh: line 126: [: missing `]‘
Hadoop version: 1.1.2
Hadoop path: /home/hadoop/hadoop-1.1.2
Extra jars:
Injecting following Hadoop JARs
/home/hadoop/hadoop-1.1.2/hadoop-core-1.1.2.jar
/home/hadoop/hadoop-1.1.2/lib/jackson-core-asl-1.8.8.jar
/home/hadoop/hadoop-1.1.2/lib/jackson-mapper-asl-1.8.8.jar
/home/hadoop/hadoop-1.1.2/lib/commons-configuration-1.6.jar
/home/hadoop/hadoop-1.1.2/lib/commons-logging-api-1.0.4.jar
/home/hadoop/hadoop-1.1.2/lib/slf4j-api-1.4.3.jar
/home/hadoop/hadoop-1.1.2/lib/slf4j-log4j12-1.4.3.jar
Backing up original WAR file to ./bin/../server/webapps/sqoop.war_2013-08-20_09:36:01.263437795
New Sqoop WAR file with added ‘Hadoop JARs’ at ./bin/../server/webapps/sqoop.war
脚本126行]有个空格问题,但不影响结果。改后:
[hadoop@hs11 sqoop-1.99.2-bin-hadoop100]$ vi ./bin/addtowar.sh
[hadoop@hs11 sqoop-1.99.2-bin-hadoop100]$ ./bin/addtowar.sh -hadoop-auto
Hadoop version: 1.1.2
Hadoop path: /home/hadoop/hadoop-1.1.2
Extra jars:
Specified Sqoop WAR ‘./bin/../server/webapps/sqoop.war’ already contains Hadoop JAR files
也可以指定,-hadoop-version指定版本,-hadoop-path 指定目录。如果各部分安装在多个目录,则用:分隔。
如:
hadoop 1.0
./bin/addtowar.sh -hadoop-version 1.0 -hadoop-path /usr/lib/hadoop-common:/usr/lib/hadoop-hdfs:/usr/lib/hadoop-mpred
hadoop 2.0
./bin/addtowar.sh -hadoop-version 2.0 -hadoop-path /usr/lib/hadoop-common:/usr/lib/hadoop-hdfs:/usr/lib/hadoop-yarn
也可以用addtowar.sh的-jars参数来绑定其他的jar文件。
绑定jdbc mysql库:
由于不同协议,sqoop没有自带mysql jdbc库,需下载,并绑定:
http://dev.mysql.com/downloads/mirror.php?id=13597
wget http://cdn.mysql.com/" rel="nofollow" target="_blank">http://dev.mysql.com/get/Downloa ... tp://cdn.mysql.com/
注意下载的是tar.gz, 里面有源码和jar包。只需解压后使用jar包即可。
[hadoop@hs11 sqoop-1.99.2-bin-hadoop100]$ ./bin/addtowar.sh -jars /home/hadoop/hive-0.11.0/lib/mysql-connector-java-5.0.8-bin.jar
Hadoop version:
Hadoop path:
Extra jars: /home/hadoop/hive-0.11.0/lib/mysql-connector-java-5.0.8-bin.jar
Injecting following additional JARs
/home/hadoop/hive-0.11.0/lib/mysql-connector-java-5.0.8-bin.jar
Backing up original WAR file to ./bin/../server/webapps/sqoop.war_2013-08-20_09:49:32.401896012
New Sqoop WAR file with added ‘JARs’ at ./bin/../server/webapps/sqoop.war
3.配置sqoop服务器
server/conf 里面存放服务器的配置文件。包括tomcat等配置。但缺省配置PropertiesConfigurationProvider足够用。如需修改,编辑sqoop_bootstrap.properties的sqoop.config.provider即可。
[hadoop@hs11 sqoop-1.99.2-bin-hadoop100]$ cd server/conf
[hadoop@hs11 conf]$ ls
catalina.policy catalina.properties context.xml logging.properties server.xml sqoop_bootstrap.properties sqoop.properties tomcat-users.xml web.xml
[hadoop@hs11 conf]$ cat sqoop_bootstrap.properties
sqoop.config.provider=org.apache.sqoop.core.PropertiesConfigurationProvider
sqoop.properties包含其他的一些修改。可能需要微调,以适应环境需要。
#org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/etc/hadoop/conf/
org.apache.sqoop.submission.engine.mapreduce.configuration.directory=/home/hadoop/hadoop-1.1.2/conf/
4. 启动和停止服务器
./bin/sqoop.sh server start
[hadoop@hs11 sqoop-1.99.2-bin-hadoop100]$ ./bin/sqoop.sh server start
Sqoop home directory: /home/hadoop/sqoop-1.99.2-bin-hadoop100…
Using CATALINA_BASE: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server
Using CATALINA_HOME: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server
Using CATALINA_TMPDIR: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server/temp
Using JRE_HOME: /usr/java/jdk1.6.0_45
Using CLASSPATH: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server/bin/bootstrap.jar
[hadoop@hs11 sqoop-1.99.2-bin-hadoop100]$ ./bin/sqoop.sh server stop
Sqoop home directory: /home/hadoop/sqoop-1.99.2-bin-hadoop100…
Using CATALINA_BASE: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server
Using CATALINA_HOME: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server
Using CATALINA_TMPDIR: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server/temp
Using JRE_HOME: /usr/java/jdk1.6.0_45
Using CLASSPATH: /home/hadoop/sqoop-1.99.2-bin-hadoop100/server/bin/bootstrap.jar
5.客户端安装
客户端无需配置,只需将下载版本解压即可。
[hadoop@hs11 sqoop-1.99.2-bin-hadoop100]$ ./bin/sqoop.sh client
Sqoop home directory: /home/hadoop/sqoop-1.99.2-bin-hadoop100…
Aug 20, 2013 10:07:54 AM java.util.prefs.FileSystemPreferences$2 run
INFO: Created user preferences directory.
Sqoop Shell: Type ‘help’ or ‘\h’ for help.
sqoop:000>
或执行sqoop脚本:
sqoop.sh client /path/to/your/script.sqoop

0 个评论

要回复文章请先登录注册