Sqoop环境搭建 · Hadoop2.x

**一、前提条件** 安装Sqoop的前提是已经具备Java和Hadoop、Zookeeper、MySQL、Hive、Hbase的环境。其它环境可以不启动，但需要启动MySQL。 **二、下载并解压** 1. 上传安装包sqoop-1.4.6-cdh5.14.2.tar.gz到虚拟机中 2. 解压sqoop安装包到指定目录 ```sql [root@hadoop101 software]# tar -zxf sqoop-1.4.6-cdh5.14.2.tar.gz -C /opt/install/ ``` 3. 创建软连接 ```sql [root@hadoop101 software]# ln -s /opt/install/sqoop-1.4.6-cdh5.14.2/ /opt/install/sqoop ``` 4. 配置环境变量 ```sql [root@hadoop101 install]# vim /etc/profile -- 添加如下内容： export SQOOP_HOME=/opt/install/sqoop export PATH=$SQOOP_HOME/bin:$PATH -- 让配置文件生效。 [root@hadoop101 install]# source /etc/profile ``` **三、修改配置文件** Sqoop的配置文件与大多数大数据框架类似，在sqoop根目录下的conf目录中。 1. 重命名配置文件 ```sql [root@hadoop101 conf]# mv sqoop-env-template.sh sqoop-env.sh ``` 2. 在{sqoop_home}/conf/sqoop-env.sh中指定hadoop、hive、zookeeper、hbase环境 ```sql [root@hadoop101 conf]# vim sqoop-env.sh -- 添加下面内容 export HADOOP_COMMON_HOME=/opt/install/hadoop export HADOOP_MAPRED_HOME=/opt/install/hadoop export HIVE_HOME=/opt/install/hive export ZOOKEEPER_HOME=/opt/install/zookeeper export ZOOCFGDIR=/opt/install/zookeeper/conf export HBASE_HOME=/opt/install/hbase ``` **四、拷贝JDBC驱动** 拷贝jdbc驱动到sqoop的lib目录下，驱动包可以到Maven仓库下载：https://mvnrepository.com/artifact/mysql/mysql-connector-java ```sql [root@hadoop101 software]# cp mysql-connector-java-5.1.27.jar /opt/install/sqoop/lib/ ``` **五、验证Sqoop** ```sql [root@hadoop101 lib]# sqoop help Warning: /opt/install/sqoop/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /opt/install/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 20/12/22 23:33:14 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.14.2 usage: sqoop COMMAND [ARGS] Available commands: codegen Generate code to interact with database records create-hive-table Import a table definition into Hive eval Evaluate a SQL statement and display the results export Export an HDFS directory to a database table help List available commands import Import a table from a database to HDFS import-all-tables Import tables from a database to HDFS import-mainframe Import datasets from a mainframe server to HDFS job Work with saved jobs list-databases List available databases on a server list-tables List available tables in a database merge Merge results of incremental imports metastore Run a standalone Sqoop metastore version Display version information See 'sqoop help COMMAND' for information on a specific command. ``` 出现上面的信息表示安装成功。 **六、测试Sqoop是否能够成功连接数据库** ```sql [root@hadoop101 lib]# sqoop list-databases --connect jdbc:mysql://hadoop101:3306/ --username root --password 123456 Warning: /opt/install/sqoop/../hcatalog does not exist! HCatalog jobs will fail. Please set $HCAT_HOME to the root of your HCatalog installation. Warning: /opt/install/sqoop/../accumulo does not exist! Accumulo imports will fail. Please set $ACCUMULO_HOME to the root of your Accumulo installation. 20/12/22 23:35:16 INFO sqoop.Sqoop: Running Sqoop version: 1.4.6-cdh5.14.2 20/12/22 23:35:16 WARN tool.BaseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead. 20/12/22 23:35:16 INFO manager.MySQLManager: Preparing to use a MySQL streaming resultset. SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/install/hadoop-2.6.0-cdh5.14.2/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/install/hbase-1.2.0-cdh5.14.2/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] information_schema hive mysql performance_schema test ``` 出现数据库信息表示连接成功。 至此，sqoop的安装就完成了！ 下面还需要配置三个jar包，在进行数据迁移时才不会报错。 ```sql （1）从网上下载java-json.jar，上传到{sqoop_home}/lib目录下 [root@hadoop101 software]# cp java-json.jar /opt/install/sqoop/lib/ （2）复制{hive_home}/lib/hive-common-*.jar到{sqoop_home}/lib目录下 [root@hadoop101 lib]# cp hive-common-1.1.0-cdh5.14.2.jar /opt/install/sqoop/lib/ （3）复制{hive_home}/lib/hive-exec-*.jar到{sqoop_home}/lib目录下 [root@hadoop101 lib]# cp hive-exec-1.1.0-cdh5.14.2.jar /opt/install/sqoop/lib/ ```