sqoop安装、配置、报错

安装配置：下载、解压到/export/servers下：

[root@gao2 /]# cd /export/soft/
[root@gao2 soft]# ls
apache-flume-1.6.0-bin.tar.gz  apache-hive-1.1.0-bin.tar.gz  mysql-connector-java-5.1.49.tar.gz  sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz
[root@gao2 soft]# tar -zxvf sqoop-1.4.7.bin__hadoop-2.6.0.tar.gz -C /export/servers/
[root@gao2 servers]# ls
flume  hadoop-2.6.0  hbase  hive  jdk  mysql  scala-2.13.3  servers  spark  sqoop-1.4.7.bin__hadoop-2.6.0  zookeeper
[root@gao2 servers]# mv sqoop-1.4.7.bin__hadoop-2.6.0 sqoop
[root@gao2 servers]# ls
hadoop-2.6.0  hbase  hive  jdk  mysql  scala-2.13.3 spark  sqoop  zookeeper

连接mysql（将mysql的连接驱动jar包复制到sqoop文件的lib文件下）：

[root@gao2 mysql-connector-java-5.1.49]# cd /export/servers/mysql/mysql-connector-java-5.1.49/
[root@gao2 mysql-connector-java-5.1.49]# ls
build.xml  CHANGES  COPYING  mysql-connector-java-5.1.49-bin.jar  mysql-connector-java-5.1.49.jar  README  README.txt  src
[root@gao2 mysql-connector-java-5.1.49]# cp mysql-connector-java-5.1.49-bin.jar /export/servers/sqoop/lib/
[root@gao2 mysql-connector-java-5.1.49]# cd /export/servers/sqoop/lib/
[root@gao2 lib]# find mysql*
mysql-connector-java-5.1.49-bin.jar

配置sqoop环境变量：

[root@gao2 sqoop]# cd /export/servers/sqoop/conf/
[root@gao2 conf]# ls
oraoop-site-template.xml  sqoop-env-template.cmd  sqoop-env-template.sh  sqoop-site-template.xml  sqoop-site.xml
[root@gao2 conf]# cp sqoop-env-template.sh sqoop-env.sh
[root@gao2 conf]# vim sqoop-env.sh

需修改配置文件如下：

#Set path to where bin/hadoop is available
export HADOOP_COMMON_HOME=/export/servers/hadoop-2.6.0

#Set path to where hadoop-*-core.jar is available
export HADOOP_MAPRED_HOME=/export/servers/hadoop-2.6.0

#set the path to where bin/hbase is available
export Hbase_HOME=/export/servers/hbase

#Set the path to where bin/hive is available
export HIVE_HOME=/export/servers/hive

#Set the path for where zookeper config dir is
export ZOOCFGDIR=/export/servers/zookeeper

[root@gao1 sqoop]# vim /etc/profile

#sqoop
export SQOOP_HOME=/export/servers/sqoop
export PATH=$PATH:$SQOOP_HOME/bin

测试：测试环境变量：

[root@gao1 sqoop]# bin/sqoop help
21/11/19 00:43:18 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
usage: sqoop COMMAND [ARGS]

Available commands:
  codegen            Generate code to interact with database records
  create-hive-table  import a table definition into Hive
  eval               evaluate a SQL statement and display the results
  export             Export an HDFS directory to a database table
  help               List available commands
  import             import a table from a database to HDFS
  import-all-tables  import tables from a database to HDFS
  import-mainframe   import datasets from a mainframe server to HDFS
  job                Work with saved jobs
  list-databases     List available databases on a server
  list-tables        List available tables in a database
  merge              Merge results of incremental imports
  metastore          Run a standalone Sqoop metastore
  version            Display version information

See 'sqoop help COMMAND' for information on a specific command.
[root@gao1 sqoop]#

测试sqoop与mysql的连接：

列出MySQL的所有数据库

[root@gao1 sqoop]# bin/sqoop eval --driver com.mysql.jdbc.Driver --connect jdbc:mysql://192.168.173.141:3306?serverTimezone=UTC&useSSL=false --username root --password 123456 --query "show databases"
21/11/19 00:40:57 INFO sqoop.Sqoop: Running Sqoop version: 1.4.7
21/11/19 00:40:57 WARN tool.baseSqoopTool: Setting your password on the command-line is insecure. Consider using -P instead.
21/11/19 00:40:57 WARN sqoop.ConnFactory: Parameter --driver is set to an explicit driver however appropriate connection manager is not being set (via --connection-manager). Sqoop is going to fall back to org.apache.sqoop.manager.GenericJdbcManager. Please specify explicitly which connection manager should be used next time.
21/11/19 00:40:57 INFO manager.SqlManager: Using default fetchSize of 1000
------------------------
| Database             | 
------------------------
| information_schema   | 
| hive                 | 
| mysql                | 
| performance_schema   | 
| sys                  |

--driver 表示数据库驱动器（注意需同mysql的配置文件一样）

--connect 表示连接对象（也就是数据库或表）的URL

--username 后面是用户名（这里是root）

--password 后面直接跟的是用户的密码（这里是123456）

--query 后面是sql语句，用双引号引用

报错： 1.在测试环境变量的时候报错：hcatalog does not exist!...accumulo does not exist

Warning: /opt/module/sqoop/bin/../../hcatalog does not exist! HCatalog jobs will fail.
Please set $HCAT_HOME to the root of your HCatalog installation.
Warning: /opt/module/sqoop/bin/../../accumulo does not exist! Accumulo imports will fail.
Please set $ACCUMULO_HOME to the root of your Accumulo installation.

解决：

https://blog.csdn.net/wx1528159409/article/details/87948866https://blog.csdn.net/wx1528159409/article/details/87948866https://blog.csdn.net/wx1528159409/article/details/87948866

sqoop安装、配置、报错

大数据系统相关栏目本月热门文章