Zookeeper-3.5.7集群安装：https://blog.csdn.net/weixin_42631782/article/details/115691252
Hadoop-3.1.3高可用集群部署
https://blog.51cto.com/simplelife/5095289
Hive-3.1.2部署文档
https://blog.51cto.com/simplelife/5100277
一文讲懂Hive高可用/HiveServer2 高可用及Metastore高可用
https://juejin.cn/post/6981353941581692936
DolphinScheudler-2.0.集群部署文档
https://blog.51cto.com/simplelife/5049665
jdk-8u331-linux-x64.rpm JDK8安装包
mysql-connector-java-5.1.49.jar MySQL5连接器
apache-zookeeper-3.5.7-bin.tar.gz ZooKeeper安装包
hadoop-3.1.3.tar.gz Hadoop安装包
apache-hive-3.1.2-bin.tar.gz Hive安装包
apache-tez-0.10.1-bin.tar.gz Tez安装包
apache-dolphinscheduler-2.0.6-bin.tar.gz DolphinScheduler安装包
注意：除JDK8的子版本外，请严格按照给定的版本号安装。
Hive下载地址：https://archive.apache.org/dist/hive/hive-3.1.2/
Hadoop下载地址：https://archive.apache.org/dist/hadoop/common/hadoop-3.1.3/
Tez下载地址：https://archive.apache.org/dist/tez/0.10.1/
Zookeeper下载地址：https://archive.apache.org/dist/zookeeper/zookeeper-3.5.7/
Dolphinscheduler下载地址：https://www.apache.org/dyn/closer.lua/dolphinscheduler/2.0.6/apache-dolphinscheduler-2.0.6-bin.tar.gz
操作系统为CentOS7。
集群中所有三台服务器执行以下操作：
配置hosts
：
sudo vim /etc/hosts
192.168.62.161 hadoop1
192.168.62.162 hadoop2
192.168.62.163 hadoop3
安装JDK1.8：
#安装jdk
#解压jdk到指定目录
tar -zxvf /tmp/jdk-8u181-linux-x64.tar.gz -C /usr/share/
对应路径如下(配置环境变量要用到该路径)
/usr/share/jdk1.8.0_181
#然后配置jdk环境变量(见子节点)
#然后把jdk目录复制到其他节点
sh /home/scp.sh /usr/share/jdk1.8.0_181
sh /home/scp.sh /etc/profile
安装hadoop及其配置
配置防火墙和SELINUX：
#关闭防火墙并查看状态(集群服务器)
systemctl stop firewalld systemctl status firewalld
配置时间同步：
yum -y install ntp ntpdate ntpdate ntp.aliyun.com hwclock –systohc
检查时间同步：
watch -t -n 1 date +%T
新增hadoop用户：
useradd hadoop passwd Hadoop
授权root权限：
vim /etc/sudoers hadoop ALL=(ALL) ALL
切换用户：
su – hadoop
注意：之后的操作尽量都在hadoop用户下完成，以避免各种权限问题。如操作需要root权限则用sudo命令。
配置免密登录：
ssh-keygen -t rsa ssh-copy-id hadoop1 ssh-copy-id hadoop2 ssh-copy-id hadoop3
创建目录：
mkdir -p /home/hadoop/data/zookeeper mkdir -p /home/hadoop/data/journalnode mkdir -p /home/hadoop/data/tmp mkdir -p /home/hadoop/data/hive mkdir -p /home/hadoop/data/dolphinscheduler mkdir -p /home/hadoop/data/dolphinscheduler-install
#主节点上传文件分发脚本
资源如下
https://download.csdn.net/download/csdn_gyb/31849299
#主节点执行同步
解压安装包后修改配置文件：
cd conf mv zoo_sample.cfg zoo.cfg dataDir=/home/hadoop/data/zookeeper server.1=hadoop1:2888:3888 server.2=hadoop2:2888:3888 server.3=hadoop3:2888:3888
cd data
vim myid
1/2/3（每台服务器不相同）
bin/zkServer.sh
export JAVA_HOME=Java安装目录
集群分发：
scp -r hadoop-3.1.3 hadoop2:/home/hadoop/ scp -r hadoop-3.1.3 hadoop3:/home/hadoop/
使用zk.sh status/start/stop来启动和管理zk集群。
六、Hadoop高可用集群部署解压hadoop-3.1.3
sudo vim /etc/profile
配置环境变量HADOOP_HOME、PATH
#配置hadoop的环境变量
vi /etc/profile
文件末尾添加
export HADOOP_HOME=/home/hadoop-3.1.3 export PATH=P A T H : PATH:PATH:HADOOP_HOME/bin:$HADOOP_HOME/sbin source /etc/profile
Hadoop配置文件：
cd etc/hadoop
hadoop-env.sh
#指定JAVA_HOME
export JAVA_HOME= Java安装目录
#指定hadoop用户，hadoop3.x之后必须配置(我的用户名就叫hadoop)
export HDFS_NAMENODE_USER=hadoop export HDFS_DATANODE_USER=hadoop export HDFS_ZKFC_USER=hadoop export HDFS_JOURNALNODE_USER=hadoop export YARN_RESOURCEMANAGER_USER=hadoop export YARN_NODEMANAGER_USER=hadoop
core-site.xml
fs.defaultFS hdfs://ns1 hadoop.tmp.dir /home/hadoop/data/tmp hadoop.http.staticuser.user hadoop ha.zookeeper.quorum hadoop1:2181,hadoop2:2181,hadoop3:2181 hadoop.proxyuser.hadoop.hosts * hadoop.proxyuser.hadoop.groups *
hdfs-site.xml
dfs.nameservices ns1 dfs.ha.namenodes.ns1 nn1,nn2 dfs.namenode.rpc-address.ns1.nn1 hadoop1:8020 dfs.namenode.rpc-address.ns1.nn2 hadoop2:8020 dfs.namenode.http-address.ns1.nn1 hadoop1:9870 dfs.namenode.http-address.ns1.nn2 hadoop2:9870 dfs.namenode.shared.edits.dir qjournal://hadoop1:8485;hadoop2:8485;hadoop3:8485/ns1 dfs.client.failover.proxy.provider.ns1 org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider dfs.ha.fencing.methods sshfence shell(/bin/true) dfs.ha.automatic-failover.enabled true dfs.ha.fencing.ssh.private-key-files /home/hadoop/.ssh/id_rsa dfs.journalnode.edits.dir /home/hadoop/data/journalnode
yarn-site.xml
yarn.resourcemanager.ha.enabled true Enable RM high-availability yarn.resourcemanager.scheduler.class org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.FairScheduler yarn.resourcemanager.cluster-id yarn_cluster1 Name of the cluster yarn.resourcemanager.ha.rm-ids rm1,rm2 The list of RM nodes in the cluster when HA is enabled yarn.resourcemanager.hostname.rm1 hadoop1 The hostname of the rm1 yarn.resourcemanager.hostname.rm2 hadoop2 The hostname of the rm2 yarn.resourcemanager.webapp.address.rm1 hadoop1:8088 yarn.resourcemanager.webapp.address.rm2 hadoop2:8088 yarn.resourcemanager.zk-address hadoop1:2181,hadoop2:2181,hadoop3:2181 yarn.resourcemanager.store.class org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore yarn.resourcemanager.recovery.enabled true yarn.resourcemanager.ha.automatic-failover.enabled true yarn.resourcemanager.ha.automatic-failover.embedded true yarn.nodemanager.aux-services mapreduce_shuffle yarn.nodemanager.aux-services.mapreduce_shuffle.class org.apache.hadoop.mapred.ShuffleHandler yarn.nodemanager.env-whitelist JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_MAPRED_HOME yarn.nodemanager.vmem-check-enabled false Whether virtual memory limits will be enforced for containers yarn.nodemanager.vmem-pmem-ratio 6 Ratio between virtual memory to physical memory when setting memory limits for containers
mapred-site.xml
mapreduce.framework.name yarn
yarn-env.sh
export JAVA_HOME=/usr/share/jdk1.8.0_181
hadoop-env.sh
export JAVA_HOME=/usr/share/jdk1.8.0_181 export HADOOP_HOME=/home/hadoop-3.1.3
workers
#配置workers
vim /home/hadoop-3.1.3/etc/hadoop/workers
#加入集群主机名称
hadoop1 hadoop2 hadoop3
注意：workers配置集群中每台服务器都一样，都要写上所有的worker主机
集群分发：
scp -r hadoop-3.1.3 hadoop2:/home/hadoop/ scp -r hadoop-3.1.3 hadoop3:/home/hadoop/
启动集群
#启动journalnode，在hadoop1,hadoop2,hadoop3上分别执行
hdfs --daemon start journalnode
格式化namenode,在hadoop1或者hadoop2上执行,我这在hadoop1上执行了
hdfs namenode -format
#启动namenode,在hadoop1上执行
hdfs --daemon start namenode
#在hadoop2上同步namenode数据后启动，在hadoop2上执行
hdfs namenode -bootstrapStandby hdfs --daemon start namenode
在hadoop1上执行
hdfs zkfc -formatZK
#至此，hadoop的hdfs服务已经启动完成。yarn服务启动可执行命令：start-yarn.sh
重启hdfs
stop-dfs.sh start-dfs.sh
重启yarn
stop-yarn.sh start-yarn.sh
直接同时重启hdfs和yarn
stop-all.sh start-all.sh
Hadoop文件系统命令：
hadoop fs -命令 参数
如查看根目录文件：hadoop fs -ls /
创建文件夹：hadoop fs -mkdir -p /test1/test2/test3
修改权限：hadoop fs -chmod -R 777 /test1
Hadoop管理网页：
http://hadoop1:8088
Yarn管理网页：
http://hadoop1:9870
解压hive-3.1.2
tar -zxvf /tmp/apache-hive-3.1.2-bin.tar.gz -C /home/ sudo /etc/profile
配置环境变量HIVE_HOME、PATH
配置文件：
cd conf/ cp hive-env.sh.template hive-env.sh HADOOP_HOME=/home/hadoop/hadoop-3.1.3 export HIVE_HOME=/home/hadoop/hive-3.1.2 export HIVE_CONF_DIR=/home/hadoop/hive-3.1.2/conf export HIVE_AUX_JARS_PATH=/home/hadoop/hive-3.1.2/lib export METASTORE_PORT=9083 export HIVESERVER2_PORT=10000
cp hive-default.xml.template hive-site.xml
该配置文件非常大，建议在官方模板中搜索以下需要修改的项目单独修改。
hive-site.xml
配置所需的环境变量
system:java.io.tmpdir /tmp/hive/java # 修改metastore数据库为mysql system:user.name ${user.name} # metastore远程节点（Hadoop） hive.metastore.db.type mysql Expects one of [derby, oracle, mysql, mssql, postgres]. Type of database used by the metastore. Information schema & JDBCStorageHandler depend on it. # metastore数据库密码（MySQL） hive.metastore.uris thrift://hadoop1:9083,thrift://hadoop2:9083 Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore. # metastore数据库地址（MySQL） javax.jdo.option.ConnectionPassword 123456789 password to use against metastore database # metastore数据库驱动（MySQL） javax.jdo.option.ConnectionURL jdbc:mysql://192.168.100.1:3307/hive?useSSL=false JDBC connect string for a JDBC metastore. To use SSL to encrypt/authenticate the connection, provide database-specific SSL flag in the connection URL. For example, jdbc:postgresql://myhost/db?ssl=true for postgres database. # metastore数据库用户名（MySQL） javax.jdo.option.ConnectionDriverName com.mysql.jdbc.Driver Driver class name for a JDBC metastore # 优化填坑配置 javax.jdo.option.ConnectionUserName root Username to use against metastore database # Zookeeper集群配置 hive.optimize.sort.dynamic.partition true When enabled dynamic partitioning column will be globally sorted. This way we can keep only one record writer open for each partition value in the reducer thereby reducing the memory pressure on reducers. # 修正官方配置文件description中的BUG hive.zookeeper.quorum hadoop1:2181,hadoop2:2181,hadoop3:2181 List of ZooKeeper servers to talk to. This is needed for: 1. Read/write locks - when hive.lock.manager is set to org.apache.hadoop.hive.ql.lockmgr.zookeeper.ZooKeeperHiveLockManager, 2. When HiveServer2 supports service discovery via Zookeeper. 3. For delegation token storage if zookeeper store is used, if hive.cluster.delegation.token.store.zookeeper.connectString is not set 4. LLAP daemon registry service 5. Leader selection for privilege synchronizer hive.txn.xlock.iow true Ensures commands with OVERWRITE (such as INSERT OVERWRITE) acquire Exclusive locks for transactional tables. This ensures that inserts (w/o overwrite) running concurrently are not hidden by the INSERT OVERWRITE. # Zookeeper注册名称 hive.server2.support.dynamic.service.discovery true Whether HiveServer2 supports dynamic service discovery for its clients. To support this, each instance of HiveServer2 currently uses ZooKeeper to register itself, when it is brought up. JDBC/ODBC clients should use the ZooKeeper ensemble: hive.zookeeper.quorum in their connection string. # 高可用配置 hive.server2.zookeeper.namespace hiveserver2 The parent node in ZooKeeper used by HiveServer2 when supporting dynamic service discovery. hive.server2.thrift.bind.host hadoop1/hadoop2 Bind host on which to run the HiveServer2 Thrift service.
lib库文件：
cd lib/ mv guava-19.0.jar guava-19.0.jar.bak
从hadoop的share/hadoop/common/lib目录复制或链接guava-27.0-jre.jar
mv log4j-slf4j-impl-2.10.0.jar log4j-slf4j-impl-2.10.0.jar.bak
解压tez-0.10.1
复制或链接根目录下的tez-api-0.10.1.jar和tez-dag-0.10.1.jar
复制mysql-connector-java-5.1.49.jar
集群分发:
scp -r hive-3.1.2 hadoop2:/home/hadoop/
初始化数据库：
新建数据库 hive
cd bin/ ./schematool -initSchema -dbType mysql -verbose
启动脚本：
hv.sh
高可用JDBC连接地址：
jdbc:hive2://hadoop1:2181,hadoop2:2181,hadoop3:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
命令行连接Hive数据库并测试：
cd bin/ ./beeline !connect jbdc:hive2://… show databases; create database testdb; use testdb; create table tb_test (id int, name, string); insert into tb_test values (1, ‘zhangsan’); select * from t_test;
管理网页：
http://hadoop1:10002
http://hadoop2:10002
所有服务器上安装：
sudo yum install -y psmisc
配置文件：
vim conf/env/dolphinscheduler_env.sh
修改以下项目：
export HADOOP_HOME=/home/hadoop/hadoop-3.1.3/ export HADOOP_CONF_DIR=/home/hadoop/hadoop-3.1.3/etc/hadoop export JAVA_HOME=/usr/java/jdk1.8.0_331-amd64 export HIVE_HOME=/home/hadoop/hive-3.1.2
vim conf/config/install_config.conf
修改以下项目：
ips="hadoop1,hadoop2,hadoop3"
masters="hadoop1,hadoop2"
workers="hadoop1:default,hadoop2:default,hadoop3:default"
alertServer="hadoop3"
apiServers="hadoop1"
pythonGatewayServers="hadoop2"
installPath="/home/hadoop/data/dolphinscheduler-install"
deployUser="hadoop"
dataBasedirPath="/home/hadoop/data/dolphinscheduler"
javaHome="/usr/soft/installs/jdk1.8.0"
DATABASE_TYPE=${DATABASE_TYPE:-"mysql"}
SPRING_DATASOURCE_URL=${SPRING_DATASOURCE_URL:-"jdbc:mysql://192.168.0.47:3306/hive?useSSL=false"}
SPRING_DATASOURCE_USERNAME=${SPRING_DATASOURCE_USERNAME:-"root"}
SPRING_DATASOURCE_PASSWORD=${SPRING_DATASOURCE_PASSWORD:-"vqn6vGJ1#"}
registryPluginName="zookeeper"
registryServers="hadoop1:2181,hadoop2:2181,hadoop3:2181"
registryNamespace="dolphinscheduler"
resourceUploadPath="/dolphinscheduler"
defaultFS="hdfs://ns1:8020"
resourceManagerHttpAddressPort="8088"
yarnHaIps="hadoop1,hadoop2"
singleYarnIp="yarnIp1"
hdfsRootUser="hadoop"
复制或链接hadoop配置文件到conf目录：
cd conf/ ln -s /home/hadoop/hadoop-3.1.3/etc/hadoop/core-site.xml core-site.xml ln -s /home/hadoop/hadoop-3.1.3/etc/hadoop/hdfs-site.xml hdfs-site.xml
lib库文件：
cd lib/
复制mysql-connector-java-5.1.49.jar
集群分发:
scp -r dolphinscheduler-2.0.6 hadoop2:/home/hadoop/
scp -r dolphinscheduler-2.0.6 hadoop3:/home/hadoop/
初始化数据库：
新建数据库 dolphinscheduler
cd /home/hadoop/dolphinscheduler-2.0.6/script/ ./create-dolphinscheduler.sh
首次启动，安装根目录下：
./install.sh
之后启动/停止
cd bin/
./stop-all.sh
./start-all.sh
网页地址：
http://hadoop1:12345/dolphinscheduler
admin/dolphinscheduler123
jps-all.sh
需要将Java环境变量配置从/etc/profile拷贝到集群所有服务器的~/.bashrc下，如：
~/.bashrc #JAVA_HOME JAVA_HOME=/usr/soft/installs/jdk1.8.0 JRE_HOME=$JAVA_HOME/jre PATH=$PATH:$JAVA_HOME/bin:$JRE_HOME/bin CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar:$JRE_HOME/lib export JAVA_HOME JRE_HOME PATH CLASSPATH
成功部署启动集群后的示例：
[hadoop@hadoop1 ~]$ ./jps-all.sh
hadoop1所有服务
