Hadoop分布式集群搭建（一）

1.集群部署准备 1.1 集群组件版本

1.2 集群部署规划

使用三台机器搭建高可用分布式集群

1.3 路径规划

/opt/module # 该目录为组件安装目录
/opt/module/software # 该目录为组件安装包上传和管理目录

1.4 上传安装包

使用root账号登录3台服务器节点，并创建目录

[root@hadoop001 ~]# mkdir -p  /opt/module/software

[root@hadoop002 ~]# mkdir -p  /opt/module/software

[root@hadoop003 ~]# mkdir -p  /opt/module/software

上传hive、hadoop和jdk安装包到hadoop001 software目录下

2.安装JDK

hadoop是用Java语言编写，因此我们需要Java运行环境，安装JDK

2.1 解压到指定目录

[root@hadoop001 software]# tar -zxvf jdk-8u231-linux-x64.tar.gz  -C  ../

2.2 配置环境变量

[root@hadoop001 software]# cd ../jdk1.8.0_231/

#配置环境变量，编辑环境变量配置文件
vim  /etc/profile


#加入如下内容
export JAVA_HOME=/opt/module/jdk1.8.0_231
export PATH=.:$PATH:$JAVA_HOME/bin

ESC退出编辑模式，Shift+zz 保存退出


#使更改生效              
[root@hadoop001 ~]# source /etc/profile


#检查JDK环境变量是否配置成功
[root@hadoop001 ~]# java -version

2.3 安装包分发并配置

将jdk安装目录拷贝到其他两个节点，并配置环境变量

[root@hadoop001 module]# scp -r jdk1.8.0_231/ hadoop002:`pwd` 
[root@hadoop001 module]# scp -r jdk1.8.0_231/ hadoop003:`pwd`

#配置环境变量，编辑环境变量配置文件
vim  /etc/profile


#加入如下内容
export JAVA_HOME=/opt/module/jdk1.8.0_231
export PATH=.:$PATH:$JAVA_HOME/bin

ESC退出编辑模式，Shift+zz 保存退出


#使更改生效              
source /etc/profile


#检查JDK环境变量是否配置成功
java -version

3.Zookeeper集群搭建 3.1 下载Zookeeper压缩包

下载地址：https://archive.apache.org/dist/zookeeper/
访问网址，搜索3.4.9 版本，点击进入，找到zookeeper-3.4.9.tar.gz,复制链接

[root@hadoop001 software]# cd /opt/module/software/
[root@hadoop001 software]# wget https://archive.apache.org/dist/zookeeper/zookeeper-3.4.9/zookeeper-3.4.9.tar.gz

3.2 解压安装包

[root@hadoop001 software]# tar -zxvf zookeeper-3.4.9.tar.gz  -C ../

3.3 创建zkdata目录

[root@hadoop001 software]# cd ../zookeeper-3.4.9/
[root@hadoop001 zookeeper-3.4.9]# mkdir zkdata


#修改配置文件

[root@hadoop001 zookeeper-3.4.9]# cd conf/
[root@hadoop001 conf]# cp zoo_sample.cfg zoo.cfg

#修改zoo.cfg配置文件
[root@hadoop001 conf]# vim zoo.cfg


#修改dataDir路径
dataDir=/opt/module/zookeeper-3.4.9/zkdata


#添加集群配置
server.1=hadoop001:2888:3888
server.2=hadoop002:2888:3888
server.3=hadoop003:2888:3888

按ESC键退出编辑模式，shift+zz保存退出

3.4 添加myid配置

[root@hadoop001 conf]# cd /opt/module/zookeeper-3.4.9/zkdata/ 
[root@hadoop001 zkdata]# echo 1 >  myid
[root@hadoop001 zkdata]# cat  myid

3.5 安装包分发并修改myid值

[root@hadoop001 zkdata]# cd  /opt/module/
[root@hadoop001 module]# scp -r zookeeper-3.4.9/ hadoop002:`pwd` 
[root@hadoop001 module]# scp -r zookeeper-3.4.9/ hadoop003:`pwd`

#在hadoop002和hadoop003分别操作
[root@hadoop002 zkdata]# echo 2 > myid 
[root@hadoop002 zkdata]# cat myid 
2

[root@hadoop003 ~]#  echo 3 > myid
[root@hadoop003 ~]# cat myid 
3

3.6 启动Zookeeper服务

三台机器都要启动zookeeper服务 ，依次登录三台机器上执行命令：
在hadoop001节点：

cd /opt/module/zookeeper-3.4.9/bin/
./zkServer.sh start

#查看各节点状态

./zkServer.sh status

#JPS命令查看进程

4.MySQL数据库安装 4.1 上传mysql安装包

下载地址：https://dev.mysql.com/downloads/mysql/5.7.html#downloads

4.2 查看系统是否安装了MariaDB，有则卸载

[root@hadoop002 software]# rpm -qa |grep mysql
[root@hadoop002 software]# rpm -qa |grep postfix
postfix-2.10.1-6.el7.x86_64
[root@hadoop002 software]# rpm -qa |grep mariadb
mariadb-libs-5.5.56-2.el7.x86_64
[root@hadoop002 software]# rpm -e postfix-2.10.1-6.el7.x86_64
[root@hadoop002 software]# rpm -e mariadb-libs-5.5.56-2.el7.x86_64

#配置yum源


#安装MySQL5.7所需要的依赖
[root@hadoop002 software]# yum -y  install libaio 
[root@hadoop002 software]# yum -y  install net-tools 
[root@hadoop002 software]# yum -y  install perl

4.3 安装MySQL 4.3.1 解压安装包

[root@hadoop002 software]# mkdir /opt/module/mysql-5.7
[root@hadoop002 software]# tar -xvf  mysql-5.7.36-1.el7.x86_64.rpm-bundle.tar -C 
/opt/module/mysql-5.7/
mysql-community-client-5.7.36-1.el7.x86_64.rpm
mysql-community-common-5.7.36-1.el7.x86_64.rpm
mysql-community-devel-5.7.36-1.el7.x86_64.rpm
mysql-community-embedded-5.7.36-1.el7.x86_64.rpm
mysql-community-embedded-compat-5.7.36-1.el7.x86_64.rpm
mysql-community-embedded-devel-5.7.36-1.el7.x86_64.rpm
mysql-community-libs-5.7.36-1.el7.x86_64.rpm
mysql-community-libs-compat-5.7.36-1.el7.x86_64.rpm
mysql-community-server-5.7.36-1.el7.x86_64.rpm
mysql-community-test-5.7.36-1.el7.x86_64.rpm

4.3.2 安装 MySQL 包（按以下顺序安装）

[root@hadoop002 mysql-5.7]# rpm  -ivh  mysql-community-common-5.7.36- 1.el7.x86_64.rpm
[root@hadoop002 mysql-5.7]# rpm  -ivh  mysql-community-libs-5.7.36-1.el7.x86_64.rpm 
[root@hadoop002 mysql-5.7]# rpm  -ivh  mysql-community-client-5.7.36-1.el7.x86_64.rpm 
[root@hadoop002 mysql-5.7]# rpm  -ivh  mysql-community-server-5.7.36-1.el7.x86_64.rpm

4.4 启动MySQL服务

[root@hadoop002 mysql-5.7]# systemctl status mysqld
● mysqld.service - MySQL Server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
   Active: inactive (dead)
     Docs: man:mysqld(8)
           http://dev.mysql.com/doc/refman/en/using-systemd.html
[root@hadoop002 mysql-5.7]# systemctl start mysqld

[root@hadoop002 mysql-5.7]# 
[root@hadoop002 mysql-5.7]# systemctl  status mysqld
● mysqld.service - MySQL Server
   Loaded: loaded (/usr/lib/systemd/system/mysqld.service; enabled; vendor preset: disabled)
   Active: active (running) since 二 2022-04-26 15:14:51 CST; 8s ago
     Docs: man:mysqld(8)
           http://dev.mysql.com/doc/refman/en/using-systemd.html
  Process: 45720 ExecStart=/usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/mysqld.pid $MYSQLD_OPTS (code=exited, status=0/SUCCESS)
  Process: 45657 ExecStartPre=/usr/bin/mysqld_pre_systemd (code=exited, status=0/SUCCESS)
 Main PID: 45724 (mysqld)
   CGroup: /system.slice/mysqld.service
           └─45724 /usr/sbin/mysqld --daemonize --pid-file=/var/run/mysqld/...

4月 26 15:14:33 hadoop002 systemd[1]: Starting MySQL Server...
4月 26 15:14:51 hadoop002 systemd[1]: Started MySQL Server.

#设置开机启动
[root@hadoop002 software]# systemctl enable mysqld 
[root@hadoop002 software]# systemctl daemon-reload

4.4 配置MySQL root密码

MySQL安装完成之后，会在/var/log/mysqld.log 文件中给 root 生成了一个临时的默认 密码，查看方式：

[root@hadoop002 mysql-5.7]# cat  /var/log/mysqld.log | grep password
2022-04-26T07:14:49.414725Z 1 [Note] A temporary password is generated for root@localhost: **-;elsW6iwNb1**

[root@hadoop002 mysql-5.7]# mysql -uroot -p
Enter password: 
Welcome to the MySQL monitor.  Commands end with ; or g.
Your MySQL connection id is 2
Server version: 5.7.36

Copyright (c) 2000, 2021, Oracle and/or its affiliates.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or 'h' for help. Type 'c' to clear the current input statement.

mysql> 

4.5 修改root密码

# 调整密码的验证强度等级，进行如下操作，如果以上步骤成功则跳过下面步骤 
mysql> set  global  validate_password_policy=LOW;
Query OK, 0 rows affected (0.00 sec) 

# 设置   5 位密码长度策略
mysql> set  global  validate_password_length=5;
Query OK, 0 rows affected (0.00 sec)

# 修改root账户密码为root,自定义
mysql> ALTER  USER 'root'@'localhost' IDENTIFIED BY 'mysql';

# 设置远程登录
mysql> GRANT  ALL PRIVILEGES  ON *.* TO  'root'@'%' IDENTIFIED BY 'mysql'  WITH GRANT  OPTION;

#退出MySQL
mysql> exit;

4.6 配置MySQL默认编码为utf-8

[root@hadoop002 software]# vim /etc/my.cnf 

添加如下代码：
character_set_server=utf8
init_connect='SET NAMES utf8'

esc退出编辑模式，shift+zz 保存退出

4.7 重启MySQL

[root@hadoop002 software]# systemctl restart mysqld

4.8 查看MySQL编码

mysql> show variables like ‘%character%’;

Hadoop分布式集群搭建（二）
Hadoop分布式集群搭建（三）