| 操作系统 | 主机名 | IP地址 |
|---|---|---|
| CentOS-7-x86_64-DVD-2009 | master | 192.168.185.150 |
| CentOS-7-x86_64-DVD-2009 | slave1 | 192.168.185.151 |
| CentOS-7-x86_64-DVD-2009 | slave2 | 192.168.185.152 |
| 主机名 | 角色 | 安装组件或服务 |
| master | 管理节点、数据节点 | Ambari-Server、Ambari-Agent、Apache、MySQL |
| slave1 | 数据节点 | Ambari-Agent |
| slave2 | 数据节点 | Ambari-Agent |
安装包文件名：MobaXterm_Personal_11.1、CentOS-7-x86_64-DVD-2009 、jdk-8u271-linux-x64.tar、mysql-connector-java-8.0.22、mysql-5.7.30-linux-glibc2.12-x86_64.tar、HDP-3.1.4.0-centos7-rpm.tar、HDP-GPL-3.1.4.0-centos7-gpl.tar、HDP-UTILS-1.1.0.22-centos7.tar，软件下载百度网盘链接：Ambari2.7.4.0+HDP3.1.4.0
提取码：DZTB
步骤较为简单，不再赘述，这里给出参考文章链接:VMware安装CentOS7超详细版_tsundere_x的博客-CSDN博客_vmware安装centos7写在前面云计算与分布式这门课程的老师让我们使用vmware安装好centos7.6并配置好Java编译环境，刚好复习一波，下面是详细的安装过程。准备工作VMware，我用的是VMware Workstation Pro 15，下载与安装方法就不提了毕竟重点在后头。CentOS7镜像文件，由于7.6版本已经停更，这里我用的是7.7版本。下载地址 http://isoredirect....https://blog.csdn.net/tsundere_x/article/details/104263100
有几点需要注意的事项：
- VMware的选择16.0的版本，否则某些情况下会与Windows10不兼容，导致电脑蓝屏重启。
- VMware的注册码网上很多，随便找个注册。
- 安装客户机操作系统选择稍后安装操作系统，安装好虚拟机之后再选择事先准备好的安装包。
- 虚拟机名称即主机名，按照自己的需求填写。
- 处理器配置与虚拟机内存按照本机性能分配。
- 网络类型选择网络地址转换（NAT）(E)。
- 磁盘容量大小按需分配，此内存为最大使用内存，未占用的部分本机仍然可以正常使用。
- CentOS7安装时，选择最小化安装。
- 使用VMware的克隆功能创建另外两个节点。
# 注意：centos自从7版本以后网卡名变成ens33而不是我这里的eth0了，我是习惯eth0了所以在安装的时候修改了网卡名，如果你的centos网卡名是ens33不要紧，就把我这里eth0的地方都换成你的ens33，对后面没影响。 [root@master ~]# vi /etc/sysconfig/network-scripts/ifcfg-ens33 TYPE="Ethernet" BOOTPROTO="static" NAME="eth0" DEVICE="eth0" onBOOT="yes" IPADDR=192.168.185.150 NETMASK=255.255.255.0 GATEWAY=192.168.185.2 [root@master ~]# vi /etc/resolv.conf nameserver 192.168.185.2 # 对其他两个slave节点也同样做上述操作，只不过在IPADDR值不一样，分别填其节点对应的ip2.3 修改每个节点主机名，添加各节点映射
# 在其他两个子节点的hostname处分别填slave1和slave2 [root@master ~]# vi /etc/hostname master [root@master ~]# vi /etc/hosts # 注意，前两行配置不要删除 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 192.168.185.150 master 192.168.185.151 slave1 192.168.185.152 slave22.4关闭防火墙
#3个节点都要做
# 关闭防火墙
sudo systemctl stop firewalld
# 禁止防火墙开机启动
sudo systemctl disable firewalld
# 查看防火墙状态
sudo systemctl status firewalld
# 确认3个节点的防火墙都是 inactive 和 disabled 状态
[root@master ~]# sudo systemctl status firewalld
● firewalld.service - firewalld - dynamic firewall daemon
Loaded: loaded (/usr/lib/systemd/system/firewalld.service; disabled; vendor preset: enabled)
Active: inactive (dead)
Docs: man:firewalld(1)
2.5禁用SELINUX
# 3个节点都要做 # 首先临时关闭SELINUX sudo setenforce 0 # 然后修改配置文件，永久关闭SELINUX # [root@master ~]# vi /etc/selinux/config # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. # 将SELINUX的值设为disabled SELINUX=disabled # SELINUXTYPE= can take one of three values: # targeted - Targeted processes are protected, # minimum - Modification of targeted policy. only selected processes are protected. # mls - Multi Level Security protection. SELINUXTYPE=targeted
先临时修改，后永久修改的好处：如果只是临时修改，那么重启机器之后设置就会失效；如果只是永久修改，那么必须重启机器才可能让设置生效，产线环境是不可能重启机器的；这样先临时修改，让其生效，再永久修改，那么在下一次服务器重启后也不会担心这个配置失效。
2.6查看各节点是否可以ping通网络#3个节点都要做 # 重启以生效 [root@master ~]# reboot [root@master ~]# ping www.baidu.com # 注意下，重启后若ping不通百度，可能是因为namesever重启后自动被改了，所以导致ping不通百度，如果这样的话就再重新写下上面的resolv.conf [root@master ~]# vim /etc/resolv.conf nameserver 192.168.185.2 # 这下应该就通了,ping下百度试试看 [root@master ~]# ping www.baidu.com PING www.a.shifen.com (119.75.217.109) 56(84) bytes of data. 64 bytes from 119.75.217.109: icmp_seq=1 ttl=128 time=30.6 ms 64 bytes from 119.75.217.109: icmp_seq=2 ttl=128 time=30.9 ms 64 bytes from 119.75.217.109: icmp_seq=3 ttl=128 time=30.9 ms ... # 按Crtl+C结束2.7使用远程连接工具Mobaxterm
各节点可以ping通外网后，连接远程连接工具Mobaxterm，可以更加轻松的使用Linux的命令。
使用方法：
- 解压后，直接双击.exe文件
- 每个节点对应创建一个新的session，连接方式选择SSH，输入节点的主机名和IP地址，首次登录需要输入密码，保存之后可以自动登录。
# 3各节点都要做 # [root@master ~]# vi /etc/profile # 在最后一行添加如下内容 umask 0022 # 修改完之后使用source命令使其生效 source /etc/profile # 在3个节点确保配置生效，0022是应该出现的结果 [root@master ~]# umask 00222.9文件描述符配置
# 3各节点都要做 # [root@master ~]# vi /etc/security/limits.conf # 在文件末尾追加如下内容 * soft nofile 131072 * hard nofile 1310722.10SSH优化
# 3个节点都要做 # [root@master ~]# vi /etc/ssh/sshd_config # 此配置默认为yes，改为no GSSAPIAuthentication no # 此配置默认是注释掉的，放开即可 UseDNS no # 保存退出之后重启ssh服务 sudo systemctl restart sshd2.11SSH免密钥配置
设置master机器远程连接自身以及远程连接其它的2个机器免密钥。
#master节点做 [root@master ~]# ssh-keygen Generating public/private rsa key pair. # 直接Enter Enter file in which to save the key (/home/admin/.ssh/id_rsa): # 直接Enter Enter passphrase (empty for no passphrase): # 直接Enter Enter same passphrase again: Your identification has been saved in /home/admin/.ssh/id_rsa. Your public key has been saved in /home/admin/.ssh/id_rsa.pub. The key fingerprint is: SHA256:tAH9HF2xMFTCpmxDN7vjXat1NI/oPJX1avACPTi0UMU root@master The key's randomart image is: +---[RSA 2048]----+ | .. **o+. | | .. + E+ . | | o* = o. | | ..oO . .| | S+ = . +o| | = *.oo=| | +.B.++| | o+ *..| | o=. | +----[SHA256]-----+ [root@master ~]$ ssh-copy-id -i ~/.ssh/id_rsa.pub root@master /usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/home/admin/.ssh/id_rsa.pub" The authenticity of host 'master (192.168.185.150)' can't be established. ECDSA key fingerprint is SHA256:/eVZsOkuW8gHUcJxKNvtRh9yJ3UYDz4p9hA4W4CpeEg. ECDSA key fingerprint is MD5:8c:e1:a8:0c:c9:80:e5:28:7a:25:96:32:c7:5f:25:47. # 输入yes Are you sure you want to continue connecting (yes/no)? yes /usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed /usr/bin/ssh-copy-id: INFO: 1 key(s) remain to be installed -- if you are prompted now it is to install the new keys # 输入root@master的密码 root@master's password: Number of key(s) added: 1 Now try logging into the machine, with: "ssh 'root@master'" and check to make sure that only the key(s) you wanted were added. # 继续给其他机器添加master节点的公钥，过程略... ssh-copy-id -i ~/.ssh/id_rsa.pub root@slave1 ssh-copy-id -i ~/.ssh/id_rsa.pub root@slave2 # 以上操作执行完成之后，修改.ssh目录和authorized_keys文件的权限，3个节点都要执行 chmod 700 ~/.ssh chmod 600 ~/.ssh/authorized_keys # 测试，每次ssh到其他机器都不需要输入密码 [root@master ~]# ssh master Last login: Thu Sep 23 15:15:08 2021 from 192.168.185.153 [root@master ~]# exit 登出 Connection to master closed. [root@master ~]# ssh slave1 Last login: Thu Sep 23 15:15:09 2021 from 192.168.185.153 [root@slave1 ~]# exit 登出 Connection to slave1 closed. [root@master ~]# ssh slave2 Last login: Thu Sep 23 15:15:13 2021 from 192.168.185.153 [root@slave2 ~]# exit 登出 Connection to slave2 closed.2.12时钟同步
使用ntp的目的是对网络内所有具有时钟的设备进行时钟同步，使网络内所有设备能够提供基于同一时间的多种应用。
#3个节点都要做 #安装 yum install ntp #开机自启动 systemctl enable ntpd #启动ntp服务 systemctl start ntpd #查看ntpd服务状态 systemctl status ntpd #修改ntp.conf配置 vi /etc/ntp.conf #server 0.centos.pool.ntp.org iburst #server 1.centos.pool.ntp.org iburst #server 2.centos.pool.ntp.org iburst #server 3.centos.pool.ntp.org iburst #表示ntp主服务器是与自身的系统时钟同步 server 127.127.1.0 #指定阶层编号为10，降低优先度 fudge 127.127.1.0 stratum 10 #时钟同步 #slave1、slave2执行以下操作 vi /etc/ntp.conf #server 0.centos.pool.ntp.org iburst #server 1.centos.pool.ntp.org iburst #server 2.centos.pool.ntp.org iburst #server 3.centos.pool.ntp.org iburst #表示ntp主服务器是与自身的系统时钟同步 server 127.127.1.0 #指定阶层编号为10，降低优先度 fudge 127.127.1.0 stratum 10 #配置时间服务器为本地的时间服务器 server master #允许上层时间服务器主动修改本机时间 restrict master nomodify notrap noquery #检查时钟同步效果 #在slave1、slave2执行以下操作 #重启服务 systemctl restart ntpd #date查看时间 date #检查slave1、slave2是否和master时间一致2.13安装JDK
#3个节点都要做 # 新建java目录，将下载好的jdk压缩包使用MobaXterm上传到java文件夹（直接鼠标左键将文件拖拽至java文件夹实现），解压 [root@master ~]# cd /usr/local [root@master local]# mkdir java [root@master local]# cd java [root@master java]# tar -zxvf jdk-8u271-linux-x64.tar.gz # 配置环境变量，在profile文件最后添加java的环境变量 [root@master ~]# vim /etc/profile export JAVA_HOME=/usr/local/java/jdk1.8.0_271 export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar export PATH=$PATH:$JAVA_HOME/bin [root@master ~]# source /etc/profile [root@master ~]# java -version java version "1.8.0_271" Java(TM) SE Runtime Environment (build 1.8.0_271-b09) Java HotSpot(TM) 64-Bit Server VM (build 25.271-b09, mixed mode)3Ambari Server安装部署 3.1安装MySQL5.7
#master节点做 # HDP3.1要求MySQL的版本为5.7 # 安装MySQL的Yum仓库 sudo yum -y localinstall https://dev.mysql.com/get/mysql57-community-release-el7-8.noarch.rpm # 安装MySQL sudo yum -y install mysql-community-server # 创建数据目录 sudo mkdir -p /data/mysql sudo chown -R mysql:mysql /data/mysql # vi /etc/my.cnf [client] default-character-set=utf8mb4 [mysql] socket=/var/lib/mysql/mysql.sock default-character-set=utf8mb4 [mysqld] socket=/var/lib/mysql/mysql.sock symbolic-links=0 log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid # 注意将数据目录修改为/data目录，/data目录的磁盘是独立挂载的大容量磁盘 datadir=/data/mysql character_set_server=utf8mb4 character-set-client-handshake=FALSE collation-server=utf8mb4_unicode_ci max_connections=800 max_connect_errors=1000 # 设置开机自启动 sudo systemctl enable mysqld # 启动MySQL sudo systemctl start mysqld # 查看运行状态 sudo systemctl status mysqld # 获取临时密码，注意每个人的临时密码是不同的，我这里是q8rR4AYaek.p [root@master ~]# sudo grep 'temporary password' /var/log/mysqld.log 2021-09-10T02:37:41.379119Z 1 [Note] A temporary password is generated for root@localhost: q8rR4AYaek.p # 进入MySQL操作 [root@master ~]# mysql -uroot -p"q8rR4AYaek.p" mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or g. Your MySQL connection id is 14 Server version: 5.7.35 MySQL Community Server (GPL) Copyright (c) 2000, 2021, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or 'h' for help. Type 'c' to clear the current input statement. mysql> set global validate_password_policy=0; Query OK, 0 rows affected (0.00 sec) mysql> set global validate_password_mixed_case_count=0; Query OK, 0 rows affected (0.00 sec) mysql> set global validate_password_number_count=3; Query OK, 0 rows affected (0.00 sec) mysql> set global validate_password_special_char_count=0; Query OK, 0 rows affected (0.00 sec) mysql> set global validate_password_length=3; Query OK, 0 rows affected (0.00 sec) # 在生产上建议把root密码设置的更加复杂，我这里设置的root用户只能本机连接 mysql> ALTER USER 'root'@'localhost' IDENTIFIED BY '123456'; Query OK, 0 rows affected (0.00 sec) mysql> exit Bye # 将mysql-connector-java-8.0.22.jar上传到/usr/share/java目录下(使用MobaXterm上传) sudo mkdir -p /usr/share/java3.2准备本地yum源
#master节点做 #安装Apache sudo yum -y install httpd #讲Apache设置为开机启动 sudo systemctl enable httpd #启动Apache sudo systemctl start httpd
访问如下链接：http://192.168.185.150/，如果可以看到如下界面httpd就安装好了(如果IP地址是自己设置的，需要修改IP地址)。
HDP安装包部署到Apache：
#master节点做 # 首先把下载好的ambari和HDP的3个安装包上传到home目录下 cd /var/www/html/ ambari-2.7.4.0-centos7.tar.gz HDP-3.1.4.0-centos7-rpm.tar.gz HDP-GPL-3.1.4.0-centos7-gpl.tar.gz HDP-UTILS-1.1.0.22-centos7.tar.gz #解压缩 sudo tar -zxvf ambari-2.7.4.0-centos7.tar.gz sudo tar -zxvf HDP-3.1.4.0-centos7-rpm.tar.gz sudo tar -zxvf HDP-GPL-3.1.4.0-centos7-gpl.tar.gz sudo tar -zxvf HDP-UTILS-1.1.0.22-centos7.tar.gz #安装包清理 sudo rm -r ambari-2.7.4.0-centos7.tar.gz sudo rm -r HDP-3.1.4.0-centos7-rpm.tar.gz sudo rm -r HDP-GPL-3.1.4.0-centos7-gpl.tar.gz sudo rm -r HDP-UTILS-1.1.0.22-centos7.tar.gz #更改目录权限 sudo chown -R admin:admin /var/www/html sudo chmod -R ugo+rX /var/www/html/
4个安装包可以在浏览器访问，其URL分别是：
| 安装包 | URL |
| ambari | http://192.168.185.150/HDP/centos7/ |
| HDP | http://hdp01/HDP/centos7/3.1.4.0-315/ |
| HDP-GPL | http://hdp01/HDP-GPL/centos7/3.1.4.0-315/ |
| HDP-UTLS | http://hdp01/HDP-UTILS/centos7/1.1.0.22/ |
配置本地yum源：
#master节点做 # vi /etc/yum.repos.d/ambari.repo，内容如下： [ambari-repo] name=ambari baseurl=http://192.168.185.150/ambari/centos7/2.7.4.0-118/ gpgcheck=0 enabled=1 # 验证，应该看到ambari-repo那一行 [root@master yum.repos.d]$ yum repolist Loaded plugins: fastestmirror Loading mirror speeds from cached hostfile repo id repo name status ambari-repo ambari 13 base/7/x86_64 CentOS-7 - base - mirrors.aliyun.com 10,070 epel/x86_64 Extra Packages for Enterprise Linux 7 - x86_64 13,453 extras/7/x86_64 CentOS-7 - Extras - mirrors.aliyun.com 413 updates/7/x86_64 CentOS-7 - Updates - mirrors.aliyun.com 1,134 repolist: 25,083
本地yum源创建好了。
3.3安装Ambari-Server# master节点做 # 安装ambari-server sudo yum -y install ambari-server # 核心日志信息 Installed: ambari-server.x86_64 0:2.7.4.0-118 # 数据库操作 [root@master ~]$ mysql -uroot -p123456 mysql> set global validate_password_policy=0; mysql> set global validate_password_mixed_case_count=0; mysql> set global validate_password_number_count=3; mysql> set global validate_password_special_char_count=0; mysql> set global validate_password_length=3; mysql> create database ambari; mysql> CREATE USER 'ambari'@'%' IDENTIFIED BY '123456'; mysql> GRANT ALL ON ambari.* TO 'ambari'@'%'; mysql> FLUSH PRIVILEGES; mysql> use ambari; mysql> source /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql; mysql> exit # 配置AmbariServer [root@master ~]$ sudo ambari-server setup Using python /usr/bin/python Setup ambari-server Checking SELinux... SELinux status is 'disabled' # 是否自定义ambari-server的运行用户，我们保持默认root用户即可，所以输入n并回车 Customize user account for ambari-server daemon [y/n] (n)? n Adjusting ambari-server permissions and ownership... Checking firewall status... Checking JDK... [1] Oracle JDK 1.8 + Java Cryptography Extension (JCE) Policy Files 8 [2] Custom JDK ============================================================================== # 选择2，代表自定义JDK，然后回车 Enter choice (1): 2 WARNING: JDK must be installed on all hosts and JAVA_HOME must be valid on all hosts. WARNING: JCE Policy files are required for configuring Kerberos security. If you plan to use Kerberos,please make sure JCE Unlimited Strength Jurisdiction Policy Files are valid on all hosts. # 指定自己安装的JDK的路径/usr/local/java/jdk1.8.0_271，然后回车 Path to JAVA_HOME: /usr/local/java/jdk1.8.0_271 Validating JDK on Ambari Server...done. Check JDK version for Ambari Server... JDK version found: 8 Minimum JDK version is 8 for Ambari. Skipping to setup different JDK for Ambari Server. Checking GPL software agreement... GPL License for LZO: https://www.gnu.org/licenses/old-licenses/gpl-2.0.en.html # 提示是否下载并安装LZO的包，我们输入y并回车 Enable Ambari Server to download and install GPL Licensed LZO packages [y/n] (n)? y Completing setup... Configuring database... # 是否进入数据库高级设置，我们输入y然后回车 Enter advanced database configuration [y/n] (n)? y Configuring database... ============================================================================== Choose one of the following options: [1] - PostgreSQL (Embedded) [2] - Oracle [3] - MySQL / MariaDB [4] - PostgreSQL [5] - Microsoft SQL Server (Tech Preview) [6] - SQL Anywhere [7] - BDB ============================================================================== # 在选择数据库的环节，我们选择3也就是MySQL，然后回车 Enter choice (1): 3 # MySQL的相关配置我们均采用默认值，默认用户和db都是ambari，密码输入123456，然后回车 Hostname (localhost): Port (3306): Database name (ambari): Username (ambari): Enter Database Password (bigdata): Re-enter password: Configuring ambari database... # 我们需要指定MySQL的驱动包所在的路径/usr/share/java/mysql-connector-java-8.0.22.jar，然后回车 Enter full path to custom jdbc driver: /usr/share/java/mysql-connector-java-8.0.22.jar Configuring remote database connection properties... WARNING: Before starting Ambari Server, you must run the following DDL directly from the database shell to create the schema: /var/lib/ambari-server/resources/Ambari-DDL-MySQL-CREATE.sql # 选择可以远程连接该数据库(y)，然后回车就配置完毕了 Proceed with configuring remote database connection properties [y/n] (y)? y Extracting system views... ambari-admin-2.7.4.0.118.jar .... Ambari repo file doesn't contain latest json url, skipping repoinfos modification Adjusting ambari-server permissions and ownership... Ambari Server 'setup' completed successfully. # 启动Ambari-Server sudo ambari-server start # 如果出现错误，查看日志/var/log/ambari-server/ambari-server.log # 出现如下信息就是启动成功了 Starting ambari-server Ambari Server running with administrator privileges. Organizing resource files at /var/lib/ambari-server/resources... Ambari database consistency check started... Server PID at: /var/run/ambari-server/ambari-server.pid Server out at: /var/log/ambari-server/ambari-server.out Server log at: /var/log/ambari-server/ambari-server.log Waiting for server start................................. Server started listening on 8080 DB configs consistency check: no errors and warnings were found. Ambari Server 'start' completed successfully.
访问WEB-UI：http://192.168.185.150:8080/#/login
如果可以正常登录就是安装好了。
4Ambari+HDP大数据平台部署 4.1部署Ambari集群和HDFS集群滚动条下拉，把其他无用的操作系统的仓库删除：
最终只保留“redhat7” 的仓库，然后填写部署yum本地仓库的地址，同时不要勾选下面的两个高级选项。由于本机已经安装完成，无法截图示意，将下面的hdp01改成master节点的IP地址。
4.2部署YARN集群 4.3部署Hbase集群也是Add Service。
4.4部署Hive# MySQL操作 mysql> create database hive; mysql> CREATE USER 'hive'@'%' IDENTIFIED BY '123456'; mysql> GRANT ALL ON hive.* TO 'hive'@'%'; mysql> FLUSH PRIVILEGES; mysql> exit; # 执行以下命令 sudo ambari-server setup --jdbc-db=mysql --jdbc-driver=/usr/share/java/mysql-connector-java-8.0.22.jar
新服务部署后，可能会提示重启其他相关服务，重启即可。
4.5部署Kafka 4.6部署Spark2+Sqoop其余的组件安装类似，直接在页面操作安装即可，需要注意的就是修改日志目录 和数据目录。
支持，Ambari2.7+HDP3.1大数据平台就安装部署完成了！
参考文献- 一文搞懂网络知识，IP、子网掩码、网关、DNS、端口号 - 知乎
- Hadoop入门(一)——CentOS7下载+VM上安装（手动分区）图文步骤详解(2021)_Leokadia Rothschild的博客-CSDN博客
- 基于CentOS7.8安装Ambari2.7+HDP3.1大数据平台 - 简书
- Ambari离线安装大数据集群 - 云+社区 - 腾讯云
- 集群搭建篇（ambari+HDP）—— 全网最详细的，没有之一 - 云+社区 - 腾讯云
- 大数据环境搭建步骤详解（Hadoop，Hive，Zookeeper，Kafka，Flume，Hbase，Spark等安装与配置）_pig2guang的博客-CSDN博客
