基于最新CDH发布版本编写安装文档，CDH6安装过程基本与CDH5一致，主要包括以下几步：
1、安装前置准备：包括安装操作系统、关闭防火墙、同步服务器时钟等；
2、mysql安装部署；
3、安装Cloudera Manager；
4、安装CDH集群；
软件要求：
CDH 集群搭建需要的安装包包括 JDK、Cloudera-Manager RPM 包（3 个），CDH parcel⽂件（3 个）、MySQL、MySQL 驱动 jar 包。
| 组件 | 版本 | 类型 | 备注 |
|---|---|---|---|
| JDK | 8u202 | tar | jdk-8u202-linux-x64.tar.gz |
| CM daemons | 6.3.1 | rpm | cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm |
| CM agent | 6.3.1 | rpm | cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm |
| CM server | 6.3.1 | rpm | cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm |
| CDH 各组件⼆ 进制服务包 | 6.3.2 | parcel | CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel、CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1、manifest.json |
| MySQL | 8.0.18 | tar | mysql-8.0.18-el7-x86_64.tar.gz |
| MySQL驱动包 | 8.0.18 | jar | mysql-connector-java-8.0.18.jar |
节点分布：
CM Server 和 Agent 分配 Cloudera Manager 由 Server 和 Agent 组成，安装 Cloudera Manager Server 的节点称为 Server 节点，安装 Cloudera Manager Agent 的节点称为 Agent 节点。
3个节点的情形
| 节点 | 安装的服务 |
|---|---|
| node1 | CM daemon，CM server，CM agent |
| node2 | CM daemon，CM agent |
| node3 | CM daemon，CM agent |
没有出现其他节点主机名的话，说明每个节点都是需要执行的。
2.1 hosts及hostname rehl7修改主机名使用hostnamectl命令
#节点一 [root@hadoop-1 ~]# hostnamectl set-hostname hadoop-1 #节点二 [root@hadoop-2 ~]# hostnamectl set-hostname hadoop-2 #节点三 [root@hadoop-3 ~]# hostnamectl set-hostname hadoop-3
修改/etc/hosts（每个节点都执行）
[root@hadoop-1 ~]# vim /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 #集群节点地址 集群主机名 172.15.10.145 hadoop-1 172.15.10.146 hadoop-2 172.15.10.147 hadoop-32.2 禁用selinux并关闭防火墙
[root@hadoop-1 ~]# vim /etc/selinux/config # This file controls the state of SELinux on the system. # SELINUX= can take one of these three values: # enforcing - SELinux security policy is enforced. # permissive - SELinux prints warnings instead of enforcing. # disabled - No SELinux policy is loaded. SELINUX=disabled # SELINUXTYPE= can take one of three values: # targeted - Targeted processes are protected, # minimum - Modification of targeted policy. only selected processes are protected. # mls - Multi Level Security protection. SELINUXTYPE=targeted ------------------------------------------------------------------------------------------------------------------- [root@hadoop-1 ~]# systemctl stop firewalld.service [root@hadoop-1 ~]# systemctl disable firewalld.service2.3 系统参数修改
swap空间设置
[root@hadoop-1 ~]# echo "vm.swappiness = 0" >> /etc/sysctl.conf [root@hadoop-1 ~]# sysctl -p
设置透明大页面
[root@hadoop-1 ~]# echo never > /sys/kernel/mm/transparent_hugepage/defrag [root@hadoop-1 ~]# echo never > /sys/kernel/mm/transparent_hugepage/enabled
增⼤系统打开⽂件数
[root@hadoop-1 ~]# vim /etc/security/limits.conf #末尾加上 * - nofile 100000 * - nproc 100000 hadoop soft memlock unlimited hadoop hard memlock unlimited2.4 ssh免密认证
[root@hadoop-1 ~]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa [root@hadoop-1 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@hadoop-1 [root@hadoop-1 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@hadoop-2 [root@hadoop-1 ~]# ssh-copy-id -i ~/.ssh/id_rsa.pub root@hadoop-3 #验证方式，ssh hostname,直接连接上了说明配置成功 [root@hadoop-1 ~]# ssh hadoop-22.5 安装并启动Apache http
在主节点执行
#安装Apache http [root@hadoop-1 ~]# yum install -y httpd #启动Apache http [root@hadoop-1 cdh]# systemctl start httpd #设置开机启动Apache http [root@hadoop-1 cdh]# systemctl enable httpd #创建安装文件http根目录 [root@hadoop-1 cdh]# mkdir -p /var/www/html/cm6 [root@hadoop-1 cdh]# mkdir -p /var/www/html/cdh6 [root@hadoop-1 cdh]# cd /usr/local/src/cdh/ #上传安装文件到http根目录 [root@hadoop-1 cdh]# cp allkeys.asc *.rpm /var/www/html/cm6 [root@hadoop-1 cdh]# ll /var/www/html/cm6 total 1199784 -rw-r--r-- 1 root root 14041 Jan 22 00:06 allkeys.asc -rw-r--r-- 1 root root 10483568 Jan 22 00:06 cloudera-manager-agent-6.3.1-1466458.el7.x86_64.rpm -rw-r--r-- 1 root root 1203832464 Jan 22 00:06 cloudera-manager-daemons-6.3.1-1466458.el7.x86_64.rpm -rw-r--r-- 1 root root 11488 Jan 22 00:06 cloudera-manager-server-6.3.1-1466458.el7.x86_64.rpm -rw-r--r-- 1 root root 10996 Jan 22 00:06 cloudera-manager-server-db-2-6.3.1-1466458.el7.x86_64.rpm -rw-r--r-- 1 root root 14209868 Jan 22 00:06 enterprise-debuginfo-6.3.1-1466458.el7.x86_64.rpm #创建yum仓库 [root@hadoop-1 cdh]# cd /var/www/html/cm6 [root@hadoop-1 cm6]# yum install -y createrepo [root@hadoop-1 cm6]# createrepo .2.6 yum源设置
yum源设置有多种方式，无网络可以采用镜像挂载，有网络可以直接用清华源、阿里源等等，此处不做展开，自行百度。这里介绍下本地挂载的方式。
[root@hadoop-1 ~]# vim /etc/yum.repos.d/rhel7.repo [base] name=rhel7 baseurl=file:///media/cdrom enabled=1 gpgcheck=0 [root@hadoop-1 ~]# mkdir -p /media/cdrom [root@hadoop-1 ~]# mount /dev/cdrom /media/cdrom mount: /media/cdrom: WARNING: device write-protected, mounted read-only. ------------------------------------------------------------------------------------------------------------------- #挂载自建镜像 [root@hadoop-1 cm6]# vim /etc/yum.repos.d/cloudera-manager.repo [cloudera-manager] name=Cloudera Manager 6.3.1 baseurl=http://hadoop-1/cm6 gpgcheck=0 enabled=1 autorefresh=0 type=rpm-md [root@hadoop-1 cm6]# yum clean all #查看是否挂载成功 [root@hadoop-1 cm6]# yum repolist
网页测试能否正常访问
2.7 时钟同步设置 检查是否安装ntp，如果没有则使用yum安装
[root@hadoop-1 ~]# rpm -qa | grep ntp [root@hadoop-1 ~]# yum install -y ntp
主节点hadoop-1配置（设置为标准时间服务器）
[root@hadoop-1 ~]# vim /etc/ntp.conf # For more information about this file, see the man pages # ntp.conf(5), ntp_acc(5), ntp_auth(5), ntp_clock(5), ntp_misc(5), ntp_mon(5). driftfile /var/lib/ntp/drift #新增:日志目录. logfile /var/log/ntpd.log # Permit time synchronization with our time source, but do not # permit the source to query or modify the service on this system. restrict default nomodify notrap nopeer noquery # Permit all access over the loopback interface. This could # be tightened as well, but to do so would effect some of # the administrative functions. restrict 127.0.0.1 restrict ::1 # Hosts on local network are less restricted. #restrict 192.168.1.0 mask 255.255.255.0 nomodify notrap #新增：授权172.15.10.0网段上的所有机器可以从这台机器上查询和同步时间 restrict 172.15.10.0 mask 255.255.255.0 nomodify # Use public servers from the pool.ntp.org project. # Please consider joining the pool (http://www.pool.ntp.org/join.html). #修改:注释外部时钟服务器配置，实际生产部署需要外部时钟服务器，否则会有告警 #server 0.centos.pool.ntp.org iburst #server 1.centos.pool.ntp.org iburst #server 2.centos.pool.ntp.org iburst #server 3.centos.pool.ntp.org iburst #broadcast 192.168.1.255 autokey # broadcast server #broadcastclient # broadcast client #broadcast 224.0.1.1 autokey # multicast server #multicastclient 224.0.1.1 # multicast client #manycastserver 239.255.254.254 # manycast server #manycastclient 239.255.254.254 autokey # manycast client #新增:当外部时间不可用时，使用本地时间. server 172.15.10.145 iburst fudge 127.0.0.1 stratum 10 # Enable public key cryptography. #crypto ------------------------------------------------------------------------------------------------------------------- [root@hadoop-1 ~]# systemctl enable ntpd [root@hadoop-1 ~]# systemctl start ntpd
其他节点配置，同步主节点时间
[root@hadoop-2 ~]# echo "server 172.15.10.145" >> /etc/ntp.conf
[root@hadoop-2 ~]# systemctl enable ntpd
[root@hadoop-2 ~]# systemctl start ntpd
#检查方式
[root@hadoop-2 ~]# ntpq -p
remote refid st t when poll reach delay offset jitter
==============================================================================
hadoop-1 .INIT. 16 u 32 64 0 0.000 0.000 0.000
2.8 jdk安装
1、卸载默认安装的jdk
[root@hadoop-1 ~]# rpm -qa|grep jdk [root@hadoop-1 ~]# rpm -e --nodeps #上一步查询出来的jdk名字
2、安装jdk
目录必须严格要求文中设置，cdh启动的时候强制找/usr/java下面的jdk
[root@hadoop-1 src]# mkdir -p /usr/java [root@hadoop-1 ~]# cd /usr/local/src #个人习惯将包上传到/usr/local/src目录 [root@hadoop-1 src]# tar -xzvf jdk-8u202-linux-x64.tar.gz -C /usr/java [root@hadoop-1 src]# cd /usr/java [root@hadoop-1 java]# ln -s jdk1.8.0_202/ default [root@hadoop-1 java]# chmod -R 755 /usr/java #设置环境变量 [root@hadoop-1 ~]# vim /etc/profile ##新增JAVA环境变量 export JAVA_HOME=/usr/java/default export CLASSPATH=.:$JAVA_HOME/lib:$JAVA_HOME/jre/lib export PATH=$JAVA_HOME/bin:$PATH ------------------------------------------------------------------------------------------------------------------- [root@hadoop-1 java]# source /etc/profile三、mysql安装部署
mysql只需要在server节点部署，以下操作只针对hadoop-1节点
3.1 卸载原有mysql#查询mysql相关rpm包 [root@hadoop-1 ~]# rpm -qa | grep -i mysql qt-mysql-4.8.7-2.el7.x86_64 akonadi-mysql-1.9.2-4.el7.x86_64 perl-DBD-MySQL-4.023-6.el7.x86_64 #将查询到的包卸载 [root@hadoop-1 ~]# rpm --nodeps -ev qt-mysql-4.8.7-2.el7.x86_64 akonadi-mysql-1.9.2-4.el7.x86_64 perl-DBD-MySQL-4.023-6.el7.x86_64 #查询mysql相关文件 [root@hadoop-1 ~]# find / -name mysql #删除mysql相关文件 [root@hadoop-1 ~]# rm -rf /var/lib/mysql [root@hadoop-1 ~]# rm -rf /usr/lib64/mysql [root@hadoop-1 ~]# rm -rf /usr/bin/mysql [root@hadoop-1 ~]# rm -rf /usr/share/mysql [root@hadoop-1 ~]# rm -rf /var/lib/pcp/config/pmlogconf/mysql [root@hadoop-1 ~]# rm -rf /etc/selinux/targeted/active/modules/100/mysql #删除mysql默认读取配置（使用tar包安装情况） [root@hadoop-1 ~]# rm -rf /etc/my.cnf /etc/my.cnf.d/3.2 安装mysql
官网地址：mysql8下载
[root@hadoop-1 ~]# cd /usr/local/src [root@hadoop-1 src]# wget https://dev.mysql.com/get/Downloads/MySQL-8.0/mysql-8.0.18-el7-x86_64.tar.gz
解压压缩包，将解压出的程序目录改名并移动到/usr/local目录下，对其进行初始化操作后便可使用。：
[root@hadoop-1 src]# tar -zxvf mysql-8.0.18-el7-x86_64.tar.gz [root@hadoop-1 src]# cp -r mysql-8.0.18-el7-x86_64/ /usr/local/mysql [root@hadoop-1 src]# cd /usr/local/mysql/ [root@hadoop-1 mysql]# mkdir data #创建mysql用户 [root@hadoop-1 mysql]# useradd mysql -M -s /sbin/nologin
修改my.cnf配置，典型的mysql8 my.cnf配置：
[root@hadoop-1 ~]# vim /etc/my.cnf [client] port = 3306 #根据实际情况调整mysql.sock配置 socket = /tmp/mysql.sock [mysqld] #Mysql服务的唯一编号 每个mysql服务Id需唯一 server-id = 1 #服务端口号 默认3306 port = 3306 #mysql安装根目录 basedir = /usr/local/mysql #mysql数据文件所在位置 datadir = /usr/local/mysql/data #pid pid-file = /usr/local/mysql/data/mysql.pid #设置socke文件所在目录 socket = /tmp/mysql.sock #设置临时目录 tmpdir = /tmp # 用户 user = mysql # 允许访问的IP网段 bind-address = 0.0.0.0 # 跳过密码登录 #skip-grant-tables #主要用于MyISAM存储引擎,如果多台服务器连接一个数据库则建议注释下面内容 skip-external-locking #只能用IP地址检查客户端的登录，不用主机名 skip_name_resolve = 1 #事务隔离级别，默认为可重复读，mysql默认可重复读级别（此级别下可能参数很多间隙锁，影响性能） transaction_isolation = READ-COMMITTED #数据库默认字符集,主流字符集支持一些特殊表情符号（特殊表情符占用4个字节） character-set-server = utf8mb4 #数据库字符集对应一些排序等规则，注意要和character-set-server对应 collation-server = utf8mb4_general_ci #设置client连接mysql时的字符集,防止乱码 init_connect='SET NAMES utf8mb4' #是否对sql语句大小写敏感，1表示不敏感 lower_case_table_names = 1 #最大连接数 max_connections = 400 #最大错误连接数 max_connect_errors = 1000 #TIMESTAMP如果没有显示声明NOT NULL，允许NULL值 explicit_defaults_for_timestamp = true #SQL数据包发送的大小，如果有BLOB对象建议修改成1G max_allowed_packet = 128M #MySQL连接闲置超过一定时间后(单位：秒)将会被强行关闭 #MySQL默认的wait_timeout 值为8个小时, interactive_timeout参数需要同时配置才能生效 interactive_timeout = 1800 wait_timeout = 1800 #内部内存临时表的最大值 ，设置成128M。 #比如大数据量的group by ,order by时可能用到临时表， #超过了这个值将写入磁盘，系统IO压力增大 tmp_table_size = 134217728 max_heap_table_size = 134217728 #数据库错误日志文件 log_error = error.log #慢查询sql日志设置 slow_query_log = 1 slow_query_log_file = slow.log #检查未使用到索引的sql log_queries_not_using_indexes = 1 #针对log_queries_not_using_indexes开启后，记录慢sql的频次、每分钟记录的条数 log_throttle_queries_not_using_indexes = 5 #作为从库时生效,从库复制中如何有慢sql也将被记录 log_slow_slave_statements = 1 #慢查询执行的秒数，必须达到此值可被记录 long_query_time = 8 #检索的行数必须达到此值才可被记为慢查询 min_examined_row_limit = 100 #mysql binlog日志文件保存的过期时间，过期后自动删除 expire_logs_days = 53.3 初始化MySQL服务程序
对目录进行授权，保证数据能够被mysql系统用户读取。在初始化阶段，应使用mysqld命令确认管理MySQL数据库服务的用户名称、数据保存目录及编码信息。在信息确认无误后开始进行初始化。在初始化的最后阶段，系统会给用户分配一个初始化的临时密码。mysql8版本只有在初始化的时候设置 lower_case_table_names=1才有效，切记。
[root@hadoop-1 mysql]# chown -R mysql:mysql /usr/local/mysql [root@hadoop-1 mysql]# cd bin [root@hadoop-1 bin]# ./mysqld --initialize --user=mysql --basedir=/usr/local/mysql --datadir=/usr/local/mysql/data --lower-case-table-names=1 #如果使用了上面的配置文件，会生成日志在data目录下，里面会有初始化生成的密码 [root@hadoop-1 bin]# vim /usr/local/mysql/data/error.log 2022-01-20T15:45:19.609453Z 0 [Warning] [MY-011068] [Server] The syntax 'expire-logs-days' is deprecated and will be removed in a future release. Please use binlog_expire_logs_seconds instead. 2022-01-20T15:45:19.609551Z 0 [System] [MY-013169] [Server] /usr/local/mysql/bin/mysqld (mysqld 8.0.18) initializing of server in progress as process 3281 2022-01-20T15:45:20.506473Z 0 [Warning] [MY-010161] [Server] You need to use --log-bin to make --expire_logs_days work. 2022-01-20T15:45:21.612476Z 5 [Note] [MY-010454] [Server] A temporary password is generated for root@localhost: ww-=dht_A0uj
添加环境变量，方便调用
[root@hadoop-1 bin]# vim ~/.bash_profile
# .bash_profile
# Get the aliases and functions
if [ -f ~/.bashrc ]; then
. ~/.bashrc
fi
# User specific environment and startup programs
PATH=$PATH:$HOME/bin:/usr/local/mysql/bin
export PATH
[root@hadoop-1 bin]# source ~/.bash_profile
3.4 配置自启动并启动MySQL
libtinfo.so.5文件是MySQL数据库在8.0版本后新添加的重要的函数库文件，但部分情况不存在，需要将libtinfo.so.6.1文件复制过来或者作为链接文件才能正常启动：
[root@hadoop-1 bin]# cd /usr/local/mysql [root@hadoop-1 mysql]# cp -a support-files/mysql.server /etc/init.d/ [root@hadoop-1 mysql]# chmod a+x /etc/init.d/mysql.server [root@hadoop-1 mysql]# ln -s /usr/lib64/libtinfo.so.6.1 /usr/lib64/libtinfo.so.5 [root@hadoop-1 mysql]# /etc/init.d/mysql.server start
登录MySQL并修改密码
[root@hadoop-1 mysql]# mysql -u root -p Enter password: 初始化的时候生成的密码 Welcome to the MySQL monitor. Commands end with ; or g. Your MySQL connection id is 8 Server version: 8.0.27 Copyright (c) 2000, 2021, Oracle and/or its affiliates. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type 'help;' or 'h' for help. Type 'c' to clear the current input statement. mysql> alter user 'root'@'localhost' identified by 'onaslowboattochina'; #mysql8之后需要额外执行此步才会将密码修改生效 mysql> use mysql; Reading table information for completion of table and column names You can turn off this feature to get a quicker startup with -A Database changed mysql> ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY 'onaslowboattochina'; #设置远程访问权限,注意Mysql8.0不支持grant all privileges on *.* to root@“%“ identified by “.“； mysql> CREATE USER 'root'@'%' IDENTIFIED BY 'onaslowboattochina'; Query OK, 0 rows affected (0.00 sec) mysql> grant all privileges on *.* to 'root'@'%' ; Query OK, 0 rows affected (0.00 sec) mysql> flush privileges; Query OK, 0 rows affected (0.00 sec)3.5 创建数据库
根据实际需要部署的服务，增加或者减少创建的数据库用户数
create database metastore default character set utf8; CREATE USER 'hive'@'%' IDENTIFIED BY 'onaslowboattochina'; GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%'; FLUSH PRIVILEGES; create database rm default character set utf8; CREATE USER 'rm'@'%' IDENTIFIED BY 'onaslowboattochina'; GRANT ALL PRIVILEGES ON rpm.* TO 'rm'@'%'; FLUSH PRIVILEGES; create database amon default character set utf8; CREATE USER 'amon'@'%' IDENTIFIED BY 'onaslowboattochina'; GRANT ALL PRIVILEGES ON amon.* TO 'amon'@'%'; FLUSH PRIVILEGES;四、CDH安装 4.1 Cloudera Manager Server 安装(hadoop-1节点)
[root@hadoop-1 ~]# yum install -y cloudera-manager-server
cloudera-manager-server安装完成后会自动生成/opt/cloudera/parcel-repo目录，将准备好的 CDH parcel 包 copy⾄/opt/cloudera/parcel-repo，拷贝后将 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 以 mv 的⽅式重命名为CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha
[root@hadoop-1 ~]# cd /usr/local/src/cdh/ [root@hadoop-1 cdh]# cp CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 manifest.json /opt/cloudera/parcel-repo/ [root@hadoop-1 cdh]# cd /opt/cloudera/parcel-repo/ [root@hadoop-1 parcel-repo]# mv CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha1 CDH-6.3.2-1.cdh6.3.2.p0.1605554-el7.parcel.sha
安装mysql驱动
[root@hadoop-1 parcel-repo]# cd /usr/local/src/cdh/ [root@hadoop-1 cdh]# mkdir -p /usr/share/java [root@hadoop-1 cdh]# cp mysql-connector-java-8.0.18.jar /usr/share/java/mysql-connector-java.jar
执行CM初始化脚本
[root@hadoop-1 ~]# /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm onaslowboattochina #释义 库 用户 密码 JAVA_HOME=/usr/java Verifying that we can write to /etc/cloudera-scm-server Creating SCM configuration file in /etc/cloudera-scm-server Executing: /usr/java/bin/java -cp /usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:/opt/cloudera/cm/schema/../lib/* com.cloudera.enterprise.dbutil.DbCommandExecutor /etc/cloudera-scm-server/db.properties com.cloudera.cmf.db. Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered via the SPI and manual loading of the driver class is generally unnecessary. [ main] DbCommandExecutor INFO Successfully connected to database. All done, your SCM database is configured correctly!
启动CMserver
[root@hadoop-1 java]# systemctl start cloudera-scm-server.service
[root@hadoop-1 java]# systemctl status cloudera-scm-server.service
● cloudera-scm-server.service - Cloudera CM Server Service
Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-server.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2022-01-21 23:14:05 CST; 4s ago
Process: 32326 ExecStartPre=/opt/cloudera/cm/bin/cm-server-pre (code=exited, status=0/SUCCESS)
Main PID: 32332 (java)
Tasks: 44
CGroup: /system.slice/cloudera-scm-server.service
└─32332 /usr/java/jdk1.8.0_202/bin/java -cp .:/usr/share/java/mysql-connector-java.jar:/usr/share/java/oracle-connector-java.jar:/usr/share/java/postgresql-connector-java.jar:lib/* -server -Dlog4...
Jan 21 23:14:05 hadoop-1 systemd[1]: Starting Cloudera CM Server Service...
Jan 21 23:14:05 hadoop-1 systemd[1]: Started Cloudera CM Server Service.
Jan 21 23:14:05 hadoop-1 cm-server[32332]: JAVA_HOME=/usr/java/jdk1.8.0_202
Jan 21 23:14:05 hadoop-1 cm-server[32332]: Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m; support was removed in 8.0
Jan 21 23:14:07 hadoop-1 cm-server[32332]: ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'org.apac...ation logging.
Jan 21 23:14:09 hadoop-1 cm-server[32332]: Loading class `com.mysql.jdbc.Driver'. This is deprecated. The new driver class is `com.mysql.cj.jdbc.Driver'. The driver is automatically registered...y unnecessary.
Hint: Some lines were ellipsized, use -l to show in full.
4.2 Cloudera Manager agent 安装（所有节点）
[root@hadoop-2 ~]# yum install -y cloudera-manager-agent
[root@hadoop-2 ~]# systemctl start cloudera-scm-agent.service
[root@hadoop-2 ~]# systemctl status cloudera-scm-agent.service
● cloudera-scm-agent.service - Cloudera Manager Agent Service
Loaded: loaded (/usr/lib/systemd/system/cloudera-scm-agent.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2022-01-21 17:03:11 CST; 12s ago
Main PID: 17500 (cmagent)
Tasks: 21
CGroup: /system.slice/cloudera-scm-agent.service
└─17500 /usr/bin/python2 /opt/cloudera/cm-agent/bin/cm agent
Jan 21 17:03:11 hadoop-2 cm[17500]: [21/Jan/2022 17:03:11 +0000] 17500 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/flood
Jan 21 17:03:11 hadoop-2 cm[17500]: [21/Jan/2022 17:03:11 +0000] 17500 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/supervisor/include
Jan 21 17:03:11 hadoop-2 cm[17500]: [21/Jan/2022 17:03:11 +0000] 17500 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/cgroups
Jan 21 17:03:11 hadoop-2 cm[17500]: [21/Jan/2022 17:03:11 +0000] 17500 MainThread agent INFO Re-using pre-existing directory: /var/run/cloudera-scm-agent/process
Jan 21 17:03:11 hadoop-2 cm[17500]: [21/Jan/2022 17:03:11 +0000] 17500 MainThread tmpfs INFO Successfully mounted tmpfs at /var/run/cloudera-scm-agent/process
Jan 21 17:03:11 hadoop-2 cm[17500]: [21/Jan/2022 17:03:11 +0000] 17500 MainThread logging INFO Logging to /var/log/cloudera-scm-agent/cloudera-scm-agent.log
Jan 21 17:03:18 hadoop-2 cm[17500]: status_server: added process group
Jan 21 17:03:18 hadoop-2 cm[17500]: flood: added process group
Jan 21 17:03:18 hadoop-2 cm[17500]: /opt/cloudera/cm-agent/lib/python2.7/site-packages/psutil/_pslinux.py:477: RuntimeWarning: dirty, writeback, mapped, commit_limit memory stats couldn't be ...d were set to 0
Jan 21 17:03:18 hadoop-2 cm[17500]: warnings.warn(msg, RuntimeWarning)
Hint: Some lines were ellipsized, use -l to show in full.
4.3 集群界面安装
输入server节点的地址加7180默认端口，进入登录界面，默认账号密码为admin;admin
登陆完成后进入欢迎界面，点击继续即可
用户许可条款和条件界面，勾选后，点击继续
测试环境使用的三个节点，目前没有使用到高级功能，点击免费版本即可，选择后点击继续
进入集群创建环境界面，点击继续
输入集群名称，自行定义，完成后点击继续
输入主机名称，完成后点击搜索，系统将查询出来的主机显示到下方，勾选需要部署的主机后点击继续按钮
进入选择存储库界面，cloudera manager agent栏选择自定义存储库，库地址为之前创建的保存rpm包的地址，CDH other software栏选择使用parcel，下方会展示你放在server节点/opt/cloudera/parcel-repo/目录下的包。确认都勾选完毕后，点击继续。
进入JDK 安装选项，如果提前每台机器都部署好了jdk，这里可以不用勾选，勾选的话，/var/www/html目录下面要有java的rpm包。根据实际情况设置，完成后点击继续。（按本文档部署无需勾选）
进入提供ssh登录凭证界面，输入密码后，修改ssh端口为实际的端口（生产环境可能端口不是22），完成后点击继续
进入 install agents 界面，此处会自动完成agents的安装，等待完成后，点击继续
进入install parcels界面，此处也是全自动化安装，等待完成即可。
inspect network performance:检测网络性能
inspect hosts：检测主机隐患
按照本文档部署，理论上此处检测都是一次性通过，如果结果如下图所示，则点击继续即可。
4.4集群界面设置 进入部署服务选择界面，根据需要添加需要部署的服务。
为勾选的服务分配主机，这个需要提前根据自己部署的服务器的性能分配好，哪些节点部署哪些服务，下图只是参考，选择完毕后点击继续。
进入数据库设置界面，将之前创建的库，用户名，密码分别配置到对应分服务下面，点击测试连接，提示都是successful后，点击继续。
进入审核更改界面，这里的datanode配置的地址的硬盘需要越大越好。其他无需变更，点击继续即可。
进入命令详细信息界面，此处为自动化部署，等待即可，出现下图界面，点击继续即可。
4.5 HDFS及YARN高可用 4. 5.1 HDFS高可用 点击集群下面的HDFS，进入HDFS管理界面
点击操作，然后选择下面的启用High Acailability(PS:这边是部署完成后写的，所以展示的是禁用High Acailability)
进入入门界面，输入需要取的集群名称，点击继续即可。
根据实际情况选择需要部署的主机，完成后点击继续
进入审核更改界面，目录配置同之前说的一样，配置在容量足够大的目录下。
进入命令详细信息界面，此处为全自动化部署，下图报错可以忽略，完成后点击继续。
根据提示完成操作，然后点击完成。
4.5.2 YARN 高可用 点击操作，然后选择下面的启用High Acailability
进入入门界面，选择好备机后，点击继续
进入命令详细信息界面，此处为全程自动化部署，等待即可，出现下图界面说明部署完成，点击完成即可。
4.6 部分报错解决 4.6.1 Erasure Coding Policy Verification Test
问题分析
纠删码策略警告是由于当前的HDFS集群DataNode节点数不满足当前所配置的纠删码策略需要的最小主机数
纠删码策略是HDFS集群提供的数据恢复的安全机制
其中
RS-3-2-1024k策略需要至少5台DataNode节点RS-6-3-1024k策略需要至少9台DataNode节点RS-10-4-1024k策略需要至少14台DataNode节点
问题解决
这里我们的集群无法满足最低的5台DataNode节点要求，所以可以选择关闭纠删策略
点击报错Erasure Coding Policy Verification Test
进入Erasure Coding Policy Verification Test界面，点击为此服务更改Fallback Etasure Coding Policy
重启HSDF生效
