hostname按表格进行修改，/etc/hosts文件添加节点 Ip+hostname
| hostname | ip | role |
|---|---|---|
| cdh-nn01 | 172.16.0.188 | HadoopMaster，CM Server，Mysql，Hue，HS2 |
| cdh-nn02 | 172.16.0.189 | Hadoop Master，CMAgent，HS2 |
| cdh-nn03 | 172.16.0.190 | Hadoop Master，CMAgent，HS2,Impala |
| cdh-dn04 | 172.16.0.193 | 数据节点，CMAgent， |
| cdh-dn05 | 172.16.0.192 | 数据节点，CMAgent， |
| cdh-dn06 | 172.16.0.191 | 数据节点，CMAgent，Impala |
172.16.0.188 cdh-nn01 namenode zkServer NodeManager JournalNode ResourceManager 172.16.0.189 cdh-nn02 namenode zkServer NodeManager JournalNode ResourceManager 172.16.0.190 cdh-nn03 namenode zkServer NodeManager JournalNode ResourceManager 172.16.0.191 cdh-dn06 datanode zkServer NodeManager JournalNode HiveServer2 Hive metastore Server Oozie Server kudu 172.16.0.192 cdh-dn05 datanode zkServer NodeManager JournalNode kudu 172.16.0.193 cdh-dn04 datanode zkServer NodeManager JournalNode kudussh免密
公私钥生成，将所有机器的公钥拷贝至cdh-nn03节点，对应好公钥编号。
[root@cdh-nn01 .ssh]# ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa 文件保存路径 /root/.ssh/ 此时有两个文件 id_rsa和id_rsa.pub [root@cdh-nn01 .ssh]# scp id_rsa.pub root@cdh-nn03:~/.ssh/id_rsa_nn01.pub [root@cdh-nn02 .ssh]# scp id_rsa.pub root@cdh-nn03:~/.ssh/id_rsa_nn02.pub [root@cdh-dn04 .ssh]# scp id_rsa.pub root@cdh-nn03:~/.ssh/id_rsa_dn04.pub [root@cdh-dn05 .ssh]# scp id_rsa.pub root@cdh-nn03:~/.ssh/id_rsa_dn05.pub [root@cdh-dn06 .ssh]# scp id_rsa.pub root@cdh-nn03:~/.ssh/id_rsa_dn06.pub [root@cdh-nn03 .ssh]# ll total 28 -rw------- 1 root root 0 Jul 21 20:47 authorized_keys -rw------- 1 root root 1675 Jul 22 15:39 id_rsa -rw-r--r-- 1 root root 395 Jul 22 15:43 id_rsa_dn04.pub -rw-r--r-- 1 root root 395 Jul 22 15:47 id_rsa_dn05.pub -rw-r--r-- 1 root root 395 Jul 22 15:47 id_rsa_dn06.pub -rw-r--r-- 1 root root 395 Jul 22 15:41 id_rsa_nn01.pub -rw-r--r-- 1 root root 395 Jul 22 15:42 id_rsa_nn02.pub -rw-r--r-- 1 root root 395 Jul 22 15:39 id_rsa.pub [root@cdh-nn03 .ssh]# cat id_rsa*.pub > authorized_keys [root@cdh-nn03 .ssh]# scp ~/.ssh/authorized_keys root@cdh-nn01:~/.ssh/authorized_keys [root@cdh-nn03 .ssh]# scp ~/.ssh/authorized_keys root@cdh-nn02:~/.ssh/authorized_keys [root@cdh-nn03 .ssh]# scp ~/.ssh/authorized_keys root@cdh-dn04:~/.ssh/authorized_keys [root@cdh-nn03 .ssh]# scp ~/.ssh/authorized_keys root@cdh-dn05:~/.ssh/authorized_keys [root@cdh-nn03 .ssh]# scp ~/.ssh/authorized_keys root@cdh-dn06:~/.ssh/authorized_keysJDK 1.8安装
解压到/usr/java （只需在nn01上执行，步骤5、6 激活环境变量和版本查看需要全节点执行）
#1.创建文件夹 /usr/java mkidr /usr/java #2.进入/opt/software目录，解压jdk压缩包到/usr/java目录 tar -zxf jdk-8u251-linux-x64.tar.gz -C /usr/java #3.修改/etc/profile添加jdk系统变量 vim etc/profile # 在文件末尾添加如下内容 # --------------- JDK --------------------------- export JAVA_HOME=/usr/java/jdk1.8.0_251 export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar #4.拷贝/usr/java目录和系统变量配置文件/etc/profile至集群其他节点 scp -r /usr/java/ root@cdh-nn02:/usr/ scp -r /usr/java/ root@cdh-nn03:/usr/ scp -r /usr/java/ root@cdh-dn04:/usr/ scp -r /usr/java/ root@cdh-dn05:/usr/ scp -r /usr/java/ root@cdh-dn06:/usr/ scp /etc/profile root@cdh-nn02:/etc/profile scp /etc/profile root@cdh-nn03:/etc/profile scp /etc/profile root@cdh-dn04:/etc/profile scp /etc/profile root@cdh-dn05:/etc/profile scp /etc/profile root@cdh-dn06:/etc/profile #5.激活系统配置 source /etc/profile #6.测试jdk,执行以下命令能看到java版本 java -version时间同步 chronycd
所有节点以cdh-nn01为主机同步 # 1.修改集群所有主机的 /etc/chrony.conf 文件 注释掉其他所有默认时间服务器 增加 server cdh-nn01 iburst ------------------------------------------------------- #server ntp.cloud.aliyuncs.com minpoll 4 maxpoll 10 iburst #server ntp.aliyun.com minpoll 4 maxpoll 10 iburst #server ntp1.aliyun.com minpoll 4 maxpoll 10 iburst #server ntp1.cloud.aliyuncs.com minpoll 4 maxpoll 10 iburst #server ntp10.cloud.aliyuncs.com minpoll 4 maxpoll 10 iburst #server ntp11.cloud.aliyuncs.com minpoll 4 maxpoll 10 iburst #server ntp12.cloud.aliyuncs.com minpoll 4 maxpoll 10 iburst #server ntp2.aliyun.com minpoll 4 maxpoll 10 iburst #server ntp2.cloud.aliyuncs.com minpoll 4 maxpoll 10 iburst #server ntp3.aliyun.com minpoll 4 maxpoll 10 iburst #server ntp3.cloud.aliyuncs.com minpoll 4 maxpoll 10 iburst #server ntp4.aliyun.com minpoll 4 maxpoll 10 iburst #server ntp4.cloud.aliyuncs.com minpoll 4 maxpoll 10 iburst #server ntp5.aliyun.com minpoll 4 maxpoll 10 iburst #server ntp5.cloud.aliyuncs.com minpoll 4 maxpoll 10 iburst #server ntp6.aliyun.com minpoll 4 maxpoll 10 iburst #server ntp6.cloud.aliyuncs.com minpoll 4 maxpoll 10 iburst #server ntp7.cloud.aliyuncs.com minpoll 4 maxpoll 10 iburst #server ntp8.cloud.aliyuncs.com minpoll 4 maxpoll 10 iburst #server ntp9.cloud.aliyuncs.com minpoll 4 maxpoll 10 iburst server cdh-nn01 iburst # 2.修改同步源主机 cdh-nn01 /etc/chrony.conf 文件，新增如下内容 ---------------------------------- # Allow NTP client access from local network. allow 172.16.0.0/23 local stratum 10 # 3.所有节点检查服务状态 ，以cdh-nn01以时间同步源 chronyc sources ------------------------------------------------- 210 Number of sources = 1 MS Name/IP address Stratum Poll Reach LastRx Last sample =============================================================================== ^* cdh-nn01 10 10 377 472m -323ns[-1947ns] +/- 6173nsMysql安装
修改密码、设置远程登录。（只需在nn01上 ）
#1.检查是否存在mysql安装包
rpm -qa|grep -i mysql
---------------------------------
mysql-community-release-el7-5.noarch
mysql-community-common-5.6.48-2.el7.x86_64
mysql-community-client-5.6.48-2.el7.x86_64
mysql-community-server-5.6.48-2.el7.x86_64
mysql-community-libs-5.6.48-2.el7.x86_64
#2.若有安装包则全部删除
rpm -ev mysql-community-client-5.6.48-2.el7.x86_64
rpm -ev mysql-community-server-5.6.48-2.el7.x86_64
rpm -ev mysql-community-release-el7-5.noarch
rpm -ev mysql-community-common-5.6.48-2.el7.x86_64
rpm -ev mysql-community-libs-5.6.48-2.el7.x86_64
#3.查询并删除mysql安装目录
#查询
find / -name mysql
----------------------------------------------
/var/lib/mysql
/var/lib/mysql/mysql
/usr/lib64/mysql
#删除
rm -rf /var/lib/mysql
rm -rf /var/lib/mysql/mysql
rm -rf /usr/lib64/mysql
#4.删除配置文件 my.cnf
rm -rf /etc/my.cnf
#5.检查是否存在mariadb
rpm -qa | grep mariadb
-----------------------------------------------
mariadb-libs-5.5.60-1.el7_5.x86_64
#6.删除mariadb文件
rpm -e --nodeps mariadb-libs-5.5.60-1.el7_5.x86_64
#7.安装wget下载器
yum install -y wget
#8.进入/opt/software目录，下载mysql
cd /opt/software
wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm
#9.安装mysql
rpm -ivh mysql-community-release-el7-5.noarch.rpm
#10.安装mysql-server
yum install mysql-server -y
#11.修改mysql用户权限
chown -R root:root /var/lib/mysql
#12.设置登录密码和远程连接用户名密码
mysql -uroot -p （密码输入直接回车）
1.use mysql;
2.用户、密码登录设置
update user set password=password('1111') where user='root';
3.当前用户对数据库的所有操作权限赋给root用户，设置远程访问密码 1111
GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' IDENTIFIED BY "1111" WITH GRANT OPTION;
4.flush privileges;
5.exit;
#13.设置数据库编码utd8
vi /etc/my.cnf
追加内容：character-set-server=utf8
#14.重启mysql服务
service mysqld restart
拷贝jdbc驱动文件到/usr/share/java目录下（只需nn01节点执行）
#1.创建目标目录/usr/share/java mkdir /usr/share/java #2.拷贝jdbc驱动文件到该目录下,更名为mysql-connector-java.jar cd /opt/software/mysql-connector-java-5.1.47.jar /usr/share/java/mysql-connector-java.jar #3.远程拷贝目录文件/usr/share/java至其他节点 scp -r /usr/share/java/ root@cdh-nn02:/usr/share scp -r /usr/share/java/ root@cdh-nn03:/usr/share scp -r /usr/share/java/ root@cdh-dn04:/usr/share scp -r /usr/share/java/ root@cdh-dn05:/usr/share scp -r /usr/share/java/ root@cdh-dn06:/usr/shareCloudera Manager 安装 cdh-nn01上操作即可
# cloudera-repos-6.2.0 parcel-6.2.0上传至cdh-nn01 /opt/software目录下 # 上传完毕后配置本地yum库 1.安装httpd和createrepo yum -y install httpd createrepo 2.启动httpd服务并设置开机自启动： systemctl start httpd systemctl enable httpd 3.进入cloudera-repos-6.2.0目录，生成RPM元数据 cd /opt/software/cloudera-repos-6.2.0 createrepo . chmod 777 -R cloudera-repos 4.然后将cloudera-repos目录移动到httpd的html目录下： mv cloudera-repos-6.2.0 /var/www/html/ 5.网页端可查看 http://cdh-nn01/cloudera-repos-6.2.0/ 如下图 # 创建cm6的repo文件 （此步骤每个节点都要执行） cd /etc/yum.repos.d vim cloudera-manager.repo ------------------------------------- [cloudera-manager] name=Cloudera Manager 6.2.0 baseurl=http://cdh-nn01/cloudera-repos-6.2.0/ gpgcheck=0 enabled=1 #退出执行 yum clean all && yum makecache #安装CM Server yum install -y cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server # 配置本地Parcel存储库 # 移动parcel下文件到/opt/cloudera/parcel-repo中 cd /opt/software/parcel-repo cp * /opt/cloudera/parcel-repo # 修改文件所有者 chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo # 进入mysql创建CDH所需要的数据课用户名和密码 # scm CREATE DATAbase scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm'; # amon CREATE DATAbase amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon'; # rman CREATE DATAbase rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman'; # hue CREATE DATAbase hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue'; # hive CREATE DATAbase metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive'; # sentry CREATE DATAbase sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry'; # navms CREATE DATAbase navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'navms'; # oozie CREATE DATAbase oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie'; # flush FLUSH PRIVILEGES; # 设置CM数据库 /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root 1111 # 修改agent连接的服务主机 vim /etc/cloudera-scm-agent/config.ini ---------------------------------------------- server_host=cdh-nn01 # 启动CM Server 服务 systemctl start cloudera-scm-server systemctl enable cloudera-scm-server # 监控服务启动日志 （启动问题看这个日志） tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log #当看到INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.日志打印出来后，说明服务启动成功，可以通过浏览器访问Cloudera Manager WEB界面了。 # 报错ParcelUpdateService:com.cloudera.parcel.components.ParcelDownloaderImpl: Unable to retrieve remote parcel repository manifest 这个问题可以忽略 # 查看服务是否启动，看到7180端口出现及成功 netstat -nultap
创建对应的数据库以及数据库用户
| 服务名 | 数据库名 | 用户名 |
|---|---|---|
| Cloudera Manager Server | scm | scm |
| Activity Monitor | amon | amon |
| Reports Manager | rman | rman |
| Hue | hue | hue |
| Hive metastore Server | metastore | hive |
| Sentry Server | sentry | sentry |
| Cloudera Navigator Audit Server | nav | nav |
| Cloudera Navigator metadata Server | navms | navms |
| Oozie | oozie | oozie |
Cloudera 服务数据包。
yum install -y cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server # cd /opt/software/parcel-repo cp * /opt/cloudera/parcel-repo chown -R cloudera-scm:cloudera-scm /opt/cloudera/parcel-repo
/opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm root 1111 vim /etc/cloudera-scm-agent/config.ini
[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-AIcrinhD-1639019891791)(C:UsersadminAppDataRoamingTyporatypora-user-imagesimage-20210724112403969.png)]
启动CM Server
systemctl start cloudera-scm-server systemctl enable cloudera-scm-server systemctl stop cloudera-scm-server systemctl disable cloudera-scm-serverCM Agnet 安装
启动CM Agent
yum install -y cloudera-manager-agent cloudera-manager-daemons systemctl start cloudera-scm-agent systemctl enable cloudera-scm-agent systemctl stop cloudera-scm-agent systemctl disable cloudera-scm-agent
监控服务启动状态
tail -f /var/log/cloudera-scm-server/cloudera-scm-server.log tail -f /var/log/cloudera-scm-agent/cloudera-scm-agent.log #当看到 Started Jetty server 说明服务启动成功，可以浏览器访问Cloudera Manager WEB界面 # CM页面 http:47.100.89.105:7180 # netstat -nultap rm -Rf /usr/share/cmf /var/lib/cloudera* /var/cache/yum/cloudera* /var/log/cloudera* /var/run/cloudera* rpm -qa | grep cloudera rpm -ev
systemctl start mysqld systemctl enable mysqld
时间同步
vim /etc/chrony.conf systemctl restart chronyd systemctl enable chronyd
输入网址http://(ip或主机名 ip为公网ip):7180 ，账号密码都为admin
进入欢迎页面，继续。
接受条款，继续。
版本选择免费版，继续。
等待页面刷新，进入集群安装页面。
自定义集群名称。
选择集群组件可选主机。
设置Cloudera服务数据包。
JDK环境集群主机已进行安装，此处可以取消勾选。
SSH免密登录已设置完毕，此处进行登录凭证设置。
Agent安装。
集群服务安装。
选择自定义安装服务，我这里选择HDFS,Hive,Hue,Impala,Kudu,Oozie,Spark,YARN,Zookeeper。
角色分配：将所要安装的服务角色分配到集群各节点。
服务组件元数据信息存储：各个组件绑定到对应的mysql数据库表进行存储。
Clouder Manager集群安装完成，可访问主节点7180端口进行访问，集群主界面如下。
可选：配置数据安装目录，数据存储到数据盘挂载目录。
至此集群安装完毕。
参考资料：
CentOS7 Cloudera Manager6 完全离线安装 CDH6 集群(https://www.cnblogs.com/songxitang/p/12344964.html)
CDH6.2安装教程（https://www.jianshu.com/p/610cce9f9026）
