CDH自动化脚本安装

一、概述

	CDH 是一个拥有集群自动化安装、中心化管理、集群监控、报警功能的一个工具（软件），使得集群的安装可以从几天的时间缩短为几个小时，运维人数也会从数十人降低到几个人，极大的提高了集群管理的效率。

cloudera官网：https://www.cloudera.com/

二、安装环境准备 1、文件准备

所有文件如下图所示：

1.1、主机信息文件

hostname.txt
文件格式：IP地址=主机名=用户名=密码

192.168.78.101=cdh-master=root=123456
192.168.78.102=cdh-slave01=root=123456
192.168.78.103=cdh-slave02=root=123456

1.2、ISO镜像文件

1.3、JDK安装文件

1.4、MYSQL安装文件

配置文件：my.cnf

# For advice on how to change settings please see
# http://dev.mysql.com/doc/refman/5.7/en/server-configuration-defaults.html

[mysqld]
#
# Remove leading # and set to the amount of RAM for the most important data
# cache in MySQL. Start at 70% of total RAM for dedicated server, else 10%.
# innodb_buffer_pool_size = 128M
#
# Remove leading # to turn on a very important data integrity option: logging
# changes to the binary log between backups.
# log_bin
#
# Remove leading # to set options mainly useful for reporting servers.
# The server defaults are faster for transactions and fast SELECTs.
# Adjust sizes as needed, experiment to find the optimal values.
# join_buffer_size = 128M
# sort_buffer_size = 2M
# read_rnd_buffer_size = 2M
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock

# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

log-error=/var/log/mysqld.log
pid-file=/var/run/mysqld/mysqld.pid

character_set_server=utf8
collation-server=utf8_general_ci
skip-character-set-client-handshake
init_connect='SET NAMES utf8'

[client]
default-character-set=utf8

1.5、CDH安装文件

下载地址：cdh软件包下载地址

cdh安装文件

cm安装文件

cdh依赖文件

1.6、CDH数据库脚本

cdh-execute.sql
备注：可根据自己实际情况更改内容

create database metastore default character set utf8;
CREATE USER 'hive'@'%' IDENTIFIED BY '123456';
GRANT ALL PRIVILEGES ON metastore. * TO 'hive'@'%';
FLUSH PRIVILEGES;

create database cm default character set utf8;
CREATE USER 'cm'@'%' IDENTIFIED BY '123456';
GRANT ALL PRIVILEGES ON cm. * TO 'cm'@'%';
FLUSH PRIVILEGES;

create database rm default character set utf8;
CREATE USER 'rm'@'%' IDENTIFIED BY '123456';
GRANT ALL PRIVILEGES ON rm. * TO 'rm'@'%';
FLUSH PRIVILEGES;

create database hue default character set utf8;
CREATE USER 'hue'@'%' IDENTIFIED BY '123456';
GRANT ALL PRIVILEGES ON hue. * TO 'hue'@'%';
FLUSH PRIVILEGES;

create database ranger default character set utf8;
CREATE USER 'rangeradmin'@'%' IDENTIFIED BY '123456';
GRANT ALL PRIVILEGES ON ranger. * TO 'rangeradmin'@'%';
FLUSH PRIVILEGES;

2、脚本准备

下载地址：cdh完整版自动化安装脚本

cdh部分安装脚本
installCDHscript.sh

#!/bin/bash
############################################################################################################
##########################################上传所有文件到/tmp/local目录#########################################
############################################################################################################

#if [ $# -ne 3 ]; then
#	echo "Usage:"
#	echo "$0  linuxuser linuxpasswd hostsFile"
#fi
#
#user=root
#master=192.168.78.101
#linuxuser=$1
#linuxpasswd=$2
#hostfilename=$3

master_ip=$(cat /tmp/local/hostname.txt | awk -F '=' 'NR==1 {print $1}')
master_hostname=$(cat /tmp/local/hostname.txt | awk -F '=' 'NR==1 {print $2}')
all_ip=$(cat /tmp/local/hostname.txt | awk -F '=' '{print $1}')
hosts=$(cat /tmp/local/hostname.txt)
user=$(cat /tmp/local/hostname.txt | awk -F '=' ' NR==1 {print $3}')
password=$(cat /tmp/local/hostname.txt | awk -F '=' 'NR==1 {print $4}')

mysqlPort="3306"
mysqlUsername="root"
mysqlNewPassword="123456"
mysqlDB="mysql"

echo -e "===================================开始检查脚本所需所有文件==================================="
isofile=$(ls /tmp/local/iso/*.iso 2>/dev/null | wc -l)
hostsFile=$(ls /tmp/local/hostname.txt 2>/dev/null | wc -l)
jdkfile=$(ls /tmp/local/jdk/jdk*.gz 2>/dev/null | wc -l)
mysqlfile=$(ls /tmp/local/mysql/mysql-community* 2>/dev/null | wc -l)
mysqlconnectorfile=$(ls /tmp/local/mysql/mysql-connector* 2>/dev/null | wc -l)
mycnffile=$(ls /tmp/local/mysql/my.cnf 2>/dev/null | wc -l)

if [[ $isofile != 0 ]] && [[ $hostsFile != 0 ]] && [[ $jdkfile != 0 ]] && [[ $mysqlfile != 0 ]] && 
   [[ $mysqlconnectorfile != 0 ]];then
     echo -e "33[32m已检测到脚本所需所有文件33[0m"
     echo -e "$(ls /tmp/local/*)"
else
     echo -e "33[31m请上传脚本所需所有文件33[0m"
	 echo -e "$(ls /tmp/local/*)"
     exit
fi


#####制作iso本地yum源
CreateYum()
{
echo -e "nnn"

mount_dir=/mnt/iso
iso=$(ls /tmp/local/iso/*.iso)

echo -e "===================================开始制作iso本地yum源==================================="
read -n1 -p "是否开始制作iso本地yum源？[y/N]：" answer
case $answer in
Y | y)
echo -e "n===================================准备挂载iso镜像文件==================================="
if [ ! -d "$mount_dir" ]; then
   mkdir $mount_dir
   mount -o loop $iso $mount_dir
   echo -e "33[32miso镜像文件已挂载33[0m"
else
   mount -o loop $iso $mount_dir
   echo -e "33[32miso镜像文件已挂载33[0m"
fi

echo -e "===================================开始安装和启动httpd==================================="
cd /mnt/iso/Packages && yum -y localinstall httpd-2* httpd-tools-2* apr-1* apr-util-1* mailcap-2*
systemctl enable httpd --now
echo -e "33[32mhttpd已完成安装和启动33[0m"

echo -e "===================================开始安装createrepo==================================="
cd /mnt/iso/Packages && yum -y localinstall createrepo* deltarpm* libxml2-python* python-deltarpm*
echo -e "33[32mcreaterepo已完成安装33[0m"

echo -e "===================================备份系统自带yum源==================================="
if [ ! -d "/etc/yum.repos.d/bak_repo" ]; then
   mkdir /etc/yum.repos.d/bak_repo && mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/bak_repo
   echo -e "33[32m系统自带yum源已备份33[0m"
else
   mv /etc/yum.repos.d/*.repo /etc/yum.repos.d/bak_repo
   echo -e "33[32m系统自带yum源已备份33[0m"
fi

echo -e "===================================准备制作iso本地yum源==================================="

rm -rf /var/www/html/iso
mkdir /var/www/html/iso && cp -r /mnt/iso/Packages /var/www/html/iso
cd /var/www/html/iso && createrepo .

cat <
echo -e "nnn"
echo -e "===================================开始配置ssh免密==================================="
echo -e "配置免密机器ip：n33[32m$all_ip33[0m"

echo -e "===================================开始检查是否安装expect==================================="
res=$(rpm -qa|grep expect |wc -l)
if [ $res -eq 0 ];then
	yum -y install expect
    echo -e "33[32mexpect已完成安装33[0m"
else
    echo -e "33[32mexpect已安装33[0m"
fi

echo -e "===================================开始配置ssh免密==================================="
if [ ! -f ~/.ssh/id_rsa ];then
	echo "ssh-keygen ..."
	ssh-keygen -P "" -f ~/.ssh/id_rsa
fi

for ip in $all_ip
do
    echo -e "33[34m===================================copy id_rsa.pub to $ip===================================33[0m"
	ping -c1 -W1 $ip &>/dev/null
	if [ $? -eq 0 ];then
		/usr/bin/expect <<-EOF
		set timeout -1
		spawn ssh-copy-id $ip
		expect {
			"yes/no" { send "yesr";exp_continue }
			"password:" { send "$passwordr" }
		}
		expect eof
		EOF
	fi
done
echo -e "===================================修改ssh配置文件ssh_config参数==================================="
sed -i 's/#   StrictHostKeyChecking ask/StrictHostKeyChecking no/' /etc/ssh/ssh_config
systemctl restart sshd
echo -e "33[32mssh免密已配置完成33[0m"

}


#####配置主机名
ChangeHostname()
{
echo -e "nnn"
echo -e "===================================开始修改主机名=================================="
for line in $hosts
do   
	ip=$(echo $line | awk -F'=' '{print $1}')
	hostname=$(echo $line | awk -F'=' '{print $2}')
	originhostsname=`ssh $ip hostnamectl --static`
    echo -e "33[34m===================================$ip===================================33[0m"
	#echo $line
	#echo $ip
	echo "hostname=$hostname"
	echo "originhostsname=$originhostsname"
	if [ ! ${hostname} == ${originhostsname} ];then

		ssh $ip "hostnamectl set-hostname $hostname"
	    echo -e "33[32m$ip 主机名已修改为$hostname33[0m"
	else
	    echo -e "33[31m$ip 不需要修改主机名33[0m"
	fi
	echo "$ip $hostname" >> /etc/hosts
done    

echo -e "===================================开始分发hosts文件==================================="
for line in $hosts
do   
	ip=$(echo $line | awk -F'=' '{print $1}')
	hostname=$(echo $line | awk -F'=' '{print $2}')
    echo -e "33[34m===================================$ip===================================33[0m"
	scp /etc/hosts root@$ip:/etc/
	ssh $ip "rm -rf /etc/yum.repos.d/CentOS* "
done   

echo -e "===================================开始分发repo文件==================================="
for line in $hosts
do   
	ip=$(echo $line | awk -F'=' '{print $1}')
	hostname=$(echo $line | awk -F'=' '{print $2}')
	
	echo -e "33[34m===================================$ip===================================33[0m"	
    if [ X${ip} == X${master_ip} ];then
		scp /etc/yum.repos.d/iso.repo  root@$ip:/etc/yum.repos.d   
		ssh $ip "yum clean all && yum makecache"
    else
		ssh $ip "rm -f /etc/yum.repos.d/*.repo"
		scp /etc/yum.repos.d/iso.repo  root@$ip:/etc/yum.repos.d
		ssh $ip "yum clean all && yum makecache"
    fi	
done 

}


######配置系统环境参数
ModifySysEnvPar()
{
echo -e "nnn"
echo -e "===================================开始配置系统环境参数=================================="
for line in $hosts
do   
	ip=$(echo $line | awk -F'=' '{print $1}')
	hostname=$(echo $line | awk -F'=' '{print $2}')
    echo -e "33[34m===================================$ip===================================33[0m"
	
	echo -e "===================================关闭防火墙==================================="
    ssh $ip "systemctl stop firewalld"
	ssh $ip "systemctl disable firewalld"
	ssh $ip "systemctl status firewalld"
    
    echo -e "===================================关闭SELINUX==================================="
    ssh $ip "setenforce 0"
    ssh $ip "sed '-i' 's/^SELINUX=enforcing$/SELINUX=disabled/' /etc/selinux/config"
  
    echo -e "===================================设置文件句柄数==================================="
    ssh $ip "rm -f /etc/security/limits.d/*"
	ssh $ip "echo '* soft nofile 200000' >> /etc/security/limits.conf"
    ssh $ip "echo '* hard nofile 200000' >> /etc/security/limits.conf"
    ssh $ip "echo '* soft nproc 262144' >> /etc/security/limits.conf"
	ssh $ip "echo '* hard nproc 262144' >> /etc/security/limits.conf"
	ssh $ip "echo '* soft memlock unlimited' >> /etc/security/limits.conf"
	ssh $ip "echo '* hard memlock unlimited' >> /etc/security/limits.conf"

	echo -e "===================================关闭透明大页面==================================="
	ssh $ip "echo never > /sys/kernel/mm/transparent_hugepage/defrag"
    ssh $ip "echo never > /sys/kernel/mm/transparent_hugepage/enabled"	
    ssh $ip "echo 'echo never > /sys/kernel/mm/transparent_hugepage/defrag' >> /etc/rc.d/rc.local"
    ssh $ip "echo 'echo never > /sys/kernel/mm/transparent_hugepage/enabled' >> /etc/rc.d/rc.local"
    ssh $ip "echo -e 'nif test -f /sys/kernel/mm/transparent_hugepage/enabled; then echo never > /sys/kernel/mm/transparent_hugepage/enabled fi if test -f /sys/kernel/mm/transparent_hugepage/defrag; then echo never > /sys/kernel/mm/transparent_hugepage/defrag fi' >>/etc/rc.d/rc.local"

    echo -e "===================================设置交换分区==================================="
    ssh $ip "echo vm.swappiness = 10 >> /etc/sysctl.conf"
    ssh $ip "sysctl vm.swappiness=10"
    ssh $ip "sysctl '-p'"
	
done	
echo -e "33[32m配置系统环境参数已完成33[0m"
}


#####调用函数
main()
{

CreateYum
CreateSSH
ChangeHostname
ModifySysEnvPar

}

main

三、安装CDH 1、执行脚本安装

chmod a+x installCDHscript.sh
sh installCDHscript.sh

2、web页面安装

1）登录：http://192.168.78.101:7180 密码：admin/admin

2）如果没有license，则选择试用60天，点击继续

3）点击继续

4）输入集群名称，可以使用默认的Cluster 1，点击继续

5）输入主机IP或者名称，点击搜索找到主机，点击继续

6）选择存储库

选择自定义存储库，输入cm的http地址

Other Software中选择Parcel Repositories & Network Settings，

点击更多选项，删除其它所有地址，输入cdh的http地址，点击Save & Verify Configuration，验证通过，点击关闭

7）选择Manually manage JDK，点击继续

备注：如果之前未安装JDK，则选择Install a Cloudera-provided version of OpenJDK
8）配置ssh账号密码，确保所有机器的root密码都一致，然后输入root的密码即可，点击继续

9）安装完Agents，点击继续

10）安装完Parcels（大约需要10~20分钟），点击继续

11）点击Inspect Network Performance和Inspect Hosts检查主机，检查完点击完成

12）选择需要安装的服务，根据需要选择，这里选择自定义服务，点击继续

13）进入集群角色分配，点击继续

14）测试数据库连接，测试完成，点击继续

15）点击继续

16）审核更改，点击继续

17）安装完成，点击继续

18）点击完成

19）查看集群主页
至此CDH安装完成！！！！！！

CDH自动化脚本安装

Java相关栏目本月热门文章