目录

一.Shell脚本中一些常用参数的解释

二.Shell脚本

1.SSH免密登录配置

2.集群分发文件脚本

3.显示当前所有java进程及其pid脚本

4.Hadoop集群启动停止脚本

5.Zookeeper集群启动停止脚本

6.Kafka集群启动停止脚本

7.Maxwell启动停止脚本 ​​​​​​​

一.Shell脚本中一些常用参数的解释

    ' ' 和 " " 都是表示字符串的意思
    ' ' 不会解析变量,直接当作字符串使用
    " " 会解析变量,并且和其他进行拼接使用  
    $n
        功能描述：n为数字, $0 代表该脚本名称, $1-$9 代表第一到第九个参数,十以上的参数需要用大括号包含,如 ${10}
    $#
        功能描述：获取所有输入参数个数，常用于循环,判断参数的个数是否正确以及加强脚本的健壮性
    $*
        功能描述：这个变量代表命令行中所有的参数, $* 把所有的参数看成一个整体
    $@
        功能描述：这个变量也代表命令行中所有的参数,不过 $@ 把每个参数区分对待

二.Shell脚本

        注意事项:

                (1)为方便使用脚本,建议编写后将脚本放在用户bin目录下(或者自定义文件夹,将其绝对路径添加至环境变量PATH中)。

                (2)hadoop102,hadoop103,hadoop104为集群中主机名称(hostname),此处举例三台主机,使用时按需更改为对应的主机名称即可。

                (3)以下脚本使用前均需配置SSH免密登录。

1.SSH免密登录配置

(1)免密登录原理

(2)生成公钥和私钥

[local@hadoop102 ~]$ cd ~/.ssh
[local@hadoop102 .ssh]$ ssh-keygen -t rsa

  然后敲三个Enter,就会生成两个文件id_rsa(私钥)、id_rsa.pub(公钥)。

[local@hadoop102 .ssh]$ ll
-rw-------. 1 local local 1679 2月  16 19:18 id_rsa
-rw-r--r--. 1 local local  399 2月  16 19:18 id_rsa.pub

(3)将公钥拷贝到要免密登录的目标机器上

[local@hadoop102 .ssh]$ ssh-copy-id hadoop102
[local@hadoop102 .ssh]$ ssh-copy-id hadoop103
[local@hadoop102 .ssh]$ ssh-copy-id hadoop104

   之后还需在hadoop103、hadoop104上用同样的步骤配置一下免密登录到hadoop102、hadoop103、hadoop104服务器上。

2.集群分发文件脚本

  vim xsync

#!/bin/bash
#1. 判断参数个数
if [ $# -lt 1 ]
then
  echo "No Args Input..."
  exit;
fi
#2. 遍历集群所有机器
for host in hadoop102 hadoop103 hadoop104
do
  echo ====================  $host  ====================
  #3. 遍历所有目录，挨个发送
  for file in $@
  do
    #4 判断文件是否存在
    if [ -e $file ]
    then
      #5. 获取父目录
      pdir=$(cd -P $(dirname $file); pwd)
      #6. 获取当前文件的名称
      fname=$(basename $file)
      ssh $host "mkdir -p $pdir"
      rsync -av $pdir/$fname $host:$pdir
    else
      echo $file does not exists!
    fi
  done
done

3.显示当前所有java进程及其pid脚本

  vim jpsall      jpsall为脚本名称,根据自己喜好定义即可

#!/bin/bash
for i in hadoop102 hadoop103 hadoop104;

do
    echo "==========$i=========="
    ssh $i "jps $@ | grep -v Jps"
done

  执行脚本结果如下:

[local@hadoop102 ~]$ jpsall
========= hadoop102 ==========
1314 QuorumPeerMain
27811 Kafka
========= hadoop103 ==========
1361 QuorumPeerMain
6933 Kafka
7191 ConsoleConsumer
========= hadoop104 ==========
2741 Kafka
1382 QuorumPeerMain

4.Hadoop集群启动停止脚本

          三台主机集群配置部署如下:

	hadoop102	hadoop103	hadoop104
HDFS	NameNode DataNode	DataNode	SecondaryNameNode DataNode
YARN	NodeManager	ResourceManager NodeManager	NodeManager

   vim myhadoop.sh

#!/bin/bash
HADOOP_HOME=/opt/module/hadoop-3.1.3
if [ $# -lt 1 ]
then
    echo "No Args Input..."
    exit;
#未输入参数时执行
fi
case $1 in
"start")
    echo "==========启动hadoop集群=========="
  
    echo "--------- 启动 hdfs ---------"
    ssh hadoop102 "$HADOOP_HOME/sbin/start-dfs.sh"
  
    echo "--------- 启动 yarn ---------"
    ssh hadoop103 "$HADOOP_HOME/sbin/start-yarn.sh"
  
    echo "--------- 启动 historyserver ---------"
    ssh hadoop102 "$HADOOP_HOME/bin/mapred --daemon start historyserver"
;;
"stop")
    echo "==========关闭hadoop集群=========="

    echo "--------- 关闭 historyserver ---------"
    ssh hadoop102 "$HADOOP_HOME/bin/mapred --daemon stop historyserver"

    echo "--------- 关闭 yarn ---------"
    ssh hadoop103 "$HADOOP_HOME/sbin/stop-yarn.sh"

    echo "--------- 关闭 hdfs ---------"
    ssh hadoop102 "$HADOOP_HOME/sbin/stop-dfs.sh"
;;
*)
    echo "Input Args Error..."
#当输入的参数不为start或stop时执行
;;
esac

   /opt/module/hadoop-3.1.3为hadoop安装目录的绝对路径,使用时更改为对应的目录即可。

  执行脚本结果如下:

[local@hadoop102 ~]$ myhadoop.sh start
 =================== 启动 hadoop集群 ===================
 --------------- 启动 hdfs ---------------
Starting namenodes on [hadoop102]
Starting datanodes
hadoop102: datanode is running as process 1389.  Stop it first.
Starting secondary namenodes [hadoop104]
 --------------- 启动 yarn ---------------
Starting resourcemanager
Starting nodemanagers
hadoop103: nodemanager is running as process 1396.  Stop it first.
 --------------- 启动 historyserver ---------------

[local@hadoop102 ~]$ jpsall
========= hadoop102 ==========
1795 NodeManager
1308 NameNode
1965 JobHistoryServer
1471 DataNode
========= hadoop103 ==========
1459 NodeManager
1139 DataNode
1319 ResourceManager
========= hadoop104 ==========
1142 DataNode
1340 NodeManager
1215 SecondaryNameNode

[local@hadoop102 ~]$ myhadoop.sh stop
 =================== 关闭 hadoop集群 ===================
 --------------- 关闭 historyserver ---------------
 --------------- 关闭 yarn ---------------
Stopping nodemanagers
Stopping resourcemanager
 --------------- 关闭 hdfs ---------------
Stopping namenodes on [hadoop102]
Stopping datanodes
Stopping secondary namenodes [hadoop104]

[local@hadoop102 ~]$ jpsall
========= hadoop102 ==========
========= hadoop103 ==========
========= hadoop104 ==========

5.Zookeeper集群启动停止脚本

                         在hadoop102、hadoop103和hadoop104三个节点上部署Zookeeper。

	服务器hadoop102	服务器hadoop103	服务器hadoop104
Zookeeper	Zookeeper	Zookeeper	Zookeeper

  vim zookeeper.sh

#!/bin/bash
ZOOKEEPER_HOME=/opt/module/zookeeper-3.5.7
if [ $# -lt 1 ]
then
    echo "No Args Input..."
    exit;
#未输入参数时执行
fi
case $1 in
"start")
    for i in hadoop102 hadoop103 hadoop104 
    do
        echo "---------- zookeeper $i 启动 ------------"
        ssh $i "$ZOOKEEPER_HOME/bin/zkServer.sh start"
    done
;;
"stop")
    for i in hadoop102 hadoop103 hadoop104 
    do
        echo "---------- zookeeper $i 停止 ------------"
        ssh $i "$ZOOKEEPER_HOME/bin/zkServer.sh stop"
    done
;;
"status")
    for i in hadoop102 hadoop103 hadoop104 
    do
        echo "---------- zookeeper $i 状态 ------------"
        ssh $i "$ZOOKEEPER_HOME/bin/zkServer.sh status"
    done
;;
*)
    echo "Input Args Error..."
#当输入的参数不为start或stop或status时执行
;;
esac

   /opt/module/zookeeper-3.5.7为zookeeper安装目录的绝对路径,使用时更改为对应的目录即可。 

  执行脚本结果如下:

[local@hadoop102 ~]$ zookeeper.sh start
========== zookeeper hadoop102 启动 ============
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
========== zookeeper hadoop103 启动 ============
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
========== zookeeper hadoop104 启动 ============
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

[local@hadoop102 ~]$ jpsall
========= hadoop102 ==========
2960 QuorumPeerMain
========= hadoop103 ==========
2563 QuorumPeerMain
========= hadoop104 ==========
1963 QuorumPeerMain

[local@hadoop102 ~]$ zookeeper.sh status
========== zookeeper hadoop102 状态 ============
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower
========== zookeeper hadoop103 状态 ============
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: leader
========== zookeeper hadoop104 状态 ============
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: follower

[local@hadoop102 ~]$ zookeeper.sh stop
========== zookeeper hadoop102 停止 ============
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED
========== zookeeper hadoop103 停止 ============
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED
========== zookeeper hadoop104 停止 ============
ZooKeeper JMX enabled by default
Using config: /opt/module/zookeeper-3.5.7/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED

[local@hadoop102 ~]$ jpsall
========= hadoop102 ==========
========= hadoop103 ==========
========= hadoop104 ==========

6.Kafka集群启动停止脚本

                        在hadoop102、hadoop103和hadoop104三个节点上部署Kafka。

hadoop102	hadoop103	hadoop104
zookeeper	zookeeper	zookeeper
Kafka	Kafka	Kafka

   vim kafka.sh

#!/bin/bash
KAFKA_HOME=/opt/module/kafka
if [ $# -lt 1 ]
then
    echo "No Args Input..."
    exit;
#未输入参数时执行
fi
case $1 in
"start")
    for i in hadoop102 hadoop103 hadoop104
    do
        echo "--------启动 $i Kafka-------"
        ssh $i "$KAFKA_HOME/bin/kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties"
    done
;;
"stop")
    for i in hadoop102 hadoop103 hadoop104
    do
        echo "--------停止 $i Kafka-------"
        ssh $i "$KAFKA_HOME/bin/kafka-server-stop.sh"
    done
;;
*)
    echo "Input Args Error..."
#当输入的参数不为start或stop时执行
;;
esac

  /opt/module/kafka为kafka安装目录的绝对路径,使用时更改为对应的目录即可。

  Kafka集群启动之前必须先启动zookeeper集群 ! ! !

  执行脚本结果如下:

[local@hadoop102 ~]$ jpsall
========= hadoop102 ==========
2960 QuorumPeerMain
========= hadoop103 ==========
2563 QuorumPeerMain
========= hadoop104 ==========
1963 QuorumPeerMain

[local@hadoop102 ~]$ kafka.sh start
 ========启动 hadoop102 Kafka=======
 ========启动 hadoop103 Kafka=======
 ========启动 hadoop104 Kafka=======

[local@hadoop102 ~]$ jpsall
========= hadoop102 ==========
2960 QuorumPeerMain
3413 Kafka
========= hadoop103 ==========
2563 QuorumPeerMain
3037 Kafka
========= hadoop104 ==========
2437 Kafka
1963 QuorumPeerMain

[local@hadoop102 ~]$ kafka.sh stop
 ========停止 hadoop102 Kafka=======
 ========停止 hadoop103 Kafka=======
 ========停止 hadoop104 Kafka=======

[local@hadoop102 ~]$ jpsall
========= hadoop102 ==========
2960 QuorumPeerMain
========= hadoop103 ==========
2563 QuorumPeerMain
========= hadoop104 ==========
1963 QuorumPeerMain

    注意：停止Kafka集群时，一定要等Kafka所有节点进程全部停止后再停止Zookeeper集群。因为Zookeeper集群当中记录着Kafka集群相关信息，Zookeeper集群一旦先停止，Kafka集群就没有办法再获取停止进程的信息，只能手动杀死Kafka进程了。

7.Maxwell启动停止脚本 

        若Maxwell发送数据的目的地为Kafka集群，则需要先确保Kafka集群为启动状态。

vim maxwell.sh

#!/bin/bash
MAXWELL_HOME=/opt/module/maxwell

maxwell_status(){
    result = `ps -ef | grep maxwell | grep -v grep | wc -l`
    return $result
}

maxwell_start(){
    maxwell_status
    if [[ $? -lt 1 ]]
    then
        echo "正在启动Maxwell..."
        $MAXWELL_HOME/bin/maxwell --config $MAXWELL_HOME/config.properties --daemon
    else
        echo "Maxwell已经启动,无需重复启动"
    fi
}

maxwell_stop(){
    maxwell_status
    if [[ $? -eq 1 ]]
    then
        echo "正在关闭Maxwell..."
        ps -ef | grep maxwell | grep -v grep | awk '{print $2}' | xargs kill -9
    else
        echo "Maxwell尚未启动,无需关闭"
    fi
}

maxwell_restart(){
    maxwell_stop
    maxwell_start
}
case $1 in
"start")
    maxwell_start
;;
"stop")
    maxwell_stop
;;
"restart")
    maxwell_restart
;;
esac

   MAXWELL_HOME=/opt/module/maxwell 为Maxwell安装目录,根据自己的目录更改此变量值。

    执行结果如下:

[local@hadoop102 ~]$ maxwell.sh start
正在启动Maxwell...
Redirecting STDOUT to /opt/module/maxwell/bin/../logs/MaxwellDaemon.out
Using kafka version: 1.0.0
[local@hadoop102 ~]$ maxwell.sh start
Maxwell已经启动,无需重复启动

[local@hadoop102 ~]$ jps
1875 JobHistoryServer
1268 NameNode
2662 QuorumPeerMain
7225 Jps
1706 NodeManager
5371 Kafka
7182 Maxwell
1390 DataNode

[local@hadoop102 ~]$ maxwell.sh stop
正在关闭Maxwell...
[local@hadoop102 ~]$ maxwell.sh stop
Maxwell尚未启动,无需关闭

[local@hadoop102 ~]$ jps
1875 JobHistoryServer
1268 NameNode
2662 QuorumPeerMain
7225 Jps
1706 NodeManager
7182 Maxwell
1390 DataNode