ZooKeeper && Kafka
环境说明
| Hostname | IP | OS | CPU | Memory | Disk |
|---|
| kc00.iam.infra.lab.ecnl | 10.0.1.10 | CentOS8 | Intel® Xeon® Gold 6212U CPU @2.40GHZ 36MB | 8G | 160G |
| kc01.iam.infra.lab.ecnl | 10.0.1.11 | CentOS8 | Intel® Xeon® Gold 6212U CPU @2.40GHZ 36MB | 8G | 160G |
| db00.iam.infra.lab.ecnl | 10.0.1.12 | CentOS8 | Intel® Xeon® Gold 6212U CPU @2.40GHZ 36MB | 8G | 160G |
软件说明
| App | Version |
|---|
| openjdk | 1.8.0_302 |
| zookeeper | 3.6.3 |
| kafka | 3.0.0 |
一、搭建zookeeper集群
1、kc00安装zookeeper
(1) 创建用户
#创建zk用户，并进入到该用户
$ useradd zk
$ passwd zk
------------------------------------------------------------------------------------------------------------
Changing password for user zk.
New password: #Ergo2021!
Retype new password: #Ergo2021!
passwd: all authentication tokens updated successfully.
------------------------------------------------------------------------------------------------------------
$ su - zk
(2) 准备zookeeper运行环境
下载JDK8
$ curl -OL https://github.com/adoptium/temurin8-binaries/releases/download/jdk8u302-b08/OpenJDK8U-jdk_x64_linux_hotspot_8u302b08.tar.gz
------------------------------------------------------------------------------------------------------------
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 655 100 655 0 0 873 0 --:--:-- --:--:-- --:--:-- 872
100 98.1M 100 98.1M 0 0 21.3M 0 0:00:04 0:00:04 --:--:-- 27.3M
安装JDK8
$ tar xvf OpenJDK8U-jdk_x64_linux_hotspot_8u302b08.tar.gz
#配置环境变量
$ vi ~/.bash_profile
------------------------------------------------------------------------------------------------------------
#在最后加入
export PATH=$PATH:/home/zk/jdk8u302-b08/bin
------------------------------------------------------------------------------------------------------------
$ source ~/.bash_profile
$ java -version
------------------------------------------------------------------------------------------------------------
openjdk version “1.8.0_302”
OpenJDK Runtime Environment （Temurin）（build 1.8.0_302-b08）
OpenJDK 64-Bit Server VM（Temurin）（build 1.8.0_302-b08,mixed mode）
(3) 安装并运行zookeeper
下载zookeeper-3.6.3
$ curl -OL https://dlcdn.apache.org/zookeeper/zookeeper-3.6.3/apache-zookeeper-3.6.3-bin.tar.gz
------------------------------------------------------------------------------------------------------------ % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11.9M 100 11.9M 0 0 13.1M 0 --:--:-- --:--:-- --:--:-- 13.0M
解压并配置
$ tar xvf apache-zookeeper-3.6.3-bin.tar.gz
$ cd apache-zookeeper-3.6.3-bin
$ vi conf/zoo.cfg
------------------------------------------------------------------------------------------------------------
tickTime=2000
dataDir=/home/zk/data/zookeeper
clientPort=2181
initLimit=5
syncLimit=2
server.1=10.0.1.10:2888:3888
server.2=10.0.1.11:2888:3888
server.3=10.0.1.12:2888:3888
------------------------------------------------------------------------------------------------------------
$ mkdir -p /home/zk/data/zookeeper
$ echo 1 > /home/zk/data/zookeeper/myid
运行
$ bin/zkServer.sh start
------------------------------------------------------------------------------------------------------------
/home/zk/jdk8u302-b08/bin/java
ZooKeeper JMX enabled by default
Using config: /home/zk/apache-zookeeper-3.6.3-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
2、kc01安装zookeeper
(1) 创建用户
#创建zk用户，并进入到该用户
$ useradd zk
$ passwd zk
------------------------------------------------------------------------------------------------------------
Changing password for user zk.
New password: #Ergo2021!
Retype new password: #Ergo2021!
passwd: all authentication tokens updated successfully.
------------------------------------------------------------------------------------------------------------
$ su - zk
(2) 准备zookeeper运行环境
下载JDK8
$ curl -OL https://github.com/adoptium/temurin8-binaries/releases/download/jdk8u302-b08/OpenJDK8U-jdk_x64_linux_hotspot_8u302b08.tar.gz
------------------------------------------------------------------------------------------------------------
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 655 100 655 0 0 873 0 --:--:-- --:--:-- --:--:-- 872
100 98.1M 100 98.1M 0 0 21.3M 0 0:00:04 0:00:04 --:--:-- 27.3M
安装JDK8
$ tar xvf OpenJDK8U-jdk_x64_linux_hotspot_8u302b08.tar.gz
#配置环境变量
$ vi ~/.bash_profile
------------------------------------------------------------------------------------------------------------
#在最后加入
export PATH=$PATH:/home/zk/jdk8u302-b08/bin
------------------------------------------------------------------------------------------------------------
$ source ~/.bash_profile
$ java -version
------------------------------------------------------------------------------------------------------------
openjdk version “1.8.0_302”
OpenJDK Runtime Environment （Temurin）（build 1.8.0_302-b08）
OpenJDK 64-Bit Server VM（Temurin）（build 1.8.0_302-b08,mixed mode）
(3) 安装并运行zookeeper
下载zookeeper-3.6.3
$ curl -OL https://dlcdn.apache.org/zookeeper/zookeeper-3.6.3/apache-zookeeper-3.6.3-bin.tar.gz
------------------------------------------------------------------------------------------------------------ % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11.9M 100 11.9M 0 0 13.1M 0 --:--:-- --:--:-- --:--:-- 13.0M
解压并配置
$ tar xvf apache-zookeeper-3.6.3-bin.tar.gz
$ cd apache-zookeeper-3.6.3-bin
$ vi conf/zoo.cfg
------------------------------------------------------------------------------------------------------------
tickTime=2000
dataDir=/home/zk/data/zookeeper
clientPort=2181
initLimit=5
syncLimit=2
server.1=10.0.1.10:2888:3888
server.2=10.0.1.11:2888:3888
server.3=10.0.1.12:2888:3888
------------------------------------------------------------------------------------------------------------
$ mkdir -p /home/zk/data/zookeeper
$ echo 2 > /home/zk/data/zookeeper/myid
运行
$ bin/zkServer.sh start
------------------------------------------------------------------------------------------------------------
/home/zk/jdk8u302-b08/bin/java
ZooKeeper JMX enabled by default
Using config: /home/zk/apache-zookeeper-3.6.3-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
3、db00安装zookeeper
(1) 创建用户
#创建zk用户，并进入到该用户
$ useradd zk
$ passwd zk
------------------------------------------------------------------------------------------------------------
Changing password for user zk.
New password: #Ergo2021!
Retype new password: #Ergo2021!
passwd: all authentication tokens updated successfully.
------------------------------------------------------------------------------------------------------------
$ su - zk
(2) 准备zookeeper运行环境
下载JDK8
$ curl -OL https://github.com/adoptium/temurin8-binaries/releases/download/jdk8u302-b08/OpenJDK8U-jdk_x64_linux_hotspot_8u302b08.tar.gz
------------------------------------------------------------------------------------------------------------
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 655 100 655 0 0 873 0 --:--:-- --:--:-- --:--:-- 872
100 98.1M 100 98.1M 0 0 21.3M 0 0:00:04 0:00:04 --:--:-- 27.3M
安装JDK8
$ tar xvf OpenJDK8U-jdk_x64_linux_hotspot_8u302b08.tar.gz
#配置环境变量
$ vi ~/.bash_profile
------------------------------------------------------------------------------------------------------------
#在最后加入
export PATH=$PATH:/home/zk/jdk8u302-b08/bin
------------------------------------------------------------------------------------------------------------
$ source ~/.bash_profile
$ java -version
------------------------------------------------------------------------------------------------------------
openjdk version “1.8.0_302”
OpenJDK Runtime Environment （Temurin）（build 1.8.0_302-b08）
OpenJDK 64-Bit Server VM（Temurin）（build 1.8.0_302-b08,mixed mode）
(3) 安装并运行zookeeper
下载zookeeper-3.6.3
$ curl -OL https://dlcdn.apache.org/zookeeper/zookeeper-3.6.3/apache-zookeeper-3.6.3-bin.tar.gz
------------------------------------------------------------------------------------------------------------ % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 11.9M 100 11.9M 0 0 13.1M 0 --:--:-- --:--:-- --:--:-- 13.0M
解压并配置
$ tar xvf apache-zookeeper-3.6.3-bin.tar.gz
$ cd apache-zookeeper-3.6.3-bin
$ vi conf/zoo.cfg
------------------------------------------------------------------------------------------------------------
tickTime=2000
dataDir=/home/zk/data/zookeeper
clientPort=2181
initLimit=5
syncLimit=2
server.1=10.0.1.10:2888:3888
server.2=10.0.1.11:2888:3888
server.3=10.0.1.12:2888:3888
------------------------------------------------------------------------------------------------------------
$ mkdir -p /home/zk/data/zookeeper
$ echo 3 > /home/zk/data/zookeeper/myid
运行
$ bin/zkServer.sh start
------------------------------------------------------------------------------------------------------------
/home/zk/jdk8u302-b08/bin/java
ZooKeeper JMX enabled by default
Using config: /home/zk/apache-zookeeper-3.6.3-bin/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
二、搭建kafka集群
1、kc00安装kafka
下载kafka-3.0.0
$ cd
$ curl -OL https://dlcdn.apache.org/kafka/3.0.0/kafka_2.13-3.0.0.tgz
------------------------------------------------------------------------------------------------------------
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 82.3M 100 82.3M 0 0 11.3M 0 0:00:07 0:00:07 --:--:-- 11.1M
解压并配置
$ tar xvf kafka_2.13-3.0.0.tgz
$ cd kafka_2.13-3.0.0/
$ vi config/server.properties
------------------------------------------------------------------------------------------------------------
...
############################# Server Basics #############################
broker.id=0
...
############################# Socket Server Settings #############################
...
listeners=PLAINTEXT://10.0.1.10:9092
...
advertised.listeners=PLAINTEXT://10.0.1.10:9092
...
############################# Log Basics #############################
...
log.dirs=/home/zk/data/kafka-logs
...
############################# Zookeeper #############################
...
zookeeper.connect=10.0.1.10:2181,10.0.1.11:2181,10.0.1.12:2181
------------------------------------------------------------------------------------------------------------
$ mkdir /home/zk/data/kafka-logs
运行
$ bin/kafka-server-start.sh config/server.properties
2、kc01安装kafka
下载kafka-3.0.0
$ cd
$ curl -OL https://dlcdn.apache.org/kafka/3.0.0/kafka_2.13-3.0.0.tgz
------------------------------------------------------------------------------------------------------------
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 82.3M 100 82.3M 0 0 11.3M 0 0:00:07 0:00:07 --:--:-- 11.1M
解压并配置
$ tar xvf kafka_2.13-3.0.0.tgz
$ cd kafka_2.13-3.0.0/
$ vi config/server.properties
------------------------------------------------------------------------------------------------------------
...
############################# Server Basics #############################
broker.id=1
...
############################# Socket Server Settings #############################
...
listeners=PLAINTEXT://10.0.1.11:9092
...
advertised.listeners=PLAINTEXT://10.0.1.11:9092
...
############################# Log Basics #############################
...
log.dirs=/home/zk/data/kafka-logs
...
############################# Zookeeper #############################
...
zookeeper.connect=10.0.1.10:2181,10.0.1.11:2181,10.0.1.12:2181
------------------------------------------------------------------------------------------------------------
$ mkdir /home/zk/data/kafka-logs
运行
$ bin/kafka-server-start.sh config/server.properties
3、db00安装kafka
下载kafka-3.0.0
$ cd
$ curl -OL https://dlcdn.apache.org/kafka/3.0.0/kafka_2.13-3.0.0.tgz
------------------------------------------------------------------------------------------------------------
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 82.3M 100 82.3M 0 0 11.3M 0 0:00:07 0:00:07 --:--:-- 11.1M
解压并配置
$ tar xvf kafka_2.13-3.0.0.tgz
$ cd kafka_2.13-3.0.0/
$ vi config/server.properties
------------------------------------------------------------------------------------------------------------
...
############################# Server Basics #############################
broker.id=2
...
############################# Socket Server Settings #############################
...
listeners=PLAINTEXT://10.0.1.12:9092
...
advertised.listeners=PLAINTEXT://10.0.1.12:9092
...
############################# Log Basics #############################
...
log.dirs=/home/zk/data/kafka-logs
...
############################# Zookeeper #############################
...
zookeeper.connect=10.0.1.10:2181,10.0.1.11:2181,10.0.1.12:2181
------------------------------------------------------------------------------------------------------------
$ mkdir /home/zk/data/kafka-logs
运行
$ bin/kafka-server-start.sh config/server.properties
