Kafka部署

一、环境准备

Kafka集群环境搭建，需要准备好一个zookeeper环境（集群），zookeeper需要java环境

tar xf /java_v1.8.0-201_zhuizhi_20200423  /usr/local/java.tar.gz 
ln -s /data/java/java_v1.8.0-201_zhuizhi_20200423  /usr/local/java
cat /etc/profile.d/environment.sh 
#Java
export JAVA_HOME=/usr/local/java
export PATH=${JAVA_HOME}/bin:$JAVA_HOME/jre/bin:$PATH
export CLASSPATH=.$CLASSPATH:$JAVA_HOME/lib:${JAVA_HOME}/jre/lib:$JAVA_HOME/lib/tools.jar

source /etc/profile.d/environment.sh

1、主机分配

10.0.0.3
kafka-node1
Kafka、zk
zookeeper-3.4.14.tar.gz
kafka_2.11-1.1.0.tgz

10.0.0.4
kafka-node2
Kafka、zk
zookeeper-3.4.14.tar.gz
kafka_2.11-1.1.0.tgz

10.0.0.5
kafka-node3
Kafka、zk
zookeeper-3.4.14.tar.gz
kafka_2.11-1.1.0.tgz

2、zk部署

wget http://archive.apache.org/dist/zookeeper/zookeeper-3.4.14/zookeeper-3.4.14.tar.gz
tar -xzf zookeeper-3.4.14.tar.gz  -C /data/soft

3、修改zk配置文件

cp /data/soft/zookeeper-3.4.14/conf/zoo_sample.cfg /data/soft/zookeeper-3.4.14/conf/zoo.cfg   
cat /data/soft/zookeeper-3.4.14/conf/zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/data/soft/zookeeper-3.4.14/ 
dataLogDir=/data/soft/zookeeper-3.4.14/logs  
clientPort=2181 
server.1=10.111.105.31:2888:3888
server.2=10.111.105.34:2888:3888
server.3=10.111.105.35:2888:3888

4、添加myid文件（在dataDir指定的目录下）

cat /data/soft/zookeeper-3.4.14/myid

每个节点的myid需要不一样（这边对应3、4、5）

5、配置系统服务

cat /usr/lib/systemd/system/zookeeper1.service     
[Unit]
Description=Zookeeper
Requires=network.target
After=network.target
 
[Service]
User=zhuizhi
Group=zhuizhi
Type=forking
Environment=JAVA_HOME=/usr/local/java
Environment=ZOO_LOG_DIR=/data/soft/zookeeper-3.4.14/logs
WorkingDirectory=/data/soft/zookeeper-3.4.14
ExecStart=/data/soft/zookeeper-3.4.14/bin/zkServer.sh start /data/soft/zookeeper-3.4.14/conf/zoo.cfg
ExecStop=/data/soft/zookeeper-3.4.14/bin/zkServer.sh stop /data/soft/zookeeper-3.4.14/conf/zoo.cfg
ExecReload=/data/soft/zookeeper-3.4.14/bin/zkServer.sh restart /data/soft/zookeeper-3.4.14/conf/zoo.cfg
 
[Install]
WantedBy=multi-user.target

##授权并启动服务
chown -R zhuizhi:zhuizhi /data/soft/zookeeper-3.4.14
systemctl daemon-reload              
systemctl restart zookeeper1.service 
systemctl enable zookeeper1.service 

##验证服务状态
systemctl status zookeeper1.service 
lsof -i:3888
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
java 4781 zhuizhi 27u IPv6 81917 0t0 TCP node4:ciphire-serv (LISTEN)
java 4781 zhuizhi 28u IPv6 82179 0t0 TCP node4:39830->node3:ciphire-serv (ESTABLISHED)
java 4781 zhuizhi 30u IPv6 83271 0t0 TCP node4:ciphire-serv->node5:47812 (ESTABLISHED)

/data/soft/zookeeper-3.4.14/bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /data/soft/zookeeper-3.4.14/bin/../conf/zoo.cfg
Mode: leader

##zk角色说明：leader(主) follower（从）

二、Kafka部署 1、下载解压Kafka

wget http://archive.apache.org/dist/kafka/1.1.0/kafka_2.11-1.1.0.tgz
tar xf kafka_2.11-1.1.0.tgz -C /data/soft/

2、修改Kafka配置文件

cp /data/soft/kafka_2.11-1.1.0/config/server.properties /data/soft/kafka_2.11-1.1.0/config/server.properties.bak 
cat /data/soft/kafka_2.11-1.1.0/config/server.properties
broker.id=3
#broker.id每个节点需要不一样
port=9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/kafka_data_9092/logs/kafka
host.name =10.0.0.5
advertised.host.name =10.0.0.5
#最新版本0.10.x broker配置弃用了advertised.host.name 和 advertised.port 这两个个配置项，就配置advertised.listeners就可以了。
num.partitions=1
num.recovery.threads.per.data.dir=1
log.retention.hours=168
message.max.byte=5242880
default.replication.factor=1
replica.fetch.max.bytes=5242880
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
listeners=PLAINTEXT://10.0.0.5:9092
zookeeper.connect=10.0.0.3:2181,10.0.0.4:2181,10.0.0.0.5:2181
zookeeper.connection.timeout.ms=6000
delete.topic.enable=true
offsets.topic.replication.factor =1

3、配置Kafka为系统服务

cat /usr/lib/systemd/system/kafka.service 
[Unit]
Description=kafka.service
After=network.target remote-fs.target zookeeper.service

[Service]
User=root
Type=forking
Environment=JAVA_HOME=/usr/local/java
ExecStart=/usr/bin/bash /data/soft/kafka_2.11-1.1.0/bin/kafka-server-start.sh -daemon /data/soft/kafka_2.11-1.1.0/config/server.properties
ExecStop=/usr/bin/bash /data/soft/kafka_2.11-1.1.0/bin/kafka-server-stop.sh
ExecReload=$ExecStop;$ExecStart
LimitCORE=infinity
LimitNOFILE=204800
LimitNPROC=204800

[Install]
WantedBy=multi-user.target

启动Kafka服务
systemctl daemon-reload          
systemctl restart kafka1.service 
systemctl enable kafka1.service
systemctl status kafka1.service

4、验证Kafka

登陆zk验证Kafka
/data/soft/zookeeper-3.4.14/bin/zkCli.sh -server 10.0.0.3:2181
[zk: 10.0.0.3:2181(CONNECTED) 1] ls /brokers
[ids, topics, seqid]
[zk: 10.0.0.3:2181(CONNECTED) 3] ls /brokers/ids
[3, 4, 5]
发现已经检查到了已经安装的三台kafka的broker.id[3, 4, 5]

Kafka部署

Java相关栏目本月热门文章