| 192.168.10.31 | |
| 192.168.10.32 | |
| 192.168.10.33 |
1、上传安装包
2、解压应用包到指定路径（3台服务器分别操作）
3、安装zookeeper （3台服务器同步操作）
[appuser@localhost app]$ cd kafka/ [appuser@localhost kafka]$ ls apache-zookeeper-3.6.3-bin kafka_2.12-2.6.0 [appuser@localhost kafka]$ mv apache-zookeeper-3.6.3-bin/ zookeeper [appuser@localhost kafka]$ mv kafka_2.12-2.6.0/ kafka
修改配置
（一下指令3台服务器同步操作）
cd /app/kafka/zookeeper mkdir data cd conf cp zoo_sample.cfg zoo.cfg
[appuser@localhost conf]$ vim zoo.cfg
172.16.10.31
echo 1 > data/myid [appuser@localhost conf]$ vim zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 dataDir=/app/kafka/zookeeper/data dataLogDir=/app/kafka/zookeeper/logs clientPort=2181 server.1=172.16.10.31:2881:3881 server.2=172.16.10.32:2881:3881 server.3=172.16.10.33:2881:3881
添加防火墙允许其他两台服务器通信
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.32" port protocol="tcp" port="2181" accept" firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.32" port protocol="tcp" port="2881" accept" firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.32" port protocol="tcp" port="3881" accept" firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.33" port protocol="tcp" port="2181" accept" firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.33" port protocol="tcp" port="2881" accept" firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.33" port protocol="tcp" port="3881" accept"
重新加载防火墙
firewall-cmd --reload
:wq! 保存退出
启动zookeeper
cd /app/kafka/zookeeper/bin ./zkServer.sh start
172.16.101.202
echo 2 > data/myid [appuser@localhost conf]$ vim zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 dataDir=/app/kafka/zookeeper/data dataLogDir=/app/kafka/zookeeper/logs clientPort=2181 server.1=172.16.10.31:2881:3881 server.2=172.16.10.32:2881:3881 server.3=172.16.10.33:2881:3881
添加防火墙允许其他两台服务器通信
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.31" port protocol="tcp" port="2181" accept" firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.31" port protocol="tcp" port="2881" accept" firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.31" port protocol="tcp" port="3881" accept" firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.33" port protocol="tcp" port="2181" accept" firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.33" port protocol="tcp" port="2881" accept" firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.33" port protocol="tcp" port="3881" accept"
重新加载防火墙
firewall-cmd --reload
:wq! 保存退出
启动zookeeper
cd /app/kafka/zookeeper/bin ./zkServer.sh start
172.16.101.203
echo 3 > data/myid [appuser@localhost conf]$ vim zoo.cfg tickTime=2000 initLimit=10 syncLimit=5 dataDir=/app/kafka/zookeeper/data dataLogDir=/app/kafka/zookeeper/logs clientPort=2181 server.1=172.16.10.31:2881:3881 server.2=172.16.10.32:2881:3881 server.3=172.16.10.33:2881:3881
添加防火墙允许其他两台服务器通信
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.31" port protocol="tcp" port="2181" accept" firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.31" port protocol="tcp" port="2881" accept" firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.31" port protocol="tcp" port="3881" accept" firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.32" port protocol="tcp" port="2181" accept" firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.32" port protocol="tcp" port="2881" accept" firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.32" port protocol="tcp" port="3881" accept"
重新加载防火墙
firewall-cmd --reload
:wq! 保存退出
启动zookeeper
cd /app/kafka/zookeeper/bin ./zkServer.sh start
查看zookeeper 启动状态 leader 为主
./zkServer.sh status Mode: leader
4、安装kafka （3台服务器同步操作）
[appuser@localhost kafka]$ cd /app/kafka/kafka mkdir data cd /app/kafka/kafka/config
注意：broker.id 唯一
172.16.10.31
修改zk路径为zk存储data的路径
vim zookeeper.properties
更改为zookeeper的data路径 dataDir=/app/kafka/zookeeper/data
vim server.properties
broker.id=0 listeners=PLAINTEXT://172.16.10.31:9092 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/app/kafka/kafka/kafka-logs num.partitions=3 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=3 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=172.16.10.31:2181,172.16.10.32:2181,172.16.10.33:2181 zookeeper.connection.timeout.ms=18000 group.initial.rebalance.delay.ms=0 # delete.topic.enable=true
./kafka-server-start.sh -daemon ../config/server.properties
[appuser@localhost bin]$ ./kafka-server-start.sh -daemon ../config/server.properties
启动后日志输出
2019-06-26 21:48:57,183] INFO Kafka commitId: fc1aaa116b661c8a (org.apache.kafka.common.utils.AppInfoParser) [2019-06-26 21:48:57,183] INFO Kafka startTimeMs: 1561531737175 (org.apache.kafka.common.utils.AppInfoParser) [2019-06-26 21:48:57,185] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
检查启动运行情况
[appuser@localhost bin]$ jps 14771 Kafka 14840 Jps 14299 QuorumPeerMain
添加防火墙
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.32" port protocol="tcp" port="9092" accept" firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.33" port protocol="tcp" port="9092" accept" firewall-cmd --reload
172.16.10.32
修改zk路径为zk存储data的路径
vim zookeeper.properties
更改为zookeeper的data路径 dataDir=/app/kafka/zookeeper/data
vim server.properties
broker.id=1 listeners=PLAINTEXT://172.16.10.32:9092 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/app/kafka/kafka/kafka-logs num.partitions=3 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=3 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=172.16.10.31:2181,172.16.10.32:2181,172.16.10.33:2181 zookeeper.connection.timeout.ms=18000 group.initial.rebalance.delay.ms=0 # delete.topic.enable=true
[appuser@localhost bin]$ ./kafka-server-start.sh -daemon ../config/server.properties
启动后日志输出
[2019-06-26 21:48:57,183] INFO Kafka commitId: fc1aaa116b661c8a (org.apache.kafka.common.utils.AppInfoParser) [2019-06-26 21:48:57,183] INFO Kafka startTimeMs: 1561531737175 (org.apache.kafka.common.utils.AppInfoParser) [2019-06-26 21:48:57,185] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
检查启动运行情况
[appuser@localhost bin]$ jps 14771 Kafka 14840 Jps 14299 QuorumPeerMain
添加防火墙
firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.31" port protocol="tcp" port="9092" accept" firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.33" port protocol="tcp" port="9092" accept" firewall-cmd --reload
172.16.10.33
修改zk路径为zk存储data的路径
vim zookeeper.properties
更改为zookeeper的data路径 dataDir=/app/kafka/zookeeper/data
vim server.properties
broker.id=2 listeners=PLAINTEXT://172.16.10.33:9092 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/app/kafka/kafka/kafka-logs num.partitions=3 num.recovery.threads.per.data.dir=1 offsets.topic.replication.factor=3 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 log.retention.hours=168 log.segment.bytes=1073741824 log.retention.check.interval.ms=300000 zookeeper.connect=172.16.10.31:2181,172.16.10.32:2181,172.16.10.33:2181 zookeeper.connection.timeout.ms=18000 group.initial.rebalance.delay.ms=0 # delete.topic.enable=true
[appuser@localhost bin]$ ./kafka-server-start.sh -daemon ../config/server.properties
启动后日志输出
[2019-06-26 21:48:57,183] INFO Kafka commitId: fc1aaa116b661c8a (org.apache.kafka.common.utils.AppInfoParser) [2019-06-26 21:48:57,183] INFO Kafka startTimeMs: 1561531737175 (org.apache.kafka.common.utils.AppInfoParser) [2019-06-26 21:48:57,185] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
检查启动运行情况
[appuser@localhost bin]$ jps 14771 Kafka 14840 Jps 14299 QuorumPeerMain
[appuser@localhost bin]$ 添加防火墙 firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.31" port protocol="tcp" port="9092" accept" firewall-cmd --permanent --add-rich-rule="rule family="ipv4" source address="172.16.10.32" port protocol="tcp" port="9092" accept" firewall-cmd --reload
登录zk验证
cd /usr/local/zookeeper/bin
zkCli.sh
get /brokers/ids/${broker.id}
可以看到三个kafka的broker均已经注册到zk中
测试：
(1) 创建topic：
bin/kafka-topics.sh --create --bootstrap-server 172.16.10.31:9092 --replication-factor 3 --partitions 1 --topic testTopic
(2) 查看topic
bin/kafka-topics.sh --list --bootstrap-server 172.16.10.31:9092
(3) 发送消息
bin/kafka-console-producer.sh --broker-list 172.16.10.31:9092 --topic testTopic
#消息内容
> test by test.io
(4) 在Kafka-2上消费Broker-0消息
bin/kafka-console-consumer.sh --bootstrap-server 172.16.10.31:9092 --topic testTopic --from-beginning
(5) 在Kafka-3上消费Broker-0消息
bin/kafka-console-consumer.sh --bootstrap-server 192.168.88.52:9092 --topic test-ken-io --from-beginning
均能收到消息 test by test.io
在其他节点测试消费
