kafka的安装
1.将kafka压缩包上传的文件夹
2.解压kafka tar -xvf---------
[root@master software]$ mv /usr、software/kafka_2.11-1.1.0 kafka
3.配置环境变量
[root@master software]$ sudo vi /etc/profile
#kafka
export KAFKA_HOME=/usr/software/kafka
export PATH=$PATH:$KAFKA_HOME/bin
[root@master software]$ source /etc/profile
4.创建节点
在config中的 vi server.properties 创建节点
master broker id =0 logs.dir=/usr/software/kafka/ogs
slave1 broker id =1
slave2 broker id =2
zookeeper.connect=master:2181,slave1:2181,slave2:2181
分发
[root@master config]$ xrsync.sh /usr/software/kafka/
修改slave1、slave2的server.properties 分别为 1， 2
启动Kafka集群
1.[root@master ~]# zkServer.sh
2.[root@master bin]# ./kafka-server-start.sh --daemon /usr/software/kafka_2.11-1.1.0/config/server.properties
kafka的基本命令
查看topic列表
[root@master bin]$ kafka-topics.sh --zookeeper master:2181 --list
__consumer_offsets
创建topic
[root@master logs]$ kafka-topics.sh --zookeeper master:2181 --create --replication-factor 3 --partitions 2 --topic mytopic1
Created topic "mytopic1".
//查看主题
[root@master ~]$ cd /usr/software/kafka/logs
查看topic
[root@master logs]$ kafka-topics.sh --zookeeper master:2181 --describe --topic mytopic2
Topic:mytopic2 PartitionCount:3 ReplicationFactor:2 Configs:
Topic: mytopic2 Partition: 0 Leader: 2 Replicas: 2,0 Isr: 2,0
Topic: mytopic2 Partition: 1 Leader: 0 Replicas: 0,1 Isr: 0,1
Topic: mytopic2 Partition: 2 Leader: 1 Replicas: 1,2 Isr: 1,2
删除topic
[root@master ~]$ kafka-topics.sh --zookeeper master:2181 --delete --topic mytopic1
Topic mytopic1 is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.
Kafka+flume集成
Kafka作为Channel（没有Sink）
[root@master bin]$ kafka-topics.sh --zookeeper master:2181 --create --topic netcat-kafka --partitions 1 --replication-factor 1
Created topic "netcat-kafka".
启动flume
[root@master flume]$ bin/flume-ng agent -c conf/ -n a1 -f conf/kafka/netcat-kafka.conf -Dflume.root.logger=INFO,console
启动kafka控制台消费者
[root@master bin]$ ./kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic netcat-kafka
测试
[hadoop@hadoop1 ~]$ netcat localhost 44444
