栏目分类:
子分类:
返回
名师互学网用户登录
快速导航关闭
当前搜索
当前分类
子分类
实用工具
热门搜索
名师互学网 > IT > 前沿技术 > 大数据 > 大数据系统

Kafka mirror

Kafka mirror

准备两个Kafka集群 (default | loganalysis)

配置文件
### loganalysis kafka-0 ###
root@kafka-0:/etc/kafka# ls
connect-console-sink.properties    connect-file-sink.properties    connect-standalone.properties  log4j.properties     server.properties           tools-log4j.properties
connect-console-source.properties  connect-file-source.properties  consumer.properties            producer.properties  source-consumer.properties  trogdor.conf
connect-distributed.properties     connect-log4j.properties        kafka.properties               secrets              target-producer.properties  zookeeper.properties

### 数据源 Kafka ###

root@kafka-0:/etc/kafka# cat source-consumer.properties
bootstrap.servers=kafka-headless:9092
zookeeper.connect=kafka-zookeeper-headless:2181
zookeeper.connection.timeout.ms=1000000
group.id=mirror-maker-group-test
message.max.bytes=7000000
fetch.message.max.bytes=7500000
max.partition.fetch.bytes=7500000
replica.fetch.max.bytes=7500000

### 目标 Kafka ###

root@kafka-0:/etc/kafka# cat target-producer.properties
bootstrap.servers=loganalysis-kafka-headless.default:9092
message.max.bytes=7000000
fetch.message.max.bytes=7500000
max.partition.fetch.bytes=7500000
replica.fetch.max.bytes=7500000
root@kafka-0:/etc/kafka# 




kafka-mirror
root@kafka-2:/# kafka-mirror-maker
Continuously copy data between two Kafka clusters.
Option                                   Description
------                                   -----------
--abort.on.send.failure 
--consumer.config   Embedded consumer config for consuming
                                           from the source cluster.
--consumer.rebalance.listener 
--help                                   Print this message.
--message.handler                producer.
--message.handler.args 
--new.consumer                           DEPRECATED Use new consumer in mirror
                                           maker (this is the default so this
                                           option will be removed in a future
                                           version).
--num.streams                                  (default: 1)
--offset.commit.interval.ms 
--producer.config   Embedded producer config.
--rebalance.listener.args 
--whitelist 
root@kafka-2:/#

root@kafka-0:/etc/kafka# kafka-mirror-maker --consumer.config source-consumer.properties --producer.config target-producer.properties  --whitelist test

kafka集群1

kafka集群2

转载请注明:文章转载自 www.mshxw.com
本文地址:https://www.mshxw.com/it/688144.html
我们一直用心在做
关于我们 文章归档 网站地图 联系我们

版权所有 (c)2021-2022 MSHXW.COM

ICP备案号:晋ICP备2021003244-6号