一、kafka节点内磁盘使用不均衡
1、原因:
每次新建partition时,会选择partition少的目录,而不会考虑磁盘的使用空间
2、解决
1)停止broker
2)对目录下的文件进行移动
kafka-logs每个dir下会有两个文件
recovery-point-offset-checkpoint
replication-offset-checkpoint
移动partition后,需要修改对应目录中的这两个文件内容
二、kafka中新扩容broker需要进行partition迁移时
1、创建move.json
多个topic,则写每个topic需要指定的分区,多副本情况,指定每个partition的存储节点
move.json如下
{"version":1,"partitions":[
{"topic":"wanghaha_ceshi","partition":3,"replicas":[2,1]},
{"topic":"wanghaha_ceshi","partition":4,"replicas":[2,1]},
{"topic":"wanghaha_ceshi","partition":5,"replicas":[2,1]},
{"topic":"wanghaha_ceshi","partition":6,"replicas":[2,1]},
{"topic":"wanghaha_ceshi","partition":7,"replicas":[2,1]},
{"topic":"wanghaha_ceshi","partition":8,"replicas":[2,1]},
{"topic":"wanghaha_ceshi","partition":9,"replicas":[2]}]}
2、分区迁移
使用命令迁移partition
./kafka-reassign-partitions.sh --zookeeper 10.24.102.1:24002/kafka --reassignment-json-file move.json --execute