Kafka14:服役、退役节点

1. 使用Prettyzoo管理

• 通关查看broker.ids，知道当前是三台机器运行。

2. redis03模拟节点上下线

2.1 查看redis03状态

• 此时redis03中kafka在运行

2.2 停止redis03，模拟下线

root@redis03:/usr/local/kafka# ./bin/kafka-server-stop.sh

2.3 开启redis03，模拟上线

root@redis03:/usr/local/kafka# ./bin/kafka-server-start.sh -daemon config/server.properties

4. 添加新节点

4.1 复制一台机器

• 我这里复制redis03机器

4.2 修改主机名

4.3 修改kafka

1. 删除datas 和 logs两个文件
1. 因保留的是redis03的，如果不删除启动redis04中kafka时，redis03中kafka下线；

root@redis04:~# cd /usr/local/kafka
root@redis04:/usr/local/kafka# ls
bin  config  datas  libs  LICENSE  licenses  logs  NOTICE  site-docs
root@redis04:/usr/local/kafka# rm -rf datas/ logs/
root@redis04:/usr/local/kafka# ls
bin  config  libs  LICENSE  licenses  NOTICE  site-docs
root@redis04:/usr/local/kafka#

1. 修改config/server.propertities ,修改broker.id=3

root@redis04:/usr/local/kafka# vim config/server.properties

5. 模拟添加节点

5.1 准备工作

5.1.1 停止所有机器Kafka、Zookeeper

# 1.先停止Kafka
root@redis01:~# /home/bigdata/bin/kf.sh stop

# 2.查看Kafka停止完成
root@redis01:~# xcall jps

# 3.停止Zookeeper
root@redis01:~# /home/bigdata/bin/zk1.sh stop

5.1.2 启动所有机器Zookeeper、Kafka

# 1.先启动Zookeeper
root@redis01:~# /home/bigdata/bin/zk1.sh start
# 2.查看ZK启动是否完成
root@redis01:~# xcall jps
# 3.启动Kafka
root@redis01:~# /home/bigdata/bin/kf.sh start

5.2 启动redis04，添加Kafka节点

root@redis04:/usr/local/kafka# bin/kafka-server-start.sh -daemon config/server.properties
root@redis04:/usr/local/kafka# jps
2641 Jps
2613 Kafka

5.3 负载均衡—服役新节点redis04

• 新添加数据可以存储

• 历史数据如何负载均衡？

5.3.1 负载均衡

1. 创建一个要均衡的主题topics-to-move.json

root@redis01:/usr/local/kafka# vim topics-to-move.json

{
    "topics": [
        {
            "topic": "first"
        }
    ],
    "version": 1
}

1. 生成一个负载均衡的计划

root@redis01:/usr/local/kafka# ./bin/kafka-reassign-partitions.sh --bootstrap-server redis01:9092 --topics-to-move-json-file topics-to-move.json --broker-list "0,1,2,3" --generate

1. 如果上述计划合适，创建副本存储计划。

root@redis01:/usr/local/kafka# vim increase-replication-factor.json

{"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[1,2,3],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[2,3,0],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[3,0,1],"log_dirs":["any","any","any"]}]}

1. 执行副本存储计划

root@redis01:/usr/local/kafka# ./bin/kafka-reassign-partitions.sh --bootstrap-server redis01:9092 --reassignment-json-file increase-replication-factor.json --execute

1. 验证副本存储计划

root@redis01:/usr/local/kafka# ./bin/kafka-reassign-partitions.sh --bootstrap-server redis01:9092 --reassignment-json-file increase-replication-factor.json --verify

• 还可以进一步验证，发现副本已经分布在0，1，2，3上了

root@redis01:/usr/local/kafka# ./bin/kafka-topics.sh --bootstrap-server redis01:9092 --topic first --describe

5.4 负载均衡—退役旧节点redis04

5.4.1 负载均衡

1. 创建一个要均衡的主题topics-to-move.json

root@redis01:/usr/local/kafka# vim topics-to-move.json

{
    "topics": [
        {
            "topic": "first"
        }
    ],
    "version": 1
}

1. 生成一个负载均衡的计划

root@redis01:/usr/local/kafka# ./bin/kafka-reassign-partitions.sh --bootstrap-server redis01:9092 --topics-to-move-json-file topics-to-move.json --broker-list "0,1,2" --generate

1. 如果上述计划合适，创建副本存储计划。

root@redis01:/usr/local/kafka# vim increase-replication-factor.json

{"version":1,"partitions":[{"topic":"first","partition":0,"replicas":[2,0,1],"log_dirs":["any","any","any"]},{"topic":"first","partition":1,"replicas":[0,1,2],"log_dirs":["any","any","any"]},{"topic":"first","partition":2,"replicas":[1,2,0],"log_dirs":["any","any","any"]}]}

1. 执行副本存储计划

root@redis01:/usr/local/kafka# ./bin/kafka-reassign-partitions.sh --bootstrap-server redis01:9092 --reassignment-json-file increase-replication-factor.json --execute

1. 验证副本存储计划

root@redis01:/usr/local/kafka# ./bin/kafka-reassign-partitions.sh --bootstrap-server redis01:9092 --reassignment-json-file increase-replication-factor.json --verify

• 还可以进一步验证，发现副本已经分布在0，1，2，3上了

root@redis01:/usr/local/kafka# ./bin/kafka-topics.sh --bootstrap-server redis01:9092 --topic first --describe

1. 停止redis04中的Kafka

root@redis04:/usr/local/kafka# ./bin/kafka-server-stop.sh