Debian上Kafka集群如何搭建配置

在Debian上搭建和配置Kafka集群涉及多个步骤，包括安装和配置Kafka、Zookeeper，以及启动和测试集群。以下是一个详细的指南：

环境准备

1. 安装Java：Kafka需要Java 8或更高版本。可以使用以下命令安装OpenJDK：

   sudo apt update
   sudo apt install openjdk-11-jdk
   

2. 安装Zookeeper：Kafka集群依赖于Zookeeper。可以从Apache官网下载并解压Zookeeper。

   wget https://downloads.apache.org/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz
   tar -xzf apache-zookeeper-3.8.0-bin.tar.gz
   sudo mv apache-zookeeper-3.8.0 /opt/zookeeper
   

3. 安装Kafka：从Apache官网下载并解压Kafka。

   wget https://downloads.apache.org/kafka/2.8.1/kafka_2.13-2.8.1.tgz
   tar -xzf kafka_2.13-2.8.1.tgz
   sudo mv kafka_2.13-2.8.1 /opt/kafka
   

配置Zookeeper

1. 修改Zookeeper配置文件：

   编辑 /opt/zookeeper/conf/zoo.cfg 文件，添加或修改以下配置：

   dataDir=/opt/zookeeper/data
   clientPort=2181
   server.1=localhost:2888:3888
   server.2=localhost:2889:3889
   server.3=localhost:2890:3890
   

2. 启动Zookeeper：

   sudo bin/zookeeper-server-start.sh config/zoo.cfg
   

配置Kafka

1. 修改Kafka配置文件：

   编辑 /opt/kafka/config/server.properties 文件，为每个Kafka broker添加或修改以下配置：

   Broker 1 (/opt/kafka/config/server-1.properties):

   broker.id=1
   listeners=PLAINTEXT://localhost:9092
   log.dirs=/tmp/kafka-logs/cluster/log1
   zookeeper.connect=localhost:2181
   advertised.listeners=PLAINTEXT://localhost:9092
   

   Broker 2 (/opt/kafka/config/server-2.properties):

   broker.id=2
   listeners=PLAINTEXT://localhost:9093
   log.dirs=/tmp/kafka-logs/cluster/log2
   zookeeper.connect=localhost:2181
   advertised.listeners=PLAINTEXT://localhost:9093
   

   Broker 3 (/opt/kafka/config/server-3.properties):

   broker.id=3
   listeners=PLAINTEXT://localhost:9094
   log.dirs=/tmp/kafka-logs/cluster/log3
   zookeeper.connect=localhost:2181
   advertised.listeners=PLAINTEXT://localhost:9094
   

2. 启动Kafka Broker：

   sudo bin/kafka-server-start.sh /opt/kafka/config/server-1.properties
   sudo bin/kafka-server-start.sh /opt/kafka/config/server-2.properties
   sudo bin/kafka-server-start.sh /opt/kafka/config/server-3.properties
   

验证集群

1. 创建测试主题：

   sudo bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 3 --partitions 3 --topic test-topic
   

2. 查看主题信息：

   sudo bin/kafka-topics.sh --describe --bootstrap-server localhost:9092 --topic test-topic
   

3. 发送和消费消息：

   # 发送消息
   sudo bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test-topic
   
   # 消费消息
   sudo bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic --from-beginning