centos

如何在centos部署kafka配置

小樊
44
2025-08-28 20:52:04
栏目: 智能运维

在CentOS上部署Kafka并配置,可以按照以下步骤进行:

1. 安装Java

Kafka需要Java环境,首先确保你的CentOS系统上已经安装了Java。

sudo yum install java-1.8.0-openjdk-devel

验证Java安装:

java -version

2. 下载并解压Kafka

从Apache Kafka官网下载最新版本的Kafka,并解压到指定目录。

wget https://downloads.apache.org/kafka/3.2.0/kafka_2.13-3.2.0.tgz
tar -xzf kafka_2.13-3.2.0.tgz
mv kafka_2.13-3.2.0 /opt/kafka

3. 配置Kafka

进入Kafka目录并进行配置。

3.1 server.properties

编辑/opt/kafka/config/server.properties文件,进行基本配置。

# Broker ID
broker.id=0

# List of directories in which log files will be stored.
log.dirs=/tmp/kafka-logs

# The number of partitions for the default topic.
num.partitions=1

# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption and production, but this will also result in
# more files across brokers.
default.replication.factor=1

# The minimum age of a log file to be eligible for deletion due to age.
log.retention.hours=168

# The maximum size of the log files.
log.segment.bytes=1073741824

# The interval at which log segments are checked to see if they can be
# deleted according to the retention policies.
log.retention.check.interval.ms=300000

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=18000

3.2 zookeeper.properties

编辑/opt/kafka/config/zookeeper.properties文件,进行基本配置。

# the directory where the snapshot and log data will be stored.
dataDir=/tmp/zookeeper

# the port at which the clients will connect
clientPort=2181

# the maximum number of client connections.
maxClientCnxns=0

# the number of milliseconds of each tick
tickTime=2000

# the number of ticks that the initial
# synchronization phase can take
initLimit=10

# the number of ticks that can pass between
# sending a request and receiving an acknowledgment
syncLimit=5

# the leader election port
electionPort=3888

# the number of elections needed to elect a leader
quorum=1

# the path to the data log directory
dataLogDir=/tmp/zookeeper/log

4. 启动Zookeeper

在Kafka目录下启动Zookeeper。

/opt/kafka/bin/zookeeper-server-start.sh /opt/kafka/config/zookeeper.properties

5. 启动Kafka Broker

在另一个终端窗口中启动Kafka Broker。

/opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/server.properties

6. 创建Topic

创建一个Topic来测试Kafka。

/opt/kafka/bin/kafka-topics.sh --create --topic test --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1

7. 验证Kafka

发送和接收消息以验证Kafka是否正常工作。

7.1 生产者

启动一个生产者来发送消息。

/opt/kafka/bin/kafka-console-producer.sh --topic test --bootstrap-server localhost:9092

输入消息并按回车键发送。

7.2 消费者

启动一个消费者来接收消息。

/opt/kafka/bin/kafka-console-consumer.sh --topic test --from-beginning --bootstrap-server localhost:9092

你应该能看到之前发送的消息。

8. 停止Kafka和Zookeeper

停止Kafka Broker和Zookeeper。

# 停止Kafka Broker
/opt/kafka/bin/kafka-server-stop.sh

# 停止Zookeeper
/opt/kafka/bin/zookeeper-server-stop.sh

以上步骤完成了在CentOS上部署和配置Kafka的基本过程。根据实际需求,你可以进一步调整配置参数和增加更多的Broker以实现高可用性和负载均衡。

0
看了该问题的人还看了