在CentOS上集成Filebeat与Kafka的过程涉及多个步骤,包括安装和配置Filebeat、Kafka以及相关的安全措施。以下是详细的步骤和实践指南:
安装和配置Filebeat:
rpm -ivh filebeat-7.4.1-x86_64.rpm
/etc/filebeat/filebeat.yml
,配置输出到Kafka:output.kafka:
hosts: ["kafka_server_ip:9092"]
topics:
- log_topic_name
filebeat modules enable system
filebeat modules enable auditd
filebeat setup --pipelines --modules system
filebeat setup --pipelines --modules auditd
安装和配置Kafka:
yum install java-1.8.0-openjdk-devel
wget kafka_2.12-2.3.0.tgz
tar -xzf kafka_2.12-2.3.0.tgz
cd kafka_2.12-2.3.0
mkdir -p /app/zookeeper/{data,logs}
echo "1" > /app/zookeeper/myid
nohup /app/kafka_2.12-2.3.0/bin/zookeeper-server-start.sh /app/kafka_2.12-2.3.0/config/zookeeper.properties >>/dev/null 2>&1 &
mkdir -p /app/kafka/logs
vi /app/kafka_2.12-2.3.0/config/server.properties
# 配置Kafka相关参数
nohup /app/kafka_2.12-2.3.0/bin/kafka-server-start.sh /app/kafka_2.12-2.3.0/config/server.properties >>/dev/null 2>&1 &
安全加固措施:
setenforce 0
sed -i 's/selinux=enforcing/selinux=disabled/g' /etc/selinux/config
systemctl stop firewalld
systemctl disable firewalld
echo "* soft nofile 65536" > /etc/security/limits.conf
echo "* hard nofile 131072" >> /etc/security/limits.conf
echo "* soft nproc 65536" >> /etc/security/limits.conf
echo "* hard nproc 131072" >> /etc/security/limits.conf
echo "vm.max_map_count=562144" >> /etc/sysctl.conf
sysctl -p
useradd elkuser
echo 123456 | passwd --stdin elkuser
ssh-keygen
ssh-copy-id elkuser@kafka_server_ip
ssh-copy-id elkuser@filebeat_server_ip