centos

Filebeat与Kafka在CentOS上的集成实践

小樊
41
2025-04-12 17:01:10
栏目: 智能运维

在CentOS上集成Filebeat与Kafka的过程涉及多个步骤,包括安装和配置Filebeat、Kafka以及相关的安全措施。以下是详细的步骤和实践指南:

  1. 安装和配置Filebeat

    • 在Filebeat服务器上安装Filebeat:
      rpm -ivh filebeat-7.4.1-x86_64.rpm
      
    • 修改Filebeat配置文件/etc/filebeat/filebeat.yml,配置输出到Kafka:
      output.kafka:
        hosts: ["kafka_server_ip:9092"]
        topics:
          - log_topic_name
      
    • 初始化并启动Filebeat模块:
      filebeat modules enable system
      filebeat modules enable auditd
      filebeat setup --pipelines --modules system
      filebeat setup --pipelines --modules auditd
      
  2. 安装和配置Kafka

    • 安装Java并配置Kafka依赖:
      yum install java-1.8.0-openjdk-devel
      
    • 上传并解压Kafka:
      wget kafka_2.12-2.3.0.tgz
      tar -xzf kafka_2.12-2.3.0.tgz
      cd kafka_2.12-2.3.0
      
    • 配置Zookeeper和Kafka:
      • 创建data和logs目录,并启动Zookeeper:
        mkdir -p /app/zookeeper/{data,logs}
        echo "1" > /app/zookeeper/myid
        nohup /app/kafka_2.12-2.3.0/bin/zookeeper-server-start.sh /app/kafka_2.12-2.3.0/config/zookeeper.properties >>/dev/null 2>&1 &
        
      • 配置Kafka并启动Kafka:
        mkdir -p /app/kafka/logs
        vi /app/kafka_2.12-2.3.0/config/server.properties
        # 配置Kafka相关参数
        nohup /app/kafka_2.12-2.3.0/bin/kafka-server-start.sh /app/kafka_2.12-2.3.0/config/server.properties >>/dev/null 2>&1 &
        
  3. 安全加固措施

    • 关闭SELinux:
      setenforce 0
      sed -i 's/selinux=enforcing/selinux=disabled/g' /etc/selinux/config
      
    • 关闭Firewalld:
      systemctl stop firewalld
      systemctl disable firewalld
      
    • 配置文件和内存限制:
      echo "* soft nofile 65536" > /etc/security/limits.conf
      echo "* hard nofile 131072" >> /etc/security/limits.conf
      echo "* soft nproc 65536" >> /etc/security/limits.conf
      echo "* hard nproc 131072" >> /etc/security/limits.conf
      echo "vm.max_map_count=562144" >> /etc/sysctl.conf
      sysctl -p
      
    • 创建用户并配置SSH免密登录:
      useradd elkuser
      echo 123456 | passwd --stdin elkuser
      ssh-keygen
      ssh-copy-id elkuser@kafka_server_ip
      ssh-copy-id elkuser@filebeat_server_ip
      

0
看了该问题的人还看了