Debian Filebeat与Elasticsearch如何集成

Prerequisites
Before integrating Filebeat with Elasticsearch on Debian, ensure your system is updated (sudo apt update && sudo apt upgrade -y) and has Java installed (required for Elasticsearch). You’ll also need the Elastic APT repository for Filebeat/Elasticsearch installation.

1. Install Elasticsearch
Elasticsearch is the target data store for Filebeat. On Debian:

• Add the Elastic GPG key and repository:

  wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
  echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
  

• Install Elasticsearch:

  sudo apt update && sudo apt install elasticsearch -y
  

• Configure Elasticsearch:
  Edit /etc/elasticsearch/elasticsearch.yml to enable networking (set network.host: 0.0.0.0 and http.port: 9200).
• Start and enable Elasticsearch:

  sudo systemctl start elasticsearch
  sudo systemctl enable elasticsearch
  

• Verify Elasticsearch is running:

  curl -X GET "localhost:9200"
  

  You should see a JSON response with cluster info.

2. Install Filebeat
Filebeat is the lightweight shipper that forwards logs to Elasticsearch.

• Add the Elastic GPG key and repository (if not already done for Elasticsearch):

  wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
  echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list
  

• Install Filebeat:

  sudo apt update && sudo apt install filebeat -y
  

• Enable Filebeat modules (optional but recommended for structured logs like system/syslog):

  sudo filebeat modules enable system
  

  This configures Filebeat to monitor system logs (e.g., /var/log/syslog).

3. Configure Filebeat for Elasticsearch Output
Edit Filebeat’s main config file (/etc/filebeat/filebeat.yml) to define the Elasticsearch output and log inputs:

• Define Log Inputs: Specify which logs Filebeat should monitor. For system logs:

  filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/*.log  # Monitor all .log files in /var/log
      - /var/log/syslog # Include syslog (optional)
  

• Configure Elasticsearch Output: Replace localhost:9200 with your Elasticsearch host/IP. For unsecured clusters:

  output.elasticsearch:
    hosts: ["localhost:9200"]
    index: "filebeat-%{[agent.version]}-%{+yyyy.MM.dd}"  # Dynamic index name (e.g., filebeat-7.17.0-2025.09.24)
  

  For secured clusters (enabled X-Pack security), add authentication and SSL:

  output.elasticsearch:
    hosts: ["https://localhost:9200"]
    username: "elastic"  # Default superuser (change in production)
    password: "your_secure_password"
    ssl.certificate_authorities: ["/etc/filebeat/certs/ca.crt"]  # Path to CA cert (if using self-signed certs)
  

• Optional: Load Kibana Dashboards (for visualization):
  If you plan to use Kibana, configure the Kibana endpoint to load default dashboards:

  setup.kibana:
    host: "localhost:5601"  # Kibana address
  

  Run sudo filebeat setup after configuration to load dashboards.

4. Start and Enable Filebeat

• Start the Filebeat service:

  sudo systemctl start filebeat
  

• Enable Filebeat to start on boot:

  sudo systemctl enable filebeat
  

• Check Filebeat status to ensure it’s running:

  sudo systemctl status filebeat
  

  Look for “active (running)” in the output.

5. Verify Integration

• Check Elasticsearch Indices:
  Run the following command to confirm Filebeat is sending data (you should see filebeat-* indices):

  curl -X GET "localhost:9200/_cat/indices?v"
  

• View Logs in Kibana (Optional):
  Open Kibana in a browser (http://localhost:5601), go to Stack Management > Index Patterns, and create a pattern for filebeat-*. Then, navigate to Discover to view log data.

6. Security Best Practices (Optional but Recommended)

• Enable Elasticsearch Security:
  Edit /etc/elasticsearch/elasticsearch.yml to enable X-Pack security:

  xpack.security.enabled: true
  

  Restart Elasticsearch and set passwords for built-in users:

  sudo systemctl restart elasticsearch
  sudo /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive
  

• Use API Keys or PKI:
  Instead of basic auth, use API keys (more secure) or PKI certificates for Filebeat-Elasticsearch communication. For API keys:

  output.elasticsearch:
    hosts: ["localhost:9200"]
    api_key: "id:api_key_value"  # Generate via Kibana or Elasticsearch API
  

• Encrypt Communications:
  Use SSL/TLS for data in transit. Configure Elasticsearch to use HTTPS and provide Filebeat with the CA cert (as shown in step 3).

By following these steps, you’ll successfully integrate Filebeat with Elasticsearch on Debian, enabling log collection, forwarding, and storage for analysis in Elasticsearch (and visualization in Kibana if configured).