如何用Ubuntu Filebeat监控系统性能

How to Monitor System Performance with Ubuntu Filebeat

1. Install Filebeat on Ubuntu

First, ensure Filebeat is installed on your Ubuntu system. You can install it via the official Elastic APT repository for the latest version:

# Add Elastic APT key and repository
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list

# Update package list and install Filebeat
sudo apt-get update && sudo apt-get install filebeat -y

This sets up Filebeat with default configurations, ready for customization.

2. Enable System Performance Modules

Filebeat includes pre-built modules for monitoring system performance metrics (e.g., CPU, memory, disk, network). These modules simplify configuration by auto-defining inputs, processors, and dashboards. To enable system monitoring:

# Navigate to the modules directory
cd /etc/filebeat/modules.d

# Enable system modules (disable "disable: true" in each file)
sudo nano system.yml  # Set "enabled: true"
sudo nano system-logs.yml  # Set "enabled: true"

# Example: Enable disk metrics collection every 10 seconds
sudo sed -i 's/enabled: false/enabled: true/' system-disk.yml
sudo sed -i 's/period: 1m/period: 10s/' system-disk.yml

Common system modules include:

• system: Collects CPU, memory, process, and load metrics.
• system-logs: Gathers system logs (e.g., /var/log/syslog, /var/log/kern.log).
• system-disk: Tracks disk I/O and usage statistics.

3. Configure Output Destination

Send collected metrics to a backend for storage and analysis. The most common setup is Elasticsearch (for indexing) + Kibana (for visualization):

# Edit the Filebeat configuration file
sudo nano /etc/filebeat/filebeat.yml

# Configure Elasticsearch output (replace with your ES host)
output.elasticsearch:
  hosts: ["localhost:9200"]
  index: "filebeat-system-%{+yyyy.MM.dd}"

# Optional: Use Logstash as an intermediary (uncomment if needed)
# output.logstash:
#   hosts: ["localhost:5044"]

For testing, you can also output to the console:

output.console:
  pretty: true

Save changes and test the configuration:

sudo filebeat test config -e

This ensures there are no syntax errors.

4. Start and Enable Filebeat

Launch Filebeat and configure it to start on boot:

# Start Filebeat service
sudo systemctl start filebeat

# Enable Filebeat to start automatically on system reboot
sudo systemctl enable filebeat

# Check Filebeat status (should show "active (running)")
sudo systemctl status filebeat

Verify logs for startup errors:

sudo tail -f /var/log/filebeat/filebeat

5. Visualize Metrics in Kibana

If using Elasticsearch and Kibana, create visualizations to monitor system performance:

1. Open Kibana: Access http://<your-server-ip>:5601 in a browser.
2. Create Index Pattern:
   • Go to Stack Management > Index Patterns.
   • Click “Create index pattern”, enter filebeat-system-*, and select the timestamp field (e.g., @timestamp).
3. Explore Metrics:
   • Navigate to Discover to view raw system logs.
   • Use Visualize Library to create dashboards (e.g., CPU usage over time, disk I/O trends).
   • Pre-built dashboards for system modules are available in Kibana’s “Dashboard” section.

6. Optimize Filebeat Performance

To handle high log volumes efficiently, adjust these key configurations in /etc/filebeat/filebeat.yml:

• Use filestream Input: More efficient than the legacy log input (recommended for Filebeat 7.0+).

  filebeat.inputs:
  - type: filestream
    enabled: true
    paths:
      - /var/log/*.log
  

• Adjust Batch Size: Increase bulk_max_size to send more data per batch (reduces network overhead).

  output.elasticsearch:
    bulk_max_size: 512  # Default is 2048; adjust based on your network
  

• Enable Persistent Queue: Ensures data is not lost if Filebeat crashes.

  queue.type: persisted
  queue.max_bytes: 10gb  # Adjust based on available disk space
  

• Limit Concurrent Harvesters: Prevent resource exhaustion on busy systems.

  filebeat.inputs:
  - type: filestream
    harvester_limit: 100  # Default is 0 (unlimited); set based on CPU cores
  

7. Monitor Filebeat Itself

Track Filebeat’s health and performance using built-in tools:

• Systemd Status: Check if Filebeat is running.

  sudo systemctl status filebeat
  

• Logs: View real-time logs for errors or warnings.

  sudo tail -f /var/log/filebeat/filebeat
  

• HTTP API: Get detailed metrics (e.g., events sent, queue size).

  curl http://localhost:8080/stats
  

• Third-Party Monitoring: Use Prometheus + Grafana to track Filebeat metrics (e.g., CPU usage, event processing rate). Configure Filebeat to expose metrics by enabling the monitoring section in filebeat.yml:

  monitoring:
    enabled: true
    elasticsearch:
      hosts: ["localhost:9200"]
  

By following these steps, you can effectively use Ubuntu Filebeat to monitor system performance, collect critical metrics, and visualize them for actionable insights. Adjust configurations based on your system’s scale and requirements to ensure optimal performance.