Logging in Kubernetes on Debian: Tools, Configurations, and Best Practices
Kubernetes clusters running on Debian require a robust logging strategy to collect, store, and analyze logs from containers, nodes, and cluster components (e.g., kubelet, kube-apiserver). This ensures observability, helps debug issues, and meets compliance requirements. Below is a structured guide to logging in Debian-based Kubernetes environments.
Kubernetes logs are generated from three primary sources:
The default log driver for containers is json-file, which writes logs to /var/log/containers on each node. For long-term storage and analysis, these logs need to be forwarded to a centralized system.
The EFK stack is the most widely used solution for Kubernetes logging. It offers scalable log collection, storage, and visualization:
/var/log/containers and forwarding them to Elasticsearch.Key Steps for Deployment:
fluent/fluentd-kubernetes-daemonset image. Mount /var/log/containers and /var/lib/docker/containers as read-only volumes to access container logs. Configure environment variables (e.g., FLUENT_ELASTICSEARCH_HOST, FLUENT_ELASTICSEARCH_PORT) to connect to Elasticsearch.elasticsearch.hosts in kibana.yml to point to your Elasticsearch cluster.Advantages: Full-text search, real-time analytics, and rich visualization. Suitable for large-scale clusters with complex log parsing needs.
Loki is a lightweight log aggregation system developed by Grafana Labs. It is optimized for cloud-native environments and integrates seamlessly with Grafana for visualization:
/var/log/containers and sending them to Loki.Key Steps for Deployment:
/var/log/containers and configure promtail.yml to scrape logs and send them to Loki. Use relabeling rules to filter logs (e.g., exclude kube-system namespace).Advantages: Lower resource usage, simpler architecture, and cost-effective for small-to-medium clusters. Ideal for teams already using Grafana for monitoring.
Filebeat is a log shipper from Elastic that runs as a DaemonSet on Debian nodes. It collects logs from /var/log/containers and forwards them to Elasticsearch or Logstash:
paths: ["/var/log/containers/*.log"]) and outputs (e.g., Elasticsearch endpoint). Use modules (e.g., kubernetes) to parse structured logs.Format logs as JSON (e.g., {"timestamp": "2025-10-04T12:00:00Z", "level": "INFO", "message": "Application started"}). Structured logs enable easier parsing, filtering, and analysis in tools like Elasticsearch and Loki. Most applications (e.g., Go, Python) support JSON logging libraries.
Prevent disk space exhaustion by configuring log rotation for container logs (stored in /var/log/containers). Use logrotate with settings like:
/var/log/containers/*.log {
daily
rotate 7
compress
missingok
notifempty
}
This keeps 7 days of compressed logs and deletes older files.
Avoid relying on local logs (e.g., /var/log/syslog) by forwarding all logs to a centralized system (Elasticsearch, Loki). This ensures logs are accessible across nodes and persist after node failures.
record_transformer or Loki’s replace filter to redact sensitive information (e.g., passwords, API keys) from logs.monitoring service account to query Elasticsearch).Use Prometheus to monitor the health of your logging stack:
Even with centralized logging, you’ll often need to retrieve logs directly from Kubernetes:
kubectl logs <pod-name> --timestamps
Add --follow (-f) to stream logs in real-time.
kubectl logs -f -l app=my-app
This retrieves logs for all pods with the label app=my-app.
kubectl logs <pod-name> -p
Useful for debugging crashed containers.
Use kubetail (install via brew install kubetail or apt-get install kubetail):
kubetail my-app
Aggregates logs from all pods with the label app=my-app into a single stream.
Check kubelet logs (on Debian nodes):
journalctl -u kubelet --no-pager -n 50
View kube-apiserver logs (if running on Debian):
kubectl logs -n kube-system kube-apiserver-<node-name> --timestamps
By following this guide, you can implement a scalable and secure logging strategy for your Debian-based Kubernetes cluster. Choose the right tools (EFK for advanced analytics, Loki for cost efficiency) and adhere to best practices to ensure your logs are actionable and reliable.