ELK Stack Log Aggregation on K8s

ELK Stack Log Aggregation on Kubernetes

Centralized logging is non-negotiable for Kubernetes environments. With pods being ephemeral, logs disappear when pods restart. The EFK stack (Elasticsearch, Fluentd/Fluent Bit, Kibana) collects, stores, and visualizes logs from every container in your cluster.

Architecture

Pods → Fluent Bit (DaemonSet) → Elasticsearch → Kibana
  └─ stdout/stderr → /var/log/containers/ → parse & enrich → index → visualize

Fluent Bit DaemonSet

Fluent Bit runs on every node and collects container logs:

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluent-bit
  namespace: logging
spec:
  template:
    spec:
      containers:
      - name: fluent-bit
        image: fluent/fluent-bit:2.2
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: config
          mountPath: /fluent-bit/etc/
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: config
        configMap:
          name: fluent-bit-config

Fluent Bit Configuration

[SERVICE]
    Flush         5
    Log_Level     info
    Parsers_File  parsers.conf

[INPUT]
    Name              tail
    Tag               kube.*
    Path              /var/log/containers/*.log
    Parser            cri
    Mem_Buf_Limit     50MB
    Refresh_Interval  10

[FILTER]
    Name                kubernetes
    Match               kube.*
    Merge_Log           On
    K8S-Logging.Parser  On
    K8S-Logging.Exclude On

[OUTPUT]
    Name            es
    Match           kube.*
    Host            elasticsearch.logging.svc
    Port            9200
    Index           k8s-logs
    Type            _doc
    Logstash_Format On
    Logstash_Prefix k8s

Elasticsearch Deployment

helm install elasticsearch elastic/elasticsearch \
  --namespace logging \
  --set replicas=3 \
  --set minimumMasterNodes=2 \
  --set resources.requests.cpu=1 \
  --set resources.requests.memory=2Gi \
  --set volumeClaimTemplate.resources.requests.storage=100Gi \
  --set persistence.enabled=true

Index Lifecycle Management

Control storage costs with automated index lifecycle:

PUT _ilm/policy/k8s-logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_size": "50GB",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": { "number_of_shards": 1 },
          "forcemerge": { "max_num_segments": 1 }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": { "delete": {} }
      }
    }
  }
}

Structured Logging Best Practices

Use JSON structured logs for better searchability:

// Instead of: "User 123 logged in from 192.168.1.1"
// Use:
{
  "level": "info",
  "message": "User login",
  "userId": "123",
  "sourceIp": "192.168.1.1",
  "timestamp": "2026-02-13T10:30:00Z",
  "service": "auth-service"
}

Kibana Dashboards

• Error Dashboard — Error rate by service, top error messages, error trends
• Access Log Dashboard — Request volume, response codes, latency distribution
• Security Dashboard — Failed login attempts, unusual access patterns
• Application Dashboard — Custom business metrics extracted from logs

Cost Optimization

• Use Fluent Bit over Fluentd — 10x less memory usage
• Filter at the edge — Drop debug/trace logs in Fluent Bit before sending to Elasticsearch
• Compress indices — Use best_compression codec for warm indices
• Right-size Elasticsearch — Start with 3 nodes, scale based on ingest rate

Eazy SaaS Tip: We deploy Fluent Bit with aggressive filtering for our clients — dropping health check logs, debug output, and known noisy services at the edge. This typically reduces Elasticsearch storage by 60% while keeping all actionable log data available for troubleshooting.