Log Aggregation at Scale: When Centralized Logging Becomes Your Bottleneck and How to Recover

The Breaking Point

Three months into a product launch, your observability stack starts sending signals. Elasticsearch queries that used to return in 200ms now timeout at 30 seconds. Your storage bill has tripled. The team stops using the logging system because waiting for results is slower than SSH-ing into servers and grepping logs manually. You're paying for centralized logging but getting worse visibility than you had before.

This isn't a capacity planning failure. It's a symptom of a more fundamental problem: you're treating all logs as equally valuable when they're not.

Diagnosis: Where the Volume Comes From

Before you optimize, you need to see what's actually happening. Most teams skip this step and jump straight to "we need a bigger cluster."

Start with a simple audit. Query your logging backend for the top 20 log sources by volume over the last 24 hours. In most systems, you'll find:

• Verbose application logs (debug-level statements in hot code paths)
• Health check spam (load balancers, Kubernetes probes, internal monitors all logging every request)
• Third-party library chatter (SDKs, frameworks, middleware all at info level)
• Transient error noise (retries, circuit breaker trips, temporary connection failures)

One team I worked with discovered that 40% of their log volume came from a single library logging every database connection pool event. Another had load balancers sending a log entry for every health check—thousands per minute per service.

The critical insight: you're not logging too much. You're logging the wrong things at the wrong levels.

The Decision Tree: Sampling vs. Filtering vs. Tiering

Once you've identified the sources, you face a choice. Each path has real trade-offs.

Option 1: Sampling

What it is: Keep only a percentage of logs. If you sample at 10%, you keep 1 in 10 log lines.

Pros:

• Simple to implement (add one line to your logging config)
• Reduces volume immediately
• Preserves log diversity (you still see all log types)

Cons:

• You lose visibility into low-frequency events. If a particular error happens 100 times a day, sampling at 10% means you might see it 10 times—or zero times
• Debugging specific user requests becomes harder if their logs weren't sampled
• False confidence: "we have centralized logging" while missing critical events

When to use: As a temporary measure while you're fixing the root cause. Not as a long-term strategy for production systems handling sensitive workloads.

Option 2: Filtering (Dropping Specific Log Types)

What it is: Prevent certain logs from reaching your aggregation system at all.

Pros:

• Surgical: you keep high-signal logs, drop the noise
• No loss of visibility for important events
• Reduces costs significantly

Cons:

• Requires knowing what to drop (the audit phase above is mandatory)
• If you drop logs you later need, they're gone—you can't retroactively search them
• Maintenance burden: as code changes, log patterns change, and your filters become stale

When to use: For logs you're confident are noise. Health check logs, connection pool events, standard library chatter. Not for application-level errors or business events.

Implementation example (using Fluent Bit):

[FILTER]
    Name grep
    Match kube.*
    Regex log health_check|kube-probe|ELB-HealthChecker
    Exclude On

This drops any log line matching those patterns before it leaves the node.

Option 3: Tiering (Different Retention for Different Logs)

What it is: Send all logs somewhere, but store high-value logs longer and cheaper logs shorter.

Pros:

• You keep everything initially (no data loss)
• You can search recent logs comprehensively
• Cost is lower because you're not storing verbose logs for months
• Supports compliance requirements (keep audit logs for 2 years, debug logs for 7 days)

Cons:

• More complex infrastructure (multiple storage tiers, routing rules)
• Requires understanding which logs have which value
• Still requires making retention decisions

When to use: For mature systems with diverse log types and regulatory requirements.

Implementation sketch:

# In your log router (Fluent Bit, Logstash, Vector)
# Route 1: High-value logs → long-term storage
<match audit.*>
  @type elasticsearch
  host elasticsearch-archive
  logstash_format true
  logstash_prefix audit
  # Rotates daily, keeps 365 days
</match>

# Route 2: Application errors → medium-term
<match app.error>
  @type elasticsearch
  host elasticsearch-main
  logstash_format true
  logstash_prefix app-error
  # Rotates daily, keeps 30 days via ILM policy
</match>

# Route 3: Debug logs → short-term, cheap storage
<match app.debug>
  @type s3
  s3_bucket logs-archive
  path logs/debug/%Y/%m/%d/
  # Rotates daily, auto-deletes after 7 days
</match>

Option 4: Structured Logging + Targeted Aggregation

What it is: Move away from unstructured text logs. Emit structured events (JSON) and aggregate only what you actually query.

Pros:

• Enables filtering at the source (don't emit verbose logs in production)
• Makes logs queryable and aggregatable (you can sum errors by endpoint)
• Reduces storage because you're not storing full stack traces for routine events

Cons:

• Requires code changes across your services
• Upfront investment in logging infrastructure
• You lose some context (full log lines are useful for debugging)

When to use: For new services or during a major refactor. Too expensive to retrofit across a large codebase.

Example (Python):

import json
import logging

class StructuredFormatter(logging.Formatter):
    def format(self, record):
        log_obj = {
            "timestamp": self.formatTime(record),
            "level": record.levelname,
            "logger": record.name,
            "message": record.getMessage(),
            "service": "payment-api",
        }
        if record.exc_info:
            log_obj["exception"] = self.formatException(record.exc_info)
        return json.dumps(log_obj)

handler = logging.StreamHandler()
handler.setFormatter(StructuredFormatter())
logger.addHandler(handler)

# Now logs are machine-readable and you can route based on level

The Kubernetes Complication

If you're running Kubernetes, you have an additional layer of complexity: every pod, every container, every namespace is generating logs. The default behavior (logging everything to stdout/stderr) is convenient but scales poorly.

What Actually Breaks

• Node-level log rotation fails: Kubelet doesn't rotate container logs aggressively. On a busy node with 100 pods, log files can consume 50GB+ in 24 hours.
• Log router becomes a bottleneck: Your single Fluent Bit daemonset can't keep up with the volume. Logs queue up, memory usage climbs, the router crashes, logs are lost.
• Namespace sprawl hides the problem: Each team runs their own services with their own logging config. Some teams log at debug level. Some teams log every request. There's no visibility into aggregate volume until the bill arrives.

Building Sustainable Log Routing in Kubernetes

Start with a simple principle: filter at the source, not at the aggregator.

Step 1: Set log levels by environment

# deployment.yaml
env:
  - name: LOG_LEVEL
    value: "INFO"  # Overridden to DEBUG in dev namespace
  - name: LOG_FORMAT
    value: "json"

Step 2: Configure Fluent Bit to drop known noise

# fluent-bit-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluent-bit-config
data:
  filter-kubernetes.conf: |
    [FILTER]
        Name kubernetes
        Match kube.*
        Kube_URL https://kubernetes.default.svc:443
        Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        Merge_Log On
        Keep_Log Off
        
    # Drop health check logs
    [FILTER]
        Name grep
        Match kube.*
        Regex log (health_check|readiness|liveness|startup)
        Exclude On

Step 3: Route based on namespace and log level

[OUTPUT]
    Name es
    Match kube.var.log.containers.*_production_*.log
    Host elasticsearch-prod
    Index prod-%Y.%m.%d

[OUTPUT]
    Name s3
    Match kube.var.log.containers.*_staging_*.log
    S3_bucket logs-staging
    # Cheaper storage for non-prod

Step 4: Monitor the router itself

This is critical. Set up alerts on:

• Fluent Bit buffer size (if it's consistently high, you're dropping logs)
• Processing latency (logs taking >5 seconds to reach Elasticsearch means you're behind)
• Dropped log count (Fluent Bit exposes a metric for this)

# prometheus rules
- alert: FluentBitBufferHigh
  expr: fluentbit_