Monitoring and Observability: Logs, Metrics, Traces

Observability is the ability to infer the internal state of a system from its external outputs alone: logs, metrics, and traces.

Observability is the ability to infer a system's internal state from its external outputs (logs, metrics, traces). Monitoring watches for problems you defined in advance; observability lets you explore problems you did not anticipate.

OpenTelemetry has become widely adopted across cloud-native observability. AIOps now reaches past simple anomaly detection into failure prediction and automated remediation. eBPF makes kernel-level telemetry collection cheap enough to run everywhere, which shifts responsibility for observability from application teams toward platform teams.

The three pillars of observability

1. Logging

Records discrete events from the system in chronological order.

Structured logging:

{
  "timestamp": "2025-01-27T09:30:00Z",
  "level": "ERROR",
  "service": "payment-service",
  "traceId": "abc123def456",
  "userId": "user-789",
  "message": "Payment processing failed",
  "error": "insufficient_funds",
  "amount": 50000,
  "currency": "KRW"
}

Unstructured vs structured logs:

Log collection tools:

Log level guidelines:

• FATAL: errors severe enough to stop the service
• ERROR: a request failed and needs immediate attention
• WARN: a potential problem worth watching
• INFO: significant business events (payment completed, user signed up)
• DEBUG: detailed development information (disabled in production)

2. Metrics

Collects and aggregates numerical measurements over time.

Metric types:

Prometheus + Grafana:

Prometheus is a pull-based time-series database that scrapes targets periodically to collect metrics. In the configuration below, job_name names the group of targets, scrape_interval sets the collection period, and targets lists the endpoints to scrape.

# Prometheus scrape configuration
scrape_configs:
  - job_name: 'payment-service'
    scrape_interval: 15s
    static_configs:
      - targets: ['payment-service:8080']

# PromQL query examples
# requests per second over the last 5 minutes
rate(http_requests_total[5m])
 
# 95th percentile response time
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
 
# error rate
rate(http_requests_total{status=~"5.."}[5m]) / rate(http_requests_total[5m])

3. Distributed tracing

Follows the full path a single request takes across multiple services in a distributed system.

User request
  └── API Gateway (Span 1: 2ms)
      └── Auth Service (Span 2: 5ms)
      └── Product Service (Span 3: 15ms)
          └── Database Query (Span 4: 10ms)
          └── Cache Lookup (Span 5: 1ms)
      └── Payment Service (Span 6: 50ms)
          └── External Payment API (Span 7: 45ms)

Core concepts:

• Trace: the complete record of one request
• Span: an individual unit of work within a trace
• Context propagation: passing TraceID/SpanID between services

Tools:

OpenTelemetry (OTel)

Concept

OpenTelemetry is a CNCF project and a unified standard for collecting, processing, and exporting logs, metrics, and traces in a vendor-neutral way. As of 2025, cloud-native organizations are adopting OTel as their default data standard at an accelerating rate.

Application (OTel SDK)
       │
       ▼
OTel Collector
  ├── Exporter → Prometheus (metrics)
  ├── Exporter → Loki (logs)
  ├── Exporter → Tempo/Jaeger (traces)
  └── Exporter → Pyroscope (profiling)

OTel Collector architecture

Collector configuration has three stages. receivers are the entry points that accept data (OTLP gRPC/HTTP), and processors transform it through steps such as batching and memory limiting. exporters are the exits that ship it to a backend. service.pipelines defines how those three stages are wired together for traces, metrics, and logs.

# OTel Collector configuration
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
 
processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  memory_limiter:
    limit_mib: 512
 
exporters:
  prometheus:
    endpoint: 0.0.0.0:8889
  otlp/tempo:
    endpoint: tempo:4317
  loki:
    endpoint: http://loki:3100/loki/api/v1/push
 
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch, memory_limiter]
      exporters: [otlp/tempo]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [loki]

What OTel buys you

1. No vendor lock-in: backends can be swapped freely
2. Unified instrumentation: one SDK collects logs, metrics, and traces
3. Auto-instrumentation: HTTP, DB, and gRPC calls are captured without code changes
4. A large ecosystem: every major language and framework is supported

SRE indicators (SLA, SLO, SLI)

Definitions

How they relate:

SLI (measured) → SLO (internal target) → SLA (external promise)
99.97%           99.95%                   99.9%
 
SLO > SLA: a safety margin that keeps the SLA satisfiable

Error budget

The amount of failure an SLO permits.

SLO = 99.95% (monthly)
Error budget = 100% - 99.95% = 0.05%
Allowed monthly downtime = 30 days x 24 hours x 60 minutes x 0.05% = about 21.6 minutes

Error budget policy:

• Budget remaining → ship new features, run experiments
• Budget exhausted → freeze new deploys, focus on reliability work
• Alert on how fast the budget is being spent → burn rate alerting

Golden signals

The four core monitoring indicators defined by the Google SRE team.

RED method (for microservices):

• Rate: requests per second
• Errors: error rate
• Duration: request processing time

USE method (for infrastructure):

• Utilization: resource utilization
• Saturation: queue depth
• Errors: error count

eBPF-based observability

Concept

eBPF (extended Berkeley Packet Filter) runs sandboxed programs inside the Linux kernel. That makes it possible to collect networking, security, and observability data without changing application code.

Traditional approach vs eBPF:

Tools

Continuous profiling

Concept

Profiling CPU, memory, and I/O continuously in production to identify performance bottlenecks. Treating profiles as a fourth observability signal alongside the three pillars is increasingly common.

Tools:

• Pyroscope: an open-source continuous profiling platform
• Parca: eBPF-based, low overhead
• Datadog Continuous Profiler: SaaS profiling integrated with the rest of the platform

Flame graph: The standard way to visualize profiling output, where stack depth (call order) and width (share of time) are both readable at a glance.

AIOps (a 2026 trend)

Concept

AIOps (Artificial Intelligence for IT Operations) applies AI and ML to automate monitoring, analysis, and optimization. By 2025 it has moved past basic anomaly detection into failure prediction, configuration drift detection, and automated remediation.

Core AIOps capabilities

Data collection → Anomaly detection → Correlation → Root cause inference → Automated action
   (OTel)           (ML models)        (topology)     (causal inference)     (self-healing)

1. Alert noise reduction: group related alerts automatically to prevent alert fatigue
2. Anomaly detection: learn historical patterns and flag abnormal metrics automatically
3. Root cause analysis: infer causes from service topology and correlations
4. Predictive observability: detect and respond before a failure occurs

AIOps tools (2026)

Real user monitoring (RUM) and synthetic monitoring

RUM (real user monitoring)

Collects performance data from real users' browsers and devices.

What it measures:

• Core Web Vitals: LCP (Largest Contentful Paint), INP (Interaction to Next Paint, which replaced FID as of 2024-03), CLS (Cumulative Layout Shift)
• Page load time, error rate, session replay

Synthetic monitoring

Generates artificial traffic to run predefined scenarios on a schedule and measure performance.

RUM: real user data → "what users are experiencing right now"
Synthetic: simulated data → "find problems even when nobody is using the site"

Controlling observability cost

One of the main challenges in 2025 is observability cost management. Data volume grows exponentially, and the resulting spend now accounts for a meaningful share of total cloud cost.

Optimization strategies

1. Sampling: do not store every trace; use head-based or tail-based sampling
2. Metric aggregation: downsample high-resolution metrics as they age
3. Log level tuning: collect only INFO and above in production
4. Retention policy: tier storage across hot, warm, and cold
5. Collect only what matters: filter unnecessary data in an OTel processor

Summary

Observability is the ability to infer internal state from three pillars: logs, metrics, and traces. OpenTelemetry has become the standard for collecting all three vendor-neutrally, and the OTel Collector is what makes swapping backends painless. Operational goals become numbers through SLA, SLO, SLI, and error budgets, while the golden signals or RED/USE decide what to look at. eBPF gathers kernel-level telemetry without code changes, and AIOps is widening its scope from anomaly detection to root cause inference and automated action. As data volume grows, sampling and retention policy have to be managed alongside everything else.

The next post covers failure response and incident operations built on these signals.

References

• Observability Engineering - Charity Majors, Liz Fong-Jones, George Miranda
• Site Reliability Engineering - Google SRE Team
• OpenTelemetry Documentation: https://opentelemetry.io/docs
• Prometheus Documentation: https://prometheus.io/docs