Monitoring and Observability

---
name: monitoring-observability
description: "Build observability with metrics, logs, and traces to understand system behavior and diagnose issues."
metadata:
  tags: "monitoring, observability, metrics, logs, tracing, slo-alerting, devops"
  source: "https://skilldb.dev/skills/devops-cloud-skills/monitoring-observability"
  pack: "devops-cloud-skills"
  category: "Technology & Engineering"
---

# Monitoring and Observability

## When to use this skill
Use when the user says things like:
- "How do I monitor my system and gain visibility?"
- "Help me set up metrics, logs, and traces for a service."
- "I need SLOs and alerting to reduce noise."
- "Explain how to build observability dashboards."
- "Assist with tracing requests across microservices."


## Core Philosophy
Observability is the ability to understand the internal state of a system by
examining its external outputs. While monitoring tells you when something is wrong,
observability helps you understand why. The three pillars — metrics, logs, and
traces — provide complementary views into system behavior. Metrics show trends and
aggregates, logs capture discrete events with context, and traces follow individual
requests across service boundaries.

## Key Techniques
- **RED Method**: Monitor Rate (requests per second), Errors (failed requests per
  second), and Duration (latency distribution) for every service. Covers the
  essential user-facing health signals.
- **USE Method**: Monitor Utilization, Saturation, and Errors for every resource
  (CPU, memory, disk, network). Covers infrastructure health.
- **Distributed Tracing**: Propagate trace context across service boundaries to
  reconstruct the full path of a request through a microservices architecture.
- **SLO-Based Alerting**: Define Service Level Objectives and alert on error
  budget burn rate rather than arbitrary thresholds, reducing alert fatigue.
- **Structured Logging**: Emit logs as structured data (JSON) with consistent
  fields (timestamp, service, trace ID, level) to enable machine parsing and
  correlation.
- **Custom Metrics**: Instrument application code to emit business-relevant metrics
  (orders processed, payments completed) alongside technical metrics.

## Best Practices
- Instrument before you need it. Adding observability after an outage is too late.
- Use consistent naming conventions for metrics across all services.
- Set alerts on symptoms (user-visible errors, latency) not causes (CPU usage).
  High CPU that does not affect users is not an emergency.
- Include runbooks with every alert that explain what the alert means and the
  first diagnostic steps to take.
- Retain high-resolution metrics for days, downsampled metrics for months, and
  logs for weeks unless compliance requires longer.
- Correlate metrics, logs, and traces using shared identifiers (trace ID, request
  ID) to enable seamless debugging workflows.
- Dashboard for understanding, alert for action. Dashboards should tell a story;
  alerts should be actionable.

## Common Patterns
- **Golden Signals Dashboard**: A single dashboard per service showing latency,
  traffic, errors, and saturation — the four signals that matter most.
- **On-Call Escalation**: Tiered alerting that pages the primary on-call, escalates
  to secondary after a timeout, and notifies management for extended incidents.
- **Anomaly Detection**: Use statistical methods or ML to detect unusual patterns
  in metrics that static thresholds would miss.
- **Chaos Engineering Validation**: Use controlled failure injection to verify
  that monitoring detects and alerts on real failure modes.

## Anti-Patterns
- Alert fatigue from too many low-priority or flapping alerts. Every ignored alert
  makes it more likely that a real incident gets missed.
- Dashboard sprawl with hundreds of charts nobody looks at. Curate dashboards
  for specific audiences and purposes.
- Logging everything at DEBUG level in production, creating massive storage costs
  and making it impossible to find signal in noise.
- Monitoring only infrastructure metrics while ignoring application-level and
  business-level signals.
- Not testing alerting pipelines. Verify that alerts actually reach on-call
  engineers through regular fire drills.
- Treating observability as a separate team's responsibility rather than an
  integral part of every developer's workflow.