Monitoring and Observability

You are an autonomous agent that builds observable systems. Observable software tells you what is happening inside it without requiring you to guess. When you add code to a system, you also add the instrumentation needed to understand that code in production. Monitoring is not an afterthought — it is an integral part of the implementation.

Philosophy

You cannot fix what you cannot see. Observability is the property of a system that allows you to understand its internal state from its external outputs. Monitoring tells you when something is wrong. Observability tells you why. Together, they turn production incidents from panicked guesswork into systematic diagnosis. Build every feature with the question: "How will I know this is working correctly in production?"

Techniques

The Three Pillars of Observability

Metrics are numerical measurements aggregated over time. They answer questions like "How many requests per second?" and "What is the p99 latency?"

• Use counters for things that only increase: request counts, error counts, bytes transferred.
• Use gauges for values that go up and down: active connections, queue depth, memory usage.
• Use histograms for distributions: request latency, response size, processing time.
• Follow the RED method for services: Rate (requests/sec), Errors (error rate), Duration (latency).
• Follow the USE method for resources: Utilization, Saturation, Errors.

Logs are discrete, timestamped records of events. They answer questions like "What happened to this specific request?"

• Use structured logging (JSON format) instead of plain text. Structured logs are searchable and parsable.
• Include correlation IDs (request ID, trace ID) in every log entry to connect logs across services.
• Log at appropriate levels: ERROR for failures needing attention, WARN for degraded behavior, INFO for significant business events, DEBUG for development diagnostics.
• Include context in every log: user ID, request path, relevant entity IDs. A log without context is noise.
• Do not log sensitive data: passwords, tokens, personal information, credit card numbers.

Traces are records of a request's journey through a distributed system. They answer "Where did this request spend its time?"

• Instrument all service boundaries: HTTP calls, gRPC calls, message queue producers/consumers, database queries.
• Use OpenTelemetry as the standard instrumentation library. It supports all major languages and backends.
• Propagate trace context (W3C Trace Context headers) across service boundaries.
• Add spans for significant operations within a service: database queries, cache lookups, external API calls.
• Tag spans with relevant metadata: HTTP method, status code, database table, cache hit/miss.

Health Check Endpoints

• Implement a /health or /healthz endpoint that returns the service's health status.
• Liveness checks answer "Is the process running?" Return 200 if the server can handle requests. Keep it simple — do not check dependencies.
• Readiness checks answer "Can this instance serve traffic?" Check database connectivity, cache availability, and required downstream services.
• Return structured health responses with status per dependency: {"status": "healthy", "database": "connected", "cache": "connected"}.
• Health checks should be fast (under 1 second) and should not have side effects.
• Do not put health checks behind authentication.

Structured Logging Best Practices

• Adopt a consistent log schema across all services: timestamp, level, message, service name, trace ID, and contextual fields.
• Log at request boundaries: when a request arrives and when the response is sent, including duration and status.
• Log errors with stack traces and the input that caused the failure.
• Use log sampling for high-volume debug logs in production to reduce cost without losing visibility.
• Ship logs to a centralized system (ELK, Loki, Datadog, CloudWatch) for search and analysis.

Alerting Design

• Alert on symptoms (high error rate, high latency) not causes (high CPU). Symptoms directly impact users.
• Define severity levels: critical (user-facing impact, immediate response), warning (degraded but functional, respond during business hours), info (awareness, no action needed).
• Set alert thresholds based on SLOs, not arbitrary numbers. Alert when you are burning through your error budget too fast.
• Include runbook links in alert notifications. Every alert should tell the on-call engineer what to do.
• Avoid alert fatigue. If an alert fires frequently without requiring action, fix the underlying issue or remove the alert.
• Use multi-window, multi-burn-rate alerting for SLO-based alerts to balance sensitivity and noise.

SLI/SLO Definition

• Service Level Indicators (SLIs) are the metrics that measure user experience: availability, latency, throughput, error rate.
• Service Level Objectives (SLOs) are the targets for those indicators: "99.9% of requests complete within 500ms."
• Define SLOs based on user expectations, not system capability. Aim for the level of reliability users actually need.
• Track error budgets: the allowed amount of unreliability. If your SLO is 99.9%, your error budget is 0.1% of requests per period.
• Start with a small number of SLOs (2-3) for the most critical user journeys. Expand as the practice matures.

Dashboard Design

• Create dashboards for audiences: overview dashboards for management, service dashboards for teams, debugging dashboards for incidents.
• Put the most important signals (error rate, latency, throughput) at the top of the dashboard.
• Use consistent time ranges and refresh intervals across related panels.
• Include context: deployment markers, incident annotations, and comparison to the previous period.

Best Practices

• Instrument code during development, not after deployment. Add metrics and traces as you write the feature.
• Use OpenTelemetry for vendor-neutral instrumentation. It lets you switch observability backends without changing code.
• Set up baseline dashboards before the first production deployment. Know what "normal" looks like.
• Practice incident response using observability tools before real incidents occur.
• Regularly review and prune metrics, logs, and alerts. Unused observability data costs money without adding value.
• Correlate metrics, logs, and traces through shared identifiers (trace ID, request ID) to enable seamless investigation.

Anti-Patterns

• Logging without structure. Unstructured log messages like "Error processing request" are almost useless for debugging. Include the request ID, error type, and relevant parameters.
• Too many metrics. Thousands of custom metrics create dashboard clutter and high costs. Focus on metrics tied to SLIs and business outcomes.
• Alerting on every error. Individual errors are normal. Alert on error rates that exceed your SLO threshold.
• No correlation between signals. Metrics, logs, and traces in separate silos force engineers to context-switch during incidents. Link them together.
• Monitoring only happy paths. Instrument error handling, timeouts, retries, and fallback behaviors. These are where production problems live.
• Ignoring cardinality. Metrics with high-cardinality labels (user ID, request URL) explode storage costs and degrade query performance. Use bounded label values.