You are an expert in OpenTelemetry distributed tracing for building observable systems.

## Key Points

- **Trace**: A directed acyclic graph of spans representing a single end-to-end operation (e.g., an API request).
- **Span**: A timed operation within a trace. Each span records a name, start/end timestamps, status, attributes, and events.
- **SpanContext**: The propagated identity of a span — contains `trace_id`, `span_id`, and `trace_flags`. This is what crosses process boundaries.
- **Context propagation**: The mechanism (usually W3C Trace Context headers `traceparent`/`tracestate`) that carries SpanContext across HTTP, gRPC, or messaging boundaries.
- **Instrumentation**: Auto-instrumentation libraries that patch popular frameworks (Flask, Express, gRPC) and manual spans for custom business logic.
- **Collector**: The OTel Collector is a standalone process that receives, processes, and exports telemetry. It decouples applications from backends.
- **Sampling**: Head-based (decide at trace start) or tail-based (decide after trace completes) strategies that control trace volume and cost.
- **Set `service.name` and `deployment.environment` on every service resource.** These are the primary grouping keys in every tracing backend.
- **Use auto-instrumentation first, then add manual spans for business logic.** Auto-instrumentation covers HTTP, DB, and messaging; manual spans capture domain-specific operations.
- **Propagate context through async boundaries.** Ensure message queues, background jobs, and event-driven flows carry the `traceparent` header.
- **Add semantic attributes following OTel conventions.** Use `http.request.method`, `db.system`, `rpc.service` etc. from the semantic conventions spec.
- **Deploy the Collector as a sidecar or daemonset,** not embedded in the application. This allows buffering, sampling, and routing changes without redeploying apps.

Distributed Tracing — Observability

You are an expert in OpenTelemetry distributed tracing for building observable systems.

Overview

Distributed tracing captures the full journey of a request as it flows through multiple services, producing a tree of spans that reveal latency, errors, and dependencies. OpenTelemetry (OTel) is the industry-standard framework providing vendor-neutral APIs, SDKs, and the Collector for trace (and metric/log) telemetry. A well-instrumented system lets engineers answer "why was this request slow?" in minutes rather than hours.

Core Concepts

• Trace: A directed acyclic graph of spans representing a single end-to-end operation (e.g., an API request).
• Span: A timed operation within a trace. Each span records a name, start/end timestamps, status, attributes, and events.
• SpanContext: The propagated identity of a span — contains trace_id, span_id, and trace_flags. This is what crosses process boundaries.
• Context propagation: The mechanism (usually W3C Trace Context headers traceparent/tracestate) that carries SpanContext across HTTP, gRPC, or messaging boundaries.
• Instrumentation: Auto-instrumentation libraries that patch popular frameworks (Flask, Express, gRPC) and manual spans for custom business logic.
• Collector: The OTel Collector is a standalone process that receives, processes, and exports telemetry. It decouples applications from backends.
• Sampling: Head-based (decide at trace start) or tail-based (decide after trace completes) strategies that control trace volume and cost.

Implementation Patterns

Python — auto + manual instrumentation

# Install: pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-otlp
#          pip install opentelemetry-instrumentation-fastapi opentelemetry-instrumentation-requests

from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.sdk.resources import Resource

resource = Resource.create({"service.name": "order-service", "deployment.environment": "production"})
provider = TracerProvider(resource=resource)
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint="http://otel-collector:4317")))
trace.set_tracer_provider(provider)

# Auto-instrument FastAPI and requests
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
FastAPIInstrumentor.instrument()
RequestsInstrumentor.instrument()

# Manual span for business logic
tracer = trace.get_tracer("order-service")

async def place_order(order):
    with tracer.start_as_current_span("place_order", attributes={"order.id": order.id}) as span:
        inventory = check_inventory(order)
        span.add_event("inventory_checked", {"available": inventory.available})
        if not inventory.available:
            span.set_status(trace.StatusCode.ERROR, "out of stock")
            raise OutOfStockError()
        charge_payment(order)
        span.add_event("payment_charged")
        return confirm_order(order)

Node.js — OTel SDK setup

// tracing.js — require this before your app starts
const { NodeSDK } = require("@opentelemetry/sdk-node");
const { OTLPTraceExporter } = require("@opentelemetry/exporter-trace-otlp-grpc");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");

const sdk = new NodeSDK({
  serviceName: "payment-service",
  traceExporter: new OTLPTraceExporter({ url: "http://otel-collector:4317" }),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

// Manual span in application code
const { trace } = require("@opentelemetry/api");
const tracer = trace.getTracer("payment-service");

async function processPayment(paymentId, amount) {
  return tracer.startActiveSpan("processPayment", { attributes: { "payment.id": paymentId } }, async (span) => {
    try {
      const result = await gateway.charge(amount);
      span.setAttribute("payment.status", result.status);
      return result;
    } catch (err) {
      span.recordException(err);
      span.setStatus({ code: trace.SpanStatusCode.ERROR, message: err.message });
      throw err;
    } finally {
      span.end();
    }
  });
}

OTel Collector configuration

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  # Tail-based sampling: keep all error traces and 10% of successful ones
  tail_sampling:
    decision_wait: 10s
    policies:
      - name: errors
        type: status_code
        status_code: { status_codes: [ERROR] }
      - name: sample-rest
        type: probabilistic
        probabilistic: { sampling_percentage: 10 }

exporters:
  otlp/jaeger:
    endpoint: jaeger:4317
    tls:
      insecure: true
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [tail_sampling, batch]
      exporters: [otlp/jaeger]

Docker Compose snippet for local dev

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.96.0
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317"
      - "4318:4318"

  jaeger:
    image: jaegertracing/all-in-one:1.54
    ports:
      - "16686:16686"   # UI
      - "4317"          # OTLP gRPC

Core Philosophy

Distributed tracing exists to answer a question that logs and metrics cannot: "what happened to this specific request as it flowed through the system?" In a monolith, a stack trace tells the full story. In a distributed system, a request touches five, ten, or fifty services, and understanding its journey requires stitching together observations from all of them. Tracing provides that stitching through context propagation — a trace ID that follows the request across every network boundary, creating a unified view of an operation that spans multiple processes, hosts, and even cloud regions.

OpenTelemetry's vendor-neutral approach is not just a technical convenience — it is a strategic decision. Tracing backends are expensive and organizations switch between them (Jaeger to Tempo, Datadog to Honeycomb) more often than they expect. By instrumenting with OTel's API and routing through the Collector, you decouple your application code from your backend choice. The Collector becomes the control plane for your telemetry: you can add sampling, filter sensitive data, route to multiple backends, or switch providers entirely — all without redeploying a single application.

The right granularity for spans is one of the hardest judgment calls in tracing. Too few spans and you cannot see where latency lives. Too many spans and traces become expensive, slow to render, and hard to interpret. The guiding principle is: create a span for every network call (HTTP, gRPC, database query, message publish) and every significant business operation (place order, process payment, validate inventory). Do not create spans for internal function calls, loop iterations, or utility operations. A trace should read like a high-level narrative of what the system did, not a line-by-line execution log.

Anti-Patterns

• Broken context propagation. If even one service in the request path fails to extract and inject the traceparent header, the trace splits into disconnected fragments. This is the single most common tracing problem and it renders traces useless for the requests that cross the broken service. Verify end-to-end trace connectivity with integration tests.

• Instrumenting every function. Creating a span for every method call produces traces with thousands of spans that are expensive to store, slow to render, and impossible to interpret. Instrument at network boundaries and significant business operations only.

• Head-based sampling at aggressive rates. Dropping 99% of traces at the point of creation means 99% of error traces are also dropped, making tracing useless for incident investigation. Use tail-based sampling in the Collector so error and high-latency traces are always retained regardless of the sampling rate for successful requests.

• Not recording errors on spans. Catching an exception, logging it, but not calling span.set_status(ERROR) and span.record_exception() means the trace backend cannot distinguish failed requests from successful ones. Error traces become invisible in trace search and analysis.

• Embedding the exporter in the application. Exporting traces directly from the application to the backend couples your app to the backend's API and makes sampling, filtering, and routing changes require application redeployments. Always export to the OTel Collector and let the Collector handle processing and routing.

Best Practices

• Set service.name and deployment.environment on every service resource. These are the primary grouping keys in every tracing backend.
• Use auto-instrumentation first, then add manual spans for business logic. Auto-instrumentation covers HTTP, DB, and messaging; manual spans capture domain-specific operations.
• Propagate context through async boundaries. Ensure message queues, background jobs, and event-driven flows carry the traceparent header.
• Add semantic attributes following OTel conventions. Use http.request.method, db.system, rpc.service etc. from the semantic conventions spec.
• Deploy the Collector as a sidecar or daemonset, not embedded in the application. This allows buffering, sampling, and routing changes without redeploying apps.
• Use tail-based sampling for cost control. Keep 100% of error and high-latency traces, sample a percentage of successful ones.
• Record exceptions with span.record_exception(). This attaches the full stack trace to the span as an event.

Common Pitfalls

• Missing context propagation. If a service does not extract and inject trace context, the trace breaks into disconnected fragments. Verify with an end-to-end test that a single trace ID appears in all participating services.
• Creating too many spans. Instrumenting every function or loop iteration produces enormous traces that are expensive and hard to read. Span per network call or significant business step is the right granularity.
• Not setting span status on errors. Without span.set_status(ERROR), the trace backend cannot distinguish failed requests from successful ones.
• Head-based sampling too aggressively. Dropping 99% of traces at the head means most error traces are also dropped. Prefer tail-based sampling or always-sample for error paths.
• Ignoring clock skew. In multi-host environments, NTP drift can make parent spans appear to end before child spans start. Ensure NTP is configured on all hosts.