Skip to main content
Technology & EngineeringObservability Services209 lines

Opentelemetry

Instrument applications with OpenTelemetry for distributed traces, metrics, and logs.

Quick Summary32 lines
You are an expert in OpenTelemetry instrumentation, the CNCF observability framework. You guide developers through SDK setup, collector configuration, context propagation, and exporter pipelines across languages and backends.

## Key Points

- **Forgetting to call `span.end()`** - Spans that never end leak memory and never export.
- **High-cardinality attributes** - Avoid unbounded values like user IDs or request bodies as metric labels; use them only on trace spans.
- **Initializing SDK after imports** - Auto-instrumentation must patch libraries before they are imported. Load tracing.ts first.
- **Ignoring the collector** - Sending directly from apps to backends couples deployment; always route through a collector in production.
- You need vendor-neutral instrumentation that works with Jaeger, Grafana, Datadog, or any OTLP backend.
- You are building microservices and need distributed tracing across service boundaries.
- You want a unified SDK for traces, metrics, and logs instead of separate libraries.
- You need to add custom business metrics alongside auto-instrumented HTTP and database spans.
- You are migrating from a proprietary agent and want portable telemetry.

## Quick Example

```bash
npm install @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-http \
  @opentelemetry/exporter-metrics-otlp-http
```

```typescript
import { SEMATTRS_HTTP_METHOD, SEMATTRS_HTTP_STATUS_CODE } from '@opentelemetry/semantic-conventions';

span.setAttribute(SEMATTRS_HTTP_METHOD, 'POST');
span.setAttribute(SEMATTRS_HTTP_STATUS_CODE, 200);
span.setAttribute('enduser.id', userId);
```
skilldb get observability-services-skills/OpentelemetryFull skill: 209 lines
Paste into your CLAUDE.md or agent config

OpenTelemetry Integration

You are an expert in OpenTelemetry instrumentation, the CNCF observability framework. You guide developers through SDK setup, collector configuration, context propagation, and exporter pipelines across languages and backends.

Core Philosophy

Vendor-Neutral Telemetry

OpenTelemetry provides a single set of APIs and SDKs to emit traces, metrics, and logs without locking into any backend. Instrument once, export anywhere.

Context Propagation

Distributed tracing depends on propagating trace context (W3C TraceContext, B3) across service boundaries via HTTP headers and messaging metadata.

Collector as Pipeline

The OTel Collector decouples instrumentation from backends. It receives, processes, and exports telemetry data, enabling filtering, batching, and routing without code changes.

Setup

Install the SDK and auto-instrumentation for Node.js:

npm install @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-http \
  @opentelemetry/exporter-metrics-otlp-http

Initialize the SDK early in your application entry point:

// tracing.ts - import BEFORE any other module
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter({
    url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4318/v1/traces',
  }),
  metricReader: new PeriodicExportingMetricReader({
    exporter: new OTLPMetricExporter({
      url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4318/v1/metrics',
    }),
    exportIntervalMillis: 30000,
  }),
  instrumentations: [getNodeAutoInstrumentations()],
  serviceName: process.env.OTEL_SERVICE_NAME || 'my-service',
});

sdk.start();
process.on('SIGTERM', () => sdk.shutdown());

Collector config (otel-collector-config.yaml):

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1024
  memory_limiter:
    check_interval: 1s
    limit_mib: 512

exporters:
  otlphttp:
    endpoint: https://your-backend.example.com
  logging:
    loglevel: debug

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp, logging]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlphttp]

Key Patterns

Do: Add custom spans for business-critical operations

import { trace } from '@opentelemetry/api';

const tracer = trace.getTracer('order-service');

async function processOrder(orderId: string) {
  return tracer.startActiveSpan('processOrder', async (span) => {
    span.setAttribute('order.id', orderId);
    try {
      const result = await chargePayment(orderId);
      span.setAttribute('order.status', 'charged');
      return result;
    } catch (err) {
      span.recordException(err as Error);
      span.setStatus({ code: 2, message: (err as Error).message });
      throw err;
    } finally {
      span.end();
    }
  });
}

Not: Creating spans for every trivial function call

// WRONG - excessive span noise
function add(a: number, b: number) {
  return tracer.startActiveSpan('add', (span) => {
    const result = a + b;
    span.end();
    return result;
  });
}

Do: Use semantic conventions for attribute names

import { SEMATTRS_HTTP_METHOD, SEMATTRS_HTTP_STATUS_CODE } from '@opentelemetry/semantic-conventions';

span.setAttribute(SEMATTRS_HTTP_METHOD, 'POST');
span.setAttribute(SEMATTRS_HTTP_STATUS_CODE, 200);
span.setAttribute('enduser.id', userId);

Common Patterns

Custom Metrics

import { metrics } from '@opentelemetry/api';

const meter = metrics.getMeter('app-metrics');
const requestCounter = meter.createCounter('http.requests.total', {
  description: 'Total HTTP requests',
});
const requestDuration = meter.createHistogram('http.request.duration_ms', {
  description: 'HTTP request duration in milliseconds',
});

function handleRequest(method: string, route: string, durationMs: number) {
  requestCounter.add(1, { method, route });
  requestDuration.record(durationMs, { method, route });
}

Baggage for Cross-Service Context

import { propagation, context, BaggageEntry } from '@opentelemetry/api';

const baggage = propagation.createBaggage({
  'tenant.id': { value: 'acme-corp' } as BaggageEntry,
});
const ctx = propagation.setBaggage(context.active(), baggage);
context.with(ctx, () => callDownstreamService());

Resource Attributes for Service Identity

import { Resource } from '@opentelemetry/resources';
import { SEMRESATTRS_SERVICE_NAME, SEMRESATTRS_DEPLOYMENT_ENVIRONMENT } from '@opentelemetry/semantic-conventions';

const resource = new Resource({
  [SEMRESATTRS_SERVICE_NAME]: 'checkout-api',
  [SEMRESATTRS_DEPLOYMENT_ENVIRONMENT]: 'production',
  'service.version': '2.1.0',
});

Anti-Patterns

  • Forgetting to call span.end() - Spans that never end leak memory and never export.
  • High-cardinality attributes - Avoid unbounded values like user IDs or request bodies as metric labels; use them only on trace spans.
  • Initializing SDK after imports - Auto-instrumentation must patch libraries before they are imported. Load tracing.ts first.
  • Ignoring the collector - Sending directly from apps to backends couples deployment; always route through a collector in production.

When to Use

  • You need vendor-neutral instrumentation that works with Jaeger, Grafana, Datadog, or any OTLP backend.
  • You are building microservices and need distributed tracing across service boundaries.
  • You want a unified SDK for traces, metrics, and logs instead of separate libraries.
  • You need to add custom business metrics alongside auto-instrumented HTTP and database spans.
  • You are migrating from a proprietary agent and want portable telemetry.

Install this skill directly: skilldb add observability-services-skills

Get CLI access →