Opentelemetry
Instrument applications with OpenTelemetry for distributed traces, metrics, and logs.
You are an expert in OpenTelemetry instrumentation, the CNCF observability framework. You guide developers through SDK setup, collector configuration, context propagation, and exporter pipelines across languages and backends.
## Key Points
- **Forgetting to call `span.end()`** - Spans that never end leak memory and never export.
- **High-cardinality attributes** - Avoid unbounded values like user IDs or request bodies as metric labels; use them only on trace spans.
- **Initializing SDK after imports** - Auto-instrumentation must patch libraries before they are imported. Load tracing.ts first.
- **Ignoring the collector** - Sending directly from apps to backends couples deployment; always route through a collector in production.
- You need vendor-neutral instrumentation that works with Jaeger, Grafana, Datadog, or any OTLP backend.
- You are building microservices and need distributed tracing across service boundaries.
- You want a unified SDK for traces, metrics, and logs instead of separate libraries.
- You need to add custom business metrics alongside auto-instrumented HTTP and database spans.
- You are migrating from a proprietary agent and want portable telemetry.
## Quick Example
```bash
npm install @opentelemetry/sdk-node \
@opentelemetry/auto-instrumentations-node \
@opentelemetry/exporter-trace-otlp-http \
@opentelemetry/exporter-metrics-otlp-http
```
```typescript
import { SEMATTRS_HTTP_METHOD, SEMATTRS_HTTP_STATUS_CODE } from '@opentelemetry/semantic-conventions';
span.setAttribute(SEMATTRS_HTTP_METHOD, 'POST');
span.setAttribute(SEMATTRS_HTTP_STATUS_CODE, 200);
span.setAttribute('enduser.id', userId);
```skilldb get observability-services-skills/OpentelemetryFull skill: 209 linesOpenTelemetry Integration
You are an expert in OpenTelemetry instrumentation, the CNCF observability framework. You guide developers through SDK setup, collector configuration, context propagation, and exporter pipelines across languages and backends.
Core Philosophy
Vendor-Neutral Telemetry
OpenTelemetry provides a single set of APIs and SDKs to emit traces, metrics, and logs without locking into any backend. Instrument once, export anywhere.
Context Propagation
Distributed tracing depends on propagating trace context (W3C TraceContext, B3) across service boundaries via HTTP headers and messaging metadata.
Collector as Pipeline
The OTel Collector decouples instrumentation from backends. It receives, processes, and exports telemetry data, enabling filtering, batching, and routing without code changes.
Setup
Install the SDK and auto-instrumentation for Node.js:
npm install @opentelemetry/sdk-node \
@opentelemetry/auto-instrumentations-node \
@opentelemetry/exporter-trace-otlp-http \
@opentelemetry/exporter-metrics-otlp-http
Initialize the SDK early in your application entry point:
// tracing.ts - import BEFORE any other module
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';
import { OTLPMetricExporter } from '@opentelemetry/exporter-metrics-otlp-http';
import { PeriodicExportingMetricReader } from '@opentelemetry/sdk-metrics';
const sdk = new NodeSDK({
traceExporter: new OTLPTraceExporter({
url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4318/v1/traces',
}),
metricReader: new PeriodicExportingMetricReader({
exporter: new OTLPMetricExporter({
url: process.env.OTEL_EXPORTER_OTLP_ENDPOINT || 'http://localhost:4318/v1/metrics',
}),
exportIntervalMillis: 30000,
}),
instrumentations: [getNodeAutoInstrumentations()],
serviceName: process.env.OTEL_SERVICE_NAME || 'my-service',
});
sdk.start();
process.on('SIGTERM', () => sdk.shutdown());
Collector config (otel-collector-config.yaml):
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 5s
send_batch_size: 1024
memory_limiter:
check_interval: 1s
limit_mib: 512
exporters:
otlphttp:
endpoint: https://your-backend.example.com
logging:
loglevel: debug
service:
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp, logging]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp]
Key Patterns
Do: Add custom spans for business-critical operations
import { trace } from '@opentelemetry/api';
const tracer = trace.getTracer('order-service');
async function processOrder(orderId: string) {
return tracer.startActiveSpan('processOrder', async (span) => {
span.setAttribute('order.id', orderId);
try {
const result = await chargePayment(orderId);
span.setAttribute('order.status', 'charged');
return result;
} catch (err) {
span.recordException(err as Error);
span.setStatus({ code: 2, message: (err as Error).message });
throw err;
} finally {
span.end();
}
});
}
Not: Creating spans for every trivial function call
// WRONG - excessive span noise
function add(a: number, b: number) {
return tracer.startActiveSpan('add', (span) => {
const result = a + b;
span.end();
return result;
});
}
Do: Use semantic conventions for attribute names
import { SEMATTRS_HTTP_METHOD, SEMATTRS_HTTP_STATUS_CODE } from '@opentelemetry/semantic-conventions';
span.setAttribute(SEMATTRS_HTTP_METHOD, 'POST');
span.setAttribute(SEMATTRS_HTTP_STATUS_CODE, 200);
span.setAttribute('enduser.id', userId);
Common Patterns
Custom Metrics
import { metrics } from '@opentelemetry/api';
const meter = metrics.getMeter('app-metrics');
const requestCounter = meter.createCounter('http.requests.total', {
description: 'Total HTTP requests',
});
const requestDuration = meter.createHistogram('http.request.duration_ms', {
description: 'HTTP request duration in milliseconds',
});
function handleRequest(method: string, route: string, durationMs: number) {
requestCounter.add(1, { method, route });
requestDuration.record(durationMs, { method, route });
}
Baggage for Cross-Service Context
import { propagation, context, BaggageEntry } from '@opentelemetry/api';
const baggage = propagation.createBaggage({
'tenant.id': { value: 'acme-corp' } as BaggageEntry,
});
const ctx = propagation.setBaggage(context.active(), baggage);
context.with(ctx, () => callDownstreamService());
Resource Attributes for Service Identity
import { Resource } from '@opentelemetry/resources';
import { SEMRESATTRS_SERVICE_NAME, SEMRESATTRS_DEPLOYMENT_ENVIRONMENT } from '@opentelemetry/semantic-conventions';
const resource = new Resource({
[SEMRESATTRS_SERVICE_NAME]: 'checkout-api',
[SEMRESATTRS_DEPLOYMENT_ENVIRONMENT]: 'production',
'service.version': '2.1.0',
});
Anti-Patterns
- Forgetting to call
span.end()- Spans that never end leak memory and never export. - High-cardinality attributes - Avoid unbounded values like user IDs or request bodies as metric labels; use them only on trace spans.
- Initializing SDK after imports - Auto-instrumentation must patch libraries before they are imported. Load tracing.ts first.
- Ignoring the collector - Sending directly from apps to backends couples deployment; always route through a collector in production.
When to Use
- You need vendor-neutral instrumentation that works with Jaeger, Grafana, Datadog, or any OTLP backend.
- You are building microservices and need distributed tracing across service boundaries.
- You want a unified SDK for traces, metrics, and logs instead of separate libraries.
- You need to add custom business metrics alongside auto-instrumented HTTP and database spans.
- You are migrating from a proprietary agent and want portable telemetry.
Install this skill directly: skilldb add observability-services-skills
Related Skills
Axiom
Integrate Axiom for log management, analytics, and real-time dashboards.
Elastic Apm
Instrument applications with Elastic APM and the ELK Stack for traces, logs, and metrics.
Grafana
Build Grafana dashboards, configure data sources, and set up alerting rules.
Honeycomb
Integrate Honeycomb for event-driven observability with high-cardinality tracing.
Jaeger
Deploy and integrate Jaeger for distributed tracing across microservices.
New Relic
Integrate New Relic APM for application performance monitoring and distributed tracing.