# OpenTelemetry setup and signals

Verified 2026-09 against <https://opentelemetry.io/docs/> and
<https://www.w3.org/TR/trace-context/>.

## Shape

Instrument every service with the OpenTelemetry SDK, export OTLP to a **collector**, and let the
collector route to the backend. Changing vendor, adding sampling or scrubbing a field then happens
in collector configuration instead of in a redeploy of every service.

```
service ──OTLP──▶ collector ──▶ traces backend
                        ├──▶ metrics backend
                        └──▶ logs backend
```

## .NET

```csharp
builder.Services.AddOpenTelemetry()
    .ConfigureResource(r => r.AddService(serviceName: "orders-api"))
    .WithTracing(t => t
        .AddAspNetCoreInstrumentation()
        .AddHttpClientInstrumentation()
        .AddOtlpExporter())
    .WithMetrics(m => m
        .AddAspNetCoreInstrumentation()
        .AddRuntimeInstrumentation()
        .AddOtlpExporter());

builder.Logging.AddOpenTelemetry(o =>
{
    o.IncludeScopes = true;
    o.AddOtlpExporter();
});
```

Database and messaging instrumentation come from their own packages; add the ones matching the
libraries actually in use.

## Node.js

```js
import { NodeSDK } from '@opentelemetry/sdk-node';
import { getNodeAutoInstrumentations } from '@opentelemetry/auto-instrumentations-node';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';

const sdk = new NodeSDK({
  serviceName: 'orders-api',
  traceExporter: new OTLPTraceExporter(),
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();
process.on('SIGTERM', () => { sdk.shutdown().finally(() => process.exit(0)); });
```

Start the SDK **before** importing instrumented libraries — auto-instrumentation patches modules at
require/import time, so anything imported first is not traced. Use a preload (`--import` / `-r`)
rather than an import at the top of the application file.

## Context propagation

- HTTP: the W3C `traceparent` header, handled by the instrumentation for both inbound and outbound.
- **Queues**: not automatic in the useful direction. Inject the current context into the message
  headers on publish and extract it on consume, or the consumer's work appears as an unrelated
  trace and you lose the causal link exactly where async debugging is hardest.
- Background jobs: start a new root span and link it to the span that scheduled the work.
- Carry a business correlation id (order id, tenant) as a span attribute and a log field, since
  that is what support actually searches by.

## Logs

- Structured JSON, one event per line, fields rather than interpolated sentences.
- Every line carries trace id, span id and correlation id. This is what makes logs and traces one
  investigation rather than two.
- Levels that mean something: `error` is something a human must look at; `warn` is a degraded but
  handled condition; `info` is a state change worth reconstructing; `debug` is off in production.
- Redact centrally in the logger — a processor that strips known-sensitive keys and obvious token
  patterns. Per-call-site discipline fails the first time someone logs a whole request object.
- Never log credentials, tokens, full card numbers, or personal data beyond the purpose. For
  regulated data, log an identifier and look it up in a system with its own access control.

## Sampling

- Fixed-rate head sampling drops the traces you need. Prefer **tail-based** sampling in the
  collector: keep everything that errored or was slow, sample the rest.
- Keep 100% of traces for low-traffic but high-value flows (checkout, payment).
- Metrics are not sampled — they are aggregate and cheap, which is why alerting is built on them.

## A starting alert set

| Alert | Signal | Why |
|---|---|---|
| Error rate | 5xx / total per endpoint over a short window | Users are seeing failures now |
| Latency | p99 against the objective | Slow is a failure with a longer timeout |
| Saturation | Pool utilisation, queue depth, event loop or thread pool lag | Warns before the first two fire |
| Queue age | Oldest unprocessed message | Consumers stopped or fell behind |
| Dead letters | Count over a window | Messages are failing permanently |
| Readiness flapping | Instances going ready/unready repeatedly | A dependency check or capacity problem |

Alert on symptoms users feel. CPU and memory belong on dashboards for diagnosis, not on pagers —
they page for things nobody noticed and stay quiet for things everybody did.