--- name: service-operability description: > Make a back-end service safe to run in production in any stack — liveness and readiness probes that mean different things, graceful shutdown and connection draining, OpenTelemetry traces, metrics and logs with correlated context, structured logging without leaking personal data, useful health and dependency checks, container image and resource limits, and the signals an on-call engineer actually needs. Use this whenever the user adds a health or readiness endpoint, handles SIGTERM or drains in-flight work, sets up OpenTelemetry or an OTLP exporter, correlates logs with a trace or request id, configures a Dockerfile, Kubernetes probes, resource requests or limits, decides what to log or alert on, or asks why a deploy drops requests or why a pod restarts under load. This skill owns these concerns; the stack skills link here and add only library specifics. metadata: technology: Backend (general) type: devops --- # Service Operability > **Targets:** any back-end stack · OpenTelemetry, Kubernetes, OCI containers · **Verified:** 2026-09 against https://opentelemetry.io/docs/ and https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ Operability is decided before an incident, not during one. The questions that matter are: can the platform tell whether this instance should receive traffic, can it be stopped without dropping work, and when something goes wrong can one person find out what happened from the outside. Most production pain in otherwise well-written services comes from three things — a readiness probe that checks the wrong thing, a shutdown that kills in-flight requests, and logs that cannot be tied to a single request. ## 1. Decide first | Question | Default | Change when | |---|---|---| | Liveness probe | Cheap and local: the process is alive and not deadlocked | Never check dependencies here — a database blip must not restart every pod | | Readiness probe | Checks the dependencies this instance needs to serve | Keep it cheap and cached; it runs constantly | | Startup probe | Add one for slow starts | So liveness does not kill a service still warming up | | Shutdown | Fail readiness, keep serving, drain, then exit | Exiting on SIGTERM immediately drops in-flight requests | | Telemetry | OpenTelemetry with OTLP to a collector | Vendor SDK only where a feature genuinely requires it | | Log format | Structured JSON, one event per line | Human-readable locally | | Correlation | Trace and span id on every log line | Also propagate a request/correlation id across service boundaries | | Sampling | Tail-based or error-biased | Head sampling at a fixed rate loses the incidents you need | | Metrics | RED (rate, errors, duration) per endpoint plus saturation | Add business metrics where they drive alerts | | Alerting | On user-visible symptoms | Not on individual resource numbers | ## 2. Health and readiness are different - **Liveness** answers "should this process be killed and restarted?". It must be local and cheap. Checking the database here means one database hiccup restarts every instance at once, turning a partial outage into a total one. - **Readiness** answers "should this instance receive traffic right now?". It checks what this instance needs to serve: a database pool that can hand out a connection, a required cache, a loaded config. Cache the result for a second or two — it is polled continuously. - **Startup** covers slow boots (warming caches, JIT, loading models) so liveness does not fire before the service is up. - A dependency being down does not always mean unready: if the service degrades gracefully without that dependency, staying ready and serving the degraded response is better than removing every instance from the load balancer. Probe shapes per platform and what to check for each dependency: `references/health-and-shutdown.md`. ## 3. Graceful shutdown The sequence that does not drop requests: 1. Receive SIGTERM. 2. **Fail readiness immediately**, so the load balancer stops sending new work — but keep serving. 3. Wait for the load balancer's propagation delay. This wait is not optional; endpoint updates are not instant, and without it a proportion of requests still arrive after step 4 begins. 4. Stop accepting new connections; finish in-flight requests up to a deadline. 5. Stop background consumers: stop prefetching, finish or nack the current message, close the connection. 6. Close database pools and flush telemetry. 7. Exit. The whole sequence must fit inside the platform's termination grace period, or the process is killed mid-drain. Match the two numbers deliberately: if draining can take 30 seconds, the grace period must exceed it. ## 4. Telemetry - **Traces**: instrument at the edges — inbound requests, outbound HTTP, database calls, message publish and consume. Propagate W3C `traceparent` across every boundary, including through queues (put it in the message headers), or the trace stops at the broker. - **Metrics**: RED per endpoint, plus saturation — pool utilisation, queue depth, event loop or thread pool lag. These are what tell you *why* latency moved. - **Logs**: structured, with trace id, span id and a correlation id on every line. One event per line, fields not sentences, so they can be queried. - Use the OpenTelemetry SDK with an OTLP exporter to a collector, and let the collector route to the backend. Then changing vendor is a collector config change, not a redeploy of every service. - **Never log** credentials, tokens, full card numbers, or personal data beyond what the purpose requires. Redact at the logger, not at each call site — someone will forget. ## 5. Containers and resources - Small base image, non-root user, no build toolchain in the runtime layer, a pinned digest. - Set memory **requests and limits**. A limit with no bounded heap means the runtime sizes itself to the node and gets OOM-killed; tell the runtime its budget explicitly. - CPU limits throttle rather than kill, which shows up as latency, not errors — check throttling metrics before concluding the code is slow. - One process per container, logs to stdout/stderr, config from the environment, no state on local disk. ## Deliverable format ``` ## Probes — liveness, readiness, startup: what each checks, timing, and why ## Shutdown — the ordered sequence with the drain deadline and grace period ## Telemetry — traces, metrics and logs: what is emitted, exporter, sampling ## Correlation — how a request is traced end to end, including across queues ## Dashboards and alerts — the signals, thresholds, and what each means for on-call ## Container — image, user, resource requests and limits ## Verification — how each was tested, including a deploy with traffic running ``` ## Checklist - [ ] Liveness is local and cheap, and checks no external dependency. - [ ] Readiness checks what this instance needs to serve, and its result is cached briefly. - [ ] A startup probe exists where boot is slow. - [ ] SIGTERM fails readiness first, waits for propagation, then drains within a deadline. - [ ] The termination grace period is longer than the worst-case drain. - [ ] Background consumers and pools are closed as part of shutdown. - [ ] `traceparent` is propagated across HTTP and through message headers. - [ ] Every log line carries trace id and correlation id, and logs are structured. - [ ] No credential, token or unnecessary personal data reaches a log, with redaction centralised. - [ ] RED metrics plus saturation exist per endpoint, and alerts are on symptoms. - [ ] Memory limits are set and the runtime is told its budget. - [ ] A rolling deploy under load was verified to drop no requests. ## Anti-patterns | Anti-pattern | Why it hurts | Fix | |---|---|---| | Liveness probe that checks the database | A database blip restarts every pod at once | Liveness local only; dependencies belong in readiness | | One endpoint serving both liveness and readiness | The two have opposite failure responses | Separate endpoints with separate checks | | Exiting immediately on SIGTERM | In-flight requests are dropped on every deploy | Fail readiness, wait for propagation, drain, exit | | No wait between failing readiness and closing the listener | Endpoint updates are not instant; requests still arrive | An explicit propagation delay before draining | | Grace period shorter than the drain | The process is killed mid-request | Set the grace period from the measured drain time | | Trace context not propagated through the broker | Traces stop at publish; the consumer looks unrelated | Carry `traceparent` in message headers | | Logging request or response bodies wholesale | Leaks personal data and tokens; costs a fortune | Log identifiers and outcomes; redact centrally | | Alerting on CPU or memory | Pages for things users never noticed, misses things they did | Alert on symptoms: error rate, latency, queue age | | Fixed-rate head sampling | The one trace you need was not sampled | Error-biased or tail-based sampling | | Memory limit with no runtime heap budget | The runtime sizes to the node and is OOM-killed | Configure the heap against the container limit | ## Go deeper - `references/health-and-shutdown.md` — probe configuration per platform, what to check per dependency, and the shutdown sequence with realistic timings. - `references/telemetry-setup.md` — OpenTelemetry bootstrap for .NET and Node, collector-based routing, context propagation through queues, sampling strategies and a starting alert set. - Sibling skills: `messaging-patterns` for consumer lifecycle and retries; `database-migrations` for observing a long backfill; the stack's performance skill when a signal shows a real problem.