Make a back-end service safe to run in production in any stack — liveness and readiness probes that mean different things, graceful shutdown and connection draining, OpenTelemetry traces, metrics and logs with correlated context, structured logging without leaking personal data, useful health and dependency checks, container image and resource limits, and the signals an on-call engineer actually needs.
When agents use itUse this whenever the user adds a health or readiness endpoint, handles SIGTERM or drains in-flight work, sets up OpenTelemetry or an OTLP exporter, correlates logs with a trace or request id, configures a Dockerfile, Kubernetes probes, resource requests or limits, decides what to log or alert on, or asks why a deploy drops requests or why a pod restarts under load. This skill owns these concerns; the stack skills link here and add only library specifics.
Install
Copilot (VS Code, Visual Studio, Copilot CLI and github.com) reads skills from the repository — commit them so the whole team gets them.
npx skills add AGCO-Global/org-skills --skill service-operability -a github-copilot
# or with the org installer (adds .github/skills/service-operability):
npx -y github:AGCO-Global/org-skills add skill service-operability
Uses the open skills CLI. Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI, OpenCode, Windsurf and 60+ others — it asks which agent to install into.
Installs the backend-shared-skills plugin, which bundles all Backend / Shared skills and keeps them updated.
/plugin marketplace add AGCO-Global/org-skills
/plugin install backend-shared-skills@org-skills
# or just this skill, in this repository:
npx skills add AGCO-Global/org-skills --skill service-operability -a claude-code
Use Download .zip above, then upload it under Customize → Skills → Upload skill. Team and Enterprise admins can sync this repo as a plugin marketplace instead.
Installs into .agents/skills/, which Codex reads.
npx skills add AGCO-Global/org-skills --skill service-operability -a codex
Installs into Cursor's skills folder.
npx skills add AGCO-Global/org-skills --skill service-operability -a cursor
Installs into Gemini CLI's skills folder.
npx skills add AGCO-Global/org-skills --skill service-operability -a gemini-cli
Any tool with rules, instructions or custom prompts: use Copy SKILL.md above and paste it in. It is plain Markdown.
Commands use your normal git sign-in to GitHub, so they work while the repository is private. Node.js 20+ required.
Try it — example prompts
Prompts this skill is tested against, and what a good answer includes.
Add health checks to our service. We have Postgres and Redis.
separate liveness and readiness endpointsliveness is local and checks no dependencyreadiness acquires a pooled connection with a short timeoutRedis only in readiness if the service cannot serve without itcache the readiness result briefly because it is polled constantly
Every deploy drops a handful of requests with connection errors. Why?
the process stops accepting connections before the load balancer has stopped sendingfail readiness first but keep serving during a propagation windowthen close the listener and drain in-flight requests to a deadlinetermination grace period must exceed the worst-case drain
Our traces stop at the publish to SQS — the consumer shows up as a separate trace.
trace context is not propagated through the broker automaticallyinject traceparent into message headers on publish and extract on consumeW3C trace contextlink the consumer span to the publishing span
Set up OpenTelemetry in our Node service.
NodeSDK with OTLP exporter to a collectorstart the SDK before instrumented libraries are imported, via a preload flagshutdown flushes telemetry on SIGTERMcollector routes to the backend so vendor changes are config
What should we alert on for this API?
symptom-based: error rate, p99 latency against an objectivesaturation signals such as pool utilisation, queue depth, event loop lagqueue age and dead-letter count for consumersCPU and memory belong on dashboards, not pagers
Our pod keeps getting OOMKilled even though the heap looks fine.
the runtime must be told its budget rather than sizing to the nodeset memory requests and limits and configure the heap against the limitCPU limits throttle rather than kill, so check throttling metrics separatelynon-heap memory counts toward the container limit
Operability is decided before an incident, not during one. The questions that matter are: can the platform tell whether this instance should receive traffic, can it be stopped without dropping work, and when something goes wrong can one person find out what happened from the outside. Most production pain in otherwise well-written services comes from three things — a readiness probe that checks the wrong thing, a shutdown that kills in-flight requests, and logs that cannot be tied to a single request.
1. Decide first
Question
Default
Change when
Liveness probe
Cheap and local: the process is alive and not deadlocked
Never check dependencies here — a database blip must not restart every pod
Readiness probe
Checks the dependencies this instance needs to serve
Keep it cheap and cached; it runs constantly
Startup probe
Add one for slow starts
So liveness does not kill a service still warming up
Shutdown
Fail readiness, keep serving, drain, then exit
Exiting on SIGTERM immediately drops in-flight requests
Telemetry
OpenTelemetry with OTLP to a collector
Vendor SDK only where a feature genuinely requires it
Log format
Structured JSON, one event per line
Human-readable locally
Correlation
Trace and span id on every log line
Also propagate a request/correlation id across service boundaries
Sampling
Tail-based or error-biased
Head sampling at a fixed rate loses the incidents you need
Metrics
RED (rate, errors, duration) per endpoint plus saturation
Add business metrics where they drive alerts
Alerting
On user-visible symptoms
Not on individual resource numbers
2. Health and readiness are different
Liveness answers "should this process be killed and restarted?". It must be local and cheap. Checking the database here means one database hiccup restarts every instance at once, turning a partial outage into a total one.
Readiness answers "should this instance receive traffic right now?". It checks what this instance needs to serve: a database pool that can hand out a connection, a required cache, a loaded config. Cache the result for a second or two — it is polled continuously.
Startup covers slow boots (warming caches, JIT, loading models) so liveness does not fire before the service is up.
A dependency being down does not always mean unready: if the service degrades gracefully without that dependency, staying ready and serving the degraded response is better than removing every instance from the load balancer.
Probe shapes per platform and what to check for each dependency: references/health-and-shutdown.md.
3. Graceful shutdown
The sequence that does not drop requests:
Receive SIGTERM.
Fail readiness immediately, so the load balancer stops sending new work — but keep serving.
Wait for the load balancer's propagation delay. This wait is not optional; endpoint updates are not instant, and without it a proportion of requests still arrive after step 4 begins.
Stop accepting new connections; finish in-flight requests up to a deadline.
Stop background consumers: stop prefetching, finish or nack the current message, close the connection.
Close database pools and flush telemetry.
Exit.
The whole sequence must fit inside the platform's termination grace period, or the process is killed mid-drain. Match the two numbers deliberately: if draining can take 30 seconds, the grace period must exceed it.
4. Telemetry
Traces: instrument at the edges — inbound requests, outbound HTTP, database calls, message publish and consume. Propagate W3C traceparent across every boundary, including through queues (put it in the message headers), or the trace stops at the broker.
Metrics: RED per endpoint, plus saturation — pool utilisation, queue depth, event loop or thread pool lag. These are what tell you why latency moved.
Logs: structured, with trace id, span id and a correlation id on every line. One event per line, fields not sentences, so they can be queried.
Use the OpenTelemetry SDK with an OTLP exporter to a collector, and let the collector route to the backend. Then changing vendor is a collector config change, not a redeploy of every service.
Never log credentials, tokens, full card numbers, or personal data beyond what the purpose requires. Redact at the logger, not at each call site — someone will forget.
5. Containers and resources
Small base image, non-root user, no build toolchain in the runtime layer, a pinned digest.
Set memory requests and limits. A limit with no bounded heap means the runtime sizes itself to the node and gets OOM-killed; tell the runtime its budget explicitly.
CPU limits throttle rather than kill, which shows up as latency, not errors — check throttling metrics before concluding the code is slow.
One process per container, logs to stdout/stderr, config from the environment, no state on local disk.
Deliverable format
## Probes — liveness, readiness, startup: what each checks, timing, and why
## Shutdown — the ordered sequence with the drain deadline and grace period
## Telemetry — traces, metrics and logs: what is emitted, exporter, sampling
## Correlation — how a request is traced end to end, including across queues
## Dashboards and alerts — the signals, thresholds, and what each means for on-call
## Container — image, user, resource requests and limits
## Verification — how each was tested, including a deploy with traffic running
Checklist
Liveness is local and cheap, and checks no external dependency.
Readiness checks what this instance needs to serve, and its result is cached briefly.
A startup probe exists where boot is slow.
SIGTERM fails readiness first, waits for propagation, then drains within a deadline.
The termination grace period is longer than the worst-case drain.
Background consumers and pools are closed as part of shutdown.
traceparent is propagated across HTTP and through message headers.
Every log line carries trace id and correlation id, and logs are structured.
No credential, token or unnecessary personal data reaches a log, with redaction centralised.
RED metrics plus saturation exist per endpoint, and alerts are on symptoms.
Memory limits are set and the runtime is told its budget.
A rolling deploy under load was verified to drop no requests.
Anti-patterns
Anti-pattern
Why it hurts
Fix
Liveness probe that checks the database
A database blip restarts every pod at once
Liveness local only; dependencies belong in readiness
One endpoint serving both liveness and readiness
The two have opposite failure responses
Separate endpoints with separate checks
Exiting immediately on SIGTERM
In-flight requests are dropped on every deploy
Fail readiness, wait for propagation, drain, exit
No wait between failing readiness and closing the listener
Endpoint updates are not instant; requests still arrive
An explicit propagation delay before draining
Grace period shorter than the drain
The process is killed mid-request
Set the grace period from the measured drain time
Trace context not propagated through the broker
Traces stop at publish; the consumer looks unrelated
Carry traceparent in message headers
Logging request or response bodies wholesale
Leaks personal data and tokens; costs a fortune
Log identifiers and outcomes; redact centrally
Alerting on CPU or memory
Pages for things users never noticed, misses things they did
Alert on symptoms: error rate, latency, queue age
Fixed-rate head sampling
The one trace you need was not sampled
Error-biased or tail-based sampling
Memory limit with no runtime heap budget
The runtime sizes to the node and is OOM-killed
Configure the heap against the container limit
Go deeper
references/health-and-shutdown.md — probe configuration per platform, what to check per dependency, and the shutdown sequence with realistic timings.
references/telemetry-setup.md — OpenTelemetry bootstrap for .NET and Node, collector-based routing, context propagation through queues, sampling strategies and a starting alert set.
Sibling skills: messaging-patterns for consumer lifecycle and retries; database-migrations for observing a long backfill; the stack's performance skill when a signal shows a real problem.
---
name: service-operability
description: >
Make a back-end service safe to run in production in any stack — liveness and readiness probes that mean different things, graceful shutdown and connection draining, OpenTelemetry traces, metrics and logs with correlated context, structured logging without leaking personal data, useful health and dependency checks, container image and resource limits, and the signals an on-call engineer actually needs. Use this whenever the user adds a health or readiness endpoint, handles SIGTERM or drains in-flight work, sets up OpenTelemetry or an OTLP exporter, correlates logs with a trace or request id, configures a Dockerfile, Kubernetes probes, resource requests or limits, decides what to log or alert on, or asks why a deploy drops requests or why a pod restarts under load. This skill owns these concerns; the stack skills link here and add only library specifics.
metadata:
technology: Backend (general)
type: devops
---
# Service Operability
> **Targets:** any back-end stack · OpenTelemetry, Kubernetes, OCI containers · **Verified:** 2026-09 against https://opentelemetry.io/docs/ and https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
Operability is decided before an incident, not during one. The questions that matter are: can the platform tell whether this instance should receive traffic, can it be stopped without dropping work, and when something goes wrong can one person find out what happened from the outside. Most production pain in otherwise well-written services comes from three things — a readiness probe that checks the wrong thing, a shutdown that kills in-flight requests, and logs that cannot be tied to a single request.
## 1. Decide first
| Question | Default | Change when |
|---|---|---|
| Liveness probe | Cheap and local: the process is alive and not deadlocked | Never check dependencies here — a database blip must not restart every pod |
| Readiness probe | Checks the dependencies this instance needs to serve | Keep it cheap and cached; it runs constantly |
| Startup probe | Add one for slow starts | So liveness does not kill a service still warming up |
| Shutdown | Fail readiness, keep serving, drain, then exit | Exiting on SIGTERM immediately drops in-flight requests |
| Telemetry | OpenTelemetry with OTLP to a collector | Vendor SDK only where a feature genuinely requires it |
| Log format | Structured JSON, one event per line | Human-readable locally |
| Correlation | Trace and span id on every log line | Also propagate a request/correlation id across service boundaries |
| Sampling | Tail-based or error-biased | Head sampling at a fixed rate loses the incidents you need |
| Metrics | RED (rate, errors, duration) per endpoint plus saturation | Add business metrics where they drive alerts |
| Alerting | On user-visible symptoms | Not on individual resource numbers |
## 2. Health and readiness are different
- **Liveness** answers "should this process be killed and restarted?". It must be local and cheap. Checking the database here means one database hiccup restarts every instance at once, turning a partial outage into a total one.
- **Readiness** answers "should this instance receive traffic right now?". It checks what this instance needs to serve: a database pool that can hand out a connection, a required cache, a loaded config. Cache the result for a second or two — it is polled continuously.
- **Startup** covers slow boots (warming caches, JIT, loading models) so liveness does not fire before the service is up.
- A dependency being down does not always mean unready: if the service degrades gracefully without that dependency, staying ready and serving the degraded response is better than removing every instance from the load balancer.
Probe shapes per platform and what to check for each dependency: `references/health-and-shutdown.md`.
## 3. Graceful shutdown
The sequence that does not drop requests:
1. Receive SIGTERM.
2. **Fail readiness immediately**, so the load balancer stops sending new work — but keep serving.
3. Wait for the load balancer's propagation delay. This wait is not optional; endpoint updates are not instant, and without it a proportion of requests still arrive after step 4 begins.
4. Stop accepting new connections; finish in-flight requests up to a deadline.
5. Stop background consumers: stop prefetching, finish or nack the current message, close the connection.
6. Close database pools and flush telemetry.
7. Exit.
The whole sequence must fit inside the platform's termination grace period, or the process is killed mid-drain. Match the two numbers deliberately: if draining can take 30 seconds, the grace period must exceed it.
## 4. Telemetry
- **Traces**: instrument at the edges — inbound requests, outbound HTTP, database calls, message publish and consume. Propagate W3C `traceparent` across every boundary, including through queues (put it in the message headers), or the trace stops at the broker.
- **Metrics**: RED per endpoint, plus saturation — pool utilisation, queue depth, event loop or thread pool lag. These are what tell you *why* latency moved.
- **Logs**: structured, with trace id, span id and a correlation id on every line. One event per line, fields not sentences, so they can be queried.
- Use the OpenTelemetry SDK with an OTLP exporter to a collector, and let the collector route to the backend. Then changing vendor is a collector config change, not a redeploy of every service.
- **Never log** credentials, tokens, full card numbers, or personal data beyond what the purpose requires. Redact at the logger, not at each call site — someone will forget.
## 5. Containers and resources
- Small base image, non-root user, no build toolchain in the runtime layer, a pinned digest.
- Set memory **requests and limits**. A limit with no bounded heap means the runtime sizes itself to the node and gets OOM-killed; tell the runtime its budget explicitly.
- CPU limits throttle rather than kill, which shows up as latency, not errors — check throttling metrics before concluding the code is slow.
- One process per container, logs to stdout/stderr, config from the environment, no state on local disk.
## Deliverable format
```
## Probes — liveness, readiness, startup: what each checks, timing, and why
## Shutdown — the ordered sequence with the drain deadline and grace period
## Telemetry — traces, metrics and logs: what is emitted, exporter, sampling
## Correlation — how a request is traced end to end, including across queues
## Dashboards and alerts — the signals, thresholds, and what each means for on-call
## Container — image, user, resource requests and limits
## Verification — how each was tested, including a deploy with traffic running
```
## Checklist
- [ ] Liveness is local and cheap, and checks no external dependency.
- [ ] Readiness checks what this instance needs to serve, and its result is cached briefly.
- [ ] A startup probe exists where boot is slow.
- [ ] SIGTERM fails readiness first, waits for propagation, then drains within a deadline.
- [ ] The termination grace period is longer than the worst-case drain.
- [ ] Background consumers and pools are closed as part of shutdown.
- [ ] `traceparent` is propagated across HTTP and through message headers.
- [ ] Every log line carries trace id and correlation id, and logs are structured.
- [ ] No credential, token or unnecessary personal data reaches a log, with redaction centralised.
- [ ] RED metrics plus saturation exist per endpoint, and alerts are on symptoms.
- [ ] Memory limits are set and the runtime is told its budget.
- [ ] A rolling deploy under load was verified to drop no requests.
## Anti-patterns
| Anti-pattern | Why it hurts | Fix |
|---|---|---|
| Liveness probe that checks the database | A database blip restarts every pod at once | Liveness local only; dependencies belong in readiness |
| One endpoint serving both liveness and readiness | The two have opposite failure responses | Separate endpoints with separate checks |
| Exiting immediately on SIGTERM | In-flight requests are dropped on every deploy | Fail readiness, wait for propagation, drain, exit |
| No wait between failing readiness and closing the listener | Endpoint updates are not instant; requests still arrive | An explicit propagation delay before draining |
| Grace period shorter than the drain | The process is killed mid-request | Set the grace period from the measured drain time |
| Trace context not propagated through the broker | Traces stop at publish; the consumer looks unrelated | Carry `traceparent` in message headers |
| Logging request or response bodies wholesale | Leaks personal data and tokens; costs a fortune | Log identifiers and outcomes; redact centrally |
| Alerting on CPU or memory | Pages for things users never noticed, misses things they did | Alert on symptoms: error rate, latency, queue age |
| Fixed-rate head sampling | The one trace you need was not sampled | Error-biased or tail-based sampling |
| Memory limit with no runtime heap budget | The runtime sizes to the node and is OOM-killed | Configure the heap against the container limit |
## Go deeper
- `references/health-and-shutdown.md` — probe configuration per platform, what to check per dependency, and the shutdown sequence with realistic timings.
- `references/telemetry-setup.md` — OpenTelemetry bootstrap for .NET and Node, collector-based routing, context propagation through queues, sampling strategies and a starting alert set.
- Sibling skills: `messaging-patterns` for consumer lifecycle and retries; `database-migrations` for observing a long backfill; the stack's performance skill when a signal shows a real problem.