Org Skills
BackendBackend (general)DevOps

service-operability

Make a back-end service safe to run in production in any stack — liveness and readiness probes that mean different things, graceful shutdown and connection draining, OpenTelemetry traces, metrics and logs with correlated context, structured logging without leaking personal data, useful health and dependency checks, container image and resource limits, and the signals an on-call engineer actually needs.

Download .zip Raw Source
When agents use itUse this whenever the user adds a health or readiness endpoint, handles SIGTERM or drains in-flight work, sets up OpenTelemetry or an OTLP exporter, correlates logs with a trace or request id, configures a Dockerfile, Kubernetes probes, resource requests or limits, decides what to log or alert on, or asks why a deploy drops requests or why a pod restarts under load. This skill owns these concerns; the stack skills link here and add only library specifics.

Install

Copilot (VS Code, Visual Studio, Copilot CLI and github.com) reads skills from the repository — commit them so the whole team gets them.

npx skills add AGCO-Global/org-skills --skill service-operability -a github-copilot
# or with the org installer (adds .github/skills/service-operability):
npx -y github:AGCO-Global/org-skills add skill service-operability

Uses the open skills CLI. Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI, OpenCode, Windsurf and 60+ others — it asks which agent to install into.

npx skills add AGCO-Global/org-skills --skill service-operability
# user-level instead of project-level:
npx skills add AGCO-Global/org-skills --skill service-operability -g

Installs the backend-shared-skills plugin, which bundles all Backend / Shared skills and keeps them updated.

/plugin marketplace add AGCO-Global/org-skills
/plugin install backend-shared-skills@org-skills
# or just this skill, in this repository:
npx skills add AGCO-Global/org-skills --skill service-operability -a claude-code

Use Download .zip above, then upload it under Customize → Skills → Upload skill. Team and Enterprise admins can sync this repo as a plugin marketplace instead.

Installs into .agents/skills/, which Codex reads.

npx skills add AGCO-Global/org-skills --skill service-operability -a codex

Installs into Cursor's skills folder.

npx skills add AGCO-Global/org-skills --skill service-operability -a cursor

Installs into Gemini CLI's skills folder.

npx skills add AGCO-Global/org-skills --skill service-operability -a gemini-cli

Any tool with rules, instructions or custom prompts: use Copy SKILL.md above and paste it in. It is plain Markdown.

Commands use your normal git sign-in to GitHub, so they work while the repository is private. Node.js 20+ required.

Try it — example prompts

Prompts this skill is tested against, and what a good answer includes.

  • Add health checks to our service. We have Postgres and Redis.

    separate liveness and readiness endpoints liveness is local and checks no dependency readiness acquires a pooled connection with a short timeout Redis only in readiness if the service cannot serve without it cache the readiness result briefly because it is polled constantly

  • Every deploy drops a handful of requests with connection errors. Why?

    the process stops accepting connections before the load balancer has stopped sending fail readiness first but keep serving during a propagation window then close the listener and drain in-flight requests to a deadline termination grace period must exceed the worst-case drain

  • Our traces stop at the publish to SQS — the consumer shows up as a separate trace.

    trace context is not propagated through the broker automatically inject traceparent into message headers on publish and extract on consume W3C trace context link the consumer span to the publishing span

  • Set up OpenTelemetry in our Node service.

    NodeSDK with OTLP exporter to a collector start the SDK before instrumented libraries are imported, via a preload flag shutdown flushes telemetry on SIGTERM collector routes to the backend so vendor changes are config

  • What should we alert on for this API?

    symptom-based: error rate, p99 latency against an objective saturation signals such as pool utilisation, queue depth, event loop lag queue age and dead-letter count for consumers CPU and memory belong on dashboards, not pagers

  • Our pod keeps getting OOMKilled even though the heap looks fine.

    the runtime must be told its budget rather than sizing to the node set memory requests and limits and configure the heap against the limit CPU limits throttle rather than kill, so check throttling metrics separately non-heap memory counts toward the container limit

Skill contents

Service Operability

Targets: any back-end stack · OpenTelemetry, Kubernetes, OCI containers · Verified: 2026-09 against https://opentelemetry.io/docs/ and https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

Operability is decided before an incident, not during one. The questions that matter are: can the platform tell whether this instance should receive traffic, can it be stopped without dropping work, and when something goes wrong can one person find out what happened from the outside. Most production pain in otherwise well-written services comes from three things — a readiness probe that checks the wrong thing, a shutdown that kills in-flight requests, and logs that cannot be tied to a single request.

1. Decide first

Question Default Change when
Liveness probe Cheap and local: the process is alive and not deadlocked Never check dependencies here — a database blip must not restart every pod
Readiness probe Checks the dependencies this instance needs to serve Keep it cheap and cached; it runs constantly
Startup probe Add one for slow starts So liveness does not kill a service still warming up
Shutdown Fail readiness, keep serving, drain, then exit Exiting on SIGTERM immediately drops in-flight requests
Telemetry OpenTelemetry with OTLP to a collector Vendor SDK only where a feature genuinely requires it
Log format Structured JSON, one event per line Human-readable locally
Correlation Trace and span id on every log line Also propagate a request/correlation id across service boundaries
Sampling Tail-based or error-biased Head sampling at a fixed rate loses the incidents you need
Metrics RED (rate, errors, duration) per endpoint plus saturation Add business metrics where they drive alerts
Alerting On user-visible symptoms Not on individual resource numbers

2. Health and readiness are different

  • Liveness answers "should this process be killed and restarted?". It must be local and cheap. Checking the database here means one database hiccup restarts every instance at once, turning a partial outage into a total one.
  • Readiness answers "should this instance receive traffic right now?". It checks what this instance needs to serve: a database pool that can hand out a connection, a required cache, a loaded config. Cache the result for a second or two — it is polled continuously.
  • Startup covers slow boots (warming caches, JIT, loading models) so liveness does not fire before the service is up.
  • A dependency being down does not always mean unready: if the service degrades gracefully without that dependency, staying ready and serving the degraded response is better than removing every instance from the load balancer.

Probe shapes per platform and what to check for each dependency: references/health-and-shutdown.md.

3. Graceful shutdown

The sequence that does not drop requests:

  1. Receive SIGTERM.
  2. Fail readiness immediately, so the load balancer stops sending new work — but keep serving.
  3. Wait for the load balancer's propagation delay. This wait is not optional; endpoint updates are not instant, and without it a proportion of requests still arrive after step 4 begins.
  4. Stop accepting new connections; finish in-flight requests up to a deadline.
  5. Stop background consumers: stop prefetching, finish or nack the current message, close the connection.
  6. Close database pools and flush telemetry.
  7. Exit.

The whole sequence must fit inside the platform's termination grace period, or the process is killed mid-drain. Match the two numbers deliberately: if draining can take 30 seconds, the grace period must exceed it.

4. Telemetry

  • Traces: instrument at the edges — inbound requests, outbound HTTP, database calls, message publish and consume. Propagate W3C traceparent across every boundary, including through queues (put it in the message headers), or the trace stops at the broker.
  • Metrics: RED per endpoint, plus saturation — pool utilisation, queue depth, event loop or thread pool lag. These are what tell you why latency moved.
  • Logs: structured, with trace id, span id and a correlation id on every line. One event per line, fields not sentences, so they can be queried.
  • Use the OpenTelemetry SDK with an OTLP exporter to a collector, and let the collector route to the backend. Then changing vendor is a collector config change, not a redeploy of every service.
  • Never log credentials, tokens, full card numbers, or personal data beyond what the purpose requires. Redact at the logger, not at each call site — someone will forget.

5. Containers and resources

  • Small base image, non-root user, no build toolchain in the runtime layer, a pinned digest.
  • Set memory requests and limits. A limit with no bounded heap means the runtime sizes itself to the node and gets OOM-killed; tell the runtime its budget explicitly.
  • CPU limits throttle rather than kill, which shows up as latency, not errors — check throttling metrics before concluding the code is slow.
  • One process per container, logs to stdout/stderr, config from the environment, no state on local disk.

Deliverable format

## Probes — liveness, readiness, startup: what each checks, timing, and why
## Shutdown — the ordered sequence with the drain deadline and grace period
## Telemetry — traces, metrics and logs: what is emitted, exporter, sampling
## Correlation — how a request is traced end to end, including across queues
## Dashboards and alerts — the signals, thresholds, and what each means for on-call
## Container — image, user, resource requests and limits
## Verification — how each was tested, including a deploy with traffic running

Checklist

  • Liveness is local and cheap, and checks no external dependency.
  • Readiness checks what this instance needs to serve, and its result is cached briefly.
  • A startup probe exists where boot is slow.
  • SIGTERM fails readiness first, waits for propagation, then drains within a deadline.
  • The termination grace period is longer than the worst-case drain.
  • Background consumers and pools are closed as part of shutdown.
  • traceparent is propagated across HTTP and through message headers.
  • Every log line carries trace id and correlation id, and logs are structured.
  • No credential, token or unnecessary personal data reaches a log, with redaction centralised.
  • RED metrics plus saturation exist per endpoint, and alerts are on symptoms.
  • Memory limits are set and the runtime is told its budget.
  • A rolling deploy under load was verified to drop no requests.

Anti-patterns

Anti-pattern Why it hurts Fix
Liveness probe that checks the database A database blip restarts every pod at once Liveness local only; dependencies belong in readiness
One endpoint serving both liveness and readiness The two have opposite failure responses Separate endpoints with separate checks
Exiting immediately on SIGTERM In-flight requests are dropped on every deploy Fail readiness, wait for propagation, drain, exit
No wait between failing readiness and closing the listener Endpoint updates are not instant; requests still arrive An explicit propagation delay before draining
Grace period shorter than the drain The process is killed mid-request Set the grace period from the measured drain time
Trace context not propagated through the broker Traces stop at publish; the consumer looks unrelated Carry traceparent in message headers
Logging request or response bodies wholesale Leaks personal data and tokens; costs a fortune Log identifiers and outcomes; redact centrally
Alerting on CPU or memory Pages for things users never noticed, misses things they did Alert on symptoms: error rate, latency, queue age
Fixed-rate head sampling The one trace you need was not sampled Error-biased or tail-based sampling
Memory limit with no runtime heap budget The runtime sizes to the node and is OOM-killed Configure the heap against the container limit

Go deeper

  • references/health-and-shutdown.md — probe configuration per platform, what to check per dependency, and the shutdown sequence with realistic timings.
  • references/telemetry-setup.md — OpenTelemetry bootstrap for .NET and Node, collector-based routing, context propagation through queues, sampling strategies and a starting alert set.
  • Sibling skills: messaging-patterns for consumer lifecycle and retries; database-migrations for observing a long backfill; the stack's performance skill when a signal shows a real problem.

References

Deeper material the agent loads only when needed.