Org Skills

service-operability

Make a back-end service safe to run in production in any stack — liveness and readiness probes that mean different things, graceful shutdown and connection draining, OpenTelemetry traces, metrics and logs with correlated context, structured logging without leaking personal data, useful health and dependency checks, container image and resource limits, and the signals an on-call engineer actually needs.

Download .zip Raw Source
When agents use itUse this whenever the user adds a health or readiness endpoint, handles SIGTERM or drains in-flight work, sets up OpenTelemetry or an OTLP exporter, correlates logs with a trace or request id, configures a Dockerfile, Kubernetes probes, resource requests or limits, decides what to log or alert on, or asks why a deploy drops requests or why a pod restarts under load. This skill owns these concerns; the stack skills link here and add only library specifics.

Install

Copilot (VS Code, Visual Studio, Copilot CLI and github.com) reads skills from the repository — commit them so the whole team gets them.

npx skills add AGCO-Global/org-skills --skill service-operability -a github-copilot
# or with the org installer (adds .github/skills/service-operability):
npx -y github:AGCO-Global/org-skills add skill service-operability

Uses the open skills CLI. Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI, OpenCode, Windsurf and 60+ others — it asks which agent to install into.

npx skills add AGCO-Global/org-skills --skill service-operability
# user-level instead of project-level:
npx skills add AGCO-Global/org-skills --skill service-operability -g

Installs the backend-shared-skills plugin, which bundles all Backend / Shared skills and keeps them updated.

/plugin marketplace add AGCO-Global/org-skills
/plugin install backend-shared-skills@org-skills
# or just this skill, in this repository:
npx skills add AGCO-Global/org-skills --skill service-operability -a claude-code

Use Download .zip above, then upload it under Customize → Skills → Upload skill. Team and Enterprise admins can sync this repo as a plugin marketplace instead.

Installs into .agents/skills/, which Codex reads.

npx skills add AGCO-Global/org-skills --skill service-operability -a codex

Installs into Cursor's skills folder.

npx skills add AGCO-Global/org-skills --skill service-operability -a cursor

Installs into Gemini CLI's skills folder.

npx skills add AGCO-Global/org-skills --skill service-operability -a gemini-cli

Nothing to install. Press Download .zip above, unzip it, and drop the service-operability folder into your tool's skills folder — .claude/skills/, .github/skills/, .cursor/skills/ or .agents/skills/. Step-by-step, per tool: How to use it. For any other tool with rules or custom prompts, Copy SKILL.md and paste it in — it is plain Markdown.

The npx commands need Node.js 20+ and use your normal git sign-in to GitHub, so they work while the repository is private. Without Node.js, see the No Node.js tab — a skill is a folder of Markdown, so copying it in works just as well.

Skill contents

Service Operability

Targets: any back-end stack · OpenTelemetry, Kubernetes, OCI containers · Verified: 2026-09 against https://opentelemetry.io/docs/ and https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/

Operability is decided before an incident, not during one. The questions that matter are: can the platform tell whether this instance should receive traffic, can it be stopped without dropping work, and when something goes wrong can one person find out what happened from the outside. Most production pain in otherwise well-written services comes from three things — a readiness probe that checks the wrong thing, a shutdown that kills in-flight requests, and logs that cannot be tied to a single request.

1. Decide first

Question Default Change when
Liveness probe Cheap and local: the process is alive and not deadlocked Never check dependencies here — a database blip must not restart every pod
Readiness probe Checks the dependencies this instance needs to serve Keep it cheap and cached; it runs constantly
Startup probe Add one for slow starts So liveness does not kill a service still warming up
Shutdown Fail readiness, keep serving, drain, then exit Exiting on SIGTERM immediately drops in-flight requests
Telemetry OpenTelemetry with OTLP to a collector Vendor SDK only where a feature genuinely requires it
Log format Structured JSON, one event per line Human-readable locally
Correlation Trace and span id on every log line Also propagate a request/correlation id across service boundaries
Sampling Tail-based or error-biased Head sampling at a fixed rate loses the incidents you need
Metrics RED (rate, errors, duration) per endpoint plus saturation Add business metrics where they drive alerts
Alerting On user-visible symptoms Not on individual resource numbers

2. Health and readiness are different

  • Liveness answers "should this process be killed and restarted?". It must be local and cheap. Checking the database here means one database hiccup restarts every instance at once, turning a partial outage into a total one.
  • Readiness answers "should this instance receive traffic right now?". It checks what this instance needs to serve: a database pool that can hand out a connection, a required cache, a loaded config. Cache the result for a second or two — it is polled continuously.
  • Startup covers slow boots (warming caches, JIT, loading models) so liveness does not fire before the service is up.
  • A dependency being down does not always mean unready: if the service degrades gracefully without that dependency, staying ready and serving the degraded response is better than removing every instance from the load balancer.

Probe shapes per platform and what to check for each dependency: references/health-and-shutdown.md.

3. Graceful shutdown

The sequence that does not drop requests:

  1. Receive SIGTERM.
  2. Fail readiness immediately, so the load balancer stops sending new work — but keep serving.
  3. Wait for the load balancer's propagation delay. This wait is not optional; endpoint updates are not instant, and without it a proportion of requests still arrive after step 4 begins.
  4. Stop accepting new connections; finish in-flight requests up to a deadline.
  5. Stop background consumers: stop prefetching, finish or nack the current message, close the connection.
  6. Close database pools and flush telemetry.
  7. Exit.

The whole sequence must fit inside the platform's termination grace period, or the process is killed mid-drain. Match the two numbers deliberately: if draining can take 30 seconds, the grace period must exceed it.

4. Telemetry

  • Traces: instrument at the edges — inbound requests, outbound HTTP, database calls, message publish and consume. Propagate W3C traceparent across every boundary, including through queues (put it in the message headers), or the trace stops at the broker.
  • Metrics: RED per endpoint, plus saturation — pool utilisation, queue depth, event loop or thread pool lag. These are what tell you why latency moved.
  • Logs: structured, with trace id, span id and a correlation id on every line. One event per line, fields not sentences, so they can be queried.
  • Use the OpenTelemetry SDK with an OTLP exporter to a collector, and let the collector route to the backend. Then changing vendor is a collector config change, not a redeploy of every service.
  • Never log credentials, tokens, full card numbers, or personal data beyond what the purpose requires. Redact at the logger, not at each call site — someone will forget.

5. Containers and resources

  • Small base image, non-root user, no build toolchain in the runtime layer, a pinned digest.
  • Set memory requests and limits. A limit with no bounded heap means the runtime sizes itself to the node and gets OOM-killed; tell the runtime its budget explicitly.
  • CPU limits throttle rather than kill, which shows up as latency, not errors — check throttling metrics before concluding the code is slow.
  • One process per container, logs to stdout/stderr, config from the environment, no state on local disk.

Deliverable format

## Probes — liveness, readiness, startup: what each checks, timing, and why
## Shutdown — the ordered sequence with the drain deadline and grace period
## Telemetry — traces, metrics and logs: what is emitted, exporter, sampling
## Correlation — how a request is traced end to end, including across queues
## Dashboards and alerts — the signals, thresholds, and what each means for on-call
## Container — image, user, resource requests and limits
## Verification — how each was tested, including a deploy with traffic running

Checklist

  • Liveness is local and cheap, and checks no external dependency.
  • Readiness checks what this instance needs to serve, and its result is cached briefly.
  • A startup probe exists where boot is slow.
  • SIGTERM fails readiness first, waits for propagation, then drains within a deadline.
  • The termination grace period is longer than the worst-case drain.
  • Background consumers and pools are closed as part of shutdown.
  • traceparent is propagated across HTTP and through message headers.
  • Every log line carries trace id and correlation id, and logs are structured.
  • No credential, token or unnecessary personal data reaches a log, with redaction centralised.
  • RED metrics plus saturation exist per endpoint, and alerts are on symptoms.
  • Memory limits are set and the runtime is told its budget.
  • A rolling deploy under load was verified to drop no requests.

Anti-patterns

Anti-pattern Why it hurts Fix
Liveness probe that checks the database A database blip restarts every pod at once Liveness local only; dependencies belong in readiness
One endpoint serving both liveness and readiness The two have opposite failure responses Separate endpoints with separate checks
Exiting immediately on SIGTERM In-flight requests are dropped on every deploy Fail readiness, wait for propagation, drain, exit
No wait between failing readiness and closing the listener Endpoint updates are not instant; requests still arrive An explicit propagation delay before draining
Grace period shorter than the drain The process is killed mid-request Set the grace period from the measured drain time
Trace context not propagated through the broker Traces stop at publish; the consumer looks unrelated Carry traceparent in message headers
Logging request or response bodies wholesale Leaks personal data and tokens; costs a fortune Log identifiers and outcomes; redact centrally
Alerting on CPU or memory Pages for things users never noticed, misses things they did Alert on symptoms: error rate, latency, queue age
Fixed-rate head sampling The one trace you need was not sampled Error-biased or tail-based sampling
Memory limit with no runtime heap budget The runtime sizes to the node and is OOM-killed Configure the heap against the container limit

Go deeper

  • references/health-and-shutdown.md — probe configuration per platform, what to check per dependency, and the shutdown sequence with realistic timings.
  • references/telemetry-setup.md — OpenTelemetry bootstrap for .NET and Node, collector-based routing, context propagation through queues, sampling strategies and a starting alert set.
  • Sibling skills: messaging-patterns for consumer lifecycle and retries; database-migrations for observing a long backfill; the stack's performance skill when a signal shows a real problem.

References

Deeper material the agent loads only when needed.