# Probes and graceful shutdown

Verified 2026-09 against
<https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/>.

## The three probes

| Probe | Question | On failure | Must be |
|---|---|---|---|
| Liveness | Is this process wedged? | Container restarted | Local, cheap, no dependencies |
| Readiness | Should it get traffic now? | Removed from endpoints | Cheap, cached, dependency-aware |
| Startup | Has it finished booting? | Container restarted | Generous; disables the other two until it passes |

Example configuration:

```yaml
startupProbe:
  httpGet: { path: /health/startup, port: 8080 }
  periodSeconds: 5
  failureThreshold: 30          # allows 150s to boot
livenessProbe:
  httpGet: { path: /health/live, port: 8080 }
  periodSeconds: 10
  failureThreshold: 3
readinessProbe:
  httpGet: { path: /health/ready, port: 8080 }
  periodSeconds: 5
  failureThreshold: 2
```

## What each dependency check should do

| Dependency | Readiness check | Not this |
|---|---|---|
| Relational database | Acquire a pooled connection and run a trivial statement, with a short timeout | A real query, or anything that touches application tables |
| Cache | A ping, only if the service cannot serve without it | Treating an optional cache as required |
| Message broker | Connection is established and the consumer is subscribed | Publishing a test message |
| Downstream HTTP API | Usually nothing | Calling it — you inherit its outage and can create a cascade |

The guiding rule: **readiness reflects whether *this instance* can serve, not whether the whole
system is healthy.** A service that degrades gracefully without a dependency should stay ready.
Otherwise every instance goes unready at once and a partial outage becomes total.

Cache the result for one or two seconds. With a 5-second period across many replicas, an uncached
readiness check becomes meaningful load on the database it is checking.

## Shutdown sequence with timings

Assume a 30-second termination grace period:

| t | Step |
|---|---|
| 0s | SIGTERM received |
| 0s | Readiness starts failing; **requests continue to be served** |
| 0–5s | Load balancer propagation window — new requests still arrive, and must succeed |
| 5s | Stop accepting new connections |
| 5–25s | Finish in-flight requests; stop consumer prefetch, finish or nack the current message |
| 25s | Close database pools, flush telemetry |
| ~26s | Exit 0 |

The propagation window at 0–5s is the step most often missed. Endpoint removal is eventually
consistent: closing the listener the moment SIGTERM arrives guarantees a burst of connection errors
on every deploy.

Sizing: the grace period must exceed the longest drain, which is bounded by your slowest normal
request. If a request can take 60 seconds, either the grace period exceeds that, or long work moves
to a queue where it can be resumed instead of finished.

## Consumers and background work

- Stop prefetching first, so no new messages are claimed.
- Finish the in-flight message, or nack it for redelivery — do not exit holding an unacknowledged
  message with a long visibility timeout, or it stays invisible until that expires.
- A scheduled job mid-run either completes within the window or records progress and stops at a
  resumable point.

## Verifying it

The only test that counts: run a steady request load, do a rolling deploy, and confirm zero failed
requests. Then repeat with a consumer running and check no message is lost or processed twice.
Reading the code is not sufficient — the propagation window and the grace period only show up under
real traffic.