Diagnose and fix performance problems in Node.js back-end services — event loop blocking and lag, high p99 latency, low throughput, CPU hot paths, memory leaks and OOM kills, stream backpressure, JSON serialisation cost, connection-pool exhaustion, missing keep-alive and container CPU throttling.
When agents use itUse this whenever the user says a Node or TypeScript API is slow, p99 spikes, CPU sits at 100%, memory keeps growing, the pod is OOMKilled, they see "JavaScript heap out of memory", event loop lag or utilisation alerts fire, requests time out under load, or the load test misses its target; or asks about --cpu-prof, --heap-prof, --heapsnapshot-signal, 0x, node --inspect, autocannon, k6, perf_hooks, worker_threads, Piscina, cluster, UV_THREADPOOL_SIZE, --max-old-space-size or undici agents. Also use it when 502s appear behind a load balancer after a deploy.
Install
Copilot (VS Code, Visual Studio, Copilot CLI and github.com) reads skills from the repository — commit them so the whole team gets them.
npx skills add AGCO-Global/org-skills --skill nodejs-performance -a github-copilot
# or with the org installer (adds .github/skills/nodejs-performance):
npx -y github:AGCO-Global/org-skills add skill nodejs-performance
Uses the open skills CLI. Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI, OpenCode, Windsurf and 60+ others — it asks which agent to install into.
Installs the backend-nodejs-skills plugin, which bundles all Backend / Node.js skills and keeps them updated.
/plugin marketplace add AGCO-Global/org-skills
/plugin install backend-nodejs-skills@org-skills
# or just this skill, in this repository:
npx skills add AGCO-Global/org-skills --skill nodejs-performance -a claude-code
Use Download .zip above, then upload it under Customize → Skills → Upload skill. Team and Enterprise admins can sync this repo as a plugin marketplace instead.
Installs into .agents/skills/, which Codex reads.
npx skills add AGCO-Global/org-skills --skill nodejs-performance -a codex
Installs into Cursor's skills folder.
npx skills add AGCO-Global/org-skills --skill nodejs-performance -a cursor
Installs into Gemini CLI's skills folder.
npx skills add AGCO-Global/org-skills --skill nodejs-performance -a gemini-cli
Nothing to install. Press Download .zip above, unzip it, and drop the nodejs-performance folder into your tool's skills folder — .claude/skills/, .github/skills/, .cursor/skills/ or .agents/skills/. Step-by-step, per tool: How to use it. For any other tool with rules or custom prompts, Copy SKILL.md and paste it in — it is plain Markdown.
The npx commands need Node.js 20+ and use your normal git sign-in to GitHub, so they work while the repository is private. Without Node.js, see the No Node.js tab — a skill is a folder of Markdown, so copying it in works just as well.
Node is fast until something occupies the single JavaScript thread; then every request waits together. Find which of four things is true — the loop is busy, the loop is idle and waiting, memory is the problem, or the bottleneck is downstream — fix that one thing, re-measure. GC flags, clustering and extra replicas before a profile are guesses that make the system harder to diagnose.
1. Decide first
Question
Default
Change when
Do we have a baseline?
Get one: p50/p95/p99 and throughput at a stated rate, against a production build with production-like data
Never skip it; "slow" without a number cannot be fixed or verified
Which of the four?
Read event loop utilisation and loop lag first — they split "busy" from "waiting" in one measurement
—
Profiler
node --cpu-prof and --heap-prof, opened in Chrome DevTools
Flame graph preferred → 0x. Not clinic.js: its README says it is not actively maintained and results may be inaccurate
Scale out or fix?
Fix the bottleneck
CPU is genuinely saturated with useful work
Processes per container
One, scaled horizontally, about 1 CPU each
A VM, or a pod that must use several cores and you accept the memory and complexity → cluster or PM2
Caching
lru-cache with maxandttl for small per-instance data; Redis when shared
Every cache needs a bound, a TTL and an invalidation story before it is added
percentile(99) is in nanoseconds, hence / 1e6. Read the pair together:
ELU
Loop lag p99
Meaning
Near 1.0
High
CPU-bound on the main thread — profile for synchronous work (§4)
Low
High
Something blocks in bursts: a big JSON.parse, a sync call, a GC pause
Low
Low, but latency is high
Waiting on the database, an upstream or a connection pool — the fix is not in Node
Rising with load, lag stable
Approaching capacity honestly — scale out
Ship this permanently, not just during an incident: loop lag and ELU are the two metrics that tell you which half of the system to look at, and they cost almost nothing (service-operability).
3. Triage by symptom
Symptom
Likely cause
Confirm with
Fix
Every endpoint slow together, ELU near 1
Synchronous work on the loop
CPU profile: JSON.parse, regex, *Sync, crypto, long loops
Stream or chunk it; move it to a worker (§4)
p99 spikes periodically
GC pauses, large allocations
--trace-gc, heap profile
Allocate less; paginate; stop building giant strings and arrays
Latency high, ELU low
Waiting on a dependency or pool
Traces, pool wait metrics
Queries and indexes, pool size, parallelise independent calls
bcrypt/fs/zlib/dns.lookup slow under load
libuv thread pool (default 4) saturated
Concurrency of those ops vs UV_THREADPOOL_SIZE
Raise UV_THREADPOOL_SIZE to match the CPU allowance; cache DNS
Backpressure with pipeline, stream instead of buffering
Many sockets, slow outbound calls
No keep-alive, or a client created per request
Socket counts, TLS handshakes in traces
One shared undici dispatcher per upstream
502s behind the load balancer
server.keepAliveTimeout shorter than the LB idle timeout
Compare the two numbers
Server timeout above the LB's; headersTimeout above that
Throughput flat as pods scale
The database is the bottleneck
DB CPU, connections, locks
Queries, caching, read replicas — more pods make it worse
CPU looks low but latency is bad in Kubernetes
CFS quota throttling — bursts hit the quota and the process is stopped
container_cpu_cfs_throttled_seconds_total
Raise the limit or lower burstiness; ELU alone will not show this
Commands, load-test setup and the heap-snapshot workflow: references/profiling-playbook.md.
4. Keep the loop free
Anything over a few milliseconds of synchronous CPU per request — image processing, PDF generation, large CSV or JSON transforms, hashing loops — belongs in worker_threads through a pool (piscina) or in a separate queue-fed worker service.
setImmediate yields inside a long loop are a stopgap that keeps the service responsive while you move the work; they do not make it faster.
Watch for catastrophic regex backtracking on user input: bound input length and prefer linear-time patterns. This is a denial-of-service vector as much as a performance one.
AsyncLocalStorage is not free — it costs on every async hop. One store holding a small request context is fine; several stores, or one holding large objects, shows up in profiles.
5. I/O, serialisation and caching
Stream large payloads with pipeline() from node:stream/promises: it propagates errors and respects backpressure. Never await res.text() on a multi-megabyte body just to forward it.
JSON is often the top CPU cost. Fastify response schemas compile a serialiser (fast-json-stringify) and drop unlisted fields — faster and safer. Return only the fields the client needs, and paginate lists.
JSON.parse(JSON.stringify(x)) as a deep clone parses twice; use structuredClone, or ask whether the clone is needed at all.
Compression is CPU work on the machine you are trying to keep fast: do it at the proxy or CDN when there is one.
Caching: lru-cache with max and ttl for hot per-instance data, Redis for shared. Add stampede protection (coalesce concurrent misses, jitter the TTLs), or the first expiry under load sends every request to the origin at once.
6. Database, outbound HTTP and memory limits
Pool size is per process: replicas × processes × pool size must stay below the database's connection limit, and serverless multiplies that by its concurrency — put a pooler in front rather than raising the ceiling. Kill N+1 with joins or DataLoader batching, add indexes from EXPLAIN ANALYZE, select only the columns you use, and paginate by keyset. Parallelise independent awaits with Promise.all, capped with p-limit.
Reuse one undici dispatcher (or agent) per upstream with keep-alive, and set timeouts on every call. A client created per request pays a TLS handshake per request and leaks sockets.
Set --max-old-space-size explicitly to roughly 70–80% of the container memory limit. Node does try to derive a default from the cgroup limit, but the heuristics have changed between versions and cgroup v2 detection has been unreliable — an explicit value means V8 collects before the kernel kills the process. Never set it above the container limit; that guarantees an OOM kill instead of a GC.
7. Deliverable format
## Baseline load profile, p50/p95/p99, RPS, error rate, ELU, loop lag, heap, tool used
## Diagnosis the triage row matched, with the evidence — profile frames, retainers, spans
## Fixes ordered by impact; the smallest diff for each, trade-off named
## Result the same load test re-run, before and after in one table
## Guardrails metrics and alerts added (loop lag, ELU, heap, pool wait) and the CI load-test threshold
Checklist
A baseline exists, taken against a production build with realistic data.
ELU and loop lag were read before anything was changed.
The diagnosis cites a profile frame, a retainer chain or a span — not a hunch.
One change at a time, re-measured with the same load profile.
Every cache added has max, ttl and a stated invalidation path.
Concurrency added anywhere is capped.
Pool arithmetic checked against the database's connection limit.
--max-old-space-size is set below the container memory limit.
Loop lag, ELU and heap are exported as metrics so the next incident starts with data.
Anti-patterns
Adding replicas or cluster before profiling → find the bottleneck; scaling a blocked loop scales the block.
Profiling tsx watch mode or the dev server → profile the built artefact; the dev loader's frames are not your code.
clinic.js as the recommended tool → unmaintained; --cpu-prof, --heap-prof and 0x.
*Sync fs, crypto or zlib on a request path → async API, or a worker thread.
Unbounded Map used as a cache → lru-cache with max and ttl.
Listener or timer added per request → add once, or remove it in a finally.
Buffering a whole file or response in memory → pipeline() and stream it.
A new HTTP client, pool or Redis connection per request → create once at start-up, inject it.
--max-old-space-size at or above the container limit → 70–80% of it.
JSON.parse(JSON.stringify(x)) to clone → structuredClone, or do not clone.
Go deeper
references/profiling-playbook.md — --cpu-prof, --heap-prof, --heapsnapshot-signal and 0x with the exact commands, reading a flame graph, the three-snapshot leak workflow, profiling inside a pod with node --inspect and port-forward, and autocannon and k6 setups that produce comparable numbers.
references/hot-path-patterns.md — a Piscina worker pool, stream backpressure, Fastify serialisation schemas, an lru-cache with stampede protection, a shared undici dispatcher, the connection-pool sizing formula worked through, and UV_THREADPOOL_SIZE against container CPU limits.
Sibling skills: nodejs-development for correct code, nodejs-testing for load and regression tests, nodejs-architecture for cache and worker placement, database-migrations for index changes on a live table, service-operability for the metrics pipeline, backend-code-review for reviews.
---
name: nodejs-performance
description: >
Diagnose and fix performance problems in Node.js back-end services — event loop blocking and lag, high p99 latency, low throughput, CPU hot paths, memory leaks and OOM kills, stream backpressure, JSON serialisation cost, connection-pool exhaustion, missing keep-alive and container CPU throttling. Use this whenever the user says a Node or TypeScript API is slow, p99 spikes, CPU sits at 100%, memory keeps growing, the pod is OOMKilled, they see "JavaScript heap out of memory", event loop lag or utilisation alerts fire, requests time out under load, or the load test misses its target; or asks about --cpu-prof, --heap-prof, --heapsnapshot-signal, 0x, node --inspect, autocannon, k6, perf_hooks, worker_threads, Piscina, cluster, UV_THREADPOOL_SIZE, --max-old-space-size or undici agents. Also use it when 502s appear behind a load balancer after a deploy.
metadata:
technology: Node.js
type: performance
---
# Node.js Performance
> **Targets:** Node 24 LTS (22 also LTS), undici, Fastify 5 · **Verified:** 2026-09 against https://nodejs.org/api/cli.html and https://nodejs.org/learn/diagnostics
Node is fast until something occupies the single JavaScript thread; then every request waits together. Find which of four things is true — the loop is busy, the loop is idle and waiting, memory is the problem, or the bottleneck is downstream — fix that one thing, re-measure. GC flags, clustering and extra replicas before a profile are guesses that make the system harder to diagnose.
## 1. Decide first
| Question | Default | Change when |
|---|---|---|
| Do we have a baseline? | **Get one**: p50/p95/p99 and throughput at a stated rate, against a production build with production-like data | Never skip it; "slow" without a number cannot be fixed or verified |
| Which of the four? | Read **event loop utilisation** and **loop lag** first — they split "busy" from "waiting" in one measurement | — |
| Profiler | `node --cpu-prof` and `--heap-prof`, opened in Chrome DevTools | Flame graph preferred → 0x. **Not clinic.js**: its README says it is not actively maintained and results may be inaccurate |
| Scale out or fix? | Fix the bottleneck | CPU is genuinely saturated with useful work |
| Processes per container | **One**, scaled horizontally, about 1 CPU each | A VM, or a pod that must use several cores and you accept the memory and complexity → `cluster` or PM2 |
| Caching | `lru-cache` with `max` **and** `ttl` for small per-instance data; Redis when shared | Every cache needs a bound, a TTL and an invalidation story before it is added |
## 2. Split "busy" from "waiting" in one measurement
```ts
import { monitorEventLoopDelay, performance } from 'node:perf_hooks';
const histogram = monitorEventLoopDelay({ resolution: 20 });
histogram.enable();
let previous = performance.eventLoopUtilization();
setInterval(() => {
const elu = performance.eventLoopUtilization(previous);
previous = performance.eventLoopUtilization();
logger.info({ elu: elu.utilization, lagP99ms: histogram.percentile(99) / 1e6 });
histogram.reset();
}, 10_000).unref();
```
`percentile(99)` is in nanoseconds, hence `/ 1e6`. Read the pair together:
| ELU | Loop lag p99 | Meaning |
|---|---|---|
| Near 1.0 | High | **CPU-bound on the main thread** — profile for synchronous work (§4) |
| Low | High | Something blocks in bursts: a big `JSON.parse`, a sync call, a GC pause |
| Low | Low, but latency is high | **Waiting** on the database, an upstream or a connection pool — the fix is not in Node |
| Rising with load, lag stable | Approaching capacity honestly — scale out |
Ship this permanently, not just during an incident: loop lag and ELU are the two metrics that tell you which half of the system to look at, and they cost almost nothing (`service-operability`).
## 3. Triage by symptom
| Symptom | Likely cause | Confirm with | Fix |
|---|---|---|---|
| Every endpoint slow together, ELU near 1 | Synchronous work on the loop | CPU profile: `JSON.parse`, regex, `*Sync`, crypto, long loops | Stream or chunk it; move it to a worker (§4) |
| p99 spikes periodically | GC pauses, large allocations | `--trace-gc`, heap profile | Allocate less; paginate; stop building giant strings and arrays |
| Latency high, ELU low | Waiting on a dependency or pool | Traces, pool wait metrics | Queries and indexes, pool size, parallelise independent calls |
| `bcrypt`/`fs`/`zlib`/`dns.lookup` slow under load | libuv thread pool (default **4**) saturated | Concurrency of those ops vs `UV_THREADPOOL_SIZE` | Raise `UV_THREADPOOL_SIZE` to match the CPU allowance; cache DNS |
| Memory floor rises until OOMKilled | Leak: unbounded cache, per-request listener, timer, retained closure | Three heap snapshots over time, compare retainers | Bound the cache, remove listeners, clear timers |
| RSS high, heap normal | Buffers, native memory, undrained streams | `memoryUsage().external` and `.arrayBuffers` | Backpressure with `pipeline`, stream instead of buffering |
| Many sockets, slow outbound calls | No keep-alive, or a client created per request | Socket counts, TLS handshakes in traces | One shared undici dispatcher per upstream |
| 502s behind the load balancer | `server.keepAliveTimeout` shorter than the LB idle timeout | Compare the two numbers | Server timeout above the LB's; `headersTimeout` above that |
| Throughput flat as pods scale | The database is the bottleneck | DB CPU, connections, locks | Queries, caching, read replicas — more pods make it worse |
| CPU looks low but latency is bad in Kubernetes | **CFS quota throttling** — bursts hit the quota and the process is stopped | `container_cpu_cfs_throttled_seconds_total` | Raise the limit or lower burstiness; ELU alone will not show this |
Commands, load-test setup and the heap-snapshot workflow: `references/profiling-playbook.md`.
## 4. Keep the loop free
- Anything over a few milliseconds of synchronous CPU per request — image processing, PDF generation, large CSV or JSON transforms, hashing loops — belongs in **`worker_threads`** through a pool (`piscina`) or in a separate queue-fed worker service.
- `setImmediate` yields inside a long loop are a stopgap that keeps the service responsive while you move the work; they do not make it faster.
- Watch for catastrophic regex backtracking on user input: bound input length and prefer linear-time patterns. This is a denial-of-service vector as much as a performance one.
- `AsyncLocalStorage` is not free — it costs on every async hop. One store holding a small request context is fine; several stores, or one holding large objects, shows up in profiles.
## 5. I/O, serialisation and caching
- Stream large payloads with `pipeline()` from `node:stream/promises`: it propagates errors and respects backpressure. Never `await res.text()` on a multi-megabyte body just to forward it.
- JSON is often the top CPU cost. Fastify response schemas compile a serialiser (`fast-json-stringify`) and drop unlisted fields — faster *and* safer. Return only the fields the client needs, and paginate lists.
- `JSON.parse(JSON.stringify(x))` as a deep clone parses twice; use `structuredClone`, or ask whether the clone is needed at all.
- Compression is CPU work on the machine you are trying to keep fast: do it at the proxy or CDN when there is one.
- Caching: `lru-cache` with `max` and `ttl` for hot per-instance data, Redis for shared. Add stampede protection (coalesce concurrent misses, jitter the TTLs), or the first expiry under load sends every request to the origin at once.
## 6. Database, outbound HTTP and memory limits
Pool size is per process: `replicas × processes × pool size` must stay below the database's connection limit, and serverless multiplies that by its concurrency — put a pooler in front rather than raising the ceiling. Kill N+1 with joins or DataLoader batching, add indexes from `EXPLAIN ANALYZE`, select only the columns you use, and paginate by keyset. Parallelise independent awaits with `Promise.all`, capped with `p-limit`.
Reuse **one** undici dispatcher (or agent) per upstream with keep-alive, and set timeouts on every call. A client created per request pays a TLS handshake per request and leaks sockets.
Set `--max-old-space-size` explicitly to roughly 70–80% of the container memory limit. Node does try to derive a default from the cgroup limit, but the heuristics have changed between versions and cgroup v2 detection has been unreliable — an explicit value means V8 collects before the kernel kills the process. Never set it *above* the container limit; that guarantees an OOM kill instead of a GC.
## 7. Deliverable format
```
## Baseline load profile, p50/p95/p99, RPS, error rate, ELU, loop lag, heap, tool used
## Diagnosis the triage row matched, with the evidence — profile frames, retainers, spans
## Fixes ordered by impact; the smallest diff for each, trade-off named
## Result the same load test re-run, before and after in one table
## Guardrails metrics and alerts added (loop lag, ELU, heap, pool wait) and the CI load-test threshold
```
## Checklist
- [ ] A baseline exists, taken against a production build with realistic data.
- [ ] ELU and loop lag were read before anything was changed.
- [ ] The diagnosis cites a profile frame, a retainer chain or a span — not a hunch.
- [ ] One change at a time, re-measured with the same load profile.
- [ ] Every cache added has `max`, `ttl` and a stated invalidation path.
- [ ] Concurrency added anywhere is capped.
- [ ] Pool arithmetic checked against the database's connection limit.
- [ ] `--max-old-space-size` is set below the container memory limit.
- [ ] Loop lag, ELU and heap are exported as metrics so the next incident starts with data.
## Anti-patterns
- **Adding replicas or `cluster` before profiling** → find the bottleneck; scaling a blocked loop scales the block.
- **Profiling `tsx` watch mode or the dev server** → profile the built artefact; the dev loader's frames are not your code.
- **clinic.js as the recommended tool** → unmaintained; `--cpu-prof`, `--heap-prof` and 0x.
- **`*Sync` fs, crypto or zlib on a request path** → async API, or a worker thread.
- **Unbounded `Map` used as a cache** → `lru-cache` with `max` and `ttl`.
- **Listener or timer added per request** → add once, or remove it in a `finally`.
- **Buffering a whole file or response in memory** → `pipeline()` and stream it.
- **A new HTTP client, pool or Redis connection per request** → create once at start-up, inject it.
- **`--max-old-space-size` at or above the container limit** → 70–80% of it.
- **`JSON.parse(JSON.stringify(x))` to clone** → `structuredClone`, or do not clone.
## Go deeper
- `references/profiling-playbook.md` — `--cpu-prof`, `--heap-prof`, `--heapsnapshot-signal` and 0x with the exact commands, reading a flame graph, the three-snapshot leak workflow, profiling inside a pod with `node --inspect` and port-forward, and autocannon and k6 setups that produce comparable numbers.
- `references/hot-path-patterns.md` — a Piscina worker pool, stream backpressure, Fastify serialisation schemas, an `lru-cache` with stampede protection, a shared undici dispatcher, the connection-pool sizing formula worked through, and `UV_THREADPOOL_SIZE` against container CPU limits.
- Sibling skills: `nodejs-development` for correct code, `nodejs-testing` for load and regression tests, `nodejs-architecture` for cache and worker placement, `database-migrations` for index changes on a live table, `service-operability` for the metrics pipeline, `backend-code-review` for reviews.