Org Skills

nodejs-performance

Diagnose and fix performance problems in Node.js back-end services — event loop blocking and lag, high p99 latency, low throughput, CPU hot paths, memory leaks and OOM kills, stream backpressure, JSON serialisation cost, connection-pool exhaustion, missing keep-alive and container CPU throttling.

Download .zip Raw Source
When agents use itUse this whenever the user says a Node or TypeScript API is slow, p99 spikes, CPU sits at 100%, memory keeps growing, the pod is OOMKilled, they see "JavaScript heap out of memory", event loop lag or utilisation alerts fire, requests time out under load, or the load test misses its target; or asks about --cpu-prof, --heap-prof, --heapsnapshot-signal, 0x, node --inspect, autocannon, k6, perf_hooks, worker_threads, Piscina, cluster, UV_THREADPOOL_SIZE, --max-old-space-size or undici agents. Also use it when 502s appear behind a load balancer after a deploy.

Install

Copilot (VS Code, Visual Studio, Copilot CLI and github.com) reads skills from the repository — commit them so the whole team gets them.

npx skills add AGCO-Global/org-skills --skill nodejs-performance -a github-copilot
# or with the org installer (adds .github/skills/nodejs-performance):
npx -y github:AGCO-Global/org-skills add skill nodejs-performance

Uses the open skills CLI. Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI, OpenCode, Windsurf and 60+ others — it asks which agent to install into.

npx skills add AGCO-Global/org-skills --skill nodejs-performance
# user-level instead of project-level:
npx skills add AGCO-Global/org-skills --skill nodejs-performance -g

Installs the backend-nodejs-skills plugin, which bundles all Backend / Node.js skills and keeps them updated.

/plugin marketplace add AGCO-Global/org-skills
/plugin install backend-nodejs-skills@org-skills
# or just this skill, in this repository:
npx skills add AGCO-Global/org-skills --skill nodejs-performance -a claude-code

Use Download .zip above, then upload it under Customize → Skills → Upload skill. Team and Enterprise admins can sync this repo as a plugin marketplace instead.

Installs into .agents/skills/, which Codex reads.

npx skills add AGCO-Global/org-skills --skill nodejs-performance -a codex

Installs into Cursor's skills folder.

npx skills add AGCO-Global/org-skills --skill nodejs-performance -a cursor

Installs into Gemini CLI's skills folder.

npx skills add AGCO-Global/org-skills --skill nodejs-performance -a gemini-cli

Nothing to install. Press Download .zip above, unzip it, and drop the nodejs-performance folder into your tool's skills folder — .claude/skills/, .github/skills/, .cursor/skills/ or .agents/skills/. Step-by-step, per tool: How to use it. For any other tool with rules or custom prompts, Copy SKILL.md and paste it in — it is plain Markdown.

The npx commands need Node.js 20+ and use your normal git sign-in to GitHub, so they work while the repository is private. Without Node.js, see the No Node.js tab — a skill is a folder of Markdown, so copying it in works just as well.

Skill contents

Node.js Performance

Targets: Node 24 LTS (22 also LTS), undici, Fastify 5 · Verified: 2026-09 against https://nodejs.org/api/cli.html and https://nodejs.org/learn/diagnostics

Node is fast until something occupies the single JavaScript thread; then every request waits together. Find which of four things is true — the loop is busy, the loop is idle and waiting, memory is the problem, or the bottleneck is downstream — fix that one thing, re-measure. GC flags, clustering and extra replicas before a profile are guesses that make the system harder to diagnose.

1. Decide first

Question Default Change when
Do we have a baseline? Get one: p50/p95/p99 and throughput at a stated rate, against a production build with production-like data Never skip it; "slow" without a number cannot be fixed or verified
Which of the four? Read event loop utilisation and loop lag first — they split "busy" from "waiting" in one measurement —
Profiler node --cpu-prof and --heap-prof, opened in Chrome DevTools Flame graph preferred → 0x. Not clinic.js: its README says it is not actively maintained and results may be inaccurate
Scale out or fix? Fix the bottleneck CPU is genuinely saturated with useful work
Processes per container One, scaled horizontally, about 1 CPU each A VM, or a pod that must use several cores and you accept the memory and complexity → cluster or PM2
Caching lru-cache with max and ttl for small per-instance data; Redis when shared Every cache needs a bound, a TTL and an invalidation story before it is added

2. Split "busy" from "waiting" in one measurement

import { monitorEventLoopDelay, performance } from 'node:perf_hooks';

const histogram = monitorEventLoopDelay({ resolution: 20 });
histogram.enable();
let previous = performance.eventLoopUtilization();

setInterval(() => {
  const elu = performance.eventLoopUtilization(previous);
  previous = performance.eventLoopUtilization();
  logger.info({ elu: elu.utilization, lagP99ms: histogram.percentile(99) / 1e6 });
  histogram.reset();
}, 10_000).unref();

percentile(99) is in nanoseconds, hence / 1e6. Read the pair together:

ELU Loop lag p99 Meaning
Near 1.0 High CPU-bound on the main thread — profile for synchronous work (§4)
Low High Something blocks in bursts: a big JSON.parse, a sync call, a GC pause
Low Low, but latency is high Waiting on the database, an upstream or a connection pool — the fix is not in Node
Rising with load, lag stable Approaching capacity honestly — scale out

Ship this permanently, not just during an incident: loop lag and ELU are the two metrics that tell you which half of the system to look at, and they cost almost nothing (service-operability).

3. Triage by symptom

Symptom Likely cause Confirm with Fix
Every endpoint slow together, ELU near 1 Synchronous work on the loop CPU profile: JSON.parse, regex, *Sync, crypto, long loops Stream or chunk it; move it to a worker (§4)
p99 spikes periodically GC pauses, large allocations --trace-gc, heap profile Allocate less; paginate; stop building giant strings and arrays
Latency high, ELU low Waiting on a dependency or pool Traces, pool wait metrics Queries and indexes, pool size, parallelise independent calls
bcrypt/fs/zlib/dns.lookup slow under load libuv thread pool (default 4) saturated Concurrency of those ops vs UV_THREADPOOL_SIZE Raise UV_THREADPOOL_SIZE to match the CPU allowance; cache DNS
Memory floor rises until OOMKilled Leak: unbounded cache, per-request listener, timer, retained closure Three heap snapshots over time, compare retainers Bound the cache, remove listeners, clear timers
RSS high, heap normal Buffers, native memory, undrained streams memoryUsage().external and .arrayBuffers Backpressure with pipeline, stream instead of buffering
Many sockets, slow outbound calls No keep-alive, or a client created per request Socket counts, TLS handshakes in traces One shared undici dispatcher per upstream
502s behind the load balancer server.keepAliveTimeout shorter than the LB idle timeout Compare the two numbers Server timeout above the LB's; headersTimeout above that
Throughput flat as pods scale The database is the bottleneck DB CPU, connections, locks Queries, caching, read replicas — more pods make it worse
CPU looks low but latency is bad in Kubernetes CFS quota throttling — bursts hit the quota and the process is stopped container_cpu_cfs_throttled_seconds_total Raise the limit or lower burstiness; ELU alone will not show this

Commands, load-test setup and the heap-snapshot workflow: references/profiling-playbook.md.

4. Keep the loop free

  • Anything over a few milliseconds of synchronous CPU per request — image processing, PDF generation, large CSV or JSON transforms, hashing loops — belongs in worker_threads through a pool (piscina) or in a separate queue-fed worker service.
  • setImmediate yields inside a long loop are a stopgap that keeps the service responsive while you move the work; they do not make it faster.
  • Watch for catastrophic regex backtracking on user input: bound input length and prefer linear-time patterns. This is a denial-of-service vector as much as a performance one.
  • AsyncLocalStorage is not free — it costs on every async hop. One store holding a small request context is fine; several stores, or one holding large objects, shows up in profiles.

5. I/O, serialisation and caching

  • Stream large payloads with pipeline() from node:stream/promises: it propagates errors and respects backpressure. Never await res.text() on a multi-megabyte body just to forward it.
  • JSON is often the top CPU cost. Fastify response schemas compile a serialiser (fast-json-stringify) and drop unlisted fields — faster and safer. Return only the fields the client needs, and paginate lists.
  • JSON.parse(JSON.stringify(x)) as a deep clone parses twice; use structuredClone, or ask whether the clone is needed at all.
  • Compression is CPU work on the machine you are trying to keep fast: do it at the proxy or CDN when there is one.
  • Caching: lru-cache with max and ttl for hot per-instance data, Redis for shared. Add stampede protection (coalesce concurrent misses, jitter the TTLs), or the first expiry under load sends every request to the origin at once.

6. Database, outbound HTTP and memory limits

Pool size is per process: replicas × processes × pool size must stay below the database's connection limit, and serverless multiplies that by its concurrency — put a pooler in front rather than raising the ceiling. Kill N+1 with joins or DataLoader batching, add indexes from EXPLAIN ANALYZE, select only the columns you use, and paginate by keyset. Parallelise independent awaits with Promise.all, capped with p-limit.

Reuse one undici dispatcher (or agent) per upstream with keep-alive, and set timeouts on every call. A client created per request pays a TLS handshake per request and leaks sockets.

Set --max-old-space-size explicitly to roughly 70–80% of the container memory limit. Node does try to derive a default from the cgroup limit, but the heuristics have changed between versions and cgroup v2 detection has been unreliable — an explicit value means V8 collects before the kernel kills the process. Never set it above the container limit; that guarantees an OOM kill instead of a GC.

7. Deliverable format

## Baseline     load profile, p50/p95/p99, RPS, error rate, ELU, loop lag, heap, tool used
## Diagnosis    the triage row matched, with the evidence — profile frames, retainers, spans
## Fixes        ordered by impact; the smallest diff for each, trade-off named
## Result       the same load test re-run, before and after in one table
## Guardrails   metrics and alerts added (loop lag, ELU, heap, pool wait) and the CI load-test threshold

Checklist

  • A baseline exists, taken against a production build with realistic data.
  • ELU and loop lag were read before anything was changed.
  • The diagnosis cites a profile frame, a retainer chain or a span — not a hunch.
  • One change at a time, re-measured with the same load profile.
  • Every cache added has max, ttl and a stated invalidation path.
  • Concurrency added anywhere is capped.
  • Pool arithmetic checked against the database's connection limit.
  • --max-old-space-size is set below the container memory limit.
  • Loop lag, ELU and heap are exported as metrics so the next incident starts with data.

Anti-patterns

  • Adding replicas or cluster before profiling → find the bottleneck; scaling a blocked loop scales the block.
  • Profiling tsx watch mode or the dev server → profile the built artefact; the dev loader's frames are not your code.
  • clinic.js as the recommended tool → unmaintained; --cpu-prof, --heap-prof and 0x.
  • *Sync fs, crypto or zlib on a request path → async API, or a worker thread.
  • Unbounded Map used as a cache → lru-cache with max and ttl.
  • Listener or timer added per request → add once, or remove it in a finally.
  • Buffering a whole file or response in memory → pipeline() and stream it.
  • A new HTTP client, pool or Redis connection per request → create once at start-up, inject it.
  • --max-old-space-size at or above the container limit → 70–80% of it.
  • JSON.parse(JSON.stringify(x)) to clone → structuredClone, or do not clone.

Go deeper

  • references/profiling-playbook.md — --cpu-prof, --heap-prof, --heapsnapshot-signal and 0x with the exact commands, reading a flame graph, the three-snapshot leak workflow, profiling inside a pod with node --inspect and port-forward, and autocannon and k6 setups that produce comparable numbers.
  • references/hot-path-patterns.md — a Piscina worker pool, stream backpressure, Fastify serialisation schemas, an lru-cache with stampede protection, a shared undici dispatcher, the connection-pool sizing formula worked through, and UV_THREADPOOL_SIZE against container CPU limits.
  • Sibling skills: nodejs-development for correct code, nodejs-testing for load and regression tests, nodejs-architecture for cache and worker placement, database-migrations for index changes on a live table, service-operability for the metrics pipeline, backend-code-review for reviews.

References

Deeper material the agent loads only when needed.