Org Skills

Profiling and load-testing Node.js

Supporting material for nodejs-performance. Agents load it on demand; it ships inside the skill folder.

RawSource

Verified 2026-09 against https://nodejs.org/api/cli.html, https://nodejs.org/learn/diagnostics/memory, https://github.com/davidmarkclements/0x, https://github.com/clinicjs/node-clinic and https://k6.io/docs/.

1. Which tool

Need Tool Why
Where is CPU going? node --cpu-prof → Chrome DevTools Built in, no dependencies, works in a container
Same, as a flame graph 0x Best single picture of a hot path
What is allocating? node --heap-prof Sampling allocation profile, cheap enough under load
What is retained? --heapsnapshot-signal=SIGUSR2 or v8.writeHeapSnapshot() The only way to find a leak's retainer chain
GC behaviour node --trace-gc Pause durations and which generation, straight to stdout
Live poking node --inspect + DevTools Interactive; use with a port-forward, never exposed

clinic.js is no longer the recommendation. Its README states the project is not actively maintained and that it may not work or may produce inaccurate results. If a runbook mentions clinic doctor, replace it with --cpu-prof plus DevTools.

2. CPU profile

node --cpu-prof --cpu-prof-dir=./profiles dist/server.js
# drive real load against it, then stop the process with SIGINT — the file is written on exit

Open the .cpuprofile in Chrome DevTools → Performance → Load profile. Keep source maps on (--enable-source-maps) so frames map back to TypeScript instead of dist.

Two rules that decide whether the profile is worth anything:

  1. Profile the built artefact, under load. A profile of tsx watch at idle shows the loader and the event loop sleeping. Neither is in production.
  2. Profile long enough to be representative — 30 seconds of steady load, not one request.

Read the flame graph bottom-up for self time: the widest frames with no children are where the CPU actually is. A wide frame with one wide child is just a call chain.

For a flame graph without DevTools:

npx 0x -- node dist/server.js

3. Memory: the three-snapshot method

One snapshot tells you nothing — everything in it looks legitimately alive. The comparison is the diagnosis.

node --heapsnapshot-signal=SIGUSR2 dist/server.js
kill -USR2 <pid>      # 1: after warm-up
# run load for several minutes
kill -USR2 <pid>      # 2
# run the same load again
kill -USR2 <pid>      # 3

Load all three into DevTools → Memory, select snapshot 3 and compare against snapshot 1. Sort by "Delta" and look for a constructor whose count grows in both intervals — that shape, growing linearly with work done, is the leak. Click an instance and follow Retainers to the root: the answer is usually a module-level Map, an EventEmitter with a listener added per request, or a closure captured by a timer.

Snapshots are the size of the heap and pause the process while they are written. On a memory-limited pod, check the ephemeral storage limit first and take them from a canary instance if you can.

For allocation rate rather than retention:

node --heap-prof --heap-prof-dir=./profiles dist/server.js

4. GC pauses

node --trace-gc dist/server.js

Each line gives the type (Scavenge is cheap gen-0 work; Mark-Compact is the expensive one), the heap before and after, and the pause in milliseconds. Frequent Mark-Compact with the heap barely shrinking means objects are surviving — a leak or a cache holding everything. Long scavenges mean a high allocation rate; --max-semi-space-size can help an allocation-heavy service, but only with a before-and-after measurement.

5. Profiling inside a pod

kubectl port-forward pod/orders-api-7d8 9229:9229
# start the process with --inspect=0.0.0.0:9229, then open chrome://inspect

Never expose the inspector port through a Service or Ingress: it is remote code execution by design. Prefer this order:

  1. --cpu-prof / --heapsnapshot-signal on one canary pod, copy the artefact out with kubectl cp, analyse locally. No open ports, no interactive session.
  2. --inspect with a port-forward, when you need to poke live and the artefact route has failed.
  3. Never leave either enabled as a default in the deployment.

Have the signal handler available in production images from the start — adding --heapsnapshot-signal requires a restart, and a restart destroys the leak you were trying to capture.

6. Load tests that produce comparable numbers

Quick check of one endpoint:

npx autocannon -c 100 -d 30 -p 10 http://localhost:3000/orders

A real scenario with thresholds, which is what belongs in CI:

// k6/orders.js
import http from 'k6/http';
import { check } from 'k6';

export const options = {
  stages: [
    { duration: '30s', target: 50 },   // ramp
    { duration: '2m', target: 50 },    // steady — measure this part
    { duration: '30s', target: 0 },
  ],
  thresholds: {
    http_req_failed: ['rate<0.01'],
    http_req_duration: ['p(95)<300', 'p(99)<800'],
  },
};

export default function () {
  const res = http.get(`${__ENV.BASE_URL}/orders?limit=20`);
  check(res, { 'status is 200': (r) => r.status === 200 });
}
k6 run -e BASE_URL=https://staging.internal k6/orders.js

What makes the result usable:

  • A production build, not the dev server, and a realistic dataset. An empty database benchmarks an empty database.
  • Report percentiles, never averages. An average hides the p99 that users complain about.
  • Discard the ramp. Compare the steady-state window before and after a change.
  • Keep the script in the repository next to the service, and put its p95 threshold in CI as the regression guard for whatever you just fixed. A performance fix without a threshold is re-broken within two quarters.