# .NET diagnostics playbook

Verified 2026-09 against <https://learn.microsoft.com/en-us/dotnet/core/diagnostics/built-in-metrics-runtime>,
<https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-counters>,
<https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-monitor> and
<https://benchmarkdotnet.org/>.

## 1. Install the tools

```bash
dotnet tool install --global dotnet-counters
dotnet tool install --global dotnet-trace
dotnet tool install --global dotnet-gcdump
dotnet tool install --global dotnet-dump
```

They attach to a running process by PID and need no changes to the application.

## 2. Live metrics: `dotnet-counters`

```bash
dotnet-counters ps                                   # find the PID
dotnet-counters monitor -p <pid> --counters System.Runtime,Microsoft.AspNetCore.Hosting
dotnet-counters collect -p <pid> --format csv -o run.csv   # for a load test
```

The `System.Runtime` meter (metric names below are the .NET 9+ `System.Diagnostics.Metrics` names)
gives you everything the triage table needs:

| Metric | Reads as | Confirms |
|---|---|---|
| `dotnet.thread_pool.queue.length` | Work items waiting for a pool thread | **Thread-pool starvation** when it is persistently above 0 and rising while CPU is low. A momentary spike is normal |
| `dotnet.thread_pool.thread.count` | Pool threads that exist | Starvation again: the pool injects roughly one thread per second, so a slow steady climb under constant load is the signature |
| `dotnet.gc.pause.time` | Total seconds paused for GC since start | Divide the delta by wall-clock time: more than a few percent is GC pressure |
| `dotnet.gc.heap.total_allocated` | Bytes allocated since start | Allocation rate — the delta per second is the number that matters, not the total |
| `dotnet.gc.collections` (attribute `gc.heap.generation`) | Collections per generation | Rising `gen2` count means objects are surviving; look for a leak or large objects |
| `dotnet.gc.last_collection.heap.size` (generation `loh`) | Large object heap size | Confirms objects ≥ 85,000 bytes are the pressure |
| `dotnet.process.memory.working_set` | Physical memory of the process | Compare against the container limit before the OOM kill |
| `dotnet.monitor.lock_contentions` | Contended lock acquisitions since start | A hot `lock` serialising requests |
| `dotnet.exceptions` (attribute `error.type`) | Exceptions thrown | Exception-driven control flow on a hot path |

ASP.NET Core adds `http.server.request.duration` (a histogram, in seconds, with `http.route` and
`http.response.status_code`), which is the metric to alert on and the one to quote as the baseline.

`dotnet-counters` also understands the older EventCounters names (`threadpool-queue-length`,
`gc-heap-size`); expect those on .NET 8 and earlier.

## 3. CPU and wall-clock profiles: `dotnet-trace`

```bash
dotnet-trace collect -p <pid> --profile cpu-sampling --duration 00:00:30
dotnet-trace collect -p <pid> --providers Microsoft-DotNETCore-SampleProfiler,Microsoft-Diagnostics-DiagnosticSource
```

Open the `.nettrace` in Visual Studio, PerfView, or convert it for speedscope:

```bash
dotnet-trace convert trace.nettrace --format Speedscope
```

Read it as: wide frames near the top of the flame graph are where the time is. Blocked time shows up
as frames in `System.Threading` and `Monitor.Wait` — that is the sync-over-async fingerprint, and it
is invisible in a CPU-percentage graph.

## 4. Memory: `dotnet-gcdump` and `dotnet-dump`

The workflow that actually finds leaks is a **diff of two snapshots**, not one snapshot:

```bash
dotnet-gcdump collect -p <pid> -o before.gcdump
# let the service run under normal load for 10-30 minutes
dotnet-gcdump collect -p <pid> -o after.gcdump
```

Open both in Visual Studio and compare; the types whose counts grew are the candidates. `gcdump` is
cheap enough for production (it triggers a gen2 GC and walks the heap) and contains type names and
sizes, not field values.

When you need the object graph and the values, take a full dump instead — it contains everything in
memory, including secrets, so treat the file as sensitive:

```bash
dotnet-dump collect -p <pid> -o core.dmp
dotnet-dump analyze core.dmp
> dumpheap -stat          # types by total size
> gcroot <address>        # what is keeping one alive
> clrthreads              # thread states; look for many BLOCKED
```

## 5. Capturing from a container or pod

Three options, in order of preference:

1. **`dotnet-monitor` as a sidecar.** It exposes metrics and on-demand traces, dumps and gcdumps over
   HTTP inside the pod, with no shell in the application container. This is the only option that
   works against a chiseled image and the only one that is reasonable to leave enabled.
2. **Ephemeral debug container** (`kubectl debug -it <pod> --image=mcr.microsoft.com/dotnet/sdk:10.0
   --target=<container>`), then install the tools and attach to PID 1. Works anywhere, leaves nothing
   behind.
3. **Tools baked into the image.** Convenient, but they enlarge the image and the attack surface;
   acceptable in a non-production environment only.

Copy artefacts out with `kubectl cp`. Remember that a dump can be as large as the process's memory —
check the ephemeral storage limit before collecting one in a pod.

## 6. Micro-benchmarks: BenchmarkDotNet

```xml
<!-- Benchmarks.csproj -->
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Optimize>true</Optimize>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.15.*" />
  </ItemGroup>
</Project>
```

```csharp
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkRunner.Run<CsvBenchmarks>();

[MemoryDiagnoser]
public class CsvBenchmarks
{
    [Params(10, 1_000)]
    public int FieldCount { get; set; }

    private string _input = string.Empty;

    [GlobalSetup]
    public void Setup() => _input = string.Join(',', Enumerable.Range(0, FieldCount));

    [Benchmark(Baseline = true)]
    public int Split() => _input.Split(',').Length;

    [Benchmark]
    public int SpanCount() => _input.AsSpan().Count(',') + 1;
}
```

Run it with `dotnet run -c Release --project Benchmarks`. BenchmarkDotNet refuses to run a Debug build
and warns when a debugger is attached — do not work around either.

Read the output as *Mean*, *Ratio* against the baseline, and *Allocated* per operation. A change
under a few percent with overlapping error bars is noise. `[Params]` matters more than people expect:
an optimisation that wins at ten items and loses at a thousand is a regression waiting for real data.

## 7. Load tests

Any of k6, NBomber, Bombardier or JMeter will do. What makes the result usable:

- A fixed, representative dataset — an empty database benchmarks nothing.
- Report percentiles (p50/p95/p99) at a stated request rate, never an average.
- Warm up first, then measure; the first seconds are JIT and cache population.
- Run against a build and topology that resembles production. A load test on a laptop against a
  Debug build measures the laptop.
- Keep the scenario in the repository next to the service, and put its p95 threshold in CI as the
  regression guard for whatever you just fixed.
