Org Skills

.NET diagnostics playbook

Supporting material for dotnet-performance. Agents load it on demand; it ships inside the skill folder.

RawSource

Verified 2026-09 against https://learn.microsoft.com/en-us/dotnet/core/diagnostics/built-in-metrics-runtime, https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-counters, https://learn.microsoft.com/en-us/dotnet/core/diagnostics/dotnet-monitor and https://benchmarkdotnet.org/.

1. Install the tools

dotnet tool install --global dotnet-counters
dotnet tool install --global dotnet-trace
dotnet tool install --global dotnet-gcdump
dotnet tool install --global dotnet-dump

They attach to a running process by PID and need no changes to the application.

2. Live metrics: dotnet-counters

dotnet-counters ps                                   # find the PID
dotnet-counters monitor -p <pid> --counters System.Runtime,Microsoft.AspNetCore.Hosting
dotnet-counters collect -p <pid> --format csv -o run.csv   # for a load test

The System.Runtime meter (metric names below are the .NET 9+ System.Diagnostics.Metrics names) gives you everything the triage table needs:

Metric Reads as Confirms
dotnet.thread_pool.queue.length Work items waiting for a pool thread Thread-pool starvation when it is persistently above 0 and rising while CPU is low. A momentary spike is normal
dotnet.thread_pool.thread.count Pool threads that exist Starvation again: the pool injects roughly one thread per second, so a slow steady climb under constant load is the signature
dotnet.gc.pause.time Total seconds paused for GC since start Divide the delta by wall-clock time: more than a few percent is GC pressure
dotnet.gc.heap.total_allocated Bytes allocated since start Allocation rate — the delta per second is the number that matters, not the total
dotnet.gc.collections (attribute gc.heap.generation) Collections per generation Rising gen2 count means objects are surviving; look for a leak or large objects
dotnet.gc.last_collection.heap.size (generation loh) Large object heap size Confirms objects ≥ 85,000 bytes are the pressure
dotnet.process.memory.working_set Physical memory of the process Compare against the container limit before the OOM kill
dotnet.monitor.lock_contentions Contended lock acquisitions since start A hot lock serialising requests
dotnet.exceptions (attribute error.type) Exceptions thrown Exception-driven control flow on a hot path

ASP.NET Core adds http.server.request.duration (a histogram, in seconds, with http.route and http.response.status_code), which is the metric to alert on and the one to quote as the baseline.

dotnet-counters also understands the older EventCounters names (threadpool-queue-length, gc-heap-size); expect those on .NET 8 and earlier.

3. CPU and wall-clock profiles: dotnet-trace

dotnet-trace collect -p <pid> --profile cpu-sampling --duration 00:00:30
dotnet-trace collect -p <pid> --providers Microsoft-DotNETCore-SampleProfiler,Microsoft-Diagnostics-DiagnosticSource

Open the .nettrace in Visual Studio, PerfView, or convert it for speedscope:

dotnet-trace convert trace.nettrace --format Speedscope

Read it as: wide frames near the top of the flame graph are where the time is. Blocked time shows up as frames in System.Threading and Monitor.Wait — that is the sync-over-async fingerprint, and it is invisible in a CPU-percentage graph.

4. Memory: dotnet-gcdump and dotnet-dump

The workflow that actually finds leaks is a diff of two snapshots, not one snapshot:

dotnet-gcdump collect -p <pid> -o before.gcdump
# let the service run under normal load for 10-30 minutes
dotnet-gcdump collect -p <pid> -o after.gcdump

Open both in Visual Studio and compare; the types whose counts grew are the candidates. gcdump is cheap enough for production (it triggers a gen2 GC and walks the heap) and contains type names and sizes, not field values.

When you need the object graph and the values, take a full dump instead — it contains everything in memory, including secrets, so treat the file as sensitive:

dotnet-dump collect -p <pid> -o core.dmp
dotnet-dump analyze core.dmp
> dumpheap -stat          # types by total size
> gcroot <address>        # what is keeping one alive
> clrthreads              # thread states; look for many BLOCKED

5. Capturing from a container or pod

Three options, in order of preference:

  1. dotnet-monitor as a sidecar. It exposes metrics and on-demand traces, dumps and gcdumps over HTTP inside the pod, with no shell in the application container. This is the only option that works against a chiseled image and the only one that is reasonable to leave enabled.
  2. Ephemeral debug container (kubectl debug -it <pod> --image=mcr.microsoft.com/dotnet/sdk:10.0 --target=<container>), then install the tools and attach to PID 1. Works anywhere, leaves nothing behind.
  3. Tools baked into the image. Convenient, but they enlarge the image and the attack surface; acceptable in a non-production environment only.

Copy artefacts out with kubectl cp. Remember that a dump can be as large as the process's memory — check the ephemeral storage limit before collecting one in a pod.

6. Micro-benchmarks: BenchmarkDotNet

<!-- Benchmarks.csproj -->
<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType>Exe</OutputType>
    <TargetFramework>net10.0</TargetFramework>
    <Optimize>true</Optimize>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="BenchmarkDotNet" Version="0.15.*" />
  </ItemGroup>
</Project>
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;

BenchmarkRunner.Run<CsvBenchmarks>();

[MemoryDiagnoser]
public class CsvBenchmarks
{
    [Params(10, 1_000)]
    public int FieldCount { get; set; }

    private string _input = string.Empty;

    [GlobalSetup]
    public void Setup() => _input = string.Join(',', Enumerable.Range(0, FieldCount));

    [Benchmark(Baseline = true)]
    public int Split() => _input.Split(',').Length;

    [Benchmark]
    public int SpanCount() => _input.AsSpan().Count(',') + 1;
}

Run it with dotnet run -c Release --project Benchmarks. BenchmarkDotNet refuses to run a Debug build and warns when a debugger is attached — do not work around either.

Read the output as Mean, Ratio against the baseline, and Allocated per operation. A change under a few percent with overlapping error bars is noise. [Params] matters more than people expect: an optimisation that wins at ten items and loses at a thousand is a regression waiting for real data.

7. Load tests

Any of k6, NBomber, Bombardier or JMeter will do. What makes the result usable:

  • A fixed, representative dataset — an empty database benchmarks nothing.
  • Report percentiles (p50/p95/p99) at a stated request rate, never an average.
  • Warm up first, then measure; the first seconds are JIT and cache population.
  • Run against a build and topology that resembles production. A load test on a laptop against a Debug build measures the laptop.
  • Keep the scenario in the repository next to the service, and put its p95 threshold in CI as the regression guard for whatever you just fixed.