Diagnose and fix performance problems in .NET and ASP.NET Core services — measure first with dotnet-counters, dotnet-trace, dotnet-gcdump, BenchmarkDotNet and OpenTelemetry, then fix thread-pool starvation and sync-over-async, slow EF Core queries, allocation-heavy hot paths, leaking caches, connection-pool exhaustion, GC pressure and slow start-up.
When agents use itUse this whenever the user says a .NET API is slow, p95 or p99 latency has climbed, CPU sits high, memory grows until the pod restarts, they see OutOfMemoryException, GC pauses, timeouts under load, socket exhaustion or slow cold start; or asks which dotnet-counters or OpenTelemetry metric to read, how to profile a running process, whether Span<T>, ArrayPool, ValueTask, HybridCache, ServerGarbageCollection, DATAS, ReadyToRun or PublishAot would help. Also use it when a load test misses its target and nobody knows which layer is at fault.
Install
Copilot (VS Code, Visual Studio, Copilot CLI and github.com) reads skills from the repository — commit them so the whole team gets them.
npx skills add AGCO-Global/org-skills --skill dotnet-performance -a github-copilot
# or with the org installer (adds .github/skills/dotnet-performance):
npx -y github:AGCO-Global/org-skills add skill dotnet-performance
Uses the open skills CLI. Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI, OpenCode, Windsurf and 60+ others — it asks which agent to install into.
Installs the backend-dotnet-skills plugin, which bundles all Backend / .NET skills and keeps them updated.
/plugin marketplace add AGCO-Global/org-skills
/plugin install backend-dotnet-skills@org-skills
# or just this skill, in this repository:
npx skills add AGCO-Global/org-skills --skill dotnet-performance -a claude-code
Use Download .zip above, then upload it under Customize → Skills → Upload skill. Team and Enterprise admins can sync this repo as a plugin marketplace instead.
Installs into .agents/skills/, which Codex reads.
npx skills add AGCO-Global/org-skills --skill dotnet-performance -a codex
Installs into Cursor's skills folder.
npx skills add AGCO-Global/org-skills --skill dotnet-performance -a cursor
Installs into Gemini CLI's skills folder.
npx skills add AGCO-Global/org-skills --skill dotnet-performance -a gemini-cli
Nothing to install. Press Download .zip above, unzip it, and drop the dotnet-performance folder into your tool's skills folder — .claude/skills/, .github/skills/, .cursor/skills/ or .agents/skills/. Step-by-step, per tool: How to use it. For any other tool with rules or custom prompts, Copy SKILL.md and paste it in — it is plain Markdown.
The npx commands need Node.js 20+ and use your normal git sign-in to GitHub, so they work while the repository is private. Without Node.js, see the No Node.js tab — a skill is a folder of Markdown, so copying it in works just as well.
Measure, change one thing, measure again. Almost all real .NET slowness is waiting — queries, chatty HTTP, blocked threads, a missing cache — not CPU micro-costs, so a Span<T> rewrite before a profile is a guess with extra steps. Refuse to optimise without a baseline, and say what evidence you need instead.
1. Decide first
Question
Default
Change when
Do we have a baseline?
Stop and get one: the operation, current p95/p99, target, and whether it is CPU, memory, I/O or waiting
Commands, counter thresholds and how to capture any of this from a container: references/diagnostics-playbook.md.
3. Async and concurrency
Sync-over-async (.Result, .Wait(), GetAwaiter().GetResult()) is the single biggest throughput killer: the request holds a pool thread while waiting for another pool thread, so load makes it worse, not linear.
Task.Run around synchronous work in a request adds a thread hop and no capacity. It does not "make it async".
Parallelise independent I/O with Task.WhenAll; bound it with Parallel.ForEachAsync (MaxDegreeOfParallelism) or a SemaphoreSlim. Unbounded fan-out moves the bottleneck into the database.
Kestrel blocks synchronous I/O on request threads by default. That default is protecting you; re-enabling AllowSynchronousIO trades throughput for convenience.
ValueTask only where the result is usually already available and a profile shows the Task allocation. Never await one twice.
4. EF Core, queries and pooling
Correctness rules (AsNoTracking + projection, raw SQL parameterisation, concurrency tokens) live in dotnet-development §7. What belongs here:
Two or more collection Includes causing a cartesian explosion; costs extra round-trips, so measure both
ExecuteUpdateAsync / ExecuteDeleteAsync
Set-based updates and deletes, no materialisation
EF.CompileAsyncQuery
A very hot, identical query where translation shows in the profile
AddDbContextPool
High-throughput service with cheap contexts and no per-tenant model (dotnet-architecture multi-tenancy)
Index matching WHERE/ORDER BY/join columns
Always verified against the real execution plan, not guessed
Count queries per request in development; more than a handful for one endpoint is N+1 until proven otherwise. Connection pool size is a capacity decision: instances × Max Pool Size must stay under the database's connection limit, and a serverless host multiplies that by its instance count.
5. Allocation reduction — only on a proven hot path
Span<T> / ReadOnlySpan<char> and bounded stackalloc for parsing and slicing instead of Substring/Split.
ArrayPool<T>.Shared.Rent/Return in try/finally for temporary buffers; RecyclableMemoryStream for pooled streams.
Objects of 85,000 bytes or more go on the large object heap and are only collected in gen2. Big arrays and buffers are the usual source; pool or stream them.
Avoid LINQ inside tight loops (delegate and enumerator allocations); it is fine everywhere else.
StringComparison.Ordinal over ToLower() comparisons; StringBuilder for building; FrozenDictionary/FrozenSet for read-mostly lookups; SearchValues<T> for character-set searches.
Source generators remove whole categories of allocation: [LoggerMessage], [GeneratedRegex], JsonSerializerContext.
// BenchmarkDotNet comparison — MemoryExtensions.Count needs .NET 8+
[Benchmark(Baseline = true)] public int Split() => Input.Split(',').Length;
[Benchmark] public int SpanCount() => Input.AsSpan().Count(',') + 1;
6. Runtime, GC and hosting
Server GC is what an ASP.NET Core service wants, and Microsoft.NET.Sdk.Web sets ServerGarbageCollection=true for you — but a container limited to about one CPU falls back to workstation GC, which is why "it is slower in the small pod" is so common. DATAS is on by default with Server GC from .NET 9 and keeps the heap proportional to live data; turn it off with <GarbageCollectionAdaptationMode>0</GarbageCollectionAdaptationMode> only with a measurement that justifies it.
Set CPU and memory limits on the container — the runtime reads them and sizes heaps accordingly. Full settings table, connection-pool arithmetic, Kestrel HTTP/2 and HTTP/3 knobs and ReadyToRun versus Native AOT: references/runtime-and-hosting.md; the Native AOT prerequisite list lives in ../dotnet-development/references/background-and-streaming.md.
7. Deliverable format
## Baseline metric, tool, conditions (build, load, data size)
## Diagnosis the evidence — counter values, trace stacks, SQL — and what it rules out
## Fix the smallest change, as a diff, with the trade-off named
## Result the same measurement after, plus a regression guard (benchmark, load-test threshold, alert)
Checklist
A baseline number exists before any change is proposed.
The diagnosis names the metric or stack that proves it, not a hunch.
One change at a time, re-measured.
Benchmarks run in Release with [MemoryDiagnoser] and a baseline case.
Query fixes are verified against a real execution plan.
Every cache added has an expiry, a size bound and a stated invalidation path.
Concurrency added anywhere is bounded.
A regression guard (test, threshold or alert) ships with the fix.
The trade-off is written down — readability, memory, complexity.
Anti-patterns
Optimising without a profile → ask for the baseline and the symptom first; refuse politely.
Benchmarking in Debug or with Stopwatch loops → BenchmarkDotNet in Release.
ThreadPool.SetMinThreads to fix starvation → remove the blocking call; the pool was the symptom.
GC.Collect() in application code → fix allocation or lifetime; forcing collection makes pauses worse.
Unbounded IMemoryCache or a static Dictionary as a cache → size limit, expiry, eviction.
Task.Run to "make it async" → make the call actually asynchronous, or leave it synchronous off the request path.
Unbounded Task.WhenAll over a collection → Parallel.ForEachAsync with a degree limit.
Span<T> and unsafe in cold code → revert; readability is a performance property of the team.
Enabling Native AOT without checking every dependency → prerequisites first, ReadyToRun as the cheap alternative.
Go deeper
references/diagnostics-playbook.md — the exact dotnet-counters, dotnet-trace, dotnet-gcdump and dotnet-dump commands, the metric names and thresholds that confirm each triage row, capturing from a Kubernetes pod with dotnet-monitor, and a BenchmarkDotNet project skeleton.
references/runtime-and-hosting.md — GC modes and DATAS, container limits, connection-pool sizing, Kestrel HTTP/2 and HTTP/3, response compression, ReadyToRun versus Native AOT.
Sibling skills: dotnet-development for correct code, dotnet-testing for load and regression tests, dotnet-architecture for caching placement, database-migrations for index changes on a live table, service-operability for the metrics pipeline, backend-code-review for reviews.
---
name: dotnet-performance
description: >
Diagnose and fix performance problems in .NET and ASP.NET Core services — measure first with dotnet-counters, dotnet-trace, dotnet-gcdump, BenchmarkDotNet and OpenTelemetry, then fix thread-pool starvation and sync-over-async, slow EF Core queries, allocation-heavy hot paths, leaking caches, connection-pool exhaustion, GC pressure and slow start-up. Use this whenever the user says a .NET API is slow, p95 or p99 latency has climbed, CPU sits high, memory grows until the pod restarts, they see OutOfMemoryException, GC pauses, timeouts under load, socket exhaustion or slow cold start; or asks which dotnet-counters or OpenTelemetry metric to read, how to profile a running process, whether Span<T>, ArrayPool, ValueTask, HybridCache, ServerGarbageCollection, DATAS, ReadyToRun or PublishAot would help. Also use it when a load test misses its target and nobody knows which layer is at fault.
metadata:
technology: .NET
type: performance
---
# .NET Performance
> **Targets:** .NET 10 LTS, ASP.NET Core 10, EF Core 10 (.NET 8 LTS noted where different) · **Verified:** 2026-09 against https://learn.microsoft.com/en-us/dotnet/core/diagnostics/
Measure, change one thing, measure again. Almost all real .NET slowness is waiting — queries, chatty HTTP, blocked threads, a missing cache — not CPU micro-costs, so a `Span<T>` rewrite before a profile is a guess with extra steps. Refuse to optimise without a baseline, and say what evidence you need instead.
## 1. Decide first
| Question | Default | Change when |
|---|---|---|
| Do we have a baseline? | **Stop and get one**: the operation, current p95/p99, target, and whether it is CPU, memory, I/O or waiting | Never skip it — "it feels slow" is not a baseline |
| Production diagnosis | Metrics (`dotnet-counters` / OpenTelemetry) → traces → targeted profile | A process is already unhealthy → capture `dotnet-gcdump` / `dotnet-dump` before it is restarted |
| Micro-benchmark | **BenchmarkDotNet**, Release, no debugger, `[MemoryDiagnoser]` | Never a `Stopwatch` loop: JIT tiering and GC make it lie |
| Fix order | Remove work → cache it → make it async → make it cheaper | Reverse this and you will micro-optimise code that should not run |
| Caching | `HybridCache` (`Microsoft.Extensions.Caching.Hybrid`, works on .NET 8+) for shared read-mostly data | Per-instance and tiny → `IMemoryCache` with `SizeLimit`; whole responses → output caching |
| Scaling out | More instances | CPU is genuinely saturated by useful work — otherwise you are paying to run the bug in parallel |
## 2. Triage by symptom
| Symptom | Likely cause | Confirm with | Fix |
|---|---|---|---|
| Latency climbs under load, CPU low, thread count rising | **Thread-pool starvation** from sync-over-async | `dotnet.thread_pool.queue.length` > 0 and rising; `dotnet.thread_pool.thread.count` climbing; `.Result`/`.Wait()` in stacks | Async all the way. Raising `SetMinThreads` hides it |
| DB time dominates one endpoint | N+1, missing index, over-fetching | Logged SQL / trace spans; query plan | Projection, `AsSplitQuery`, index, pagination (§4) |
| p99 spikes every few seconds | GC pauses from allocation-heavy paths | `dotnet.gc.pause.time` rate, `dotnet.gc.collections{gen2}`, `dotnet.gc.heap.total_allocated` | Cut allocations on the hot path (§5); check for large object allocations |
| Memory grows until restart | Leak: static or unbounded cache, un-removed event handler, captured scope | `dotnet-gcdump` diff of two snapshots, then `gcroot` | Bound the cache, unsubscribe, fix lifetimes |
| `Timeout expired… from the pool` | Connection-pool exhaustion | Pool counters; count concurrent queries per request | Shorten transactions, stop leaking contexts, size `Max Pool Size` deliberately |
| Sockets in TIME_WAIT, stale DNS | `new HttpClient()` per call, or a static client forever | Socket counts | `IHttpClientFactory`, `PooledConnectionLifetime` |
| High steady CPU | Hot loop, regex, serialisation, logging | CPU trace hot stacks | Source-generated regex/JSON/logging, better algorithm |
| Slow cold start | JIT, reflection-heavy start-up, big DI graph | Start-up trace, `dotnet.jit.compilation.time` | ReadyToRun; trim start-up work; AOT only if compatible |
| Retry storms on a slow dependency | No timeout, unjittered retries | Resilience telemetry | Timeouts, jittered capped retries, circuit breaker (`dotnet-development` §8) |
Commands, counter thresholds and how to capture any of this from a container: `references/diagnostics-playbook.md`.
## 3. Async and concurrency
- Sync-over-async (`.Result`, `.Wait()`, `GetAwaiter().GetResult()`) is the single biggest throughput killer: the request holds a pool thread while waiting for another pool thread, so load makes it worse, not linear.
- `Task.Run` around synchronous work in a request adds a thread hop and no capacity. It does not "make it async".
- Parallelise independent I/O with `Task.WhenAll`; **bound** it with `Parallel.ForEachAsync` (`MaxDegreeOfParallelism`) or a `SemaphoreSlim`. Unbounded fan-out moves the bottleneck into the database.
- Kestrel blocks synchronous I/O on request threads by default. That default is protecting you; re-enabling `AllowSynchronousIO` trades throughput for convenience.
- `ValueTask` only where the result is usually already available *and* a profile shows the `Task` allocation. Never await one twice.
## 4. EF Core, queries and pooling
Correctness rules (`AsNoTracking` + projection, raw SQL parameterisation, concurrency tokens) live in `dotnet-development` §7. What belongs here:
| Technique | Use when |
|---|---|
| Keyset pagination (`WHERE (created, id) < (@c, @i)`) | Deep pages — `Skip(n)` re-reads every skipped row |
| `AsSplitQuery()` | Two or more collection `Include`s causing a cartesian explosion; costs extra round-trips, so measure both |
| `ExecuteUpdateAsync` / `ExecuteDeleteAsync` | Set-based updates and deletes, no materialisation |
| `EF.CompileAsyncQuery` | A very hot, identical query where translation shows in the profile |
| `AddDbContextPool` | High-throughput service with cheap contexts and no per-tenant model (`dotnet-architecture` multi-tenancy) |
| Index matching `WHERE`/`ORDER BY`/join columns | Always verified against the real execution plan, not guessed |
Count queries per request in development; more than a handful for one endpoint is N+1 until proven otherwise. Connection pool size is a capacity decision: instances × `Max Pool Size` must stay under the database's connection limit, and a serverless host multiplies that by its instance count.
## 5. Allocation reduction — only on a proven hot path
- `Span<T>` / `ReadOnlySpan<char>` and bounded `stackalloc` for parsing and slicing instead of `Substring`/`Split`.
- `ArrayPool<T>.Shared.Rent`/`Return` in `try/finally` for temporary buffers; `RecyclableMemoryStream` for pooled streams.
- Objects of 85,000 bytes or more go on the large object heap and are only collected in gen2. Big arrays and buffers are the usual source; pool or stream them.
- Avoid LINQ inside tight loops (delegate and enumerator allocations); it is fine everywhere else.
- `StringComparison.Ordinal` over `ToLower()` comparisons; `StringBuilder` for building; `FrozenDictionary`/`FrozenSet` for read-mostly lookups; `SearchValues<T>` for character-set searches.
- Source generators remove whole categories of allocation: `[LoggerMessage]`, `[GeneratedRegex]`, `JsonSerializerContext`.
```csharp
// BenchmarkDotNet comparison — MemoryExtensions.Count needs .NET 8+
[Benchmark(Baseline = true)] public int Split() => Input.Split(',').Length;
[Benchmark] public int SpanCount() => Input.AsSpan().Count(',') + 1;
```
## 6. Runtime, GC and hosting
Server GC is what an ASP.NET Core service wants, and `Microsoft.NET.Sdk.Web` sets `ServerGarbageCollection=true` for you — but a container limited to about one CPU falls back to workstation GC, which is why "it is slower in the small pod" is so common. DATAS is on by default with Server GC from .NET 9 and keeps the heap proportional to live data; turn it off with `<GarbageCollectionAdaptationMode>0</GarbageCollectionAdaptationMode>` only with a measurement that justifies it.
Set CPU and memory limits on the container — the runtime reads them and sizes heaps accordingly. Full settings table, connection-pool arithmetic, Kestrel HTTP/2 and HTTP/3 knobs and ReadyToRun versus Native AOT: `references/runtime-and-hosting.md`; the Native AOT prerequisite list lives in `../dotnet-development/references/background-and-streaming.md`.
## 7. Deliverable format
```
## Baseline metric, tool, conditions (build, load, data size)
## Diagnosis the evidence — counter values, trace stacks, SQL — and what it rules out
## Fix the smallest change, as a diff, with the trade-off named
## Result the same measurement after, plus a regression guard (benchmark, load-test threshold, alert)
```
## Checklist
- [ ] A baseline number exists before any change is proposed.
- [ ] The diagnosis names the metric or stack that proves it, not a hunch.
- [ ] One change at a time, re-measured.
- [ ] Benchmarks run in Release with `[MemoryDiagnoser]` and a baseline case.
- [ ] Query fixes are verified against a real execution plan.
- [ ] Every cache added has an expiry, a size bound and a stated invalidation path.
- [ ] Concurrency added anywhere is bounded.
- [ ] A regression guard (test, threshold or alert) ships with the fix.
- [ ] The trade-off is written down — readability, memory, complexity.
## Anti-patterns
- **Optimising without a profile** → ask for the baseline and the symptom first; refuse politely.
- **Benchmarking in Debug or with `Stopwatch` loops** → BenchmarkDotNet in Release.
- **`ThreadPool.SetMinThreads` to fix starvation** → remove the blocking call; the pool was the symptom.
- **`GC.Collect()` in application code** → fix allocation or lifetime; forcing collection makes pauses worse.
- **Unbounded `IMemoryCache` or a `static Dictionary` as a cache** → size limit, expiry, eviction.
- **`Task.Run` to "make it async"** → make the call actually asynchronous, or leave it synchronous off the request path.
- **Unbounded `Task.WhenAll` over a collection** → `Parallel.ForEachAsync` with a degree limit.
- **`Span<T>` and `unsafe` in cold code** → revert; readability is a performance property of the team.
- **Enabling Native AOT without checking every dependency** → prerequisites first, ReadyToRun as the cheap alternative.
## Go deeper
- `references/diagnostics-playbook.md` — the exact `dotnet-counters`, `dotnet-trace`, `dotnet-gcdump` and `dotnet-dump` commands, the metric names and thresholds that confirm each triage row, capturing from a Kubernetes pod with `dotnet-monitor`, and a BenchmarkDotNet project skeleton.
- `references/runtime-and-hosting.md` — GC modes and DATAS, container limits, connection-pool sizing, Kestrel HTTP/2 and HTTP/3, response compression, ReadyToRun versus Native AOT.
- Sibling skills: `dotnet-development` for correct code, `dotnet-testing` for load and regression tests, `dotnet-architecture` for caching placement, `database-migrations` for index changes on a live table, `service-operability` for the metrics pipeline, `backend-code-review` for reviews.