# Runners, coverage, CI and flaky tests

Verified 2026-09 against <https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-with-dotnet-test>,
<https://xunit.net/docs/getting-started/v3/migration>,
<https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-code-coverage> and
<https://stryker-mutator.io/docs/stryker-net/introduction/>.

## 1. Two `dotnet test` modes

The .NET 10 SDK can drive tests two ways, and the difference decides your coverage command.

| | VSTest mode | Microsoft.Testing.Platform (MTP) mode |
|---|---|---|
| Status | **The SDK default**, and the only mode before .NET 10 | Opt-in, introduced in the .NET 10 SDK |
| Opt in with | Nothing | `global.json` (below) |
| Test project | A library run by `vstest.console` | A stand-alone executable that runs itself |
| Coverage | `--collect:"XPlat Code Coverage"` (coverlet.collector) | `--coverage` from `Microsoft.Testing.Extensions.CodeCoverage` |
| TRX report | `--logger "trx"` | `--report-trx`, with `Microsoft.Testing.Extensions.TrxReport` referenced |

```json
{
  "test": {
    "runner": "Microsoft.Testing.Platform"
  }
}
```

xUnit v3 test projects are MTP applications natively, so MTP mode is the coherent choice for a new
solution on the .NET 10 SDK. Existing solutions can stay on VSTest mode indefinitely — but pick one
per repository. A solution where some projects run under VSTest and others under MTP is explicitly
unsupported and fails in confusing ways.

## 2. Coverage

```bash
# VSTest mode — needs a PackageReference to coverlet.collector
dotnet test --collect:"XPlat Code Coverage"

# MTP mode — needs Microsoft.Testing.Extensions.CodeCoverage
dotnet test --coverage --coverage-output-format cobertura
```

Turn either into a readable report with ReportGenerator:

```bash
dotnet tool install --global dotnet-reportgenerator-globaltool
reportgenerator -reports:"**/coverage.cobertura.xml" -targetdir:coveragereport -reporttypes:Html
```

Gate on **changed-line coverage**, not a global percentage. A global gate that only ratchets upward
produces assertion-free tests written to satisfy it, which is worse than no gate: it adds maintenance
and removes the signal. Exclude generated code (`[ExcludeFromCodeCoverage]`, migrations, `*.g.cs`)
so the number means something.

## 3. Mutation testing

Coverage says a line ran. Mutation testing says an assertion would have caught it being wrong.

```bash
dotnet tool install --global dotnet-stryker
dotnet stryker --project Orders.csproj
```

Run it on the domain project, not the whole solution, and not on every pull request — it is slow by
construction (it runs the suite once per mutant). A quarterly run over the modules that hold the
money rules finds the tests that assert nothing, which is exactly where it pays.

## 4. Contract tests

- **PactNet** for consumer-driven contracts between services owned by different teams: the consumer
  publishes expectations, the provider verifies them in its own pipeline. Worth the setup only when
  the two sides really do deploy independently.
- **OpenAPI diff** for a public HTTP API: generate the document in the build, compare it to the
  committed one, fail on a breaking change (`api-design`).
- **`Verify` snapshots** for message and JSON contracts, so an accidental property rename shows up as
  a diff rather than a silently ignored field downstream:

```csharp
[Fact]
public Task OrderPlaced_contract() =>
    Verifier.Verify(new OrderPlaced(Guid.Empty, "GBP", 42.00m));
```

Review the `.received.` file before accepting it into `.verified.` — the value of a snapshot test is
entirely in the review step.

## 5. CI pipeline shape

```
restore → build -warnaserror → unit tests → integration tests (Docker available) → publish TRX + coverage
```

- Build once, then `dotnet test --no-build -c Release`. Tests that run against a Debug build measure
  a build nobody ships.
- Set culture and time zone explicitly in CI (`DOTNET_SYSTEM_GLOBALIZATION_INVARIANT` is *not* the
  way — set `TZ` and use `CultureInfo.InvariantCulture` in code) so locale bugs surface in a
  controlled way rather than only on a developer's machine in another country.
- Publish test results and coverage as artefacts even on failure; a red build whose report was not
  uploaded costs a full re-run to diagnose.
- Split long integration suites with `--filter` or per-project jobs rather than raising the timeout.

## 6. Flaky test triage

| Symptom | Likely cause | Fix |
|---|---|---|
| Passes alone, fails in the suite | Shared rows or static state; parallel collections touching one database | Reset per test (Respawn), isolate collections, delete the statics |
| Fails only in CI | Time zone, culture, a slower machine, container not ready | `FakeTimeProvider`, invariant culture, a Testcontainers wait strategy instead of a sleep |
| Intermittent timeout | Un-awaited task, a real delay, a hosted service racing the test | Await everything, remove hosted services from the test host, poll with a deadline |
| Order-dependent | A test relies on data another created | Each test arranges its own data |
| Fails once per day at about the same time | A date boundary, a token expiry, a scheduled job | Control time; do not retry the test |
| Fails right after a dependency upgrade | Behaviour change, not flakiness | Read the changelog before adding a retry |

Two rules that keep the suite trustworthy: **never add an automatic retry to make a test green** — it
converts a real intermittent bug into a slow one — and give every quarantined test an owner and an
issue link, with a date at which it is deleted if nobody has fixed it.
