Org Skills

Runners, coverage, CI and flaky tests

Supporting material for dotnet-testing. Agents load it on demand; it ships inside the skill folder.

RawSource

Verified 2026-09 against https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-with-dotnet-test, https://xunit.net/docs/getting-started/v3/migration, https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-code-coverage and https://stryker-mutator.io/docs/stryker-net/introduction/.

1. Two dotnet test modes

The .NET 10 SDK can drive tests two ways, and the difference decides your coverage command.

VSTest mode Microsoft.Testing.Platform (MTP) mode
Status The SDK default, and the only mode before .NET 10 Opt-in, introduced in the .NET 10 SDK
Opt in with Nothing global.json (below)
Test project A library run by vstest.console A stand-alone executable that runs itself
Coverage --collect:"XPlat Code Coverage" (coverlet.collector) --coverage from Microsoft.Testing.Extensions.CodeCoverage
TRX report --logger "trx" --report-trx, with Microsoft.Testing.Extensions.TrxReport referenced
{
  "test": {
    "runner": "Microsoft.Testing.Platform"
  }
}

xUnit v3 test projects are MTP applications natively, so MTP mode is the coherent choice for a new solution on the .NET 10 SDK. Existing solutions can stay on VSTest mode indefinitely — but pick one per repository. A solution where some projects run under VSTest and others under MTP is explicitly unsupported and fails in confusing ways.

2. Coverage

# VSTest mode — needs a PackageReference to coverlet.collector
dotnet test --collect:"XPlat Code Coverage"

# MTP mode — needs Microsoft.Testing.Extensions.CodeCoverage
dotnet test --coverage --coverage-output-format cobertura

Turn either into a readable report with ReportGenerator:

dotnet tool install --global dotnet-reportgenerator-globaltool
reportgenerator -reports:"**/coverage.cobertura.xml" -targetdir:coveragereport -reporttypes:Html

Gate on changed-line coverage, not a global percentage. A global gate that only ratchets upward produces assertion-free tests written to satisfy it, which is worse than no gate: it adds maintenance and removes the signal. Exclude generated code ([ExcludeFromCodeCoverage], migrations, *.g.cs) so the number means something.

3. Mutation testing

Coverage says a line ran. Mutation testing says an assertion would have caught it being wrong.

dotnet tool install --global dotnet-stryker
dotnet stryker --project Orders.csproj

Run it on the domain project, not the whole solution, and not on every pull request — it is slow by construction (it runs the suite once per mutant). A quarterly run over the modules that hold the money rules finds the tests that assert nothing, which is exactly where it pays.

4. Contract tests

  • PactNet for consumer-driven contracts between services owned by different teams: the consumer publishes expectations, the provider verifies them in its own pipeline. Worth the setup only when the two sides really do deploy independently.
  • OpenAPI diff for a public HTTP API: generate the document in the build, compare it to the committed one, fail on a breaking change (api-design).
  • Verify snapshots for message and JSON contracts, so an accidental property rename shows up as a diff rather than a silently ignored field downstream:
[Fact]
public Task OrderPlaced_contract() =>
    Verifier.Verify(new OrderPlaced(Guid.Empty, "GBP", 42.00m));

Review the .received. file before accepting it into .verified. — the value of a snapshot test is entirely in the review step.

5. CI pipeline shape

restore → build -warnaserror → unit tests → integration tests (Docker available) → publish TRX + coverage
  • Build once, then dotnet test --no-build -c Release. Tests that run against a Debug build measure a build nobody ships.
  • Set culture and time zone explicitly in CI (DOTNET_SYSTEM_GLOBALIZATION_INVARIANT is not the way — set TZ and use CultureInfo.InvariantCulture in code) so locale bugs surface in a controlled way rather than only on a developer's machine in another country.
  • Publish test results and coverage as artefacts even on failure; a red build whose report was not uploaded costs a full re-run to diagnose.
  • Split long integration suites with --filter or per-project jobs rather than raising the timeout.

6. Flaky test triage

Symptom Likely cause Fix
Passes alone, fails in the suite Shared rows or static state; parallel collections touching one database Reset per test (Respawn), isolate collections, delete the statics
Fails only in CI Time zone, culture, a slower machine, container not ready FakeTimeProvider, invariant culture, a Testcontainers wait strategy instead of a sleep
Intermittent timeout Un-awaited task, a real delay, a hosted service racing the test Await everything, remove hosted services from the test host, poll with a deadline
Order-dependent A test relies on data another created Each test arranges its own data
Fails once per day at about the same time A date boundary, a token expiry, a scheduled job Control time; do not retry the test
Fails right after a dependency upgrade Behaviour change, not flakiness Read the changelog before adding a retry

Two rules that keep the suite trustworthy: never add an automatic retry to make a test green — it converts a real intermittent bug into a slow one — and give every quarantined test an owner and an issue link, with a date at which it is deleted if nobody has fixed it.