novotnyllc/dotnet-artisan · GitHub
Defines .NET test strategy and implementation patterns across xUnit v3 (Facts, Theories, fixtures, IAsyncLifetime), integration testing (WebApplicationFactory, Testcontainers), Aspire testing (DistributedApplicationTestingBuilder), snapshot testing (Verify, scrubbing), Playwright E2E browser automation, BenchmarkDotNet microbenchmarks, code coverage (Coverlet), mutation testing (Stryker.NET), UI testing (page objects, selectors), and AOT WASM test compilation. Spans 13 topic areas. Do not use for production API architecture or CI workflow authoring.
프로젝트 폴더에서 아래 명령어를 실행하고, 설치할 에이전트를 선택하세요.
npx skills add novotnyllc/dotnet-artisan --skill dotnet-testing설치 명령을 직접 실행해야 적용됩니다. 지원 에이전트와 필요한 권한·라이선스는 제작자의 안내를 확인하세요.
agents/openai.yamlinterface: display_name: "dotnet-testing" short_description: "xUnit, integration, E2E, and benchmark strategy" default_prompt: "Use $dotnet-advisor to route this testing request, then load $dotnet-testing for strategy and quality gates." policy: allow_implicit_invocation: true
references/add-testing.md# Add Testing
Add test infrastructure scaffolding to an existing .NET project. Creates test projects with xUnit, configures code coverage with coverlet, and sets up the conventional directory structure.
## Test Project Structure
Follow the convention of mirroring `src/` project names under `tests/`:
```
MyApp/
├── src/
│ ├── MyApp.Core/
│ ├── MyApp.Api/
│ └── MyApp.Infrastructure/
└── tests/
├── MyApp.Core.UnitTests/
├── MyApp.Api.UnitTests/
├── MyApp.Api.IntegrationTests/
└── Directory.Build.props # Test-specific build settings
```
Naming conventions:
- `*.UnitTests` -- isolated tests with no external dependencies
- `*.IntegrationTests` -- tests that use real infrastructure (database, HTTP, file system)
- `*.FunctionalTests` -- end-to-end tests through the full application stack
---
## Step 1: Create the Test Project
```bash
# Create xUnit test project
dotnet new xunit -n MyApp.Core.UnitTests -o tests/MyApp.Core.UnitTests
# Add to solution
dotnet sln add tests/MyApp.Core.UnitTests/MyApp.Core.UnitTests.csproj
# Add reference to the project under test
dotnet add tests/MyApp.Core.UnitTests/MyApp.Core.UnitTests.csproj \
reference src/MyApp.Core/MyApp.Core.csproj
```
### Clean Up Generated Project
Remove properties already defined in `Directory.Build.props`:
```xml
<!-- tests/MyApp.Core.UnitTests/MyApp.Core.UnitTests.csproj -->
<Project Sdk="Microsoft.NET.Sdk">
<ItemGroup>
<PackageReference Include="Microsoft.NET.Test.Sdk" />
<PackageReference Include="xunit.v3" />
<PackageReference Include="xunit.runner.visualstudio" />
<PackageReference Include="coverlet.collector" />
</ItemGroup>
<ItemGroup>
<ProjectReference Include="..\..\src\MyApp.Core\MyApp.Core.csproj" />
</ItemGroup>
</Project>
```
With CPM, `Version` attributes are managed in `Directory.Packages.props`. Remove them from the generated `.csproj`.
---
## Step 2: Add Test-Specific Build Properties
Create `tests/Directory.Build.props` to customize settings for all test projects:
```xml
<!-- tests/Directory.Build.props -->
<Project>
<Import Project="$([MSBuild]::GetPathOfFileAbove('Directory.Build.props', '$(MSBuildThisFileDirectory)../'))" />
<PropertyGroup>
<IsPackable>false</IsPackable>
<IsTestProject>true</IsTestProject>
<!-- Use Microsoft.Testing.Platform v2 runner (requires Microsoft.NET.Test.Sdk 17.13+/18.x) -->
<UseMicrosoftTestingPlatformRunner>true</UseMicrosoftTestingPlatformRunner>
<!-- Relax strictness for test projects -->
<TreatWarningsAsErrors>false</TreatWarningsAsErrors>
</PropertyGroup>
</Project>
```
This imports the root `Directory.Build.props` (for shared settings like `Nullable`, `ImplicitUsings`, `LangVersion`) and overrides test-specific properties.
---
## Step 3: Register Test Packages in CPM
Add test package versions to `Directory.Packages.props`:
```xml
<!-- In Directory.Packages.props -->
<ItemGroup>
<!-- Test packages -->
<PackageVersion Include="Microsoft.NET.Test.Sdk" Version="18.0.1" />
<PackageVersion Include="xunit.v3" Version="3.2.2" />
<PackageVersion Include="xunit.runner.visualstudio" Version="3.1.5" />
<PackageVersion Include="coverlet.collector" Version="8.0.0" />
</ItemGroup>
```
### Optional: Mocking Library
Add a mocking library if the project needs test doubles:
```xml
<PackageVersion Include="NSubstitute" Version="5.3.0" />
```
---
## Step 4: Configure Code Coverage
### Coverlet (Collector Mode)
The `coverlet.collector` package integrates with `dotnet test` via the data collector. No additional configuration is needed for basic coverage.
Generate coverage reports:
```bash
# Collect coverage (Cobertura format by default)
dotnet test --collect:"XPlat Code Coverage"
# Results appear in TestResults/*/coverage.cobertura.xml
```
### Coverage Thresholds
For CI enforcement, use `coverlet.msbuild` for threshold checks:
```xml
<!-- In test csproj or tests/Directory.Build.props -->
<PackageReference Include="coverlet.msbuild" />
```
```bash
# Enforce minimum coverage threshold
dotnet test /p:CollectCoverage=true \
/p:CoverageOutputFormat=cobertura \
/p:Threshold=80 \
/p:ThresholdType=line
```
### Coverage Report Generation
Use `reportgenerator` for human-readable HTML reports:
```bash
# Install globally
dotnet tool install -g dotnet-reportgenerator-globaltool
# Generate HTML report
reportgenerator \
-reports:"tests/**/coverage.cobertura.xml" \
-targetdir:coverage-report \
-reporttypes:Html
```
---
## Step 5: Add EditorConfig Overrides for Tests
In the root `.editorconfig`, add test-specific relaxations:
```ini
[tests/**.cs]
# Allow underscores in test method names (Given_When_Then or Should_Behavior)
dotnet_diagnostic.CA1707.severity = none
# Test parameters are validated by the framework
dotnet_diagnostic.CA1062.severity = none
# ConfigureAwait not relevant in test context
dotnet_diagnostic.CA2007.severity = none
# Tests often have intentionally unused variables for assertions
dotnet_diagnostic.IDE0059.severity = suggestion
```
---
## Step 6: Write a Starter Test
Replace the template-generated `UnitTest1.cs` with a properly structured test:
```csharp
namespace MyApp.Core.UnitTests;
public class SampleServiceTests
{
[Fact]
public void Method_Condition_ExpectedResult()
{
// Arrange
var sut = new SampleService();
// Act
var result = sut.DoWork();
// Assert
Assert.NotNull(result);
}
[Theory]
[InlineData(1, 2, 3)]
[InlineData(0, 0, 0)]
[InlineData(-1, 1, 0)]
public void Add_TwoNumbers_ReturnsSum(int a, int b, int expected)
{
var result = Calculator.Add(a, b);
Assert.Equal(expected, result);
}
}
```
### Test Naming Convention
Use the pattern `Method_Condition_ExpectedResult`:
- `CreateUser_WithValidInput_ReturnsUser`
- `GetById_WhenNotFound_ReturnsNull`
- `Delete_WithoutPermission_ThrowsUnauthorized`
---
## Verify
After adding test infrastructure, verify everything works:
```bash
# Restore (regenerate lock files if using CPM)
dotnet restore
# Build (verifies project references and analyzer config)
dotnet build --no-restore
# Run tests
dotnet test --no-build
# Run with coverage
dotnet test --collect:"XPlat Code Coverage"
```
---
## Adding Integration Test Projects
For integration tests that need `WebApplicationFactory` or database access:
```bash
dotnet new xunit -n MyApp.Api.IntegrationTests -o tests/MyApp.Api.IntegrationTests
dotnet sln add tests/MyApp.Api.IntegrationTests/MyApp.Api.IntegrationTests.csproj
dotnet add tests/MyApp.Api.IntegrationTests/MyApp.Api.IntegrationTests.csproj \
reference src/MyApp.Api/MyApp.Api.csproj
```
Add integration test packages to CPM (match the `Microsoft.AspNetCore.Mvc.Testing` major version to the target framework -- e.g., `8.x` for `net8.0`, `9.x` for `net9.0`, `10.x` for `net10.0`):
```xml
<!-- Version must match the project's target framework major version -->
<PackageVersion Include="Microsoft.AspNetCore.Mvc.Testing" Version="10.0.0" />
<PackageVersion Include="Testcontainers" Version="4.3.0" />
```
Integration test depth (WebApplicationFactory patterns, test containers, database fixtures) -- see [skill:dotnet-testing] `references/integration-testing.md`.
---
## What's Next
This skill covers test project scaffolding. For deeper testing guidance:
- **xUnit v3 features and patterns** -- [skill:dotnet-testing] `references/xunit.md`
- **Integration testing with WebApplicationFactory** -- [skill:dotnet-testing] `references/integration-testing.md`
- **UI testing (Blazor, MAUI, Uno)** -- [skill:dotnet-ui] `references/blazor-testing.md`, `references/maui-testing.md`, `references/uno-testing.md`
- **Snapshot testing** -- [skill:dotnet-testing] `references/snapshot-testing.md`
- **Test quality and coverage enforcement** -- [skill:dotnet-testing] `references/test-quality.md`
- **CI test reporting** -- [skill:dotnet-devops] `references/add-ci.md` for starter, `references/gha-build-test.md` and `references/ado-build-test.md` for advanced
---
## References
- [xUnit Documentation](https://xunit.net/)
- [Coverlet](https://github.com/coverlet-coverage/coverlet)
- [.NET Testing Best Practices](https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-best-practices)
- [Microsoft.NET.Test.Sdk](https://www.nuget.org/packages/Microsoft.NET.Test.Sdk)
- [ReportGenerator](https://github.com/danielpalme/ReportGenerator)
references/aot-wasm.md# AOT and WASM Testing
WebAssembly AOT compilation for Blazor WASM and Uno WASM applications: compilation pipeline, download size vs runtime speed tradeoffs, trimming interplay, lazy loading assemblies, and Brotli pre-compression for download optimization.
**Version assumptions:** .NET 8.0+ baseline. Blazor WASM AOT shipped in .NET 6 and has been refined through .NET 8-10. Uno WASM uses a similar compilation pipeline with Uno-specific tooling.
**Important tradeoff:** Trimming and AOT have **opposite effects** on WASM artifact size. Trimming reduces download size by removing unused code. AOT **increases** artifact size (native WASM code is larger than IL) but **improves** runtime execution speed. Use both together for the best balance.
## Download Size vs Runtime Speed
Understanding the size/speed tradeoff is critical for WASM AOT decisions:
| Compilation Mode | Download Size | Runtime Speed | Startup Time |
|-----------------|---------------|---------------|-------------|
| IL interpreter (no AOT) | Smallest | Slowest | Fastest startup |
| AOT (all assemblies) | **Largest** | Fastest | Slower startup |
| AOT (selective) + trimming | Balanced | Good | Moderate |
| Trimmed only (no AOT) | Small | Moderate (JIT interpretation) | Fast |
**Key insight:** Trimming reduces size by removing unused IL. AOT **increases** total artifact size because compiled native WASM code is larger than the equivalent IL bytecode. However, AOT-compiled code executes significantly faster because it skips IL interpretation at runtime.
### When to Use WASM AOT
- **CPU-intensive workloads:** Image processing, complex calculations, data transformation
- **Predictable performance:** Consistent execution speed without JIT pauses
- **Hot paths:** AOT-compile only performance-critical assemblies (selective AOT)
### When to Skip WASM AOT
- **Bandwidth-constrained users:** AOT increases download size significantly
- **Simple CRUD apps:** IL interpretation is fast enough for UI interactions and API calls
- **Rapid iteration:** AOT compilation adds significant publish time
---
## Blazor WASM AOT
### Enabling AOT
```xml
<!-- Blazor WASM .csproj -->
<PropertyGroup>
<RunAOTCompilation>true</RunAOTCompilation>
</PropertyGroup>
```
```bash
# Publish with AOT (required -- AOT only applies during publish)
dotnet publish -c Release
```
Note: `RunAOTCompilation` is the Blazor WASM property (not `PublishAot` which is for server-side Native AOT). AOT compilation only happens during `dotnet publish`, not during `dotnet run` or `dotnet build`.
### Selective AOT via Lazy Loading
Blazor WASM AOT compiles all non-lazy-loaded assemblies. To control which assemblies are AOT-compiled, mark non-critical assemblies as lazy-loaded -- they will use IL interpretation instead:
```xml
<PropertyGroup>
<RunAOTCompilation>true</RunAOTCompilation>
</PropertyGroup>
<ItemGroup>
<!-- These assemblies are NOT AOT-compiled (loaded on demand via IL interpreter) -->
<BlazorWebAssemblyLazyLoad Include="MyApp.Reporting.wasm" />
<BlazorWebAssemblyLazyLoad Include="MyApp.Admin.wasm" />
<!-- All other assemblies (MyApp.Core, MyApp.Calculations, etc.) ARE AOT-compiled -->
</ItemGroup>
```
### Trimming + AOT Together
For the best balance, use both trimming and AOT:
```xml
<PropertyGroup>
<!-- Trimming reduces unused code (smaller download) -->
<PublishTrimmed>true</PublishTrimmed>
<!-- AOT compiles remaining code to native WASM (faster execution) -->
<RunAOTCompilation>true</RunAOTCompilation>
<!-- Detailed warnings during development -->
<EnableTrimAnalyzer>true</EnableTrimAnalyzer>
</PropertyGroup>
```
The publish pipeline runs: trim unused IL first, then AOT-compile the remaining assemblies to native WASM. This produces an artifact that is larger than trimmed-only but smaller than AOT-without-trimming, with the best runtime performance.
---
## Uno WASM AOT
Uno Platform 5+ with .NET 8+ uses the standard .NET WASM workload, so the AOT configuration is the same as Blazor WASM.
### Enabling AOT (Uno 5+ / .NET 8+)
```xml
<!-- Uno WASM head .csproj -->
<PropertyGroup Condition="'$(TargetFramework)' == 'net8.0-browserwasm'">
<RunAOTCompilation>true</RunAOTCompilation>
</PropertyGroup>
```
Older Uno versions using `Uno.Wasm.Bootstrap` had a separate `WasmShellMonoRuntimeExecutionMode` property with `Interpreter`, `InterpreterAndAOT`, and `FullAOT` modes. On .NET 8+, use `RunAOTCompilation` instead.
### Trimming in Uno WASM
```xml
<PropertyGroup>
<PublishTrimmed>true</PublishTrimmed>
<TrimMode>link</TrimMode>
</PropertyGroup>
```
See [skill:dotnet-ui] `references/uno-platform.md` for Uno Platform architecture patterns.
---
## Lazy Loading Assemblies
Lazy loading defers downloading assemblies until they are needed, reducing initial download size. This is especially effective when combined with AOT (which increases per-assembly size).
### Blazor WASM Lazy Loading
```xml
<!-- Mark assemblies for lazy loading in .csproj -->
<ItemGroup>
<BlazorWebAssemblyLazyLoad Include="MyApp.Reporting.wasm" />
<BlazorWebAssemblyLazyLoad Include="MyApp.Admin.wasm" />
<BlazorWebAssemblyLazyLoad Include="ChartLibrary.wasm" />
</ItemGroup>
```
```csharp
// Load assemblies on demand in a component or router
@inject LazyAssemblyLoader LazyLoader
@code {
private List<Assembly> _lazyLoadedAssemblies = new();
private async Task LoadReportingModule()
{
var assemblies = await LazyLoader.LoadAssembliesAsync(new[]
{
"MyApp.Reporting.wasm"
});
_lazyLoadedAssemblies.AddRange(assemblies);
}
}
```
### Router-Based Lazy Loading
```csharp
<!-- App.razor -->
@inject LazyAssemblyLoader LazyLoader
<Router AppAssembly="typeof(App).Assembly"
AdditionalAssemblies="@_lazyLoadedAssemblies"
OnNavigateAsync="@OnNavigateAsync">
<Navigating>
<div class="loading">Loading module...</div>
</Navigating>
</Router>
@code {
private List<Assembly> _lazyLoadedAssemblies = new();
private async Task OnNavigateAsync(NavigationContext context)
{
if (context.Path.StartsWith("admin"))
{
var assemblies = await LazyLoader.LoadAssembliesAsync(new[]
{
"MyApp.Admin.wasm"
});
_lazyLoadedAssemblies.AddRange(assemblies);
}
else if (context.Path.StartsWith("reports"))
{
var assemblies = await LazyLoader.LoadAssembliesAsync(new[]
{
"MyApp.Reporting.wasm"
});
_lazyLoadedAssemblies.AddRange(assemblies);
}
}
}
```
### Lazy Loading Strategy
| Strategy | Initial Load | Feature Load | Best For |
|----------|-------------|-------------|----------|
| No lazy loading | All at once | Instant | Small apps (<5 MB total) |
| Route-based lazy loading | Core only | On navigation | Multi-module apps |
| Feature-based lazy loading | Core only | On demand | Apps with optional features |
---
## Brotli Pre-Compression
Brotli pre-compression reduces WASM download size by 60-80%. Blazor WASM automatically generates Brotli-compressed files during publish.
### How It Works
During `dotnet publish`, Blazor WASM generates `.br` (Brotli) and `.gz` (gzip) compressed versions of all static files in `_framework/`. The web server serves the pre-compressed file when the browser supports it.
```bash
# After publish, check compressed sizes
ls -la bin/Release/net8.0/publish/wwwroot/_framework/
# You'll see:
# MyApp.wasm (original)
# MyApp.wasm.br (Brotli compressed, ~60-80% smaller)
# MyApp.wasm.gz (gzip compressed, ~50-70% smaller)
```
### Server Configuration
The web server must be configured to serve pre-compressed files. Most Blazor hosting setups handle this automatically.
**ASP.NET Core hosting:**
```csharp
// In the server project hosting Blazor WASM
app.UseBlazorFrameworkFiles();
app.UseStaticFiles();
// Blazor framework files are served with compression headers automatically
```
**Nginx:**
```nginx
location /_framework/ {
# Serve Brotli-compressed files when available
gzip_static on;
brotli_static on;
# Set correct MIME types
types {
application/wasm wasm;
}
# Cache aggressively (files are content-hashed)
add_header Cache-Control "public, max-age=31536000, immutable";
}
```
**Azure Static Web Apps / GitHub Pages:**
Pre-compressed `.br` files are served automatically when the `Accept-Encoding: br` header is present.
### Compression Impact
| Content | Original | Brotli (.br) | Reduction |
|---------|----------|-------------|-----------|
| .NET WASM runtime | ~2.5 MB | ~0.8 MB | ~68% |
| App assemblies (IL) | varies | ~70% smaller | ~70% |
| App assemblies (AOT) | varies | ~65% smaller | ~65% |
| JavaScript glue code | ~100 KB | ~25 KB | ~75% |
### Disabling Compression (Rarely Needed)
```xml
<!-- Disable Brotli pre-compression -->
<PropertyGroup>
<BlazorEnableCompression>false</BlazorEnableCompression>
</PropertyGroup>
```
---
## WASM Size Optimization Checklist
1. **Enable trimming** -- removes unused IL before AOT compilation
2. **Use lazy loading** -- defer non-critical assemblies
3. **Enable Brotli pre-compression** -- 60-80% reduction in transfer size (on by default)
4. **Use selective AOT** -- only AOT-compile performance-critical assemblies
5. **Enable invariant globalization** if culture-specific formatting is not needed:
```xml
<PropertyGroup>
<InvariantGlobalization>true</InvariantGlobalization>
</PropertyGroup>
```
6. **Remove unused framework features:**
```xml
<PropertyGroup>
<!-- Disable features you don't use -->
<EventSourceSupport>false</EventSourceSupport>
<HttpActivityPropagationSupport>false</HttpActivityPropagationSupport>
</PropertyGroup>
```
7. **Verify compression is served** -- check browser DevTools Network tab for `content-encoding: br`
---
## Agent Gotchas
1. **Do not confuse `RunAOTCompilation` with `PublishAot`.** Blazor WASM uses `RunAOTCompilation` for WASM AOT. `PublishAot` is for server-side Native AOT and produces a different kind of binary.
2. **Do not assume AOT reduces WASM download size.** AOT **increases** artifact size because native WASM code is larger than IL bytecode. Use trimming to reduce size and AOT to improve runtime speed.
3. **Do not forget to publish when testing AOT.** WASM AOT only runs during `dotnet publish`, not `dotnet run`. Debug builds always use IL interpretation.
4. **Do not lazy-load assemblies that are needed at startup.** Only lazy-load assemblies for features accessed after initial navigation. Loading a lazy assembly triggers a network request.
5. **Do not skip Brotli compression verification.** Ensure your web server serves `.br` files. Without compression, WASM downloads are 3-5x larger than necessary. Check browser DevTools for `content-encoding: br` header.
6. **Do not AOT-compile all assemblies when download size matters.** Use `BlazorWebAssemblyLazyLoad` to defer non-critical assemblies -- lazy-loaded assemblies use IL interpretation instead of AOT.
---
## References
- [ASP.NET Core Blazor WebAssembly AOT](https://learn.microsoft.com/en-us/aspnet/core/blazor/host-and-deploy/webassembly-aot)
- [Lazy load assemblies in Blazor WASM](https://learn.microsoft.com/en-us/aspnet/core/blazor/webassembly-lazy-load-assemblies)
- [Host and deploy Blazor WASM](https://learn.microsoft.com/en-us/aspnet/core/blazor/host-and-deploy/webassembly)
- [Trim self-contained applications](https://learn.microsoft.com/en-us/dotnet/core/deploying/trimming/trimming-options)
- [Uno Platform WASM](https://platform.uno/docs/articles/features/using-wasm-aot.html)
references/aspire-testing.md# Aspire Testing
Integration testing for .NET Aspire distributed applications using `DistributedApplicationTestingBuilder` from `Aspire.Hosting.Testing`. Covers creating test hosts, getting HTTP clients, waiting for resource health, environment customization, container cleanup, and when to choose Aspire testing vs WebApplicationFactory.
Cross-references: `references/integration-testing.md` for WebApplicationFactory and Testcontainers, `references/xunit.md` for xUnit fixture lifecycle.
---
## Package
```xml
<PackageReference Include="Aspire.Hosting.Testing" Version="9.*" />
```
The test project must reference the AppHost project directly:
```xml
<ProjectReference Include="..\MyApp.AppHost\MyApp.AppHost.csproj" />
```
Version follows the Aspire SDK release cadence. The API surface is stable from 9.0 onward.
---
## Basic Test Lifecycle
```csharp
public class AspireIntegrationTests
{
private static readonly TimeSpan DefaultTimeout = TimeSpan.FromSeconds(30);
[Fact]
public async Task WebFrontend_ReturnsOk()
{
// Create builder from AppHost entry point
var builder = await DistributedApplicationTestingBuilder
.CreateAsync<Projects.MyApp_AppHost>();
builder.Services.ConfigureHttpClientDefaults(http =>
http.AddStandardResilienceHandler());
// Build and start
await using var app = await builder.BuildAsync();
await app.StartAsync();
// Wait for resource health
await app.ResourceNotifications
.WaitForResourceHealthyAsync("webfrontend")
.WaitAsync(DefaultTimeout);
// Act + Assert
using var httpClient = app.CreateHttpClient("webfrontend");
using var response = await httpClient.GetAsync("/");
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
}
}
```
`app.CreateHttpClient("resource-name")` returns an `HttpClient` with the correct base address via Aspire service discovery. The resource name must exactly match the string in `AddProject`/`AddContainer` in the AppHost.
---
## Waiting for Resources
Always wait for health before making assertions. Always apply `.WaitAsync(timeout)` to prevent indefinite hangs.
```csharp
// Single resource
await app.ResourceNotifications
.WaitForResourceHealthyAsync("apiservice")
.WaitAsync(TimeSpan.FromSeconds(60));
// Multiple resources in parallel
await Task.WhenAll(
app.ResourceNotifications.WaitForResourceHealthyAsync("apiservice")
.WaitAsync(TimeSpan.FromSeconds(60)),
app.ResourceNotifications.WaitForResourceHealthyAsync("postgres")
.WaitAsync(TimeSpan.FromSeconds(60)));
```
---
## Configuring the Test Host
```csharp
var builder = await DistributedApplicationTestingBuilder
.CreateAsync<Projects.MyApp_AppHost>();
// Logging -- control verbosity
builder.Services.AddLogging(logging =>
{
logging.SetMinimumLevel(LogLevel.Debug);
logging.AddFilter("Aspire.", LogLevel.Debug);
});
// HTTP resilience -- retries transient 503s during startup
builder.Services.ConfigureHttpClientDefaults(http =>
http.AddStandardResilienceHandler());
```
For xUnit, use `MartinCostello.Logging.XUnit` to route logs to `ITestOutputHelper`.
---
## xUnit Integration with IAsyncLifetime
Share an Aspire app across multiple tests in a class:
```csharp
public class OrderServiceTests : IAsyncLifetime
{
private DistributedApplication _app = null!;
private HttpClient _apiClient = null!;
public async ValueTask InitializeAsync()
{
var builder = await DistributedApplicationTestingBuilder
.CreateAsync<Projects.MyApp_AppHost>();
builder.Services.ConfigureHttpClientDefaults(http =>
http.AddStandardResilienceHandler());
_app = await builder.BuildAsync();
await _app.StartAsync();
await _app.ResourceNotifications
.WaitForResourceHealthyAsync("apiservice")
.WaitAsync(TimeSpan.FromSeconds(60));
_apiClient = _app.CreateHttpClient("apiservice");
}
[Fact]
public async Task GetOrders_ReturnsOk()
{
var response = await _apiClient.GetAsync("/api/orders");
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
}
[Fact]
public async Task CreateOrder_Returns201()
{
var response = await _apiClient.PostAsJsonAsync("/api/orders",
new { CustomerId = "cust-1", Items = new[] { new { Sku = "X", Qty = 1 } } });
Assert.Equal(HttpStatusCode.Created, response.StatusCode);
}
public async ValueTask DisposeAsync()
{
_apiClient.Dispose();
await _app.DisposeAsync();
}
}
```
---
## Testing Environment Variables
Validate orchestration wiring without starting resources:
```csharp
[Fact]
public async Task WebFrontend_HasApiServiceReference()
{
var builder = await DistributedApplicationTestingBuilder
.CreateAsync<Projects.MyApp_AppHost>();
var frontend = builder.CreateResourceBuilder<ProjectResource>("webfrontend");
var config = await ExecutionConfigurationBuilder
.Create(frontend.Resource)
.WithEnvironmentVariablesConfig()
.BuildAsync(new(DistributedApplicationOperation.Publish),
NullLogger.Instance, CancellationToken.None);
var envVars = config.EnvironmentVariables.ToDictionary();
Assert.Contains(envVars, kvp =>
kvp.Key == "APISERVICE_HTTPS" &&
kvp.Value == "{apiservice.bindings.https.url}");
}
```
---
## Testing with Database and Cache Resources
Aspire manages container lifecycle automatically. Tests wait for readiness and interact through service endpoints.
```csharp
[Fact]
public async Task ApiWithPostgres_RoundTrip()
{
var builder = await DistributedApplicationTestingBuilder
.CreateAsync<Projects.MyApp_AppHost>();
builder.Services.ConfigureHttpClientDefaults(http =>
http.AddStandardResilienceHandler());
await using var app = await builder.BuildAsync();
await app.StartAsync();
await Task.WhenAll(
app.ResourceNotifications.WaitForResourceHealthyAsync("apiservice")
.WaitAsync(TimeSpan.FromSeconds(60)),
app.ResourceNotifications.WaitForResourceHealthyAsync("postgres")
.WaitAsync(TimeSpan.FromSeconds(60)));
using var client = app.CreateHttpClient("apiservice");
var createResp = await client.PostAsJsonAsync("/api/orders",
new { CustomerId = "test-1", Total = 42.0 });
Assert.Equal(HttpStatusCode.Created, createResp.StatusCode);
var getResp = await client.GetAsync(createResp.Headers.Location!.ToString());
getResp.EnsureSuccessStatusCode();
}
```
---
## When to Use Aspire Testing vs WebApplicationFactory
| Scenario | Recommendation |
|----------|---------------|
| Single API, no distributed dependencies | **WebApplicationFactory** -- lighter, no Docker |
| Multiple services with inter-service communication | **Aspire testing** -- validates full topology |
| Testing service discovery and resource wiring | **Aspire testing** |
| Unit-testing a single endpoint with mocks | **WebApplicationFactory** |
| CI without Docker available | **WebApplicationFactory** |
---
## Container Cleanup and Test Isolation
`DistributedApplication` implements `IAsyncDisposable`. Disposing stops and removes all containers. Always use `await using` or explicit `DisposeAsync`. Each test run gets fresh containers with no shared state. To share across test classes, use `ICollectionFixture<T>` and manage data isolation at the application level.
---
## Agent Gotchas
1. **Do not forget that Docker is required.** Aspire testing starts real containers. Tests fail immediately without Docker. Guard CI with availability checks.
2. **Do not omit `.WaitAsync(timeout)` on resource waits.** Without a timeout, `WaitForResourceHealthyAsync` hangs forever if a container fails to start.
3. **Do not hardcode resource names.** Names must exactly match the AppHost's `AddProject`/`AddContainer` string. A typo causes `CreateHttpClient` to throw with an unhelpful error.
4. **Do not skip `ConfigureHttpClientDefaults` with resilience.** Without the standard resilience handler, first requests often hit 503 or connection refused during startup.
5. **Do not assume fast container startup.** Database containers take 10-30 seconds; Redis is 2-5 seconds. Set timeouts accordingly to avoid flaky CI.
6. **Do not forget to dispose `DistributedApplication`.** Undisposed apps leave orphaned containers. Always use `await using`.
7. **Do not run Aspire tests in parallel without isolation.** Port conflicts and data races occur. Use `[Collection]` to serialize test classes sharing the same AppHost.
---
## References
- [Write your first .NET Aspire test](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/testing)
- [Aspire.Hosting.Testing on NuGet](https://www.nuget.org/packages/Aspire.Hosting.Testing)
- [DistributedApplicationTestingBuilder API](https://learn.microsoft.com/en-us/dotnet/api/aspire.hosting.testing.distributedapplicationtestingbuilder)
- [ResourceNotificationService API](https://learn.microsoft.com/en-us/dotnet/api/aspire.hosting.applicationmodel.resourcenotificationservice)
references/benchmarkdotnet.md# BenchmarkDotNet
Microbenchmarking guidance for .NET using BenchmarkDotNet v0.14+. Covers benchmark class setup, memory and disassembly diagnosers, exporters for CI artifact collection, baseline comparisons, and common pitfalls that invalidate measurements.
**Version assumptions:** BenchmarkDotNet v0.14+ on .NET 8.0+ baseline. Examples use current stable APIs.
## Package Setup
```xml
<!-- Benchmarks.csproj -->
<Project Sdk="Microsoft.NET.Sdk">
<PropertyGroup>
<OutputType>Exe</OutputType>
<TargetFramework>net8.0</TargetFramework>
</PropertyGroup>
<ItemGroup>
<PackageReference Include="BenchmarkDotNet" Version="0.14.*" />
</ItemGroup>
</Project>
```
Keep benchmark projects separate from production code. Use a `benchmarks/` directory at the solution root.
---
## Benchmark Class Setup
### Basic Benchmark with [Benchmark] Attribute
```csharp
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Running;
[MemoryDiagnoser]
public class StringConcatBenchmarks
{
private readonly string[] _items = Enumerable.Range(0, 100)
.Select(i => i.ToString())
.ToArray();
[Benchmark(Baseline = true)]
public string StringConcat()
{
var result = string.Empty;
foreach (var item in _items)
result += item;
return result;
}
[Benchmark]
public string StringBuilder()
{
var sb = new System.Text.StringBuilder();
foreach (var item in _items)
sb.Append(item);
return sb.ToString();
}
[Benchmark]
public string StringJoin() => string.Join(string.Empty, _items);
}
```
### Running Benchmarks
```csharp
// Program.cs
using BenchmarkDotNet.Running;
BenchmarkRunner.Run<StringConcatBenchmarks>();
```
Run in Release mode (mandatory for valid results):
```bash
dotnet run -c Release
```
### Parameterized Benchmarks
```csharp
[MemoryDiagnoser]
public class CollectionBenchmarks
{
[Params(10, 100, 1000)]
public int Size { get; set; }
private int[] _data = null!;
[GlobalSetup]
public void Setup()
{
_data = Enumerable.Range(0, Size).ToArray();
}
[Benchmark(Baseline = true)]
public int ForLoop()
{
var sum = 0;
for (var i = 0; i < _data.Length; i++)
sum += _data[i];
return sum;
}
[Benchmark]
public int LinqSum() => _data.Sum();
}
```
---
## Memory Diagnosers
### MemoryDiagnoser
Tracks GC allocations and collection counts per benchmark invocation. Apply at class level to all benchmarks:
```csharp
[MemoryDiagnoser]
public class AllocationBenchmarks
{
[Benchmark]
public byte[] AllocateArray() => new byte[1024];
[Benchmark]
public int UseStackalloc()
{
Span<byte> buffer = stackalloc byte[1024];
buffer[0] = 42;
return buffer[0];
}
}
```
Output columns:
| Column | Meaning |
|--------|---------|
| `Allocated` | Bytes allocated per operation |
| `Gen0` | Gen 0 GC collections per 1000 operations |
| `Gen1` | Gen 1 GC collections per 1000 operations |
| `Gen2` | Gen 2 GC collections per 1000 operations |
Zero in `Allocated` column confirms zero-allocation code paths.
### DisassemblyDiagnoser
Inspects JIT-compiled assembly to verify optimizations (devirtualization, inlining):
```csharp
[DisassemblyDiagnoser(maxDepth: 2)]
[MemoryDiagnoser]
public class DevirtualizationBenchmarks
{
// sealed enables JIT devirtualization -- verify in disassembly output
// See [skill:dotnet-csharp] `references/coding-standards.md` for sealed class conventions
[Benchmark]
public int SealedCall()
{
var obj = new SealedService();
return obj.Calculate(42);
}
[Benchmark]
public int VirtualCall()
{
IService obj = new SealedService();
return obj.Calculate(42);
}
}
public interface IService { int Calculate(int x); }
public sealed class SealedService : IService
{
public int Calculate(int x) => x * 2;
}
```
Use `DisassemblyDiagnoser` to verify that `sealed` classes receive devirtualization from the JIT, confirming the performance rationale documented in [skill:dotnet-csharp] `references/coding-standards.md`.
---
## Exporters for CI Integration
### Configuring Exporters
```csharp
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Exporters;
using BenchmarkDotNet.Exporters.Json;
[MemoryDiagnoser]
[JsonExporterAttribute.Full]
[HtmlExporter]
[MarkdownExporter]
public class CiBenchmarks
{
[Benchmark]
public void MyOperation()
{
// benchmark code
}
}
```
### Exporter Output
| Exporter | File | Use Case |
|----------|------|----------|
| `JsonExporterAttribute.Full` | `BenchmarkDotNet.Artifacts/results/*-report-full.json` | CI regression comparison (machine-readable) |
| `HtmlExporter` | `BenchmarkDotNet.Artifacts/results/*-report.html` | Human-readable PR review artifact |
| `MarkdownExporter` | `BenchmarkDotNet.Artifacts/results/*-report-github.md` | Paste into PR comments |
### Custom Config for CI
```csharp
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Exporters.Json;
using BenchmarkDotNet.Jobs;
var config = ManualConfig.Create(DefaultConfig.Instance)
.AddJob(Job.ShortRun) // fewer iterations for CI speed
.AddExporter(JsonExporter.Full)
.WithArtifactsPath("./benchmark-results");
BenchmarkRunner.Run<CiBenchmarks>(config);
```
### GitHub Actions Artifact Upload
```yaml
- name: Run benchmarks
run: dotnet run -c Release --project benchmarks/MyBenchmarks.csproj
- name: Upload benchmark results
uses: actions/upload-artifact@v4
with:
name: benchmark-results
path: benchmarks/BenchmarkDotNet.Artifacts/results/
retention-days: 30
```
---
## Baseline Comparison
### Setting a Baseline
Mark one benchmark as the baseline for ratio comparison:
```csharp
[MemoryDiagnoser]
public class SerializationBenchmarks
{
// Serialization format choice -- see [skill:dotnet-csharp] `references/serialization.md` for API details
private readonly JsonSerializerOptions _options = new()
{
PropertyNamingPolicy = JsonNamingPolicy.CamelCase,
};
private readonly WeatherForecast _data = new()
{
Date = DateOnly.FromDateTime(DateTime.Now),
TemperatureC = 25,
Summary = "Warm"
};
[Benchmark(Baseline = true)]
public string SystemTextJson()
=> System.Text.Json.JsonSerializer.Serialize(_data, _options);
[Benchmark]
public byte[] Utf8Serialization()
=> System.Text.Json.JsonSerializer.SerializeToUtf8Bytes(_data, _options);
}
public record WeatherForecast
{
public DateOnly Date { get; init; }
public int TemperatureC { get; init; }
public string? Summary { get; init; }
}
```
The `Ratio` column in output shows performance relative to the baseline (1.00). Values below 1.00 indicate faster than baseline; above 1.00 indicate slower.
### Benchmark Categories
Group benchmarks with `[BenchmarkCategory]` and filter at runtime:
```csharp
[MemoryDiagnoser]
[GroupBenchmarksBy(BenchmarkLogicalGroupRule.ByCategory)]
public class CategorizedBenchmarks
{
[Benchmark, BenchmarkCategory("Serialization")]
public string JsonSerialize() => "...";
[Benchmark, BenchmarkCategory("Allocation")]
public byte[] ArrayAlloc() => new byte[1024];
}
```
Run a specific category:
```bash
dotnet run -c Release -- --filter *Serialization*
```
---
## BenchmarkRunner.Run Patterns
### Running Specific Benchmarks
```csharp
// Run a single benchmark class
BenchmarkRunner.Run<StringConcatBenchmarks>();
// Run all benchmarks in assembly
BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args);
```
### Command-Line Filtering
```bash
# Run benchmarks matching a pattern
dotnet run -c Release -- --filter *StringBuilder*
# List all available benchmarks without running
dotnet run -c Release -- --list flat
# Dry run (validates setup without full benchmark)
dotnet run -c Release -- --filter *StringBuilder* --job Dry
```
### AOT Benchmark Considerations
When benchmarking Native AOT scenarios, the JIT diagnosers are not available (there is no JIT). Use wall-clock time and memory comparisons instead. See [skill:dotnet-tooling] `references/native-aot.md` for AOT compilation setup:
```csharp
[MemoryDiagnoser]
// Do NOT use DisassemblyDiagnoser with AOT -- no JIT to disassemble
public class AotBenchmarks
{
[Benchmark]
public string SourceGenSerialize()
=> System.Text.Json.JsonSerializer.Serialize(
new { Value = 42 },
AppJsonContext.Default.Options);
}
```
---
## Common Pitfalls
### Dead Code Elimination
The JIT may eliminate benchmark code whose result is unused. Always **return** or **consume** the result:
```csharp
// BAD: JIT may eliminate the entire loop
[Benchmark]
public void DeadCode()
{
var sum = 0;
for (var i = 0; i < 1000; i++)
sum += i;
// sum is never used -- JIT removes the loop
}
// GOOD: return the value to prevent elimination
[Benchmark]
public int LiveCode()
{
var sum = 0;
for (var i = 0; i < 1000; i++)
sum += i;
return sum;
}
```
### Measurement Bias
| Pitfall | Cause | Fix |
|---------|-------|-----|
| Running in Debug mode | No JIT optimizations applied | Always use `-c Release` |
| Shared mutable state | Benchmarks interfere with each other | Use `[IterationSetup]` or immutable data |
| Cold-start measurement | First run includes JIT compilation | BenchmarkDotNet handles warmup automatically -- do not add manual warmup |
| Allocations in setup | Setup allocations inflate `Allocated` column | Use `[GlobalSetup]` (runs once) vs `[IterationSetup]` (runs per iteration) |
| Environment noise | Background processes skew results | BenchmarkDotNet detects and warns about environment issues; use `Job.MediumRun` for noisy environments |
### Setup vs Iteration Lifecycle
```csharp
[MemoryDiagnoser]
public class LifecycleBenchmarks
{
private byte[] _data = null!;
[GlobalSetup] // Runs once before all benchmark iterations
public void GlobalSetup() => _data = new byte[1024];
[IterationSetup] // Runs before each benchmark iteration
public void IterationSetup() => Array.Fill(_data, (byte)0);
[Benchmark]
public int Process()
{
// uses _data
return _data.Length;
}
[GlobalCleanup] // Runs once after all iterations
public void GlobalCleanup() { /* dispose resources */ }
}
```
Prefer `[GlobalSetup]` over `[IterationSetup]` unless the benchmark mutates shared state. `[IterationSetup]` adds overhead that BenchmarkDotNet excludes from timing, but it still affects GC pressure measurement.
---
## Agent Gotchas
1. **Always run benchmarks in Release mode** -- `dotnet run -c Release`. Debug mode disables JIT optimizations and produces meaningless results.
2. **Never benchmark in a test project** -- xUnit/NUnit test runners interfere with BenchmarkDotNet's measurement harness. Use a standalone console project.
3. **Return values from benchmark methods** to prevent dead code elimination. The JIT will remove computation whose result is discarded.
4. **Do not add manual Thread.Sleep or Task.Delay in benchmarks** -- BenchmarkDotNet manages warmup and iteration timing automatically.
5. **Use `[GlobalSetup]` not constructor** for initialization -- BenchmarkDotNet creates benchmark instances multiple times during a run; constructor code runs repeatedly.
6. **Prefer `[Params]` over manual loops** for parameterized benchmarks. BenchmarkDotNet runs each parameter combination independently with proper statistical analysis.
7. **Export JSON for CI** -- use `[JsonExporterAttribute.Full]` to produce machine-readable artifacts for regression detection, not just Markdown.
references/ci-benchmarking.md# CI Benchmarking
Continuous benchmarking guidance for detecting performance regressions in CI pipelines. Covers baseline file management with BenchmarkDotNet JSON exporters, GitHub Actions workflows for artifact-based baseline comparison, regression detection patterns with configurable thresholds, and alerting strategies for performance degradation.
**Version assumptions:** BenchmarkDotNet v0.14+ for JSON export, GitHub Actions runner environment. Examples use `actions/upload-artifact@v4` and `actions/download-artifact@v4`.
## Baseline File Management
### BenchmarkDotNet JSON Export
BenchmarkDotNet's JSON exporter produces machine-readable results for automated comparison. Configure the exporter in benchmark classes:
```csharp
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Exporters.Json;
[JsonExporterAttribute.Full]
[MemoryDiagnoser]
public class CriticalPathBenchmarks
{
[Benchmark(Baseline = true)]
public void ProcessOrder() { /* ... */ }
[Benchmark]
public void ProcessOrderOptimized() { /* ... */ }
}
```
Or configure via custom config for all benchmark classes:
```csharp
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Exporters.Json;
using BenchmarkDotNet.Jobs;
using BenchmarkDotNet.Running;
var config = ManualConfig.Create(DefaultConfig.Instance)
.AddJob(Job.ShortRun) // fewer iterations for CI speed
.AddExporter(JsonExporter.Full)
.WithArtifactsPath("./benchmark-results");
BenchmarkSwitcher.FromAssembly(typeof(Program).Assembly).Run(args, config);
```
### JSON Export Structure
The exported JSON file (`*-report-full.json`) contains structured benchmark results:
```json
{
"Title": "CriticalPathBenchmarks",
"Benchmarks": [
{
"FullName": "MyApp.Benchmarks.CriticalPathBenchmarks.ProcessOrder",
"Statistics": {
"Mean": 1234.5678,
"Median": 1230.1234,
"StandardDeviation": 15.234,
"StandardError": 4.812
},
"Memory": {
"BytesAllocatedPerOperation": 1024,
"Gen0Collections": 0.0012,
"Gen1Collections": 0,
"Gen2Collections": 0
}
}
]
}
```
Key fields for regression comparison:
| Field | Purpose |
|-------|---------|
| `Statistics.Mean` | Average execution time (nanoseconds) |
| `Statistics.Median` | Middle execution time (more robust to outliers) |
| `Statistics.StandardDeviation` | Measurement variability |
| `Memory.BytesAllocatedPerOperation` | GC allocation per operation |
### Baseline Storage Strategies
| Strategy | Pros | Cons | Best For |
|----------|------|------|----------|
| Git-committed baseline file | Versioned, auditable, no external deps | Repo size grows; must update deliberately | Small benchmark suites, stable hardware |
| GitHub Actions artifacts | No repo bloat; automatic retention | 90-day default retention; cross-workflow access requires tokens | Large benchmark suites, shared runners |
| External storage (S3/Azure Blob) | Unlimited history; cross-repo sharing | Extra infrastructure; credential management | Multi-repo benchmark comparison |
This skill focuses on the **GitHub Actions artifact** strategy as the default. For composable workflow patterns and reusable actions, see [skill:dotnet-devops] `references/gha-patterns.md`.
---
## GitHub Actions Benchmark Workflow
### Basic Benchmark Workflow
```yaml
name: Benchmarks
on:
pull_request:
paths:
- 'src/**'
- 'benchmarks/**'
workflow_dispatch:
permissions:
contents: read
actions: read # required for artifact download
jobs:
benchmark:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup .NET
uses: actions/setup-dotnet@v4
with:
dotnet-version: '8.0.x'
- name: Run benchmarks
run: dotnet run -c Release --project benchmarks/MyBenchmarks.csproj -- --exporters json
- name: Upload benchmark results
uses: actions/upload-artifact@v4
with:
name: benchmark-results-${{ github.sha }}
path: benchmarks/BenchmarkDotNet.Artifacts/results/
retention-days: 90
```
### Baseline Comparison Workflow
This workflow downloads the baseline from a previous run and compares against current results:
```yaml
name: Benchmark Regression Check
on:
pull_request:
paths:
- 'src/**'
- 'benchmarks/**'
permissions:
contents: read
actions: read
env:
BENCHMARK_PROJECT: benchmarks/MyBenchmarks.csproj
RESULTS_DIR: benchmarks/BenchmarkDotNet.Artifacts/results
jobs:
benchmark:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup .NET
uses: actions/setup-dotnet@v4
with:
dotnet-version: '8.0.x'
- name: Download baseline results
uses: actions/download-artifact@v4
with:
name: benchmark-baseline
path: ./baseline-results
continue-on-error: true
id: download-baseline
- name: Run benchmarks
run: dotnet run -c Release --project ${{ env.BENCHMARK_PROJECT }} -- --exporters json
- name: Compare with baseline
if: steps.download-baseline.outcome == 'success'
shell: bash
run: |
set -euo pipefail
python3 scripts/compare-benchmarks.py \
--baseline ./baseline-results \
--current "${{ env.RESULTS_DIR }}" \
--threshold 10 \
--output benchmark-comparison.md
- name: Upload current results as new baseline
if: github.ref == 'refs/heads/main'
uses: actions/upload-artifact@v4
with:
name: benchmark-baseline
path: ${{ env.RESULTS_DIR }}/
retention-days: 90
overwrite: true
- name: Upload comparison report
if: steps.download-baseline.outcome == 'success'
uses: actions/upload-artifact@v4
with:
name: benchmark-comparison-${{ github.sha }}
path: benchmark-comparison.md
retention-days: 30
```
**Key design decisions:**
- `continue-on-error: true` on baseline download handles first-run (no baseline exists yet)
- Baseline is only updated from `main` branch merges to prevent PR branches from polluting the baseline
- `overwrite: true` replaces the previous baseline artifact
For converting these inline workflows into reusable `workflow_call` patterns, see [skill:dotnet-devops] `references/gha-patterns.md`.
---
## Regression Detection Patterns
### Threshold-Based Comparison
Compare current benchmark results against baseline using percentage thresholds. A regression is flagged when the current mean exceeds the baseline mean by more than the configured threshold:
```python
#!/usr/bin/env python3
"""compare-benchmarks.py -- Detect benchmark regressions from BenchmarkDotNet JSON exports."""
import json
import sys
from pathlib import Path
def load_benchmarks(results_dir: str) -> dict:
"""Load benchmark results from BenchmarkDotNet JSON export files."""
benchmarks = {}
for json_file in Path(results_dir).glob("*-report-full.json"):
with open(json_file) as f:
data = json.load(f)
for bm in data.get("Benchmarks", []):
name = bm["FullName"]
benchmarks[name] = {
"mean": bm["Statistics"]["Mean"],
"median": bm["Statistics"]["Median"],
"stddev": bm["Statistics"]["StandardDeviation"],
"allocated": bm.get("Memory", {}).get("BytesAllocatedPerOperation", 0),
}
return benchmarks
def compare(baseline_dir: str, current_dir: str, threshold_pct: float) -> list:
"""Compare current results against baseline. Returns list of regressions."""
baseline = load_benchmarks(baseline_dir)
current = load_benchmarks(current_dir)
regressions = []
for name, curr in current.items():
if name not in baseline:
continue # new benchmark, no comparison possible
base = baseline[name]
if base["mean"] == 0:
continue # avoid division by zero
time_change_pct = ((curr["mean"] - base["mean"]) / base["mean"]) * 100
alloc_change = curr["allocated"] - base["allocated"]
if time_change_pct > threshold_pct:
regressions.append({
"name": name,
"baseline_mean": base["mean"],
"current_mean": curr["mean"],
"change_pct": time_change_pct,
"alloc_change": alloc_change,
})
return regressions
if __name__ == "__main__":
import argparse
parser = argparse.ArgumentParser(description="Compare BenchmarkDotNet results")
parser.add_argument("--baseline", required=True, help="Path to baseline results directory")
parser.add_argument("--current", required=True, help="Path to current results directory")
parser.add_argument("--threshold", type=float, default=10.0,
help="Regression threshold percentage (default: 10)")
parser.add_argument("--output", default="comparison.md", help="Output markdown file")
args = parser.parse_args()
regressions = compare(args.baseline, args.current, args.threshold)
with open(args.output, "w") as f:
if regressions:
f.write("## Benchmark Regressions Detected\n\n")
f.write("| Benchmark | Baseline (ns) | Current (ns) | Change | Alloc Delta |\n")
f.write("|-----------|--------------|-------------|--------|-------------|\n")
for r in regressions:
f.write(f"| `{r['name']}` | {r['baseline_mean']:.2f} | "
f"{r['current_mean']:.2f} | +{r['change_pct']:.1f}% | "
f"{r['alloc_change']:+d} B |\n")
f.write(f"\nThreshold: {args.threshold}%\n")
else:
f.write("## Benchmark Results\n\nNo regressions detected ")
f.write(f"(threshold: {args.threshold}%).\n")
if regressions:
print(f"REGRESSION: {len(regressions)} benchmark(s) exceeded "
f"{args.threshold}% threshold", file=sys.stderr)
sys.exit(1)
```
### Choosing Thresholds
| Environment | Suggested Threshold | Rationale |
|-------------|-------------------|-----------|
| Dedicated benchmark hardware | 5% | Low noise floor; small regressions are signal |
| GitHub Actions shared runners | 10-15% | Shared runners introduce 5-10% variance from noisy neighbors |
| Self-hosted runners | 5-10% | More stable than shared, but still monitor variance |
**Calibrate thresholds empirically:** Run the same benchmark suite 5-10 times on your CI environment without code changes. The maximum observed variance sets your noise floor. Set the threshold above this noise floor (typically 2x the observed variance).
### Allocation Regression Detection
Memory allocation regressions are more reliable signals than timing regressions because allocations are deterministic (not affected by noisy neighbors):
```python
# Add to the compare function:
if alloc_change > 0:
regressions.append({
"name": name,
"type": "allocation",
"baseline_alloc": base["allocated"],
"current_alloc": curr["allocated"],
"alloc_change": alloc_change,
})
```
Use allocation changes as a **hard gate** (zero tolerance for new allocations in zero-alloc paths) and timing changes as a **soft gate** (warning with threshold).
---
## Alerting Strategies
### PR Comment with Regression Summary
Post benchmark comparison results as a PR comment for reviewer visibility:
```yaml
- name: Comment PR with results
if: steps.download-baseline.outcome == 'success' && github.event_name == 'pull_request'
uses: actions/github-script@v7
with:
script: |
const fs = require('fs');
const body = fs.readFileSync('benchmark-comparison.md', 'utf8');
await github.rest.issues.createComment({
owner: context.repo.owner,
repo: context.repo.repo,
issue_number: context.issue.number,
body: body
});
```
### Fail the Build on Regression
Exit with non-zero status from the comparison script to fail the GitHub Actions job. This prevents merging PRs that introduce performance regressions:
```yaml
- name: Check for regressions
if: steps.download-baseline.outcome == 'success'
shell: bash
run: |
set -euo pipefail
python3 scripts/compare-benchmarks.py \
--baseline ./baseline-results \
--current "${{ env.RESULTS_DIR }}" \
--threshold 10
# Script exits non-zero if regressions found -- fails the job
```
For required status checks and branch protection integration with benchmark gates, see [skill:dotnet-devops] `references/gha-patterns.md`.
### Trend Tracking
For long-term trend analysis beyond single-PR comparison, upload results to a persistent store and track metrics over time:
| Approach | Tool | Complexity |
|----------|------|------------|
| GitHub Actions artifacts | Built-in, 90-day retention | Low -- artifact download/upload only |
| GitHub Pages with benchmark-action | `benchmark-action/github-action-benchmark@v1` | Medium -- auto-generates trend charts |
| External time-series DB | InfluxDB, Prometheus + Grafana | High -- full observability stack |
The simplest approach for most projects is the artifact-based baseline comparison shown in this skill. Graduate to trend tracking when you need historical regression analysis across many releases.
---
## CI-Specific BenchmarkDotNet Configuration
### ShortRun for CI Speed
Full benchmark runs take 10-30+ minutes. Use `Job.ShortRun` in CI to reduce iteration counts while retaining regression detection capability:
```csharp
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Jobs;
public class CiConfig : ManualConfig
{
public CiConfig()
{
AddJob(Job.ShortRun
.WithWarmupCount(3)
.WithIterationCount(5)
.WithInvocationCount(1));
AddExporter(BenchmarkDotNet.Exporters.Json.JsonExporter.Full);
}
}
```
Apply conditionally based on environment:
```csharp
var config = Environment.GetEnvironmentVariable("CI") is not null
? new CiConfig()
: DefaultConfig.Instance;
BenchmarkRunner.Run<CriticalPathBenchmarks>(config);
```
### Filtering Benchmarks for CI
Run only critical-path benchmarks in CI to reduce pipeline duration:
```bash
# Run only benchmarks in the "Critical" category
dotnet run -c Release --project benchmarks/MyBenchmarks.csproj -- \
--filter *Critical* --exporters json
```
```csharp
[BenchmarkCategory("Critical")]
[MemoryDiagnoser]
[JsonExporterAttribute.Full]
public class CriticalPathBenchmarks
{
[Benchmark]
public void ProcessOrder() { /* ... */ }
}
[BenchmarkCategory("Extended")]
[MemoryDiagnoser]
public class ExtendedBenchmarks
{
[Benchmark]
public void RareCodePath() { /* ... */ }
}
```
Run `Critical` benchmarks on every PR; run `Extended` benchmarks on a nightly schedule.
### Nightly Benchmark Schedule
```yaml
name: Nightly Benchmarks (Full Suite)
on:
schedule:
- cron: '0 3 * * *' # 3 AM UTC daily
workflow_dispatch:
jobs:
benchmark-full:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup .NET
uses: actions/setup-dotnet@v4
with:
dotnet-version: '8.0.x'
- name: Run full benchmark suite
run: dotnet run -c Release --project benchmarks/MyBenchmarks.csproj -- --exporters json
# No --filter: runs all benchmarks including Extended category
- name: Upload full results
uses: actions/upload-artifact@v4
with:
name: benchmark-full-${{ github.run_number }}
path: benchmarks/BenchmarkDotNet.Artifacts/results/
retention-days: 90
```
For scheduled workflow patterns and matrix builds across TFMs, see [skill:dotnet-devops] `references/gha-patterns.md`.
---
## Agent Gotchas
1. **Use `Job.ShortRun` in CI, not `Job.Default`** -- default benchmark jobs run many iterations for statistical precision, taking 10-30+ minutes per benchmark class. CI pipelines need faster feedback with `ShortRun` (3 warmup, 5 iteration).
2. **Set threshold above measured noise floor** -- shared CI runners introduce 5-10% timing variance from noisy neighbors. A 5% threshold on shared runners produces false positives. Calibrate by running the same code multiple times and measuring variance.
3. **Use allocation changes as hard gates** -- allocation counts are deterministic and unaffected by runner noise. A zero-to-nonzero allocation change is always a real regression, unlike timing variations.
4. **Only update baselines from main branch** -- if PR branches can update the baseline, a regression in one PR becomes the new baseline, masking it from subsequent comparisons.
5. **Always set `set -euo pipefail` in bash steps** -- without `pipefail`, a regression detection script that exits non-zero in a pipeline (e.g., `script | tee`) does not fail the GitHub Actions step.
6. **Handle missing baselines gracefully** -- the first CI run has no baseline to compare against. Use `continue-on-error: true` on the baseline download step and skip comparison when no baseline exists.
7. **Export JSON, not just Markdown** -- Markdown reports are human-readable but not machine-parseable for automated regression detection. Always include `[JsonExporterAttribute.Full]` or `JsonExporter.Full` in the config.
references/integration-testing.md# Integration Testing
Integration testing patterns for .NET applications using WebApplicationFactory, Testcontainers, and .NET Aspire testing. Covers in-process API testing, disposable infrastructure via containers, database fixture management, and test isolation strategies.
**Version assumptions:** .NET 8.0+ baseline, Testcontainers 3.x+, .NET Aspire 9.0+. Package versions for `Microsoft.AspNetCore.Mvc.Testing` must match the project's target framework major version (e.g., 8.x for net8.0, 9.x for net9.0, 10.x for net10.0). Examples below use Testcontainers 4.x APIs; the patterns apply equally to 3.x with minor namespace differences.
## WebApplicationFactory
`WebApplicationFactory<TEntryPoint>` creates an in-process test server for ASP.NET Core applications. Tests send HTTP requests without network overhead, exercising the full middleware pipeline, routing, model binding, and serialization.
### Package
```xml
<!-- Version must match target framework: 8.x for net8.0, 9.x for net9.0, etc. -->
<PackageReference Include="Microsoft.AspNetCore.Mvc.Testing" />
```
### Basic Usage
```csharp
public class OrdersApiTests : IClassFixture<WebApplicationFactory<Program>>
{
private readonly HttpClient _client;
public OrdersApiTests(WebApplicationFactory<Program> factory)
{
_client = factory.CreateClient();
}
[Fact]
public async Task GetOrders_ReturnsOkWithJsonArray()
{
var response = await _client.GetAsync("/api/orders");
response.EnsureSuccessStatusCode();
var orders = await response.Content
.ReadFromJsonAsync<List<OrderDto>>();
Assert.NotNull(orders);
}
[Fact]
public async Task CreateOrder_ValidPayload_Returns201()
{
var request = new CreateOrderRequest
{
CustomerId = "cust-123",
Items = [new("SKU-001", Quantity: 2)]
};
var response = await _client.PostAsJsonAsync("/api/orders", request);
Assert.Equal(HttpStatusCode.Created, response.StatusCode);
Assert.NotNull(response.Headers.Location);
}
}
```
**Important:** The `Program` class must be accessible to the test project. Either make it public or add an `InternalsVisibleTo` attribute:
```csharp
// In the API project (e.g., Program.cs or a separate file)
[assembly: InternalsVisibleTo("MyApp.Api.IntegrationTests")]
```
Or in the csproj:
```xml
<ItemGroup>
<InternalsVisibleTo Include="MyApp.Api.IntegrationTests" />
</ItemGroup>
```
### Customizing the Test Server
Override services, configuration, or middleware using `WebApplicationFactory<T>.WithWebHostBuilder`:
```csharp
public class CustomWebAppFactory : WebApplicationFactory<Program>
{
// Provide connection string from test fixture (e.g., Testcontainers)
public string ConnectionString { get; set; } = "";
protected override void ConfigureWebHost(IWebHostBuilder builder)
{
builder.UseEnvironment("Testing");
builder.ConfigureAppConfiguration((context, config) =>
{
config.AddInMemoryCollection(new Dictionary<string, string?>
{
["ConnectionStrings:Default"] = ConnectionString,
["Features:EnableNewCheckout"] = "true"
});
});
builder.ConfigureTestServices(services =>
{
// Replace real services with test doubles
services.RemoveAll<IEmailSender>();
services.AddSingleton<IEmailSender, FakeEmailSender>();
// Replace database context with test database
services.RemoveAll<DbContextOptions<AppDbContext>>();
services.AddDbContext<AppDbContext>(options =>
options.UseNpgsql(ConnectionString));
});
}
}
```
### Authenticated Requests
Test authenticated endpoints by configuring an authentication handler:
```csharp
public class AuthenticatedWebAppFactory : WebApplicationFactory<Program>
{
protected override void ConfigureWebHost(IWebHostBuilder builder)
{
builder.ConfigureTestServices(services =>
{
services.AddAuthentication("Test")
.AddScheme<AuthenticationSchemeOptions, TestAuthHandler>(
"Test", options => { });
});
}
}
public class TestAuthHandler : AuthenticationHandler<AuthenticationSchemeOptions>
{
public TestAuthHandler(
IOptionsMonitor<AuthenticationSchemeOptions> options,
ILoggerFactory logger,
UrlEncoder encoder)
: base(options, logger, encoder) { }
protected override Task<AuthenticateResult> HandleAuthenticateAsync()
{
var claims = new[]
{
new Claim(ClaimTypes.NameIdentifier, "test-user-id"),
new Claim(ClaimTypes.Name, "Test User"),
new Claim(ClaimTypes.Role, "Admin")
};
var identity = new ClaimsIdentity(claims, "Test");
var principal = new ClaimsPrincipal(identity);
var ticket = new AuthenticationTicket(principal, "Test");
return Task.FromResult(AuthenticateResult.Success(ticket));
}
}
```
---
## Testcontainers
Testcontainers spins up real infrastructure (databases, message brokers, caches) in Docker containers for tests. Each test run gets a fresh, disposable environment.
### Packages
```xml
<PackageReference Include="Testcontainers" Version="4.*" />
<!-- Database-specific modules -->
<PackageReference Include="Testcontainers.PostgreSql" Version="4.*" />
<PackageReference Include="Testcontainers.MsSql" Version="4.*" />
<PackageReference Include="Testcontainers.Redis" Version="4.*" />
```
### PostgreSQL Example
```csharp
public class PostgresFixture : IAsyncLifetime
{
private readonly PostgreSqlContainer _container = new PostgreSqlBuilder()
.WithImage("postgres:16-alpine")
.WithDatabase("testdb")
.WithUsername("test")
.WithPassword("test")
.Build();
public string ConnectionString => _container.GetConnectionString();
public async ValueTask InitializeAsync()
{
await _container.StartAsync();
}
public async ValueTask DisposeAsync()
{
await _container.DisposeAsync();
}
}
[CollectionDefinition("Postgres")]
public class PostgresCollection : ICollectionFixture<PostgresFixture> { }
[Collection("Postgres")]
public class OrderRepositoryTests
{
private readonly PostgresFixture _postgres;
public OrderRepositoryTests(PostgresFixture postgres)
{
_postgres = postgres;
}
[Fact]
public async Task Insert_ValidOrder_CanBeRetrieved()
{
await using var context = CreateContext(_postgres.ConnectionString);
await context.Database.EnsureCreatedAsync();
var order = new Order { CustomerId = "cust-1", Total = 99.99m };
context.Orders.Add(order);
await context.SaveChangesAsync();
var retrieved = await context.Orders.FindAsync(order.Id);
Assert.NotNull(retrieved);
Assert.Equal(99.99m, retrieved.Total);
}
private static AppDbContext CreateContext(string connectionString)
{
var options = new DbContextOptionsBuilder<AppDbContext>()
.UseNpgsql(connectionString)
.Options;
return new AppDbContext(options);
}
}
```
### SQL Server Example
```csharp
public class SqlServerFixture : IAsyncLifetime
{
private readonly MsSqlContainer _container = new MsSqlBuilder()
.WithImage("mcr.microsoft.com/mssql/server:2022-latest")
.Build();
public string ConnectionString => _container.GetConnectionString();
public async ValueTask InitializeAsync()
{
await _container.StartAsync();
}
public async ValueTask DisposeAsync()
{
await _container.DisposeAsync();
}
}
```
### Combining WebApplicationFactory with Testcontainers
The most common pattern: use Testcontainers for the database and WebApplicationFactory for the API:
```csharp
public class ApiTestFactory : WebApplicationFactory<Program>, IAsyncLifetime
{
private readonly PostgreSqlContainer _postgres = new PostgreSqlBuilder()
.WithImage("postgres:16-alpine")
.Build();
protected override void ConfigureWebHost(IWebHostBuilder builder)
{
builder.ConfigureTestServices(services =>
{
services.RemoveAll<DbContextOptions<AppDbContext>>();
services.AddDbContext<AppDbContext>(options =>
options.UseNpgsql(_postgres.GetConnectionString()));
});
}
public async ValueTask InitializeAsync()
{
await _postgres.StartAsync();
}
public new async ValueTask DisposeAsync()
{
await _postgres.DisposeAsync();
await base.DisposeAsync();
}
}
public class OrdersApiIntegrationTests : IClassFixture<ApiTestFactory>
{
private readonly HttpClient _client;
private readonly ApiTestFactory _factory;
public OrdersApiIntegrationTests(ApiTestFactory factory)
{
_factory = factory;
_client = factory.CreateClient();
}
[Fact]
public async Task CreateAndRetrieveOrder_RoundTrip()
{
// Ensure schema exists
using var scope = _factory.Services.CreateScope();
var db = scope.ServiceProvider.GetRequiredService<AppDbContext>();
await db.Database.EnsureCreatedAsync();
// Create
var createResponse = await _client.PostAsJsonAsync("/api/orders",
new { CustomerId = "cust-1", Items = new[] { new { Sku = "SKU-1", Quantity = 2 } } });
createResponse.EnsureSuccessStatusCode();
var location = createResponse.Headers.Location!.ToString();
// Retrieve
var getResponse = await _client.GetAsync(location);
getResponse.EnsureSuccessStatusCode();
var order = await getResponse.Content.ReadFromJsonAsync<OrderDto>();
Assert.Equal("cust-1", order!.CustomerId);
}
}
```
---
## .NET Aspire Testing
.NET Aspire provides `DistributedApplicationTestingBuilder` for testing multi-service applications orchestrated with Aspire. This tests the actual distributed topology including service discovery, configuration, and health checks.
### Package
```xml
<PackageReference Include="Aspire.Hosting.Testing" Version="9.*" />
```
### Basic Aspire Test
```csharp
public class AspireIntegrationTests
{
[Fact]
public async Task ApiService_ReturnsHealthy()
{
var builder = await DistributedApplicationTestingBuilder
.CreateAsync<Projects.MyApp_AppHost>();
await using var app = await builder.BuildAsync();
await app.StartAsync();
var httpClient = app.CreateHttpClient("api-service");
var response = await httpClient.GetAsync("/health");
Assert.Equal(HttpStatusCode.OK, response.StatusCode);
}
[Fact]
public async Task ApiService_WithDatabase_ReturnsOrders()
{
var builder = await DistributedApplicationTestingBuilder
.CreateAsync<Projects.MyApp_AppHost>();
await using var app = await builder.BuildAsync();
await app.StartAsync();
// Wait for resources to be healthy
var resourceNotification = app.Services
.GetRequiredService<ResourceNotificationService>();
await resourceNotification
.WaitForResourceHealthyAsync("api-service")
.WaitAsync(TimeSpan.FromSeconds(60));
var httpClient = app.CreateHttpClient("api-service");
var response = await httpClient.GetAsync("/api/orders");
response.EnsureSuccessStatusCode();
}
}
```
### Aspire with Service Overrides
Replace services in the Aspire app model for testing:
```csharp
[Fact]
public async Task ApiService_WithMockedExternalDependency()
{
var builder = await DistributedApplicationTestingBuilder
.CreateAsync<Projects.MyApp_AppHost>();
// Override configuration for the API service
builder.Services.ConfigureHttpClientDefaults(http =>
{
http.AddStandardResilienceHandler();
});
await using var app = await builder.BuildAsync();
await app.StartAsync();
var httpClient = app.CreateHttpClient("api-service");
var response = await httpClient.GetAsync("/api/orders");
response.EnsureSuccessStatusCode();
}
```
---
## Database Fixture Patterns
### Per-Test Isolation with Transactions
Roll back each test's changes using a transaction scope:
```csharp
public class TransactionalTestBase : IClassFixture<PostgresFixture>, IAsyncLifetime
{
private readonly PostgresFixture _postgres;
private AppDbContext _context = null!;
private IDbContextTransaction _transaction = null!;
public TransactionalTestBase(PostgresFixture postgres)
{
_postgres = postgres;
}
protected AppDbContext Context => _context;
public async ValueTask InitializeAsync()
{
var options = new DbContextOptionsBuilder<AppDbContext>()
.UseNpgsql(_postgres.ConnectionString)
.Options;
_context = new AppDbContext(options);
await _context.Database.EnsureCreatedAsync();
_transaction = await _context.Database.BeginTransactionAsync();
}
public async ValueTask DisposeAsync()
{
await _transaction.RollbackAsync();
await _transaction.DisposeAsync();
await _context.DisposeAsync();
}
}
public class OrderTests : TransactionalTestBase
{
public OrderTests(PostgresFixture postgres) : base(postgres) { }
[Fact]
public async Task Insert_ValidOrder_Persists()
{
Context.Orders.Add(new Order { CustomerId = "cust-1", Total = 50m });
await Context.SaveChangesAsync();
var count = await Context.Orders.CountAsync();
Assert.Equal(1, count);
// Transaction rolls back after test -- database stays clean
}
}
```
### Per-Test Isolation with Respawn
Use Respawn to reset database state between tests by deleting data instead of rolling back transactions. This is useful when transaction rollback is not feasible (e.g., testing code that commits its own transactions):
```csharp
// NuGet: Respawn
// Combined fixture: owns the container AND the respawner
public class RespawnablePostgresFixture : IAsyncLifetime
{
private readonly PostgreSqlContainer _container = new PostgreSqlBuilder()
.WithImage("postgres:16-alpine")
.Build();
private Respawner _respawner = null!;
private NpgsqlConnection _connection = null!;
public string ConnectionString => _container.GetConnectionString();
public async ValueTask InitializeAsync()
{
await _container.StartAsync();
_connection = new NpgsqlConnection(ConnectionString);
await _connection.OpenAsync();
// Run migrations or EnsureCreated before creating respawner
// so it knows which tables to clean
_respawner = await Respawner.CreateAsync(_connection, new RespawnerOptions
{
DbAdapter = DbAdapter.Postgres,
TablesToIgnore = ["__EFMigrationsHistory"]
});
}
public async Task ResetDatabaseAsync()
{
await _respawner.ResetAsync(_connection);
}
public async ValueTask DisposeAsync()
{
await _connection.DisposeAsync();
await _container.DisposeAsync();
}
}
```
---
## Test Isolation Strategies
### Strategy Comparison
| Strategy | Speed | Isolation | Complexity | Best For |
|----------|-------|-----------|------------|----------|
| **Transaction rollback** | Fastest | High | Low | Tests that use a single DbContext |
| **Respawn (data deletion)** | Fast | High | Medium | Tests where code commits its own transactions |
| **Fresh container per class** | Slow | Highest | Low | Tests that modify schema or need complete isolation |
| **Shared container + cleanup** | Moderate | Medium | Medium | Test suites with many classes sharing infrastructure |
### Container Lifecycle Recommendations
```
Per-test: Too slow. Never spin up a container per test.
Per-class: Good isolation, acceptable speed with ICollectionFixture.
Per-collection: Best balance -- share one container across related test classes.
Per-assembly: Fastest but requires careful cleanup between tests.
```
Use `ICollectionFixture<T>` (see [skill:dotnet-testing] `references/xunit.md`) to share a single container across multiple test classes while running those classes sequentially to avoid data conflicts.
---
## Testing with Redis
```csharp
public class RedisFixture : IAsyncLifetime
{
private readonly RedisContainer _container = new RedisBuilder()
.WithImage("redis:7-alpine")
.Build();
public string ConnectionString => _container.GetConnectionString();
public async ValueTask InitializeAsync() => await _container.StartAsync();
public async ValueTask DisposeAsync() => await _container.DisposeAsync();
}
[CollectionDefinition("Redis")]
public class RedisCollection : ICollectionFixture<RedisFixture> { }
[Collection("Redis")]
public class CacheServiceTests
{
private readonly RedisFixture _redis;
public CacheServiceTests(RedisFixture redis) => _redis = redis;
[Fact]
public async Task SetAndGet_RoundTrip_ReturnsOriginalValue()
{
var multiplexer = await ConnectionMultiplexer.ConnectAsync(
_redis.ConnectionString);
var cache = new RedisCacheService(multiplexer);
await cache.SetAsync("key-1", new Order { Id = 1, Total = 99m });
var result = await cache.GetAsync<Order>("key-1");
Assert.NotNull(result);
Assert.Equal(99m, result.Total);
}
}
```
---
## Key Principles
- **Use WebApplicationFactory for API tests.** It is faster, more reliable, and more deterministic than testing against a deployed instance.
- **Use Testcontainers for real infrastructure.** Do not mock `DbContext` -- test against a real database to verify LINQ-to-SQL translation and constraint enforcement.
- **Share containers across test classes** via `ICollectionFixture` to avoid the overhead of starting a new container per class.
- **Choose the right isolation strategy.** Transaction rollback is fastest and simplest; use Respawn when you cannot control transaction boundaries.
- **Always clean up test data.** Leftover data from one test causes flaky failures in another. Use transaction rollback, Respawn, or fresh containers.
- **Match `Microsoft.AspNetCore.Mvc.Testing` version to TFM.** Using the wrong version causes runtime binding failures.
---
## Agent Gotchas
1. **Do not hardcode `Microsoft.AspNetCore.Mvc.Testing` versions.** The package version must match the project's target framework major version. Specifying e.g. `Version="8.0.0"` breaks net9.0 projects.
2. **Do not forget `InternalsVisibleTo` for the `Program` class.** Without it, `WebApplicationFactory<Program>` cannot access the entry point and tests fail at compile time.
3. **Do not use `EnsureCreated()` with Respawn.** `EnsureCreated()` does not track migrations. Use `Database.MigrateAsync()` for production schemas, or `EnsureCreated()` only for simple test schemas.
4. **Do not dispose `WebApplicationFactory` before `HttpClient`.** The factory owns the test server; disposing it invalidates all clients. Let xUnit manage disposal via `IClassFixture`.
5. **Do not use `localhost` ports with Testcontainers.** Testcontainers maps random host ports to container ports. Always use the connection string from the container object (e.g., `_container.GetConnectionString()`), never hardcoded ports.
6. **Do not skip Docker availability checks in CI.** Testcontainers requires a running Docker daemon. Ensure your CI environment has Docker available, or use conditional test skipping when Docker is unavailable.
---
## References
- [Integration tests in ASP.NET Core](https://learn.microsoft.com/en-us/aspnet/core/test/integration-tests)
- [WebApplicationFactory](https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.mvc.testing.webapplicationfactory-1)
- [Testcontainers for .NET](https://dotnet.testcontainers.org/)
- [.NET Aspire testing](https://learn.microsoft.com/en-us/dotnet/aspire/fundamentals/testing)
- [Respawn](https://github.com/jbogard/Respawn)
- [Testcontainers PostgreSQL module](https://dotnet.testcontainers.org/modules/postgres/)
references/playwright.md# Playwright
Playwright for .NET: browser automation and end-to-end testing. Covers browser lifecycle management, page interactions, assertions, CI caching of browser binaries, trace viewer for debugging failures, and codegen for rapid test scaffolding.
**Version assumptions:** Playwright 1.40+ for .NET, .NET 8.0+ baseline. Playwright supports Chromium, Firefox, and WebKit browsers.
## Package Setup
```xml
<PackageReference Include="Microsoft.Playwright" Version="1.*" />
<!-- For xUnit integration: -->
<PackageReference Include="Microsoft.Playwright.Xunit" Version="1.*" />
<!-- For NUnit integration: -->
<!-- <PackageReference Include="Microsoft.Playwright.NUnit" Version="1.*" /> -->
```
### Installing Browsers
Playwright requires downloading browser binaries before tests can run:
```bash
# After building the test project:
pwsh bin/Debug/net8.0/playwright.ps1 install
# Or install specific browsers:
pwsh bin/Debug/net8.0/playwright.ps1 install chromium
pwsh bin/Debug/net8.0/playwright.ps1 install firefox
# Using dotnet tool:
dotnet tool install --global Microsoft.Playwright.CLI
playwright install
```
---
## Basic Test Structure
### With Playwright xUnit Base Class
```csharp
using Microsoft.Playwright;
using Microsoft.Playwright.Xunit;
// PageTest provides Page, Browser, BrowserContext, and Playwright properties
public class HomePageTests : PageTest
{
[Fact]
public async Task HomePage_Title_ContainsAppName()
{
await Page.GotoAsync("https://localhost:5001");
await Expect(Page).ToHaveTitleAsync(new Regex("My App"));
}
[Fact]
public async Task HomePage_NavLinks_AreVisible()
{
await Page.GotoAsync("https://localhost:5001");
var nav = Page.Locator("nav");
await Expect(nav.GetByRole(AriaRole.Link, new() { Name = "Home" }))
.ToBeVisibleAsync();
await Expect(nav.GetByRole(AriaRole.Link, new() { Name = "About" }))
.ToBeVisibleAsync();
}
}
```
### Manual Setup (Without Base Class)
```csharp
public class ManualSetupTests : IAsyncLifetime
{
private IPlaywright _playwright = null!;
private IBrowser _browser = null!;
private IBrowserContext _context = null!;
private IPage _page = null!;
public async ValueTask InitializeAsync()
{
_playwright = await Playwright.CreateAsync();
_browser = await _playwright.Chromium.LaunchAsync(new BrowserTypeLaunchOptions
{
Headless = true
});
_context = await _browser.NewContextAsync(new BrowserNewContextOptions
{
ViewportSize = new ViewportSize { Width = 1280, Height = 720 },
Locale = "en-US"
});
_page = await _context.NewPageAsync();
}
public async ValueTask DisposeAsync()
{
await _page.CloseAsync();
await _context.CloseAsync();
await _browser.CloseAsync();
_playwright.Dispose();
}
[Fact]
public async Task Login_ValidUser_RedirectsToDashboard()
{
await _page.GotoAsync("https://localhost:5001/login");
await _page.FillAsync("[data-testid='email']", "user@example.com");
await _page.FillAsync("[data-testid='password']", "P@ssw0rd!");
await _page.ClickAsync("[data-testid='login-btn']");
await Expect(_page).ToHaveURLAsync(new Regex("/dashboard"));
}
}
```
---
## Locators and Interactions
### Recommended Locator Strategies
```csharp
// BEST: Role-based (accessible and semantic)
var submitBtn = Page.GetByRole(AriaRole.Button, new() { Name = "Submit Order" });
// GOOD: Test ID (stable, explicit)
var emailInput = Page.Locator("[data-testid='email-input']");
// GOOD: Label text (user-visible, accessible)
var nameField = Page.GetByLabel("Full Name");
// GOOD: Placeholder (user-visible)
var searchBox = Page.GetByPlaceholder("Search products...");
// AVOID: CSS class (fragile, changes with styling)
var card = Page.Locator(".card-primary");
// AVOID: XPath (brittle, hard to read)
var cell = Page.Locator("//table/tbody/tr[1]/td[2]");
```
### Common Interactions
```csharp
// Text input
await Page.FillAsync("[data-testid='name']", "Alice Johnson");
// Click
await Page.ClickAsync("[data-testid='submit']");
// Select dropdown
await Page.SelectOptionAsync("[data-testid='country']", "US");
// Checkbox / radio
await Page.CheckAsync("[data-testid='agree-terms']");
// File upload
await Page.SetInputFilesAsync("[data-testid='avatar']", "testdata/photo.jpg");
// Keyboard
await Page.Keyboard.PressAsync("Enter");
await Page.Keyboard.TypeAsync("search query");
// Hover (for dropdowns, tooltips)
await Page.HoverAsync("[data-testid='user-menu']");
```
### Assertions (Expect API)
Playwright assertions auto-retry until the condition is met or the timeout expires:
```csharp
// Element visibility
await Expect(Page.Locator("[data-testid='success']")).ToBeVisibleAsync();
await Expect(Page.Locator("[data-testid='spinner']")).ToBeHiddenAsync();
// Text content
await Expect(Page.Locator("[data-testid='total']")).ToHaveTextAsync("$99.99");
await Expect(Page.Locator("[data-testid='status']")).ToContainTextAsync("Completed");
// Attribute
await Expect(Page.Locator("[data-testid='submit']")).ToBeEnabledAsync();
await Expect(Page.Locator("[data-testid='email']")).ToHaveValueAsync("user@example.com");
// Page-level
await Expect(Page).ToHaveURLAsync(new Regex("/orders/\\d+"));
await Expect(Page).ToHaveTitleAsync("Order Details - My App");
// Count
await Expect(Page.Locator("[data-testid='order-row']")).ToHaveCountAsync(5);
```
---
## Network Interception
### Mocking API Responses
```csharp
[Fact]
public async Task OrderList_WithMockedApi_DisplaysOrders()
{
// Intercept API calls and return mock data
await Page.RouteAsync("**/api/orders", async route =>
{
var json = JsonSerializer.Serialize(new[]
{
new { Id = 1, CustomerName = "Alice", Total = 99.99 },
new { Id = 2, CustomerName = "Bob", Total = 149.50 }
});
await route.FulfillAsync(new RouteFulfillOptions
{
Status = 200,
ContentType = "application/json",
Body = json
});
});
await Page.GotoAsync("https://localhost:5001/orders");
await Expect(Page.Locator("[data-testid='order-row']")).ToHaveCountAsync(2);
}
```
### Waiting for Network Requests
```csharp
[Fact]
public async Task CreateOrder_SubmitForm_WaitsForApiResponse()
{
await Page.GotoAsync("https://localhost:5001/orders/new");
await Page.FillAsync("[data-testid='customer']", "Alice");
await Page.FillAsync("[data-testid='amount']", "99.99");
// Wait for the API call triggered by form submission
var responseTask = Page.WaitForResponseAsync(
response => response.Url.Contains("/api/orders") && response.Status == 201);
await Page.ClickAsync("[data-testid='submit']");
var response = await responseTask;
Assert.Equal(201, response.Status);
}
```
---
## CI Browser Caching
Downloading browser binaries on every CI run is slow (500MB+). Cache them to speed up builds.
### GitHub Actions Caching
```yaml
# .github/workflows/e2e-tests.yml
jobs:
e2e:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup .NET
uses: actions/setup-dotnet@v4
with:
dotnet-version: '8.0.x'
- name: Build
run: dotnet build tests/MyApp.E2E/
- name: Cache Playwright browsers
id: playwright-cache
uses: actions/cache@v4
with:
path: ~/.cache/ms-playwright
key: playwright-${{ runner.os }}-${{ hashFiles('tests/MyApp.E2E/MyApp.E2E.csproj') }}
- name: Install Playwright browsers
if: steps.playwright-cache.outputs.cache-hit != 'true'
run: pwsh tests/MyApp.E2E/bin/Debug/net8.0/playwright.ps1 install --with-deps
- name: Install Playwright system deps
if: steps.playwright-cache.outputs.cache-hit == 'true'
run: pwsh tests/MyApp.E2E/bin/Debug/net8.0/playwright.ps1 install-deps
- name: Run E2E tests
run: dotnet test tests/MyApp.E2E/
```
### Azure DevOps Caching
```yaml
# azure-pipelines.yml
steps:
- task: Cache@2
inputs:
key: 'playwright | "$(Agent.OS)" | tests/MyApp.E2E/MyApp.E2E.csproj'
path: $(HOME)/.cache/ms-playwright
restoreKeys: |
playwright | "$(Agent.OS)"
cacheHitVar: PLAYWRIGHT_CACHE_RESTORED
displayName: Cache Playwright browsers
- script: pwsh tests/MyApp.E2E/bin/Debug/net8.0/playwright.ps1 install --with-deps
condition: ne(variables.PLAYWRIGHT_CACHE_RESTORED, 'true')
displayName: Install Playwright browsers
- script: pwsh tests/MyApp.E2E/bin/Debug/net8.0/playwright.ps1 install-deps
condition: eq(variables.PLAYWRIGHT_CACHE_RESTORED, 'true')
displayName: Install Playwright system deps (cached browsers)
- script: dotnet test tests/MyApp.E2E/
displayName: Run E2E tests
```
### Cache Key Strategy
The cache key should include:
- **OS:** Browser binaries are platform-specific
- **Project file hash:** Playwright version determines browser versions; changing the package version invalidates the cache
- **Fallback key:** Allows partial cache restoration when the project file changes
---
## Trace Viewer
Playwright's trace viewer captures a full recording of test execution for debugging failures. Each trace includes screenshots, DOM snapshots, network logs, and console output.
### Enabling Traces
```csharp
public class TracedTests : IAsyncLifetime
{
private IPlaywright _playwright = null!;
private IBrowser _browser = null!;
private IBrowserContext _context = null!;
public IPage Page { get; private set; } = null!;
public async ValueTask InitializeAsync()
{
_playwright = await Playwright.CreateAsync();
_browser = await _playwright.Chromium.LaunchAsync();
_context = await _browser.NewContextAsync();
// Start tracing before each test
await _context.Tracing.StartAsync(new TracingStartOptions
{
Screenshots = true,
Snapshots = true,
Sources = true
});
Page = await _context.NewPageAsync();
}
public async ValueTask DisposeAsync()
{
// Save trace on failure (check test result in xUnit requires custom wrapper)
await _context.Tracing.StopAsync(new TracingStopOptions
{
Path = Path.Combine("test-results", "traces",
$"trace-{DateTime.UtcNow:yyyyMMdd-HHmmss}.zip")
});
await Page.CloseAsync();
await _context.CloseAsync();
await _browser.CloseAsync();
_playwright.Dispose();
}
}
```
### Viewing Traces
```bash
# Open trace file in browser
pwsh bin/Debug/net8.0/playwright.ps1 show-trace test-results/traces/trace-20260101-120000.zip
# Or use the online trace viewer
# Upload the .zip to https://trace.playwright.dev/
```
### Trace on Failure Only
Save traces only when tests fail to reduce storage:
```csharp
// In a custom test class or middleware
public async Task RunWithTrace(Func<IPage, Task> testAction, string testName)
{
await _context.Tracing.StartAsync(new TracingStartOptions
{
Screenshots = true,
Snapshots = true,
Sources = true
});
try
{
await testAction(Page);
// Test passed -- discard trace
await _context.Tracing.StopAsync();
}
catch
{
// Test failed -- save trace for debugging
await _context.Tracing.StopAsync(new TracingStopOptions
{
Path = $"test-results/traces/{testName}.zip"
});
throw;
}
}
```
---
## Codegen
Playwright's code generator records browser interactions and generates test code. Use it to scaffold tests quickly, then refine the generated code.
### Running Codegen
```bash
# Open codegen with your app URL
pwsh bin/Debug/net8.0/playwright.ps1 codegen https://localhost:5001
# With specific browser
pwsh bin/Debug/net8.0/playwright.ps1 codegen --browser firefox https://localhost:5001
# With device emulation
pwsh bin/Debug/net8.0/playwright.ps1 codegen --device "iPhone 15" https://localhost:5001
# With saved authentication state
pwsh bin/Debug/net8.0/playwright.ps1 codegen --save-storage auth.json https://localhost:5001
```
### Codegen Best Practices
1. **Use codegen as a starting point,** not the final test. Generated code often uses fragile selectors and lacks proper assertions.
2. **Replace generated selectors** with `data-testid` or role-based locators immediately after generating.
3. **Add meaningful assertions.** Codegen records actions but does not know what to verify. Add `Expect()` calls for expected outcomes.
4. **Extract page objects** from generated code. Group related interactions into page object methods.
### Before and After Codegen Refinement
```csharp
// GENERATED by codegen (fragile, no assertions):
await page.GotoAsync("https://localhost:5001/orders");
await page.Locator("#root > div > main > div:nth-child(2) > button").ClickAsync();
await page.GetByPlaceholder("Customer name").FillAsync("Alice");
await page.GetByPlaceholder("Amount").FillAsync("99.99");
await page.Locator("form > button[type='submit']").ClickAsync();
// REFINED (stable selectors, proper assertions):
await Page.GotoAsync("https://localhost:5001/orders");
await Page.ClickAsync("[data-testid='new-order-btn']");
await Page.FillAsync("[data-testid='customer-name']", "Alice");
await Page.FillAsync("[data-testid='amount']", "99.99");
await Page.ClickAsync("[data-testid='submit-order']");
await Expect(Page.Locator("[data-testid='success-toast']"))
.ToBeVisibleAsync();
await Expect(Page).ToHaveURLAsync(new Regex("/orders/\\d+"));
```
---
## Multi-Browser Testing
### Running Tests Across Browsers
```csharp
// Using Playwright xUnit base class with environment variable
// Set BROWSER=chromium|firefox|webkit via CLI or CI config
public class CrossBrowserTests : PageTest
{
[Fact]
public async Task OrderFlow_WorksAcrossBrowsers()
{
// This test runs in whichever browser BROWSER env var specifies
await Page.GotoAsync("https://localhost:5001/orders/new");
await Page.FillAsync("[data-testid='customer']", "Alice");
await Page.ClickAsync("[data-testid='submit']");
await Expect(Page.Locator("[data-testid='success']")).ToBeVisibleAsync();
}
}
```
```bash
# Run tests in each browser
BROWSER=chromium dotnet test
BROWSER=firefox dotnet test
BROWSER=webkit dotnet test
```
### CI Matrix Strategy
```yaml
# GitHub Actions matrix for multi-browser
strategy:
matrix:
browser: [chromium, firefox, webkit]
steps:
- name: Run E2E tests
run: dotnet test tests/MyApp.E2E/
env:
BROWSER: ${{ matrix.browser }}
```
---
## Key Principles
- **Use Playwright assertions (`Expect`) instead of raw xUnit `Assert`.** Playwright assertions auto-retry with configurable timeouts, eliminating flaky timing issues.
- **Cache browser binaries in CI.** Downloading 500MB+ of browsers per run wastes time and bandwidth. Cache by OS + Playwright version.
- **Enable trace viewer for debugging CI failures.** Traces capture everything needed to reproduce a failure without re-running the test.
- **Use codegen to bootstrap tests, then refine.** Generated code gets you started fast; manual refinement makes tests maintainable.
- **Prefer role-based or `data-testid` locators** over CSS classes or XPath. See [skill:dotnet-testing] `references/ui-testing-core.md` for the full selector priority guide.
---
## Agent Gotchas
1. **Do not forget to install browsers after adding the Playwright package.** The NuGet package does not include browser binaries. Run the install script after building.
2. **Do not use `Task.Delay` for waiting.** Playwright's auto-waiting and `Expect` assertions handle timing automatically. Adding delays makes tests slow and still flaky.
3. **Do not hardcode `localhost` ports.** Use configuration or environment variables for the base URL. CI environments may use different ports than local development.
4. **Do not skip `--with-deps` on first CI install.** Playwright browsers need system libraries (libgbm, libasound, etc.) on Linux. The `--with-deps` flag installs them. Subsequent cached runs only need `install-deps`.
5. **Do not store trace files in the repository.** Traces are large binary files. Write them to a `test-results/` directory that is git-ignored, and upload them as CI artifacts.
6. **Do not create a new browser instance per test.** Browser launch is expensive. Use `IClassFixture` or the Playwright xUnit base class to share a browser across tests in a class. Create a new `BrowserContext` per test for isolation.
---
## References
- [Playwright for .NET Documentation](https://playwright.dev/dotnet/)
- [Playwright Locators](https://playwright.dev/dotnet/docs/locators)
- [Playwright Assertions](https://playwright.dev/dotnet/docs/test-assertions)
- [Playwright Trace Viewer](https://playwright.dev/dotnet/docs/trace-viewer)
- [Playwright Codegen](https://playwright.dev/dotnet/docs/codegen)
- [Playwright CI Configuration](https://playwright.dev/dotnet/docs/ci)
- [Playwright Browser Downloads](https://playwright.dev/dotnet/docs/browsers)
references/slopwatch.md# Slopwatch
Slopwatch: LLM Anti-Cheat Quality Gate for .NET
Run the `Slopwatch.Cmd` dotnet tool as an automated quality gate after code modifications to detect "slop" -- shortcuts that make builds/tests pass without fixing real problems.
## Prerequisites
- .NET 8.0+ SDK
- `Slopwatch.Cmd` NuGet package (v0.3.3+)
Cross-references: [skill:dotnet-tooling] `references/tool-management.md` for general dotnet tool installation mechanics.
---
## Installation
### Local Tool (Recommended)
Add to `.config/dotnet-tools.json`:
```json
{
"version": 1,
"isRoot": true,
"tools": {
"slopwatch.cmd": {
"version": "0.3.3",
"commands": ["slopwatch"],
"rollForward": false
}
}
}
```
Then restore:
```bash
dotnet tool restore
```
### Global Tool
```bash
dotnet tool install --global Slopwatch.Cmd
```
See [skill:dotnet-tooling] `references/tool-management.md` for tool manifest conventions and restore patterns.
---
## Usage
### Basic Analysis
```bash
# Analyze current directory for slop
slopwatch analyze
# Analyze specific directory
slopwatch analyze -d ./src
# Strict mode -- fail on warnings too
slopwatch analyze --fail-on warning
# JSON output for tooling integration
slopwatch analyze --output json
# Show performance stats
slopwatch analyze --stats
```
### First-Time Setup: Establish a Baseline
For existing projects with pre-existing issues, create a baseline so slopwatch only catches **new** slop. The `init` command scans all files and records current findings as the accepted baseline:
```bash
slopwatch init
git add .slopwatch/baseline.json
git commit -m "Add slopwatch baseline"
```
### Updating the Baseline (Rare)
Only update when slop is **truly justified** and documented:
```bash
slopwatch analyze --update-baseline
```
Valid reasons: third-party library forces a pattern, intentional rate-limiting delay (not test flakiness), generated code that cannot be modified. Always add a code comment explaining the justification.
---
## Configuration
Create `.slopwatch/slopwatch.json` to customize rules and exclusions:
```json
{
"minSeverity": "warning",
"rules": {
"SW001": { "enabled": true, "severity": "error" },
"SW002": { "enabled": true, "severity": "warning" },
"SW003": { "enabled": true, "severity": "error" },
"SW004": { "enabled": true, "severity": "warning" },
"SW005": { "enabled": true, "severity": "warning" },
"SW006": { "enabled": true, "severity": "warning" }
},
"exclude": [
"**/Generated/**",
"**/obj/**",
"**/bin/**"
]
}
```
### Strict Mode (Recommended for LLM Sessions)
Elevate all rules to errors during LLM coding sessions:
```json
{
"minSeverity": "warning",
"rules": {
"SW001": { "enabled": true, "severity": "error" },
"SW002": { "enabled": true, "severity": "error" },
"SW003": { "enabled": true, "severity": "error" },
"SW004": { "enabled": true, "severity": "error" },
"SW005": { "enabled": true, "severity": "error" },
"SW006": { "enabled": true, "severity": "error" }
}
}
```
---
## Detection Rules
| Rule | Severity | What It Catches |
|------|----------|-----------------|
| SW001 | Error | Disabled tests (`Skip=`, `Ignore`, `#if false`) |
| SW002 | Warning | Warning suppression (`#pragma warning disable`, `SuppressMessage`) |
| SW003 | Error | Empty catch blocks that swallow exceptions |
| SW004 | Warning | Arbitrary delays in tests (`Task.Delay`, `Thread.Sleep`) |
| SW005 | Warning | Project file slop (`NoWarn`, `TreatWarningsAsErrors=false`) |
| SW006 | Warning | CPM bypass (`VersionOverride`, inline `Version` attributes) |
### When Slopwatch Flags an Issue
1. **Understand why** the shortcut was taken
2. **Request a proper fix** -- be specific about what's wrong
3. **Verify the fix** doesn't introduce different slop
```
# Example output
❌ SW001 [Error]: Disabled test detected
File: tests/MyApp.Tests/OrderTests.cs:45
Pattern: [Fact(Skip="Test is flaky")]
```
**Never disable tests to achieve a green build.** Fix the underlying issue.
---
## Claude Code Hook Integration
Add slopwatch as a `PostToolUse` hook to automatically validate every edit. Create or update `.claude/settings.json`:
```json
{
"hooks": {
"PostToolUse": [
{
"matcher": "Write|Edit|MultiEdit",
"hooks": [
{
"type": "command",
"command": "slopwatch analyze -d . --hook",
"timeout": 60000
}
]
}
]
}
}
```
The `--hook` flag:
- Only analyzes **git dirty files** (fast, even on large repos)
- Outputs errors to stderr in readable format
- Blocks the edit on warnings/errors (exit code 2)
- Claude sees the error and can fix it immediately
This is the pattern used by projects like BrighterCommand/Brighter.
---
## CI/CD Integration
### GitHub Actions
```yaml
jobs:
slopwatch:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup .NET
uses: actions/setup-dotnet@v4
with:
dotnet-version: '8.0.x' # any .NET 8+ SDK works
- name: Install Slopwatch
run: dotnet tool install --global Slopwatch.Cmd
- name: Run Slopwatch
run: slopwatch analyze -d . --fail-on warning
```
### Azure Pipelines
```yaml
- task: DotNetCoreCLI@2
displayName: 'Install Slopwatch'
inputs:
command: 'custom'
custom: 'tool'
arguments: 'install --global Slopwatch.Cmd'
- script: slopwatch analyze -d . --fail-on warning
displayName: 'Slopwatch Analysis'
```
---
## Agent Gotchas
- **Do not suppress slopwatch findings.** If slopwatch flags an issue, fix the code -- do not update the baseline or disable the rule without explicit user approval.
- **Run after every code change**, not just at the end. Catching slop early prevents cascading shortcuts.
- **Use `--hook` flag in Claude Code hooks**, not bare `analyze`. The hook flag restricts analysis to dirty files for performance.
- **Baseline is not a wastebasket.** Adding items to the baseline requires documented justification. Never bulk-update baseline to silence warnings.
- **Local tool preferred over global.** Use `.config/dotnet-tools.json` so the version is pinned and reproducible across team members.
---
## Quick Reference
```bash
# First time setup
slopwatch init
git add .slopwatch/baseline.json
# After every code change
slopwatch analyze
# Strict mode (recommended)
slopwatch analyze --fail-on warning
# Hook mode (for Claude Code integration)
slopwatch analyze -d . --hook
# JSON output for tooling
slopwatch analyze --output json
# Update baseline (rare, document why)
slopwatch analyze --update-baseline
```
---
## References
- [Slopwatch NuGet Package](https://www.nuget.org/packages/Slopwatch.Cmd)
- [skill:dotnet-tooling] `references/tool-management.md` -- dotnet tool installation and manifest conventions
- [skill:dotnet-api] `references/agent-gotchas.md` -- manual slop pattern recognition (visual detection counterpart)
- [skill:dotnet-testing] `references/test-quality.md` -- test coverage and quality measurement
references/snapshot-testing.md# Snapshot Testing
Snapshot (approval) testing with the Verify library for .NET. Covers verifying API responses, serialized objects, rendered emails, and other complex outputs by comparing them against approved baseline files. Includes scrubbing and filtering patterns to handle non-deterministic values (dates, GUIDs, timestamps), custom converters for domain-specific types, and strategies for organizing and reviewing snapshot files.
**Version assumptions:** Verify 20.x+ (.NET 8.0+ baseline). Examples use the `Verify.Xunit` integration package; equivalent packages exist for NUnit (`Verify.NUnit`) and MSTest (`Verify.MSTest`). Verify auto-discovers the test framework from the referenced package.
## Setup
### Packages
```xml
<PackageReference Include="Verify.Xunit" Version="20.*" />
<!-- For HTTP response verification -->
<PackageReference Include="Verify.Http" Version="6.*" />
```
### Module Initializer
Verify requires a one-time initialization per test assembly. Place this in a file at the root of your test project:
```csharp
// ModuleInitializer.cs
using System.Runtime.CompilerServices;
public static class ModuleInitializer
{
[ModuleInitializer]
public static void Init() =>
VerifySourceGenerators.Initialize();
}
```
### Source Control
Add to `.gitignore`:
```gitignore
# Verify received files (test failures)
*.received.*
```
Add to `.gitattributes` so verified files diff cleanly:
```gitattributes
*.verified.txt text eol=lf
*.verified.xml text eol=lf
*.verified.json text eol=lf
```
---
## Basic Usage
### Verifying Objects
Verify serializes the object to JSON and compares against a `.verified.txt` file:
```csharp
[UsesVerify]
public class OrderSerializationTests
{
[Fact]
public Task Serialize_CompletedOrder_MatchesSnapshot()
{
var order = new Order
{
Id = 1,
CustomerId = "cust-123",
Status = OrderStatus.Completed,
Items =
[
new OrderItem("SKU-001", Quantity: 2, UnitPrice: 29.99m),
new OrderItem("SKU-002", Quantity: 1, UnitPrice: 49.99m)
],
Total = 109.97m
};
return Verify(order);
}
}
```
First run creates `OrderSerializationTests.Serialize_CompletedOrder_MatchesSnapshot.verified.txt`:
```txt
{
Id: 1,
CustomerId: cust-123,
Status: Completed,
Items: [
{
Sku: SKU-001,
Quantity: 2,
UnitPrice: 29.99
},
{
Sku: SKU-002,
Quantity: 1,
UnitPrice: 49.99
}
],
Total: 109.97
}
```
### Verifying Strings and Streams
```csharp
[Fact]
public Task RenderInvoice_MatchesExpectedHtml()
{
var html = invoiceRenderer.Render(order);
return Verify(html, extension: "html");
}
[Fact]
public Task ExportReport_MatchesExpectedXml()
{
var stream = reportExporter.Export(report);
return Verify(stream, extension: "xml");
}
```
---
## Scrubbing and Filtering
Non-deterministic values (dates, GUIDs, auto-incremented IDs) change between test runs. Scrubbing replaces them with stable placeholders so snapshots remain comparable.
### Built-In Scrubbers
Verify includes scrubbers for common non-deterministic types that are active by default:
```csharp
[Fact]
public Task CreateOrder_ScrubsNonDeterministicValues()
{
var order = new Order
{
Id = Guid.NewGuid(), // Scrubbed to Guid_1
CreatedAt = DateTime.UtcNow, // Scrubbed to DateTime_1
TrackingNumber = Guid.NewGuid().ToString() // Scrubbed to Guid_2
};
return Verify(order);
}
```
Produces stable output:
```txt
{
Id: Guid_1,
CreatedAt: DateTime_1,
TrackingNumber: Guid_2
}
```
### Custom Scrubbers
When built-in scrubbing is not sufficient, add custom scrubbers:
```csharp
[Fact]
public Task AuditLog_ScrubsTimestampsAndMachineNames()
{
var log = auditService.GetRecentEntries();
return Verify(log)
.ScrubLinesWithReplace(line =>
Regex.Replace(line, @"Machine:\s+\w+", "Machine: Scrubbed"))
.ScrubLinesContaining("CorrelationId:");
}
```
### Ignoring Members
Exclude specific properties from verification:
```csharp
[Fact]
public Task OrderSnapshot_IgnoresVolatileFields()
{
var order = orderService.CreateOrder(request);
return Verify(order)
.IgnoreMember("CreatedAt")
.IgnoreMember("UpdatedAt")
.IgnoreMember("ETag");
}
```
Or ignore by type across all verifications:
```csharp
// In ModuleInitializer
[ModuleInitializer]
public static void Init()
{
VerifierSettings.IgnoreMembersWithType<DateTime>();
VerifierSettings.IgnoreMembersWithType<DateTimeOffset>();
}
```
### Scrubbing Inline Values
Replace specific patterns in the serialized output:
```csharp
[Fact]
public Task ApiResponse_ScrubsTokens()
{
var response = authService.GenerateTokenResponse(user);
return Verify(response)
.ScrubLinesWithReplace(line =>
Regex.Replace(line, @"Bearer [A-Za-z0-9\-._~+/]+=*", "Bearer {scrubbed}"));
}
```
---
## Verifying HTTP Responses
Verify HTTP responses from WebApplicationFactory integration tests to lock down API contracts.
### Setup
```xml
<PackageReference Include="Verify.Http" Version="6.*" />
```
### Verifying Full HTTP Responses
```csharp
[UsesVerify]
public class OrdersApiSnapshotTests : IClassFixture<WebApplicationFactory<Program>>
{
private readonly HttpClient _client;
public OrdersApiSnapshotTests(WebApplicationFactory<Program> factory)
{
_client = factory.CreateClient();
}
[Fact]
public async Task GetOrders_ResponseMatchesSnapshot()
{
var response = await _client.GetAsync("/api/orders");
await Verify(response);
}
}
```
The verified file captures status code, headers, and body:
```txt
{
Status: 200 OK,
Headers: {
Content-Type: application/json; charset=utf-8
},
Body: [
{
Id: 1,
CustomerId: cust-123,
Status: Pending,
Total: 109.97
}
]
}
```
### Verifying Specific Response Parts
```csharp
[Fact]
public async Task CreateOrder_VerifyResponseBody()
{
var response = await _client.PostAsJsonAsync("/api/orders", request);
var body = await response.Content.ReadFromJsonAsync<OrderDto>();
await Verify(body)
.IgnoreMember("Id")
.IgnoreMember("CreatedAt");
}
```
---
## Verifying Rendered Emails
Snapshot-test email templates by verifying the rendered HTML output:
```csharp
[UsesVerify]
public class EmailTemplateTests
{
private readonly EmailRenderer _renderer = new();
[Fact]
public Task OrderConfirmation_MatchesSnapshot()
{
var model = new OrderConfirmationModel
{
CustomerName = "Alice Johnson",
OrderNumber = "ORD-001",
Items =
[
new("Widget A", Quantity: 2, Price: 29.99m),
new("Widget B", Quantity: 1, Price: 49.99m)
],
Total = 109.97m
};
var html = _renderer.RenderOrderConfirmation(model);
return Verify(html, extension: "html");
}
[Fact]
public Task PasswordReset_MatchesSnapshot()
{
var model = new PasswordResetModel
{
UserName = "alice",
ResetLink = "https://example.com/reset?token=test-token"
};
var html = _renderer.RenderPasswordReset(model);
return Verify(html, extension: "html")
.ScrubLinesWithReplace(line =>
Regex.Replace(line, @"token=[^""&]+", "token={scrubbed}"));
}
}
```
---
## Custom Converters
Custom converters control how specific types are serialized for verification. Use them for domain types that need a readable, stable representation.
### Writing a Custom Converter
```csharp
public class MoneyConverter : WriteOnlyJsonConverter<Money>
{
public override void Write(VerifyJsonWriter writer, Money value)
{
writer.WriteStartObject();
writer.WriteMember(value, value.Amount, "Amount");
writer.WriteMember(value, value.Currency.Code, "Currency");
writer.WriteEndObject();
}
}
```
Register in the module initializer:
```csharp
[ModuleInitializer]
public static void Init()
{
VerifierSettings.AddExtraSettings(settings =>
settings.Converters.Add(new MoneyConverter()));
}
```
### Converter for Complex Domain Types
```csharp
public class AddressConverter : WriteOnlyJsonConverter<Address>
{
public override void Write(VerifyJsonWriter writer, Address value)
{
// Single-line summary for compact snapshots
writer.WriteValue($"{value.Street}, {value.City}, {value.State} {value.Zip}");
}
}
public class DateRangeConverter : WriteOnlyJsonConverter<DateRange>
{
public override void Write(VerifyJsonWriter writer, DateRange value)
{
writer.WriteStartObject();
writer.WriteMember(value, value.Start.ToString("yyyy-MM-dd"), "Start");
writer.WriteMember(value, value.End.ToString("yyyy-MM-dd"), "End");
writer.WriteMember(value, value.Duration.Days, "DurationDays");
writer.WriteEndObject();
}
}
```
Usage in tests:
```csharp
[Fact]
public Task Customer_WithAddress_MatchesSnapshot()
{
var customer = new Customer
{
Name = "Alice Johnson",
Address = new Address("123 Main St", "Springfield", "IL", "62701"),
MemberSince = new DateRange(
new DateTime(2020, 1, 15),
new DateTime(2025, 1, 15))
};
return Verify(customer);
}
```
Produces:
```txt
{
Name: Alice Johnson,
Address: 123 Main St, Springfield, IL 62701,
MemberSince: {
Start: 2020-01-15,
End: 2025-01-15,
DurationDays: 1827
}
}
```
---
## Snapshot File Organization
### Default Naming
Verify names snapshot files based on the test class and method:
```
TestClassName.MethodName.verified.txt
```
Files are placed next to the test source file by default.
### Unique Directory
Move verified files into a dedicated directory to reduce clutter:
```csharp
// ModuleInitializer.cs
[ModuleInitializer]
public static void Init()
{
Verifier.DerivePathInfo(
(sourceFile, projectDirectory, type, method) =>
new PathInfo(
directory: Path.Combine(projectDirectory, "Snapshots"),
typeName: type.Name,
methodName: method.Name));
}
```
### Parameterized Tests
For `[Theory]` tests, Verify appends parameter values to the file name:
```csharp
[Theory]
[InlineData("en-US")]
[InlineData("de-DE")]
[InlineData("ja-JP")]
public Task FormatCurrency_ByLocale_MatchesSnapshot(string locale)
{
var formatted = currencyFormatter.Format(1234.56m, locale);
return Verify(formatted)
.UseParameters(locale);
}
```
Creates separate files:
```
FormatCurrencyTests.FormatCurrency_ByLocale_MatchesSnapshot_locale=en-US.verified.txt
FormatCurrencyTests.FormatCurrency_ByLocale_MatchesSnapshot_locale=de-DE.verified.txt
FormatCurrencyTests.FormatCurrency_ByLocale_MatchesSnapshot_locale=ja-JP.verified.txt
```
---
## Workflow: Accepting Changes
When a snapshot test fails, Verify creates a `.received.txt` file alongside the `.verified.txt` file. Review the diff and accept or reject:
### Diff Tool Integration
Verify launches a diff tool automatically when a test fails. Configure the preferred tool:
```csharp
[ModuleInitializer]
public static void Init()
{
// Verify auto-detects installed diff tools
// Override if needed:
DiffTools.UseOrder(DiffTool.VisualStudioCode, DiffTool.Rider);
}
```
### CLI Acceptance
Install the Verify CLI tool (one-time setup), then accept pending changes after review:
```bash
# Install the Verify CLI tool (one-time)
dotnet tool install -g verify.tool
# Accept all received files in the solution
verify accept
# Accept for a specific test project
verify accept --project tests/MyApp.Tests
```
### CI Behavior
In CI, Verify should fail tests without launching a diff tool. Set the environment variable:
```yaml
env:
DiffEngine_Disabled: true
```
Or in the module initializer:
```csharp
[ModuleInitializer]
public static void Init()
{
if (Environment.GetEnvironmentVariable("CI") is not null)
{
DiffRunner.Disabled = true;
}
}
```
---
## Key Principles
- **Snapshot test complex outputs, not simple values.** If the expected value fits in a single `Assert.Equal`, prefer that over a snapshot. Snapshots shine for multi-field objects, API responses, and rendered content.
- **Scrub all non-deterministic values.** Dates, GUIDs, timestamps, and machine-specific values must be scrubbed or ignored. Unscrubbed snapshots cause flaky tests.
- **Commit `.verified.txt` files to source control.** These are the approved baselines. Never add `.received.txt` files -- they represent unapproved changes.
- **Review snapshot diffs carefully.** Accepting a snapshot change without review can silently approve regressions. Treat snapshot diffs like code review.
- **Use custom converters for domain readability.** Default JSON serialization may be verbose or unclear for domain types. Converters produce focused, human-readable snapshots.
- **Keep snapshots focused.** Verify only the parts that matter. Use `IgnoreMember` to exclude volatile or irrelevant fields rather than verifying the entire object graph.
---
## Agent Gotchas
1. **Do not forget `[UsesVerify]` on the test class.** Without this attribute, `Verify()` calls compile but fail at runtime with an initialization error. Every test class using Verify must have this attribute.
2. **Do not commit `.received.txt` files.** These represent test failures and unapproved changes. Add `*.received.*` to `.gitignore` to prevent accidental commits.
3. **Do not skip `UseParameters()` in parameterized tests.** Without it, all parameter combinations write to the same snapshot file, overwriting each other. Always call `UseParameters()` with the theory data values.
4. **Do not scrub values that are part of the contract.** If an API always returns a specific date format or a known GUID, verify those values rather than scrubbing them. Only scrub values that are genuinely non-deterministic between runs.
5. **Do not use snapshot testing for rapidly evolving APIs.** During early development when the API shape changes frequently, snapshot tests create excessive churn. Wait until the API stabilizes.
6. **Do not hardcode Verify package versions across different test frameworks.** `Verify.Xunit`, `Verify.NUnit`, and `Verify.MSTest` have independent version lines. Always use version ranges (e.g., `20.*`) rather than pinning to a specific version.
---
## References
- [Verify GitHub repository](https://github.com/VerifyTests/Verify)
- [Verify documentation](https://github.com/VerifyTests/Verify/blob/main/docs/readme.md)
- [Verify.Http for HTTP response testing](https://github.com/VerifyTests/Verify.Http)
- [Scrubbing and filtering](https://github.com/VerifyTests/Verify/blob/main/docs/scrubbers.md)
- [Custom converters](https://github.com/VerifyTests/Verify/blob/main/docs/converters.md)
- [DiffEngine (diff tool integration)](https://github.com/VerifyTests/DiffEngine)
references/test-quality.md# Test Quality
Test quality analysis for .NET projects. Covers code coverage collection with coverlet, human-readable coverage reports with ReportGenerator, CRAP (Change Risk Anti-Patterns) score analysis to identify undertested complex code, mutation testing with Stryker.NET to evaluate test suite effectiveness, and strategies for detecting and managing flaky tests.
**Version assumptions:** Coverlet 6.x+, ReportGenerator 5.x+, Stryker.NET 4.x+ (.NET 8.0+ baseline). Coverlet supports both the MSBuild integration (`coverlet.msbuild`) and the `coverlet.collector` data collector; examples use `coverlet.collector` as the recommended approach.
## Code Coverage with Coverlet
Coverlet is the standard open-source code coverage library for .NET. It instruments assemblies at build time or via a data collector and produces coverage reports in multiple formats.
### Packages
```xml
<!-- Data collector approach (recommended) -->
<PackageReference Include="coverlet.collector" Version="8.0.0">
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
<PrivateAssets>all</PrivateAssets>
</PackageReference>
```
### Collecting Coverage
```bash
# Collect coverage with Cobertura output (default for ReportGenerator)
dotnet test --collect:"XPlat Code Coverage"
# Specify output format explicitly
dotnet test --collect:"XPlat Code Coverage" \
-- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura
# Multiple formats
dotnet test --collect:"XPlat Code Coverage" \
-- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=cobertura,opencover
```
Coverage results are written to `TestResults/<guid>/coverage.cobertura.xml` under each test project's output directory.
### Filtering Coverage
Exclude generated code, test projects, or specific namespaces:
```bash
dotnet test --collect:"XPlat Code Coverage" \
-- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Exclude="[*.Tests]*,[*.IntegrationTests]*" \
DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.ExcludeByAttribute="GeneratedCodeAttribute,ObsoleteAttribute,ExcludeFromCodeCoverageAttribute"
```
Or configure via a `runsettings` file for repeatability:
```xml
<!-- coverlet.runsettings -->
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<DataCollectionRunSettings>
<DataCollectors>
<DataCollector friendlyName="XPlat Code Coverage">
<Configuration>
<Format>cobertura</Format>
<Exclude>[*.Tests]*,[*.IntegrationTests]*</Exclude>
<ExcludeByAttribute>
GeneratedCodeAttribute,ObsoleteAttribute,ExcludeFromCodeCoverageAttribute
</ExcludeByAttribute>
<ExcludeByFile>**/Migrations/**</ExcludeByFile>
<IncludeTestAssembly>false</IncludeTestAssembly>
</Configuration>
</DataCollector>
</DataCollectors>
</DataCollectionRunSettings>
</RunSettings>
```
```bash
dotnet test --settings coverlet.runsettings
```
### Merge Coverage from Multiple Test Projects
When a solution has multiple test projects, merge their coverage into a single report:
```bash
# Run all tests, collecting coverage per project
dotnet test --collect:"XPlat Code Coverage"
# Find all coverage files and merge via ReportGenerator (see next section)
```
---
## Coverage Reports with ReportGenerator
ReportGenerator converts raw coverage data (Cobertura, OpenCover) into human-readable HTML reports with line-level highlighting.
### Installation
```bash
# Install as a global tool
dotnet tool install -g dotnet-reportgenerator-globaltool
# Or as a local tool
dotnet tool install dotnet-reportgenerator-globaltool
```
### Generating Reports
```bash
# Single coverage file
reportgenerator \
-reports:"tests/MyApp.Tests/TestResults/*/coverage.cobertura.xml" \
-targetdir:"coverage-report" \
-reporttypes:"Html;TextSummary"
# Multiple test projects (glob pattern merges automatically)
reportgenerator \
-reports:"**/TestResults/*/coverage.cobertura.xml" \
-targetdir:"coverage-report" \
-reporttypes:"Html;Cobertura;TextSummary"
```
### Report Types
| Type | Description | Use Case |
|------|-------------|----------|
| `Html` | Interactive HTML with line highlighting | Local developer review |
| `HtmlInline_AzurePipelines` | HTML optimized for Azure DevOps | CI artifact |
| `Cobertura` | Merged Cobertura XML | Input for other tools |
| `TextSummary` | Plain text summary | CLI/CI output |
| `Badges` | SVG coverage badges | README badges |
| `MarkdownSummaryGithub` | GitHub-flavored markdown | PR comments |
### Example: Full Coverage Pipeline
```bash
#!/bin/bash
# clean previous results
rm -rf coverage-report TestResults
# run tests with coverage
dotnet test --collect:"XPlat Code Coverage" --results-directory TestResults
# generate merged HTML report
reportgenerator \
-reports:"**/TestResults/*/coverage.cobertura.xml" \
-targetdir:"coverage-report" \
-reporttypes:"Html;TextSummary;Badges"
# display summary
cat coverage-report/Summary.txt
```
### Setting Coverage Thresholds
Enforce minimum coverage in CI by parsing the text summary or using a threshold parameter:
```bash
# ReportGenerator does not enforce thresholds directly.
# Parse the summary or use dotnet-coverage (Microsoft) for threshold enforcement.
# Alternative: use coverlet's built-in threshold via MSBuild
dotnet test /p:CollectCoverage=true \
/p:Threshold=80 \
/p:ThresholdType=line \
/p:ThresholdStat=total
```
**Note:** The `/p:Threshold` parameter requires the `coverlet.msbuild` package (not `coverlet.collector`). For `coverlet.collector` workflows, enforce thresholds by parsing the ReportGenerator text summary in your CI script.
---
## CRAP Analysis
CRAP (Change Risk Anti-Patterns) scores identify methods that are both complex and poorly tested. A high CRAP score means the method has high cyclomatic complexity and low code coverage -- a risky combination.
### Formula
```
CRAP(m) = complexity(m)^2 * (1 - coverage(m)/100)^3 + complexity(m)
```
Where:
- `complexity(m)` = cyclomatic complexity of method m
- `coverage(m)` = code coverage percentage of method m (0-100)
### Interpreting CRAP Scores
| CRAP Score | Risk Level | Action |
|------------|------------|--------|
| < 5 | Low | Method is simple or well-tested |
| 5-15 | Moderate | Review -- may need additional tests |
| 15-30 | High | Prioritize: add tests or reduce complexity |
| > 30 | Critical | Refactor and add tests immediately |
### Generating CRAP Reports
ReportGenerator includes CRAP analysis when using OpenCover format as input:
```bash
# Step 1: Collect coverage in OpenCover format
dotnet test --collect:"XPlat Code Coverage" \
-- DataCollectionRunSettings.DataCollectors.DataCollector.Configuration.Format=opencover
# Step 2: Generate report with risk hotspot analysis
reportgenerator \
-reports:"**/TestResults/*/coverage.opencover.xml" \
-targetdir:"coverage-report" \
-reporttypes:"Html;RiskHotspots"
```
The Risk Hotspots report highlights methods sorted by CRAP score, showing:
- Method name and containing class
- Cyclomatic complexity
- Code coverage percentage
- Computed CRAP score
### Using CRAP Scores Effectively
```csharp
// Example: a method with high complexity and low coverage
// Cyclomatic complexity: 12, Coverage: 20%
// CRAP = 12^2 * (1 - 0.20)^3 + 12 = 144 * 0.512 + 12 = 85.7 (Critical)
public decimal CalculateShipping(Order order)
{
if (order.Items.Count == 0) return 0;
decimal baseRate = order.DestinationCountry switch
{
"US" => 5.99m,
"CA" => 9.99m,
"UK" => 12.99m,
_ => 19.99m
};
if (order.Total > 100) baseRate *= 0.5m;
if (order.IsPriority) baseRate *= 2.0m;
if (order.Items.Any(i => i.IsFragile)) baseRate += 4.99m;
if (order.Items.Any(i => i.IsOversized)) baseRate += 14.99m;
if (order.HasInsurance) baseRate += order.Total * 0.02m;
if (order.IsExpedited && order.DestinationCountry != "US") baseRate *= 1.5m;
return Math.Round(baseRate, 2);
}
```
Address high CRAP scores by:
1. **Adding targeted tests** for uncovered branches to reduce the score via higher coverage
2. **Reducing complexity** by extracting methods (e.g., separate `CalculateBaseRate` and `ApplySurcharges` methods)
3. **Both** -- the most effective approach combines better coverage with simpler methods
---
## Mutation Testing with Stryker.NET
Mutation testing evaluates test suite quality by introducing small changes (mutations) to production code and checking whether tests detect them. If a mutation survives (tests still pass), the test suite has a gap.
### Installation
```bash
# Install as a global tool
dotnet tool install -g dotnet-stryker
# Or as a local tool (recommended for team consistency)
dotnet tool install dotnet-stryker
```
### Running Stryker.NET
```bash
# From the test project directory
cd tests/MyApp.Tests
dotnet stryker
# Specify the source project explicitly
dotnet stryker --project MyApp.csproj
# Target specific files
dotnet stryker --mutate "src/Services/**/*.cs"
```
### Configuration File
Create `stryker-config.json` in the test project directory:
```json
{
"$schema": "https://raw.githubusercontent.com/stryker-mutator/stryker-net/master/src/Stryker.Core/Stryker.Core/stryker-config.schema.json",
"stryker-config": {
"project": "MyApp.csproj",
"reporters": ["html", "progress", "cleartext"],
"mutation-level": "Standard",
"thresholds": {
"high": 80,
"low": 60,
"break": 50
},
"mutate": [
"src/Services/**/*.cs",
"!src/Services/Migrations/**/*.cs"
],
"ignore-mutations": [
"string",
"linq"
]
}
}
```
### Understanding Mutation Results
Stryker reports mutations in four categories:
| Status | Meaning | Action |
|--------|---------|--------|
| **Killed** | A test detected the mutation (failed) | Good -- test suite caught the defect |
| **Survived** | No test detected the mutation (all passed) | Gap -- add or strengthen tests |
| **No Coverage** | No test covers the mutated code | Gap -- add tests for this code |
| **Timeout** | Mutation caused an infinite loop or timeout | Usually killed (counts as detected) |
### Mutation Score
```
Mutation Score = Killed / (Killed + Survived + NoCoverage) * 100
```
A mutation score of 80%+ indicates a strong test suite. Below 60% suggests significant gaps.
### Example: Identifying Test Gaps
Given this production code:
```csharp
public class PricingService
{
public decimal CalculateDiscount(decimal price, CustomerTier tier) =>
tier switch
{
CustomerTier.Bronze => price * 0.05m,
CustomerTier.Silver => price * 0.10m,
CustomerTier.Gold => price * 0.15m,
CustomerTier.Platinum => price * 0.20m,
_ => 0m
};
}
```
If tests only verify `Gold` tier, Stryker generates mutations like:
- Replace `0.05m` with `0.06m` (survived -- no Bronze test)
- Replace `0.10m` with `0.11m` (survived -- no Silver test)
- Replace `0.15m` with `0.16m` (killed -- Gold test catches this)
- Replace `0.20m` with `0.21m` (survived -- no Platinum test)
- Replace `0m` with `1m` (survived -- no default test)
The HTML report highlights each surviving mutation with the exact code change, guiding where to add tests.
### Stryker Thresholds
```json
{
"thresholds": {
"high": 80, // Green: mutation score >= 80%
"low": 60, // Yellow: 60% <= mutation score < 80%
"break": 50 // Red: mutation score < 50% -> exit code 1
}
}
```
The `break` threshold causes Stryker to return a non-zero exit code, useful for CI gates.
---
## Flaky Test Detection
Flaky tests pass and fail intermittently without code changes. They erode trust in the test suite and slow development.
### Common Causes
| Cause | Symptom | Fix |
|-------|---------|-----|
| **Shared mutable state** | Tests fail when run in specific order | Use proper test isolation (see [skill:dotnet-testing] `references/xunit.md` for fixtures) |
| **Time-dependent logic** | Tests fail near midnight or at specific times | Inject `TimeProvider` (or `ISystemClock`) instead of using `DateTime.Now` |
| **Race conditions** | Tests fail intermittently under parallel execution | Use `ICollectionFixture` for shared resources; avoid shared static state |
| **External dependencies** | Tests fail when network/services unavailable | Mock external calls; use Testcontainers for infrastructure |
| **Port conflicts** | Tests fail when another process uses the same port | Use dynamic port allocation (WebApplicationFactory handles this) |
| **File system contention** | Tests fail under parallel execution | Use unique temp directories per test (see [skill:dotnet-testing] `references/xunit.md` `IAsyncLifetime` patterns) |
### Detecting Flaky Tests
#### Repeated Runs
```bash
# Run tests multiple times to surface flakiness
for i in $(seq 1 10); do
dotnet test --logger "trx;LogFileName=run-$i.trx" || echo "Run $i failed"
done
```
#### xUnit Conditional Skip
**xUnit v3** has built-in conditional skip via `Skip` on `[Fact]`:
```csharp
// xUnit v3 — built-in conditional skip
[Fact(Skip = "Requires external service")]
public async Task ExternalApi_ReturnsData()
{
var result = await _client.GetDataAsync();
Assert.NotEmpty(result);
}
// xUnit v3 — runtime skip via Assert.Skip
[Fact]
public async Task ExternalApi_ReturnsData()
{
if (!await IsServiceAvailable())
Assert.Skip("External service unavailable");
var result = await _client.GetDataAsync();
Assert.NotEmpty(result);
}
```
### Time-Dependent Tests
Replace `DateTime.Now`/`DateTime.UtcNow` with .NET 8's `TimeProvider`:
```csharp
// Production code
public class SubscriptionService(TimeProvider timeProvider)
{
public bool IsExpired(Subscription sub)
{
var now = timeProvider.GetUtcNow();
return sub.ExpiresAt < now;
}
}
// Test code
[Fact]
public void IsExpired_PastExpiry_ReturnsTrue()
{
var fakeTime = new FakeTimeProvider(
new DateTimeOffset(2025, 6, 15, 0, 0, 0, TimeSpan.Zero));
var service = new SubscriptionService(fakeTime);
var sub = new Subscription
{
ExpiresAt = new DateTimeOffset(2025, 6, 14, 0, 0, 0, TimeSpan.Zero)
};
Assert.True(service.IsExpired(sub));
}
[Fact]
public void IsExpired_FutureExpiry_ReturnsFalse()
{
var fakeTime = new FakeTimeProvider(
new DateTimeOffset(2025, 6, 15, 0, 0, 0, TimeSpan.Zero));
var service = new SubscriptionService(fakeTime);
var sub = new Subscription
{
ExpiresAt = new DateTimeOffset(2025, 6, 16, 0, 0, 0, TimeSpan.Zero)
};
Assert.False(service.IsExpired(sub));
}
```
**Note:** `FakeTimeProvider` is available in `Microsoft.Extensions.TimeProvider.Testing` (NuGet).
### Quarantine Strategy
When a flaky test cannot be fixed immediately:
```csharp
// Mark as skipped with a tracking issue
[Fact(Skip = "Flaky: tracking in #1234 -- race condition in event handler")]
public async Task EventHandler_ConcurrentEvents_ProcessesAll()
{
// ...
}
```
Do not delete flaky tests. Skip them with an issue reference and fix them systematically.
---
## Key Principles
- **Coverage is a lagging indicator, not a target.** High coverage does not guarantee good tests. A test suite with 90% coverage can still miss critical bugs if the assertions are weak.
- **Use CRAP scores to prioritize.** Focus testing effort on methods with high complexity and low coverage rather than chasing overall coverage percentage.
- **Run mutation testing on critical paths.** Mutation testing is computationally expensive. Focus on business-critical code (pricing, authentication, data validation) rather than running it on the entire codebase.
- **Fix flaky tests immediately or quarantine them.** A flaky test that remains in the suite trains developers to ignore failures, undermining the entire test suite's value.
- **Measure trends, not snapshots.** Track coverage and mutation scores over time. A declining trend indicates test quality erosion even if absolute numbers look acceptable.
- **Exclude generated code from coverage.** Migrations, generated clients, and scaffolded code inflate or deflate coverage numbers without reflecting actual test quality.
---
## Agent Gotchas
1. **Do not confuse `coverlet.collector` with `coverlet.msbuild`.** The `coverlet.collector` package uses the `--collect:"XPlat Code Coverage"` CLI flag. The `coverlet.msbuild` package uses `/p:CollectCoverage=true` MSBuild properties. Do not mix flags across packages -- they are independent integration points.
2. **Do not hardcode coverage result paths.** The GUID in `TestResults/<guid>/coverage.cobertura.xml` changes every run. Always use glob patterns (`**/TestResults/*/coverage.cobertura.xml`) when referencing coverage output files.
3. **Do not set coverage thresholds too high initially.** Starting with 90%+ thresholds on an existing project blocks all PRs. Begin with the current baseline and increase incrementally (e.g., 5% per quarter).
4. **Do not run Stryker.NET on the entire solution for CI.** Mutation testing is CPU-intensive. In CI, limit mutations to changed files (`--since:main`) or critical paths. Reserve full runs for nightly builds.
5. **Do not ignore survived mutations in trivial code.** While some survived mutations are in code that does not warrant testing (logging, `ToString()`), review each one. Configure `ignore-mutations` in `stryker-config.json` for categories you have consciously decided not to test.
6. **Do not use `[ExcludeFromCodeCoverage]` as a blanket fix for low coverage.** This attribute hides the problem rather than solving it. Use it only for genuinely untestable code (platform interop, generated code) and ensure the reason is documented.
---
## References
- [Coverlet GitHub repository](https://github.com/coverlet-coverage/coverlet)
- [ReportGenerator GitHub repository](https://github.com/danielpalme/ReportGenerator)
- [ReportGenerator usage guide](https://github.com/danielpalme/ReportGenerator/wiki)
- [Stryker.NET documentation](https://stryker-mutator.io/docs/stryker-net/introduction/)
- [Stryker.NET configuration](https://stryker-mutator.io/docs/stryker-net/configuration/)
- [TimeProvider in .NET 8](https://learn.microsoft.com/en-us/dotnet/api/system.timeprovider)
- [Microsoft.Extensions.TimeProvider.Testing](https://www.nuget.org/packages/Microsoft.Extensions.TimeProvider.Testing)
- [CRAP metric explanation](https://testing.googleblog.com/2011/02/this-code-is-crap.html)
references/testing-strategy.md# Testing Strategy
Decision framework for choosing the right test type, organizing test projects, and selecting test doubles in .NET applications. Covers unit vs integration vs E2E trade-offs with concrete criteria, naming conventions, and when to use mocks vs fakes vs stubs.
## Test Type Decision Tree
Use this decision tree to determine which test type fits a given scenario. Start at the top and follow the first matching criterion.
```
Does the code under test depend on external infrastructure?
(database, HTTP service, file system, message broker)
|
+-- YES --> Is the infrastructure behavior critical to correctness?
| |
| +-- YES --> Does it need the full application stack (middleware, auth, routing)?
| | |
| | +-- YES --> E2E / Functional Test
| | | (WebApplicationFactory or Playwright)
| | |
| | +-- NO --> Integration Test
| | (WebApplicationFactory or Testcontainers)
| |
| +-- NO --> Unit Test with test doubles
| (mock the infrastructure boundary)
|
+-- NO --> Is this pure logic (calculations, transformations, validation)?
|
+-- YES --> Unit Test (no test doubles needed)
|
+-- NO --> Unit Test with test doubles
(mock collaborator interfaces)
```
### Concrete Criteria by Test Type
| Test Type | Infrastructure | Speed | Scope | When to Use |
|-----------|---------------|-------|-------|-------------|
| **Unit** | None (mocked/faked) | <10ms per test | Single class/method | Pure logic, domain rules, value objects, transformations, validators |
| **Integration** | Real (DB, HTTP) | 100ms-5s per test | Multiple components | Repository queries, API contract verification, serialization round-trips, middleware behavior |
| **E2E / Functional** | Full stack | 1-30s per test | Entire request pipeline | Critical user flows, auth + routing + middleware combined, cross-cutting concern verification |
### Cost-Benefit Guidance
- **Prefer unit tests** for business logic. They run fast, pinpoint failures precisely, and have no infrastructure requirements.
- **Use integration tests** to verify infrastructure boundaries work correctly. A repository unit test with a mocked `DbContext` proves nothing about actual SQL generation -- use a real database via Testcontainers.
- **Use E2E tests sparingly** for critical paths only. They are slow, brittle, and expensive to maintain. Cover the happy path and one or two critical failure scenarios.
- **The testing pyramid is a guideline, not a rule.** Some applications (CRUD APIs with minimal logic) benefit from more integration tests than unit tests. Match the strategy to the application's complexity profile.
---
## Test Organization
### Project Naming Convention
Mirror the `src/` project structure under `tests/` with a suffix indicating test type:
```
MyApp/
src/
MyApp.Domain/
MyApp.Application/
MyApp.Api/
MyApp.Infrastructure/
tests/
MyApp.Domain.UnitTests/
MyApp.Application.UnitTests/
MyApp.Api.IntegrationTests/
MyApp.Api.FunctionalTests/
MyApp.Infrastructure.IntegrationTests/
```
- `*.UnitTests` -- isolated tests, no external dependencies
- `*.IntegrationTests` -- real infrastructure (database, HTTP, file system)
- `*.FunctionalTests` -- full application stack via `WebApplicationFactory`
See [skill:dotnet-testing] `references/add-testing.md` for creating these projects with proper package references and build configuration.
### Test Class Organization
One test class per production class. Place test files in a namespace that mirrors the production namespace:
```csharp
// Production: src/MyApp.Domain/Orders/OrderService.cs
// Test: tests/MyApp.Domain.UnitTests/Orders/OrderServiceTests.cs
namespace MyApp.Domain.UnitTests.Orders;
public class OrderServiceTests
{
// Group by method, then by scenario
}
```
For large production classes, split test classes by method:
```csharp
// OrderService_CreateTests.cs
// OrderService_CancelTests.cs
// OrderService_RefundTests.cs
```
---
## Test Naming Conventions
Use the `Method_Scenario_ExpectedBehavior` pattern. This reads naturally in test explorer output and makes failures self-documenting:
```csharp
public class OrderServiceTests
{
[Fact]
public void CalculateTotal_WithDiscountCode_AppliesPercentageDiscount()
{
// ...
}
[Fact]
public void CalculateTotal_WithExpiredDiscount_ThrowsInvalidOperationException()
{
// ...
}
[Fact]
public async Task SubmitOrder_WhenInventoryInsufficient_ReturnsOutOfStockError()
{
// ...
}
}
```
Alternative naming styles (choose one per project and stay consistent):
| Style | Example |
|-------|---------|
| `Method_Scenario_Expected` | `CalculateTotal_EmptyCart_ReturnsZero` |
| `Should_Expected_When_Scenario` | `Should_ReturnZero_When_CartIsEmpty` |
| `Given_When_Then` | `GivenEmptyCart_WhenCalculatingTotal_ThenReturnsZero` |
---
## Arrange-Act-Assert Pattern
Every test follows the AAA structure. Keep each section clearly separated:
```csharp
[Fact]
public async Task CreateOrder_WithValidItems_PersistsAndReturnsOrder()
{
// Arrange
var repository = new FakeOrderRepository();
var service = new OrderService(repository);
var request = new CreateOrderRequest
{
CustomerId = "cust-123",
Items = [new OrderItem("SKU-001", Quantity: 2, UnitPrice: 29.99m)]
};
// Act
var result = await service.CreateAsync(request);
// Assert
Assert.NotNull(result);
Assert.Equal("cust-123", result.CustomerId);
Assert.Single(result.Items);
Assert.True(repository.SavedOrders.ContainsKey(result.Id));
}
```
**Guideline:** If you cannot clearly label the three sections, the test may be doing too much. Split into multiple tests.
---
## Test Doubles: When to Use What
### Terminology
| Double Type | Behavior | State Verification | Use When |
|-------------|----------|-------------------|----------|
| **Stub** | Returns canned data | No | You need a dependency to return specific values so the code under test can proceed |
| **Mock** | Verifies interactions | Yes (interaction) | You need to verify that the code under test called a dependency in a specific way |
| **Fake** | Working implementation | Yes (state) | You need a lightweight but functional substitute (in-memory repository, in-memory message bus) |
| **Spy** | Records calls for later assertion | Yes (interaction) | You need to verify calls happened without prescribing them upfront |
### Decision Guidance
```
Do you need to verify HOW a dependency was called?
|
+-- YES --> Do you need a working implementation too?
| |
| +-- YES --> Spy (record calls on a fake)
| +-- NO --> Mock (NSubstitute / Moq)
|
+-- NO --> Do you need the dependency to DO something realistic?
|
+-- YES --> Fake (in-memory implementation)
+-- NO --> Stub (return canned values)
```
### Example: Stub vs Mock vs Fake
```csharp
// STUB: Returns canned data -- verifying the code under test's logic
var priceService = Substitute.For<IPriceService>();
priceService.GetPriceAsync("SKU-001").Returns(29.99m); // canned return
var total = await calculator.CalculateTotalAsync(items);
Assert.Equal(59.98m, total); // assert on the result, not the call
// MOCK: Verifies interaction -- ensuring a side effect happened
var emailSender = Substitute.For<IEmailSender>();
await orderService.CompleteAsync(order);
await emailSender.Received(1).SendAsync( // assert on the call
Arg.Is<string>(to => to == order.CustomerEmail),
Arg.Any<string>(),
Arg.Any<string>());
// FAKE: In-memory implementation -- realistic behavior without infrastructure
public class FakeOrderRepository : IOrderRepository
{
public Dictionary<Guid, Order> Orders { get; } = new();
public Task<Order?> GetByIdAsync(Guid id, CancellationToken ct = default)
=> Task.FromResult(Orders.GetValueOrDefault(id));
public Task SaveAsync(Order order, CancellationToken ct = default)
{
Orders[order.Id] = order;
return Task.CompletedTask;
}
}
```
### When to Prefer Fakes Over Mocks
- **Domain-heavy applications:** Fakes give more realistic behavior for complex interactions. An in-memory repository catches bugs that mocks miss (e.g., duplicate key violations).
- **Overuse of mocks is a test smell.** If a test has more mock setup than actual assertions, consider whether a fake would be clearer and more maintainable.
- **Integration boundaries are better tested with real infrastructure** via [skill:dotnet-testing] `references/integration-testing.md` than with mocks. A mocked `DbContext` does not verify that your LINQ translates to valid SQL.
---
## Testing Anti-Patterns
### 1. Testing Implementation Details
```csharp
// BAD: Breaks when refactoring internals
repository.Received(1).GetByIdAsync(Arg.Is<Guid>(id => id == orderId));
repository.Received(1).SaveAsync(Arg.Any<Order>());
// ... five more Received() calls verifying the exact call sequence
// GOOD: Test the observable outcome
var result = await service.ProcessAsync(orderId);
Assert.Equal(OrderStatus.Completed, result.Status);
```
### 2. Excessive Mock Setup
```csharp
// BAD: Mock setup is longer than the actual test
var repo = Substitute.For<IOrderRepository>();
var pricing = Substitute.For<IPricingService>();
var inventory = Substitute.For<IInventoryService>();
var shipping = Substitute.For<IShippingService>();
var notification = Substitute.For<INotificationService>();
var audit = Substitute.For<IAuditService>();
// ... 20 lines of .Returns() setup
// BETTER: Use a builder or fake that encapsulates setup
var fixture = new OrderServiceFixture()
.WithOrder(testOrder)
.WithPrice("SKU-001", 29.99m);
var result = await fixture.Service.ProcessAsync(testOrder.Id);
```
### 3. Non-Deterministic Tests
Tests must not depend on system clock, random values, or external network. Inject abstractions:
```csharp
// BAD: Uses DateTime.UtcNow directly
public bool IsExpired() => ExpiresAt < DateTime.UtcNow;
// GOOD: Inject TimeProvider (.NET 8+)
public bool IsExpired(TimeProvider time) => ExpiresAt < time.GetUtcNow();
// In test
var fakeTime = new FakeTimeProvider(new DateTimeOffset(2025, 6, 15, 0, 0, 0, TimeSpan.Zero));
Assert.True(order.IsExpired(fakeTime));
```
---
## Key Principles
- **Test behavior, not implementation.** Assert on observable outcomes (return values, state changes, published events), not internal method calls.
- **One logical assertion per test.** Multiple `Assert` calls are fine if they verify one logical concept (e.g., all properties of a returned object). Multiple unrelated assertions indicate the test should be split.
- **Keep tests independent.** No test should depend on another test's execution or ordering. Use fresh fixtures for each test.
- **Name tests so failures are self-documenting.** A failing test name should tell you what broke without reading the test body.
- **Match test type to risk.** High-risk code (payments, auth) deserves integration and E2E coverage. Low-risk code (simple mapping) needs only unit tests.
- **Use `TimeProvider` for time-dependent logic** (.NET 8+). It is the framework-provided abstraction; do not create custom `IClock` interfaces.
---
## Agent Gotchas
1. **Do not mock types you do not own.** Mocking `HttpClient`, `DbContext`, or framework types leads to brittle tests that do not reflect real behavior. Use `WebApplicationFactory` or Testcontainers instead -- see [skill:dotnet-testing] `references/integration-testing.md`.
2. **Do not create test projects without checking for existing structure.** Run [skill:dotnet-tooling] `references/project-analysis.md` first; duplicating test infrastructure causes build conflicts.
3. **Do not use `Thread.Sleep` in tests.** Use `Task.Delay` with a cancellation token, or better, use `FakeTimeProvider.Advance()` to control time deterministically.
4. **Do not test private methods directly.** If a private method needs its own tests, it should be extracted into its own class. Test through the public API.
5. **Do not hard-code connection strings in integration tests.** Use Testcontainers for disposable infrastructure or `WebApplicationFactory` for in-process testing -- see [skill:dotnet-testing] `references/integration-testing.md`.
---
## References
- [.NET Testing Best Practices](https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-best-practices)
- [Unit testing C# with xUnit](https://learn.microsoft.com/en-us/dotnet/core/testing/unit-testing-with-dotnet-test)
- [Integration tests in ASP.NET Core](https://learn.microsoft.com/en-us/aspnet/core/test/integration-tests)
- [NSubstitute documentation](https://nsubstitute.github.io/help/getting-started/)
- [TimeProvider in .NET 8](https://learn.microsoft.com/en-us/dotnet/api/system.timeprovider)
references/ui-testing-core.md# UI Testing Core
Core UI testing patterns applicable across .NET UI frameworks (Blazor, MAUI, Uno Platform). Covers the page object model for maintainable test structure, test selector strategies for reliable element identification, async wait patterns for non-deterministic UI, and accessibility testing approaches.
**Version assumptions:** .NET 8.0+ baseline. Framework-specific details are delegated to dedicated skills.
## Page Object Model
The page object model (POM) encapsulates page structure and interactions behind a class, isolating tests from UI implementation details. When the UI changes, only the page object needs updating -- not every test that touches that page.
### Structure
```
PageObjects/
LoginPage.cs -- login form interactions
DashboardPage.cs -- dashboard navigation + widgets
OrderListPage.cs -- order list filtering + selection
Components/
NavigationMenu.cs -- shared nav component
ConfirmDialog.cs -- reusable confirmation modal
```
### Example: Generic Page Object Base
```csharp
/// <summary>
/// Base class for page objects. Subclass per framework:
/// Playwright uses IPage, bUnit uses IRenderedComponent, Appium uses AppiumDriver.
/// </summary>
public abstract class PageObjectBase<TDriver>
{
protected TDriver Driver { get; }
protected PageObjectBase(TDriver driver)
{
Driver = driver;
}
/// <summary>
/// Verifies the page/component is in the expected state after navigation.
/// Call this in the constructor or after navigation to fail fast on wrong pages.
/// </summary>
protected abstract void VerifyLoaded();
}
```
### Example: Playwright Page Object
```csharp
public class LoginPage : PageObjectBase<IPage>
{
public LoginPage(IPage page) : base(page)
{
VerifyLoaded();
}
protected override void VerifyLoaded()
{
// Fail fast if not on the login page
Driver.WaitForSelectorAsync("[data-testid='login-form']")
.GetAwaiter().GetResult();
}
public async Task<DashboardPage> LoginAsync(string email, string password)
{
await Driver.FillAsync("[data-testid='email-input']", email);
await Driver.FillAsync("[data-testid='password-input']", password);
await Driver.ClickAsync("[data-testid='login-button']");
await Driver.WaitForURLAsync("**/dashboard");
return new DashboardPage(Driver);
}
public async Task<string> GetErrorMessageAsync()
{
var error = Driver.Locator("[data-testid='login-error']");
return await error.TextContentAsync() ?? "";
}
}
// Usage in test
[Fact]
public async Task Login_ValidCredentials_RedirectsToDashboard()
{
var loginPage = new LoginPage(Page);
var dashboard = await loginPage.LoginAsync("user@example.com", "P@ssw0rd!");
Assert.NotNull(dashboard);
}
```
### Page Object Principles
- **Return the next page object from navigation actions.** `LoginAsync` returns `DashboardPage`, guiding test authors through the application flow.
- **Never expose raw selectors from page objects.** Tests call `LoginAsync()`, not `ClickAsync("#submit")`.
- **Keep assertions in tests, not page objects.** Page objects provide data (e.g., `GetErrorMessageAsync()`); tests make assertions on that data.
- **Compose page objects from reusable components.** A `NavigationMenu` component object can be embedded in every page that has a nav bar.
---
## Test Selector Strategies
Selectors determine how tests find UI elements. Fragile selectors are the leading cause of flaky UI tests.
### Selector Priority (Most to Least Reliable)
| Priority | Selector Type | Example | Reliability |
|----------|--------------|---------|-------------|
| 1 | **`data-testid`** | `[data-testid='submit-btn']` | Highest -- survives CSS/layout changes |
| 2 | **Accessibility role + name** | `GetByRole(AriaRole.Button, new() { Name = "Submit" })` | High -- tied to visible behavior |
| 3 | **Label text** | `GetByLabel("Email address")` | High -- changes when copy changes |
| 4 | **Placeholder text** | `GetByPlaceholder("Enter email")` | Medium -- often localized |
| 5 | **CSS class** | `.btn-primary` | Low -- changes with styling |
| 6 | **XPath / DOM structure** | `//div[3]/button[1]` | Lowest -- breaks on any layout change |
### Adding Test IDs
Add `data-testid` attributes to elements that tests interact with. They are invisible to users and stable across refactors:
**Blazor:**
```razor
<button data-testid="submit-order" @onclick="SubmitOrder">Place Order</button>
<input data-testid="search-input" @bind="SearchTerm" />
```
**MAUI XAML:**
```xml
<Button AutomationId="submit-order" Text="Place Order" Clicked="OnSubmit" />
<Entry AutomationId="search-input" Text="{Binding SearchTerm}" />
```
**Uno Platform XAML:**
```xml
<Button AutomationProperties.AutomationId="submit-order" Content="Place Order" />
```
### Selector Anti-Patterns
```csharp
// BAD: Tied to CSS implementation
await page.ClickAsync(".MuiButton-root.MuiButton-containedPrimary");
// BAD: Tied to DOM structure
await page.ClickAsync("div > form > div:nth-child(3) > button");
// BAD: Tied to dynamic content
await page.ClickAsync($"text=Order #{orderId}");
// GOOD: Stable test identifier
await page.ClickAsync("[data-testid='submit-order']");
// GOOD: Accessibility-driven (Playwright)
await page.GetByRole(AriaRole.Button, new() { Name = "Place Order" }).ClickAsync();
```
---
## Async Wait Strategies
UI tests deal with asynchronous rendering, network requests, and animations. Hardcoded delays cause flaky tests and slow suites.
### Wait Strategy Decision Tree
```
Is the element already in the DOM?
|
+-- YES --> Is it visible and actionable?
| |
| +-- YES --> Interact immediately
| +-- NO --> Wait for visibility/enabled state
|
+-- NO --> Wait for element to appear in DOM
|
Is it loaded via network request?
|
+-- YES --> Wait for network idle or specific API response
+-- NO --> Wait for render cycle to complete
```
### Framework-Specific Wait Patterns
**Playwright (browser-based):**
```csharp
// Auto-waiting: Playwright waits for actionability by default
await page.ClickAsync("[data-testid='submit']"); // waits until visible + enabled
// Explicit wait for network-loaded content
await page.WaitForResponseAsync(
response => response.Url.Contains("/api/orders") && response.Status == 200);
// Wait for element state
await page.Locator("[data-testid='results']")
.WaitForAsync(new() { State = WaitForSelectorState.Visible });
// Wait for specific text content
await Expect(page.Locator("[data-testid='status']")).ToHaveTextAsync("Completed");
```
**bUnit (Blazor component testing):**
```csharp
// Wait for async state changes to render
var cut = RenderComponent<OrderList>();
// Wait for component to finish async operations
cut.WaitForState(() => cut.Instance.Orders.Count > 0,
timeout: TimeSpan.FromSeconds(5));
// Wait for specific markup
cut.WaitForAssertion(() =>
Assert.NotEmpty(cut.FindAll("[data-testid='order-row']")),
timeout: TimeSpan.FromSeconds(5));
```
### Wait Anti-Patterns
```csharp
// BAD: Hardcoded delay -- slow and still flaky
await Task.Delay(3000);
await page.ClickAsync("[data-testid='results']");
// BAD: Polling with Thread.Sleep
while (!element.IsVisible)
{
Thread.Sleep(100); // blocks thread, no timeout safety
}
// GOOD: Framework-native wait
await page.Locator("[data-testid='results']")
.WaitForAsync(new() { State = WaitForSelectorState.Visible, Timeout = 5000 });
// GOOD: Assertion with retry (Playwright)
await Expect(page.Locator("[data-testid='count']")).ToHaveTextAsync("5");
```
---
## Accessibility Testing
Accessibility testing verifies that UI components are usable by people with disabilities and compatible with assistive technologies. Automated checks catch common issues; manual review is still needed for subjective criteria.
### Automated Accessibility Checks with Playwright
```csharp
// NuGet: Deque.AxeCore.Playwright
[Fact]
public async Task HomePage_PassesAccessibilityAudit()
{
await Page.GotoAsync("/");
var results = await Page.RunAxe();
Assert.Empty(results.Violations);
}
[Fact]
public async Task OrderForm_NoAccessibilityViolations()
{
await Page.GotoAsync("/orders/new");
// Scope to specific component
var form = Page.Locator("[data-testid='order-form']");
var results = await Page.RunAxe(new AxeRunOptions
{
// Focus on WCAG 2.1 AA rules
RunOnly = new RunOnlyOptions
{
Type = "tag",
Values = ["wcag2a", "wcag2aa", "wcag21aa"]
}
});
// Report violations with details for debugging
foreach (var violation in results.Violations)
{
// Log: violation.Id, violation.Description, violation.Nodes
}
Assert.Empty(results.Violations);
}
```
### Accessibility Checklist for UI Tests
| Check | How to Test | Tool |
|-------|------------|------|
| **Color contrast** | Automated axe-core rule | Deque.AxeCore.Playwright |
| **Keyboard navigation** | Tab through all interactive elements | Playwright `page.Keyboard` |
| **ARIA labels** | Verify `aria-label` / `aria-labelledby` present | Playwright locators + assertions |
| **Focus management** | Verify focus moves to dialogs/modals | Playwright `page.Locator(':focus')` |
| **Screen reader text** | Verify `aria-live` regions update | Manual + assertion on ARIA attributes |
### Keyboard Navigation Test Example
```csharp
[Fact]
public async Task OrderForm_TabOrder_FollowsLogicalSequence()
{
await Page.GotoAsync("/orders/new");
// Tab through form fields and verify focus order
await Page.Keyboard.PressAsync("Tab");
await Expect(Page.Locator("[data-testid='customer-name']")).ToBeFocusedAsync();
await Page.Keyboard.PressAsync("Tab");
await Expect(Page.Locator("[data-testid='customer-email']")).ToBeFocusedAsync();
await Page.Keyboard.PressAsync("Tab");
await Expect(Page.Locator("[data-testid='order-items']")).ToBeFocusedAsync();
// Verify Enter submits the form
await Page.Keyboard.PressAsync("Tab"); // focus submit button
await Expect(Page.Locator("[data-testid='submit-order']")).ToBeFocusedAsync();
}
```
---
## Key Principles
- **Use the page object model for any UI test suite with more than a handful of tests.** The upfront cost pays for itself quickly in reduced maintenance.
- **Prefer `data-testid` or accessibility-based selectors over CSS or DOM-structure selectors.** Stable selectors are the single most effective defense against flaky tests.
- **Never use `Thread.Sleep` or `Task.Delay` as a wait strategy.** Use framework-native waits that poll for conditions with timeouts.
- **Run accessibility checks as part of the standard test suite,** not as a separate audit. Catching violations early prevents accessibility debt.
- **Keep page objects framework-agnostic where possible.** The patterns (POM, selector strategy, wait patterns) are universal; only the driver API changes between Playwright, bUnit, and Appium.
---
## Agent Gotchas
1. **Do not add `data-testid` attributes to production code without team agreement.** Some teams strip them in production builds; others keep them. Check the project's conventions first.
2. **Do not use `WaitForTimeout` (hardcoded delay) in Playwright tests.** It masks timing issues and makes tests slow. Use `WaitForSelectorAsync`, `Expect(...).ToBeVisibleAsync()`, or `WaitForResponseAsync` instead.
3. **Do not assert on element count without waiting for the list to load.** `FindAll("[data-testid='row']").Count` returns zero if the component has not finished rendering. Use `WaitForState` or `WaitForAssertion` first.
4. **Do not skip accessibility testing because "it's not a requirement."** WCAG compliance is increasingly a legal requirement. Automated checks catch the low-hanging fruit at near-zero cost.
5. **Do not create deeply nested page objects.** If a page object has page objects inside page objects, flatten the hierarchy. One level of component composition (page -> components) is sufficient.
---
## References
- [Page Object Model (Martin Fowler)](https://martinfowler.com/bliki/PageObject.html)
- [Playwright Locators Best Practices](https://playwright.dev/dotnet/docs/locators)
- [axe-core Accessibility Rules](https://dequeuniversity.com/rules/axe/)
- [WCAG 2.1 Guidelines](https://www.w3.org/WAI/WCAG21/quickref/)
- [Testing Library Guiding Principles](https://testing-library.com/docs/guiding-principles)
references/xunit.md# xUnit
xUnit v3 testing framework features for .NET. Covers `[Fact]` and `[Theory]` attributes, test fixtures (`IClassFixture`, `ICollectionFixture`), parallel execution configuration, `IAsyncLifetime` for async setup/teardown, custom assertions, and xUnit analyzers.
**Version assumptions:** xUnit v3 primary (.NET 8.0+ baseline). Where v3 behavior differs from v2, compatibility notes are provided inline.
## Facts and Theories
### `[Fact]` -- Single Test Case
Use `[Fact]` for tests with no parameters:
```csharp
public class DiscountCalculatorTests
{
[Fact]
public void Apply_NegativePercentage_ThrowsArgumentOutOfRangeException()
{
var calculator = new DiscountCalculator();
var ex = Assert.Throws<ArgumentOutOfRangeException>(
() => calculator.Apply(100m, percentage: -5));
Assert.Equal("percentage", ex.ParamName);
}
}
```
### `[Theory]` -- Parameterized Tests
Use `[Theory]` to run the same test logic with different inputs.
#### `[InlineData]`
Best for simple value types:
```csharp
[Theory]
[InlineData(100, 10, 90)]
[InlineData(200, 25, 150)]
[InlineData(50, 0, 50)]
public void Apply_VariousInputs_ReturnsExpectedPrice(
decimal price, decimal percentage, decimal expected)
{
var calculator = new DiscountCalculator();
Assert.Equal(expected, calculator.Apply(price, percentage));
}
```
#### `[MemberData]` with `TheoryData<T>`
Best for complex data or shared datasets:
```csharp
public class OrderValidatorTests
{
public static TheoryData<Order, bool> ValidationCases => new()
{
{ new Order { Items = [new("SKU-1", 1)], CustomerId = "C1" }, true },
{ new Order { Items = [], CustomerId = "C1" }, false },
};
[Theory]
[MemberData(nameof(ValidationCases))]
public void IsValid_VariousOrders_ReturnsExpected(Order order, bool expected)
{
Assert.Equal(expected, new OrderValidator().IsValid(order));
}
}
```
#### `[ClassData]` (xUnit v3)
For data shared across multiple test classes. v3 uses `TheoryDataRow<T>` for strongly-typed rows (v2 used `IEnumerable<object[]>`):
```csharp
public class CurrencyConversionData : IEnumerable<TheoryDataRow<string, string, decimal>>
{
public IEnumerator<TheoryDataRow<string, string, decimal>> GetEnumerator()
{
yield return new("USD", "EUR", 0.92m);
yield return new("GBP", "USD", 1.27m);
}
IEnumerator IEnumerable.GetEnumerator() => GetEnumerator();
}
[Theory]
[ClassData(typeof(CurrencyConversionData))]
public void Convert_KnownPairs_ReturnsExpectedRate(
string from, string to, decimal expectedRate)
{
Assert.Equal(expectedRate, new CurrencyConverter().GetRate(from, to), precision: 2);
}
```
---
## Fixtures: Shared Setup and Teardown
### `IClassFixture<T>` -- Shared Per Test Class
Use when multiple tests in the same class share an expensive resource:
```csharp
public class DatabaseFixture : IAsyncLifetime
{
public string ConnectionString { get; private set; } = "";
// xUnit v3: IAsyncLifetime returns ValueTask (v2 returns Task)
public ValueTask InitializeAsync()
{
ConnectionString = $"Host=localhost;Database=test_{Guid.NewGuid():N}";
return ValueTask.CompletedTask;
}
public ValueTask DisposeAsync() => ValueTask.CompletedTask;
}
public class OrderRepositoryTests : IClassFixture<DatabaseFixture>
{
private readonly DatabaseFixture _db;
public OrderRepositoryTests(DatabaseFixture db) => _db = db;
[Fact]
public async Task GetById_ExistingOrder_ReturnsOrder()
{
var repo = new OrderRepository(_db.ConnectionString);
Assert.NotNull(await repo.GetByIdAsync(KnownOrderId));
}
}
```
### `ICollectionFixture<T>` -- Shared Across Test Classes
Use when multiple test classes need the same expensive resource:
```csharp
[CollectionDefinition("Database")]
public class DatabaseCollection : ICollectionFixture<DatabaseFixture> { }
[Collection("Database")]
public class OrderRepositoryTests(DatabaseFixture db)
{
[Fact]
public async Task Insert_ValidOrder_Persists() { /* uses db */ }
}
[Collection("Database")]
public class CustomerRepositoryTests(DatabaseFixture db) { }
```
---
## Parallel Execution
xUnit runs test classes within a collection sequentially but runs different collections in parallel. Each test class without an explicit `[Collection]` is its own implicit collection.
### Disable Parallelism for Specific Tests
```csharp
[CollectionDefinition("Sequential", DisableParallelization = true)]
public class SequentialCollection { }
[Collection("Sequential")]
public class StatefulServiceTests { /* runs sequentially */ }
```
### Assembly-Level Configuration
Create `xunit.runner.json` in the test project root (copy to output via `<Content Include="xunit.runner.json" CopyToOutputDirectory="PreserveNewest" />`):
```json
{
"$schema": "https://xunit.net/schema/current/xunit.runner.schema.json",
"parallelizeAssembly": false,
"parallelizeTestCollections": true,
"maxParallelThreads": 4
}
```
---
## Custom Assertions and Assert.Multiple
Create domain-specific assertions for cleaner test code:
```csharp
public static class OrderAssert
{
public static void HasStatus(Order order, OrderStatus expected)
{
Assert.NotNull(order);
if (order.Status != expected)
throw Xunit.Sdk.EqualException.ForMismatchedValues(expected, order.Status);
}
}
```
Use `Assert.Multiple` (xUnit v3 only) to evaluate all assertions even if one fails:
```csharp
[Fact]
public void CreateOrder_ValidRequest_SetsAllProperties()
{
var order = OrderFactory.Create(request);
Assert.Multiple(
() => Assert.Equal("cust-123", order.CustomerId),
() => Assert.Equal(OrderStatus.Pending, order.Status),
() => Assert.NotEqual(Guid.Empty, order.Id)
);
}
```
---
## xUnit Analyzers
The `xunit.analyzers` package (included with xUnit v3) catches common mistakes at compile time. Key rules:
| Rule | What it catches |
|------|----------------|
| `xUnit1025` | Duplicate `[InlineData]` within a `[Theory]` |
| `xUnit2000` | Constants should be the expected (first) argument in `Assert.Equal` |
| `xUnit2013` | Do not use equality check to verify collection size (use `Assert.Single`, `Assert.Empty`) |
Suppress per-project in `.editorconfig`:
```ini
[tests/**.cs]
dotnet_diagnostic.xUnit1004.severity = suggestion
```
---
## Key Principles
- **One fact per `[Fact]`, one concept per `[Theory]`.** Split fundamentally different scenarios into separate methods.
- **Use `IClassFixture` for expensive shared resources** within a class, `ICollectionFixture` when multiple classes share the same resource.
- **Do not disable parallelism globally.** Group tests sharing mutable state into named collections instead.
- **Use `IAsyncLifetime` for async setup/teardown** instead of constructors and `IDisposable`.
- **Keep test data close to the test.** Prefer `[InlineData]` for simple cases, `[MemberData]`/`[ClassData]` only when data is complex or shared.
---
## Agent Gotchas
1. **Do not use constructor-injected `ITestOutputHelper` in static methods.** It is per-test-instance; store in an instance field.
2. **Fixture classes must be `public` with a public parameterless constructor** (or `IAsyncLifetime`). Non-public fixtures cause silent failures.
3. **Do not mix `[Fact]` and `[Theory]` on the same method.** A method is either a fact or a theory.
4. **Async test methods must return `Task` or `ValueTask`, never `async void`.** `async void` tests report false success.
5. **`[Collection]` without a matching `[CollectionDefinition]`** silently creates an implicit collection with default behavior.
---
## References
- [xUnit Documentation](https://xunit.net/)
- [xUnit v3 migration guide](https://xunit.net/docs/getting-started/v3/migration)
- [xUnit analyzers](https://xunit.net/xunit.analyzers/rules/)
- [Shared context in xUnit](https://xunit.net/docs/shared-context)
SKILL.md--- name: dotnet-testing description: Defines .NET test strategy and implementation patterns across xUnit v3 (Facts, Theories, fixtures, IAsyncLifetime), integration testing (WebApplicationFactory, Testcontainers), Aspire testing (DistributedApplicationTestingBuilder), snapshot testing (Verify, scrubbing), Playwright E2E browser automation, BenchmarkDotNet microbenchmarks, code coverage (Coverlet), mutation testing (Stryker.NET), UI testing (page objects, selectors), and AOT WASM test compilation. Spans 13 topic areas. Do not use for production API architecture or CI workflow authoring. license: MIT user-invocable: false --- # dotnet-testing ## Overview Testing strategy, frameworks, and quality tooling for .NET applications. This consolidated skill spans 13 topic areas. Load the appropriate companion file from `references/` based on the routing table below. Baseline dependency: `references/testing-strategy.md` defines the unit vs integration vs E2E decision tree and test doubles selection that inform all testing decisions. Load it by default whenever a testing approach needs to be chosen. Most-shared companion: `references/xunit.md` covers xUnit v3 framework features used by integration, snapshot, and UI testing companions. ## Routing Table | Topic | Keywords | Description | Companion File | |-------|----------|-------------|----------------| | Strategy | unit vs integration vs E2E, test doubles | Unit vs integration vs E2E decision tree, test doubles selection | references/testing-strategy.md | | xUnit | Facts, Theories, fixtures, parallelism | xUnit v3 Facts, Theories, fixtures, parallelism, IAsyncLifetime | references/xunit.md | | Integration | WebApplicationFactory, Testcontainers, Aspire | WebApplicationFactory, Testcontainers, Aspire, database fixtures | references/integration-testing.md | | Snapshot | Verify, scrubbing, API responses | Verify library, scrubbing, custom converters, HTTP response snapshots | references/snapshot-testing.md | | Playwright | E2E browser, CI caching, trace viewer | Playwright E2E browser automation, CI caching, trace viewer, codegen | references/playwright.md | | BenchmarkDotNet | microbenchmarks, memory diagnosers | BenchmarkDotNet microbenchmarks, memory diagnosers, baselines | references/benchmarkdotnet.md | | CI benchmarking | threshold alerts, baseline tracking | CI benchmark regression detection, threshold alerts, baseline tracking | references/ci-benchmarking.md | | Test quality | Coverlet, Stryker.NET, flaky tests | Coverlet code coverage, Stryker.NET mutation testing, flaky tests | references/test-quality.md | | Add testing | scaffold xUnit project, coverlet, layout | Scaffold xUnit project, coverlet setup, directory layout | references/add-testing.md | | Slopwatch | LLM reward hacking detection | Slopwatch CLI for LLM reward hacking detection | references/slopwatch.md | | AOT WASM | Blazor/Uno WASM AOT, size, lazy loading | Blazor/Uno WASM AOT compilation, size vs speed, lazy loading, Brotli | references/aot-wasm.md | | UI testing core | page objects, selectors, async waits | Page object model, test selectors, async waits, accessibility testing | references/ui-testing-core.md | | Aspire testing | DistributedApplicationTestingBuilder, Aspire test host | Aspire test host, service HTTP clients, resource health | references/aspire-testing.md | ## Scope - Test strategy and architecture (unit, integration, E2E) - xUnit v3 test authoring - Integration testing (WebApplicationFactory, Testcontainers) - E2E browser testing (Playwright) - Snapshot testing (Verify) - Benchmarking (BenchmarkDotNet, CI gating) - Quality (coverage, mutation testing) - Cross-framework UI testing patterns - Test scaffolding ## Out of scope - UI framework-specific testing (bUnit, Appium) -> `dotnet-ui` - CI/CD pipeline configuration -> `dotnet-devops` - Performance profiling -> `dotnet-tooling`