qa-testing-strategy

어떤 스킬인가요?

Risk-based test strategy for software delivery. Use when defining coverage, setting CI gates, managing flaky tests, choosing test layers, or establishing release criteria.

설치 방법

프로젝트 폴더에서 아래 명령어를 실행하고, 설치할 에이전트를 선택하세요.

설치 명령을 직접 실행해야 적용됩니다. 지원 에이전트와 필요한 권한·라이선스는 제작자의 안내를 확인하세요.

스킬 원문 파일

interface:
  display_name: "Testing Strategy"
  short_description: "Risk-based test strategy for software delivery"
  default_prompt: "Use $qa-testing-strategy for Risk-based test strategy for software delivery. Use when defining coverage, setting CI gates, managing flaky tests, choosing test layers, or establishing release criteria."

# Automation Pipeline Template

- **Triggering events:** PR open/update, nightly, release branch, hotfix path  
- **Stages:** Lint → unit → component/contract → integration → E2E → performance → security scans  
- **Parallelization:** Which suites can parallelize; shard strategy; caching plan  
- **Environment setup:** Containers/services required, seeding scripts, secrets handling  
- **Quality gates:** Required checks, coverage thresholds, allowed flake rate, blocking vs warning jobs  
- **Artifacts:** Test reports, coverage, screenshots/videos, traces, SBOMs  
- **Rollbacks:** What to do on failure; auto-revert, feature flag toggles, chat notifications  
- **Governance:** Owners, escalation path, maintenance cadence for dependencies and flaky tests  

# BDD Testing Template: Cucumber & Gherkin

Use this template for behavior-driven development (BDD) with Cucumber and Gherkin syntax to create executable specifications.

## Why BDD

**Benefits**:
- Living documentation (scenarios are always up-to-date)
- Collaboration between technical and non-technical stakeholders
- Clear acceptance criteria before development
- Executable specifications
- Shared understanding of requirements

**When to use**: Acceptance tests, E2E critical paths, stakeholder-facing features

**When NOT to use**: Unit tests (use code directly), implementation details

## Basic Gherkin Syntax

```gherkin
# features/user-login.feature
Feature: User Login
  As a registered user
  I want to log in to my account
  So that I can access my personalized dashboard

  Background:
    Given the application is running
    And I am on the login page

  Scenario: Successful login with valid credentials
    When I enter email "user@example.com"
    And I enter password "SecurePass123"
    And I click the "Login" button
    Then I should see my dashboard
    And I should see "Welcome back, John"

  Scenario: Failed login with invalid password
    When I enter email "user@example.com"
    And I enter password "WrongPassword"
    And I click the "Login" button
    Then I should see an error "Invalid credentials"
    And I should remain on the login page

  Scenario: Account lockout after multiple failed attempts
    When I enter email "user@example.com"
    And I enter password "WrongPassword"
    And I click the "Login" button 3 times
    Then I should see an error "Account temporarily locked"
    And I should not be able to log in for 15 minutes
```

## Step Definitions (TypeScript)

```typescript
// step-definitions/login.steps.ts
import { Given, When, Then, Before, After } from '@cucumber/cucumber'
import { expect } from '@playwright/test'
import { LoginPage } from '../pages/login.page'

let page: Page
let loginPage: LoginPage

Before(async function () {
  page = await this.browser.newPage()
  loginPage = new LoginPage(page)
})

After(async function () {
  await page.close()
})

Given('the application is running', async function () {
  // Verify app health endpoint
  const response = await page.request.get('https://api.example.com/health')
  expect(response.status()).toBe(200)
})

Given('I am on the login page', async function () {
  await page.goto('/login')
  await expect(page.getByRole('heading', { name: 'Login' })).toBeVisible()
})

When('I enter email {string}', async function (email: string) {
  await page.getByLabel('Email').fill(email)
})

When('I enter password {string}', async function (password: string) {
  await page.getByLabel('Password').fill(password)
})

When('I click the {string} button', async function (buttonText: string) {
  await page.getByRole('button', { name: buttonText }).click()
})

When('I click the {string} button {int} times', async function (buttonText: string, times: number) {
  for (let i = 0; i < times; i++) {
    await page.getByRole('button', { name: buttonText }).click()
    await page.waitForTimeout(1000)
  }
})

Then('I should see my dashboard', async function () {
  await expect(page).toHaveURL('/dashboard')
  await expect(page.getByTestId('dashboard')).toBeVisible()
})

Then('I should see {string}', async function (text: string) {
  await expect(page.getByText(text)).toBeVisible()
})

Then('I should see an error {string}', async function (errorMessage: string) {
  await expect(page.getByRole('alert')).toContainText(errorMessage)
})

Then('I should remain on the login page', async function () {
  await expect(page).toHaveURL('/login')
})

Then('I should not be able to log in for {int} minutes', async function (minutes: number) {
  // Store context for future validation
  this.lockoutDuration = minutes
  const lockoutMessage = await page.getByTestId('lockout-message').textContent()
  expect(lockoutMessage).toContain(`${minutes} minutes`)
})
```

## Scenario Outlines (Data-Driven Tests)

```gherkin
Feature: Shopping Cart

  Scenario Outline: Apply discount codes
    Given I have "<item>" in my cart with price <price>
    When I apply discount code "<code>"
    Then the total should be <total>
    And I should see discount message "<message>"

    Examples:
      | item      | price | code     | total | message                |
      | Laptop    | 1000  | SAVE10   | 900   | 10% discount applied   |
      | Mouse     | 50    | SAVE10   | 45    | 10% discount applied   |
      | Laptop    | 1000  | SAVE50   | 500   | 50% discount applied   |
      | Mouse     | 50    | INVALID  | 50    | Invalid discount code  |
      | Keyboard  | 100   |          | 100   | No discount applied    |

  Scenario Outline: Validate product search
    Given I am on the products page
    When I search for "<query>"
    Then I should see <result_count> results
    And the first result should be "<first_result>"

    Examples:
      | query     | result_count | first_result       |
      | laptop    | 15           | MacBook Pro        |
      | mouse     | 42           | Logitech MX Master |
      | keyboard  | 28           | Mechanical Keyboard|
      | monitor   | 31           | Dell UltraSharp    |
      | invalid   | 0            |                    |
```

## Tags for Organization

```gherkin
@smoke @critical
Feature: User Authentication

  @happy-path
  Scenario: Successful login
    # ...

  @error-handling
  Scenario: Invalid credentials
    # ...

  @security @slow
  Scenario: Account lockout
    # ...

  @wip
  Scenario: Two-factor authentication
    # Work in progress
```

```bash
# Run specific tags
npm run test:cucumber -- --tags "@smoke"
npm run test:cucumber -- --tags "@critical and not @slow"
npm run test:cucumber -- --tags "@smoke or @regression"
```

## Best Practices: Writing Good Gherkin

### GOOD: Declarative (Focus on WHAT, not HOW)

```gherkin
# Good - Describes behavior from user perspective
Scenario: User completes checkout
  Given I have items in my cart
  When I complete the checkout process
  Then my order should be confirmed

# Bad - Implementation details (HOW)
Scenario: User completes checkout
  Given I click the cart icon
  And I see the cart page
  When I click the "Checkout" button
  And I fill in field "address" with "123 Main St"
  And I fill in field "city" with "San Francisco"
  And I click the "Submit" button
  Then I should see element with id "confirmation"
```

### GOOD: Independent Scenarios

```gherkin
# Good - Self-contained
Scenario: Delete user account
  Given I have a user account
  When I request account deletion
  Then my account should be deleted

# Bad - Depends on previous scenario
Scenario: Delete user account
  # Assumes account was created in previous scenario
  When I request account deletion
  Then my account should be deleted
```

### GOOD: Use Background for Common Setup

```gherkin
Feature: Product Management

  Background:
    Given I am logged in as an admin
    And I am on the products page

  Scenario: Add new product
    When I create a product with name "Laptop"
    Then I should see "Laptop" in the product list

  Scenario: Edit product
    Given I have a product "Mouse"
    When I edit the product name to "Wireless Mouse"
    Then I should see "Wireless Mouse" in the product list
```

## Data Tables

```gherkin
Scenario: Create user with complete profile
  When I create a user with the following details:
    | field      | value                |
    | name       | John Doe             |
    | email      | john@example.com     |
    | age        | 30                   |
    | country    | USA                  |
    | role       | admin                |
  Then the user should be created successfully

Scenario: Bulk create users
  When I create the following users:
    | name       | email                | role   |
    | Alice      | alice@example.com    | user   |
    | Bob        | bob@example.com      | admin  |
    | Charlie    | charlie@example.com  | user   |
  Then all users should be created successfully
```

```typescript
// Step definition for data tables
When('I create a user with the following details:', async function (dataTable) {
  const userData = dataTable.rowsHash()
  await this.api.post('/users', userData)
})

When('I create the following users:', async function (dataTable) {
  const users = dataTable.hashes()
  for (const user of users) {
    await this.api.post('/users', user)
  }
})
```

## Hooks for Setup/Teardown

```typescript
// support/hooks.ts
import { Before, After, BeforeAll, AfterAll, Status } from '@cucumber/cucumber'

BeforeAll(async function () {
  // Global setup (runs once before all scenarios)
  console.log('Starting test suite')
})

AfterAll(async function () {
  // Global teardown (runs once after all scenarios)
  console.log('Test suite completed')
})

Before(async function () {
  // Setup before each scenario
  this.startTime = Date.now()
})

After(async function (scenario) {
  // Teardown after each scenario
  const duration = Date.now() - this.startTime
  console.log(`Scenario "${scenario.pickle.name}" took ${duration}ms`)

  // Take screenshot on failure
  if (scenario.result?.status === Status.FAILED) {
    const screenshot = await this.page.screenshot()
    this.attach(screenshot, 'image/png')
  }
})

// Tagged hooks
Before({ tags: '@database' }, async function () {
  await this.db.clear()
})

After({ tags: '@database' }, async function () {
  await this.db.close()
})
```

## Custom World (Shared Context)

```typescript
// support/world.ts
import { setWorldConstructor, World, IWorldOptions } from '@cucumber/cucumber'
import { chromium, Browser, Page } from '@playwright/test'

export class CustomWorld extends World {
  browser?: Browser
  page?: Page
  apiResponse?: any
  testData: Map<string, any>

  constructor(options: IWorldOptions) {
    super(options)
    this.testData = new Map()
  }

  async init() {
    this.browser = await chromium.launch()
    const context = await this.browser.newContext()
    this.page = await context.newPage()
  }

  async cleanup() {
    await this.page?.close()
    await this.browser?.close()
  }

  // Helper methods
  async login(email: string, password: string) {
    await this.page!.goto('/login')
    await this.page!.getByLabel('Email').fill(email)
    await this.page!.getByLabel('Password').fill(password)
    await this.page!.getByRole('button', { name: 'Login' }).click()
  }

  storeData(key: string, value: any) {
    this.testData.set(key, value)
  }

  getData(key: string) {
    return this.testData.get(key)
  }
}

setWorldConstructor(CustomWorld)
```

## Configuration

```typescript
// cucumber.config.ts
export default {
  require: ['step-definitions/**/*.ts'],
  requireModule: ['ts-node/register'],
  format: [
    'progress-bar',
    'html:test-results/cucumber-report.html',
    'json:test-results/cucumber-report.json',
    'junit:test-results/cucumber-report.xml'
  ],
  formatOptions: {
    snippetInterface: 'async-await'
  },
  parallel: 2,
  retry: 1,
  retryTagFilter: '@flaky'
}
```

## Integration with CI/CD

```yaml
# .github/workflows/bdd-tests.yml
name: BDD Tests

on: [push, pull_request]

jobs:
  cucumber-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '20'

      - name: Install dependencies
        run: npm ci

      - name: Run Cucumber tests
        run: npm run test:cucumber

      - name: Publish test results
        uses: EnricoMi/publish-unit-test-result-action@v2
        if: always()
        with:
          files: test-results/cucumber-report.xml

      - name: Upload HTML report
        uses: actions/upload-artifact@v3
        if: always()
        with:
          name: cucumber-report
          path: test-results/cucumber-report.html
```

## Common Patterns Checklist

- [ ] Write scenarios from user perspective (declarative)
- [ ] Keep scenarios independent (no dependencies)
- [ ] Use Background for common setup
- [ ] Use Scenario Outline for data-driven tests
- [ ] Use tags for organization (@smoke, @regression, @wip)
- [ ] Implement reusable step definitions
- [ ] Use Custom World for shared context
- [ ] Add screenshots on failure
- [ ] Write one assertion per Then step
- [ ] Avoid brittle implementation details

## Anti-Patterns to Avoid

[FAIL] **Overly specific scenarios**:
```gherkin
# Too detailed
When I click the button with id "submit-btn-123"
And I wait 2 seconds
Then I should see element with class "success-message"
```

[OK] **User-focused scenarios**:
```gherkin
# Better
When I submit the form
Then I should see a success message
```

[FAIL] **Reusing steps inappropriately**:
```gherkin
# Confusing reuse
Given I am on the login page
And I am on the products page  # Which page am I on?
```

[FAIL] **Testing too much in one scenario**:
```gherkin
# Too much in one scenario (split into 3 scenarios)
Scenario: Complete user journey
  Given I register a new account
  And I log in
  And I add products to cart
  And I checkout
  And I view order history
  And I update my profile
  # ... 20 more steps
```

## Related Resources

See [../../references/shift-left-testing.md](../../references/shift-left-testing.md) for writing scenarios in requirements phase, and [../e2e/template-playwright.md](../e2e/template-playwright.md) for implementing step definitions with Playwright.

# Component Testing Template: Vitest Browser Mode

Use this template when you want real-browser component coverage without paying full E2E cost.

## Typical use cases

- Form validation and state transitions
- Keyboard navigation and focus behavior
- Loading, empty, and error states
- Stable visual regression for design-system components

## Example

```typescript
import { describe, expect, it, vi } from 'vitest'
import { render, screen } from 'vitest-browser-react'
import userEvent from '@testing-library/user-event'
import { SignupForm } from './SignupForm'

describe('SignupForm', () => {
  it('shows validation and submits valid input', async () => {
    const onSubmit = vi.fn()
    const user = userEvent.setup()

    render(<SignupForm onSubmit={onSubmit} />)

    await user.click(screen.getByRole('button', { name: 'Create account' }))
    await expect.element(screen.getByText('Email is required')).toBeVisible()

    await user.type(screen.getByLabelText('Email'), 'user@example.com')
    await user.type(screen.getByLabelText('Password'), 'correct horse battery staple')
    await user.click(screen.getByRole('button', { name: 'Create account' }))

    expect(onSubmit).toHaveBeenCalledWith({
      email: 'user@example.com',
      password: 'correct horse battery staple'
    })
  })
})
```

## Accessibility smoke

- Add an accessibility pass in the same browser harness you use for the component.
- Prefer role, label, and keyboard-flow assertions alongside automated axe checks.
- Keep the automation narrow and repeatable; manual review is still required for WCAG 2.2 coverage.

## Defaults

- Prefer role, label, and text queries over CSS selectors
- Keep mocks at the component boundary
- Use screenshot diffs only for stable states
- Escalate to E2E only when the risk crosses page boundaries

# Playwright E2E Template — Moved

This template previously duplicated the dedicated `qa-testing-playwright` skill and risked drift.

For Playwright E2E patterns, templates, and CI integration, use the dedicated skill:

- **`frameworks/shared-skills/skills/qa-testing-playwright/`**

That skill owns: locator priority, three-tier suite topology (smoke / targeted-batch / deploy-gate), stateful app failure classification, auth setup, Page Object Model, sharding, trace/video artifacts, and CI integration.

This stub remains so existing references in plans and PRDs resolve. Remove the link to this file from your skill instructions and point directly at `qa-testing-playwright/SKILL.md` instead.

# Integration Testing Template: API Integration Tests

Use this template for testing API integrations, service communication, and database interactions.

## Framework Selection

**Supertest + Jest/Vitest** - Best for:
- REST API testing with Express/Fastify
- HTTP request/response validation
- Middleware testing
- Integration with existing Jest/Vitest setup

**Playwright/Puppeteer** - Best for:
- Full-stack integration tests
- Browser-based API interactions
- Testing with real authentication flows
- Visual verification alongside API calls

**Testcontainers** - Best for:
- Testing with real databases (PostgreSQL, MongoDB, Redis)
- Message queue integration (RabbitMQ, Kafka)
- Isolated test environments
- CI/CD compatibility

## Basic API Integration Test

```typescript
// api/users.integration.test.ts
import request from 'supertest'
import { app } from '../app'
import { db } from '../database'
import { UserFactory } from '../test-factories/user.factory'

describe('User API Integration', () => {
  beforeAll(async () => {
    // Setup: Start test database
    await db.connect()
    await db.migrate.latest()
  })

  afterAll(async () => {
    // Teardown: Close connections
    await db.destroy()
  })

  beforeEach(async () => {
    // Reset database state before each test
    await db('users').truncate()
  })

  describe('POST /api/users', () => {
    it('should create user and return 201', async () => {
      // Arrange
      const userData = {
        email: 'test@example.com',
        password: 'SecurePass123!',
        name: 'Test User'
      }

      // Act
      const response = await request(app)
        .post('/api/users')
        .send(userData)
        .expect('Content-Type', /json/)
        .expect(201)

      // Assert
      expect(response.body).toMatchObject({
        id: expect.any(String),
        email: 'test@example.com',
        name: 'Test User'
      })
      expect(response.body.password).toBeUndefined() // Never return password

      // Verify database state
      const dbUser = await db('users').where({ id: response.body.id }).first()
      expect(dbUser).toBeDefined()
      expect(dbUser.email).toBe('test@example.com')
    })

    it('should return 409 for duplicate email', async () => {
      // Arrange
      const userData = { email: 'test@example.com', password: 'pass', name: 'Test' }
      await request(app).post('/api/users').send(userData)

      // Act
      const response = await request(app)
        .post('/api/users')
        .send(userData)
        .expect(409)

      // Assert
      expect(response.body.error).toBe('Email already exists')
    })

    it('should validate required fields', async () => {
      // Act
      const response = await request(app)
        .post('/api/users')
        .send({ email: 'test@example.com' }) // Missing password and name
        .expect(400)

      // Assert
      expect(response.body.errors).toContainEqual(
        expect.objectContaining({ field: 'password', message: expect.any(String) })
      )
      expect(response.body.errors).toContainEqual(
        expect.objectContaining({ field: 'name', message: expect.any(String) })
      )
    })
  })

  describe('GET /api/users/:id', () => {
    it('should return user by id', async () => {
      // Arrange
      const user = await UserFactory.createInDb(db)

      // Act
      const response = await request(app)
        .get(`/api/users/${user.id}`)
        .expect(200)

      // Assert
      expect(response.body).toMatchObject({
        id: user.id,
        email: user.email,
        name: user.name
      })
    })

    it('should return 404 for non-existent user', async () => {
      // Act
      const response = await request(app)
        .get('/api/users/non-existent-id')
        .expect(404)

      // Assert
      expect(response.body.error).toBe('User not found')
    })
  })

  describe('PUT /api/users/:id', () => {
    it('should update user', async () => {
      // Arrange
      const user = await UserFactory.createInDb(db)
      const updates = { name: 'Updated Name' }

      // Act
      const response = await request(app)
        .put(`/api/users/${user.id}`)
        .send(updates)
        .expect(200)

      // Assert
      expect(response.body.name).toBe('Updated Name')

      // Verify database state
      const dbUser = await db('users').where({ id: user.id }).first()
      expect(dbUser.name).toBe('Updated Name')
    })

    it('should not allow email update', async () => {
      // Arrange
      const user = await UserFactory.createInDb(db)

      // Act
      const response = await request(app)
        .put(`/api/users/${user.id}`)
        .send({ email: 'newemail@example.com' })
        .expect(400)

      // Assert
      expect(response.body.error).toContain('cannot change email')
    })
  })

  describe('DELETE /api/users/:id', () => {
    it('should soft delete user', async () => {
      // Arrange
      const user = await UserFactory.createInDb(db)

      // Act
      await request(app)
        .delete(`/api/users/${user.id}`)
        .expect(204)

      // Assert - User should still exist but be marked deleted
      const dbUser = await db('users').where({ id: user.id }).first()
      expect(dbUser.deleted_at).toBeDefined()
    })
  })
})
```

## Authentication Integration Tests

```typescript
describe('Authentication Flow', () => {
  let authToken: string

  describe('POST /api/auth/login', () => {
    it('should authenticate user and return token', async () => {
      // Arrange
      const user = await UserFactory.createInDb(db, { password: 'TestPass123!' })

      // Act
      const response = await request(app)
        .post('/api/auth/login')
        .send({ email: user.email, password: 'TestPass123!' })
        .expect(200)

      // Assert
      expect(response.body).toMatchObject({
        token: expect.any(String),
        user: {
          id: user.id,
          email: user.email
        }
      })

      authToken = response.body.token
    })

    it('should reject invalid credentials', async () => {
      // Arrange
      const user = await UserFactory.createInDb(db, { password: 'TestPass123!' })

      // Act
      const response = await request(app)
        .post('/api/auth/login')
        .send({ email: user.email, password: 'WrongPassword' })
        .expect(401)

      // Assert
      expect(response.body.error).toBe('Invalid credentials')
    })

    it('should lock account after 5 failed attempts', async () => {
      // Arrange
      const user = await UserFactory.createInDb(db, { password: 'TestPass123!' })

      // Act - 5 failed attempts
      for (let i = 0; i < 5; i++) {
        await request(app)
          .post('/api/auth/login')
          .send({ email: user.email, password: 'WrongPassword' })
      }

      // Act - 6th attempt should be locked
      const response = await request(app)
        .post('/api/auth/login')
        .send({ email: user.email, password: 'TestPass123!' }) // Even correct password
        .expect(423)

      // Assert
      expect(response.body.error).toContain('Account locked')
    })
  })

  describe('Protected Routes', () => {
    it('should allow access with valid token', async () => {
      // Arrange
      const user = await UserFactory.createInDb(db)
      const token = await generateAuthToken(user)

      // Act
      const response = await request(app)
        .get('/api/users/me')
        .set('Authorization', `Bearer ${token}`)
        .expect(200)

      // Assert
      expect(response.body.id).toBe(user.id)
    })

    it('should reject invalid token', async () => {
      // Act
      const response = await request(app)
        .get('/api/users/me')
        .set('Authorization', 'Bearer invalid-token')
        .expect(401)

      // Assert
      expect(response.body.error).toBe('Invalid token')
    })

    it('should reject expired token', async () => {
      // Arrange
      const user = await UserFactory.createInDb(db)
      const expiredToken = await generateAuthToken(user, { expiresIn: '-1h' })

      // Act
      const response = await request(app)
        .get('/api/users/me')
        .set('Authorization', `Bearer ${expiredToken}`)
        .expect(401)

      // Assert
      expect(response.body.error).toContain('expired')
    })
  })
})
```

## Database Integration Tests

```typescript
import { PostgreSqlContainer } from '@testcontainers/postgresql'

describe('Database Integration', () => {
  let container: PostgreSqlContainer
  let testDb: Database

  beforeAll(async () => {
    // Start PostgreSQL container
    container = await new PostgreSqlContainer('postgres:15')
      .withDatabase('test_db')
      .withUsername('test_user')
      .withPassword('test_pass')
      .start()

    // Connect to test database
    testDb = await connectToDatabase({
      host: container.getHost(),
      port: container.getPort(),
      database: container.getDatabase(),
      username: container.getUsername(),
      password: container.getPassword()
    })

    // Run migrations
    await testDb.migrate.latest()
  }, 60000) // Increased timeout for container startup

  afterAll(async () => {
    await testDb.destroy()
    await container.stop()
  })

  describe('Transaction Handling', () => {
    it('should commit transaction on success', async () => {
      // Act
      await testDb.transaction(async (trx) => {
        await trx('users').insert({ email: 'test@example.com', name: 'Test' })
        await trx('profiles').insert({ user_email: 'test@example.com', bio: 'Test bio' })
      })

      // Assert
      const user = await testDb('users').where({ email: 'test@example.com' }).first()
      const profile = await testDb('profiles').where({ user_email: 'test@example.com' }).first()

      expect(user).toBeDefined()
      expect(profile).toBeDefined()
    })

    it('should rollback transaction on error', async () => {
      // Act
      await expect(
        testDb.transaction(async (trx) => {
          await trx('users').insert({ email: 'test@example.com', name: 'Test' })
          throw new Error('Simulated error')
        })
      ).rejects.toThrow()

      // Assert - User should not exist
      const user = await testDb('users').where({ email: 'test@example.com' }).first()
      expect(user).toBeUndefined()
    })
  })

  describe('Complex Queries', () => {
    it('should perform join queries', async () => {
      // Arrange
      await testDb('users').insert([
        { id: '1', email: 'user1@example.com', name: 'User 1' },
        { id: '2', email: 'user2@example.com', name: 'User 2' }
      ])
      await testDb('posts').insert([
        { id: '1', user_id: '1', title: 'Post 1' },
        { id: '2', user_id: '1', title: 'Post 2' },
        { id: '3', user_id: '2', title: 'Post 3' }
      ])

      // Act
      const results = await testDb('users')
        .select('users.name', testDb.raw('COUNT(posts.id) as post_count'))
        .leftJoin('posts', 'users.id', 'posts.user_id')
        .groupBy('users.id')
        .orderBy('post_count', 'desc')

      // Assert
      expect(results).toHaveLength(2)
      expect(results[0]).toMatchObject({ name: 'User 1', post_count: '2' })
      expect(results[1]).toMatchObject({ name: 'User 2', post_count: '1' })
    })
  })
})
```

## External Service Integration Tests

```typescript
import { WireMock } from 'wiremock'

describe('External Service Integration', () => {
  let wireMock: WireMock

  beforeAll(async () => {
    // Start WireMock server for stubbing external APIs
    wireMock = new WireMock({ host: 'localhost', port: 8080 })
    await wireMock.start()
  })

  afterAll(async () => {
    await wireMock.stop()
  })

  beforeEach(async () => {
    await wireMock.resetAll()
  })

  describe('Payment Service Integration', () => {
    it('should process payment successfully', async () => {
      // Arrange - Stub external payment API
      await wireMock.stub({
        request: {
          method: 'POST',
          url: '/api/payments'
        },
        response: {
          status: 200,
          jsonBody: {
            transactionId: 'TX123456',
            status: 'approved'
          }
        }
      })

      const order = await OrderFactory.createInDb(db)

      // Act
      const response = await request(app)
        .post(`/api/orders/${order.id}/pay`)
        .send({ amount: 100, currency: 'USD' })
        .expect(200)

      // Assert
      expect(response.body).toMatchObject({
        transactionId: 'TX123456',
        status: 'approved'
      })

      // Verify WireMock received request
      const requests = await wireMock.getRequests()
      expect(requests).toHaveLength(1)
      expect(requests[0].body).toContain('amount')
    })

    it('should handle payment service timeout', async () => {
      // Arrange - Stub with delay
      await wireMock.stub({
        request: {
          method: 'POST',
          url: '/api/payments'
        },
        response: {
          status: 200,
          fixedDelayMilliseconds: 10000 // 10 second delay
        }
      })

      const order = await OrderFactory.createInDb(db)

      // Act
      const response = await request(app)
        .post(`/api/orders/${order.id}/pay`)
        .send({ amount: 100, currency: 'USD' })
        .expect(504)

      // Assert
      expect(response.body.error).toContain('timeout')
    })

    it('should retry on service failure', async () => {
      // Arrange - First call fails, second succeeds
      await wireMock.stub({
        request: {
          method: 'POST',
          url: '/api/payments'
        },
        response: {
          status: 500
        }
      })

      setTimeout(async () => {
        await wireMock.resetAll()
        await wireMock.stub({
          request: {
            method: 'POST',
            url: '/api/payments'
          },
          response: {
            status: 200,
            jsonBody: { transactionId: 'TX123456', status: 'approved' }
          }
        })
      }, 1000)

      const order = await OrderFactory.createInDb(db)

      // Act
      const response = await request(app)
        .post(`/api/orders/${order.id}/pay`)
        .send({ amount: 100, currency: 'USD' })
        .expect(200)

      // Assert
      expect(response.body.transactionId).toBe('TX123456')

      // Verify retries happened
      const requests = await wireMock.getRequests()
      expect(requests.length).toBeGreaterThan(1)
    })
  })
})
```

## Message Queue Integration Tests

```typescript
import { RabbitMQContainer } from '@testcontainers/rabbitmq'
import amqp from 'amqplib'

describe('Message Queue Integration', () => {
  let container: RabbitMQContainer
  let connection: amqp.Connection
  let channel: amqp.Channel

  beforeAll(async () => {
    // Start RabbitMQ container
    container = await new RabbitMQContainer().start()

    // Connect to RabbitMQ
    connection = await amqp.connect(container.getAmqpUrl())
    channel = await connection.createChannel()
  }, 60000)

  afterAll(async () => {
    await channel.close()
    await connection.close()
    await container.stop()
  })

  describe('Order Processing Queue', () => {
    it('should publish and consume messages', async () => {
      // Arrange
      const queueName = 'order_processing'
      await channel.assertQueue(queueName, { durable: false })

      const orderData = {
        orderId: '123',
        userId: 'user-456',
        total: 99.99
      }

      // Act - Publish message
      channel.sendToQueue(queueName, Buffer.from(JSON.stringify(orderData)))

      // Assert - Consume message
      const message = await new Promise<any>((resolve) => {
        channel.consume(queueName, (msg) => {
          if (msg) {
            resolve(JSON.parse(msg.content.toString()))
            channel.ack(msg)
          }
        })
      })

      expect(message).toMatchObject(orderData)
    })

    it('should handle message rejection and retry', async () => {
      // Arrange
      const queueName = 'order_processing_retry'
      await channel.assertQueue(queueName, { durable: true })
      await channel.assertQueue(`${queueName}_dlq`, { durable: true })

      const invalidOrder = { orderId: 'invalid' }
      channel.sendToQueue(queueName, Buffer.from(JSON.stringify(invalidOrder)))

      // Act - Consumer rejects invalid message
      let attempts = 0
      await new Promise<void>((resolve) => {
        channel.consume(queueName, (msg) => {
          if (msg) {
            attempts++
            if (attempts < 3) {
              channel.nack(msg, false, true) // Requeue
            } else {
              channel.sendToQueue(`${queueName}_dlq`, msg.content) // Dead letter
              channel.ack(msg)
              resolve()
            }
          }
        })
      })

      // Assert
      expect(attempts).toBe(3)
    })
  })
})
```

## Best Practices Checklist

- [ ] Test the entire request/response cycle (not just business logic)
- [ ] Use real database instances (Testcontainers) for accuracy
- [ ] Reset database state between tests (truncate or transactions)
- [ ] Test authentication and authorization flows
- [ ] Verify database state after operations (not just API responses)
- [ ] Stub external services (WireMock, MSW) for reliability
- [ ] Test error scenarios (timeouts, retries, failures)
- [ ] Test transaction rollbacks and commits
- [ ] Use factories for complex test data setup
- [ ] Test message queue processing (if applicable)
- [ ] Validate response headers and status codes
- [ ] Test rate limiting and throttling
- [ ] Verify side effects (emails sent, events published)

## Common Pitfalls

[FAIL] **Using in-memory databases for integration tests**:
```typescript
// Bad - SQLite in-memory doesn't match production PostgreSQL
const db = new SQLite(':memory:')

// Good - Use Testcontainers with real PostgreSQL
const container = await new PostgreSqlContainer('postgres:15').start()
```

[FAIL] **Not cleaning up between tests**:
```typescript
// Bad - Tests interfere with each other
beforeAll(async () => {
  await db.seed.run() // Only runs once
})

// Good - Fresh state for each test
beforeEach(async () => {
  await db('users').truncate()
})
```

[FAIL] **Testing external APIs directly**:
```typescript
// Bad - Tests depend on external service availability
await fetch('https://api.stripe.com/v1/charges')

// Good - Stub external services
await wireMock.stub({ ... })
```

## Configuration

### package.json

```json
{
  "scripts": {
    "test:integration": "vitest run --config vitest.integration.config.ts",
    "test:integration:watch": "vitest --config vitest.integration.config.ts"
  },
  "devDependencies": {
    "@testcontainers/postgresql": "^10.0.0",
    "@testcontainers/rabbitmq": "^10.0.0",
    "supertest": "^6.3.3",
    "wiremock": "^3.0.0"
  }
}
```

### vitest.integration.config.ts

```typescript
import { defineConfig } from 'vitest/config'

export default defineConfig({
  test: {
    include: ['**/*.integration.test.ts'],
    testTimeout: 30000, // Longer timeout for containers
    hookTimeout: 60000, // Container startup can be slow
    globalSetup: './test/integration-setup.ts',
    pool: 'forks', // Isolation for database tests
    poolOptions: {
      forks: {
        singleFork: false // Run tests in parallel
      }
    }
  }
})
```

## Related Resources

See [../../references/comprehensive-testing-guide.md](../../references/comprehensive-testing-guide.md) for complete testing guide across all layers.

# k6 Load Testing Template — Moved

This template previously duplicated the dedicated `qa-testing-performance` skill and risked drift.

For k6 load/stress/soak/spike/capacity patterns, templates, and CI gates, use the dedicated skill:

- **`frameworks/shared-skills/skills/qa-testing-performance/`**

That skill owns: test-type taxonomy, percentile-over-average rule, k6 1.x and 2.x scripting differences, perf budget checker script, and CI gate design.

This stub remains so existing references in plans and PRDs resolve. Remove the link to this file from your skill instructions and point directly at `qa-testing-performance/SKILL.md` instead.

# Flaky Test Triage & Deflake Runbook

Use this runbook to reduce CI noise, prevent silent regressions, and restore confidence.

## Core

### Definitions

- Flaky test: fails without product change and passes on rerun.
- “Rerun-pass” is a defect signal, not a success.

### Flake SLOs (Example Targets)

- Suite flake rate <= 1% weekly.
- Time-to-deflake: p50 <= 2 business days, p95 <= 7 business days.
- Mainline health: >= 99% green builds/day.

### Intake Checklist (First 5 Minutes)

- Identify the failing test(s): name/path, suite, owner.
- Collect context:
  - Build URL, commit SHA, branch, runner type (self-hosted vs hosted)
  - Timestamp, region, parallel shard/worker ID
  - Retry count and whether it passed on retry
  - Correlation IDs (request/trace IDs) and artifacts (logs/screenshots/traces/error-context)
- Record the execution topology:
  - shared dev stack or test-managed server
  - exact repro command
  - cleanup command if a shared stack is in use

### Triage Flow (Reproduce → Classify → Fix → Prevent)

Reproduce:
- Run one exact spec or one named batch with `--workers=1`.
- If CI-only: reproduce in a container/runner that matches CI resources.
- Do not widen to full-suite replay until the targeted scope is understood.

Classify (pick the dominant class):

| Class | Signals | Typical fixes |
|------|---------|--------------|
| Timing/race | “Sometimes element not ready”, async hazards | event-based waits, remove sleeps, wait on the real readiness signal |
| Data/state | ordering dependency, shared accounts, leaked DB rows | isolate data, reset state, unique IDs, cleanup |
| Auth-state | protected route falls back to login, storage state missing, session not restored | auth-aware navigation, re-seed auth, verify storage/session lifecycle |
| State-sync | backend reset/webhook finished but UI has not converged | assert the convergence signal, not immediate copy; tighten cleanup sequencing |
| Environment | low CPU/memory, timezone/locale, stale port/PID/lock | pin locale/tz, clear stale processes, remove env assumptions |
| Dependency | third-party API, unstable backend | mock boundary, contract tests, test doubles |
| Optional-network | incidental side request failed but visible journey may still succeed | remove incidental request waits; keep the user-facing oracle |
| Degraded-mode | 429, fallback UX, partial content | assert rate-limit/fallback behavior intentionally |
| Test design | brittle selectors/assertions | assert user intent, stable selectors, stronger oracles |
| Product bug | genuine race in product | fix race; add regression at lowest layer |

Fix:
- Prefer product fixes for real races over test-only band-aids.
- Add or upgrade observability for the failing path (logs/traces) to catch it next time.

Prevent:
- Add a pre-merge check that would have caught the issue earlier (unit/integration/contract).
- If the fix is localized, rerun the smallest affected scope first; only then trigger the deploy-gate replay.

### Quarantine Policy (If You Must)

Quarantine is a temporary safety valve, not a solution.

REQUIRED fields:
- Owner: ______________________
- Ticket: _____________________
- Reason: _____________________
- Expiry date: ________________
- Impact: blocks PRs? yes/no

Rules:
- No quarantines without expiry and an assigned owner.
- Quarantined tests must still run and report; they just don’t block merges.
- If a quarantined test starts failing consistently, escalate as a product defect.

### CI Economics (Contain Blast Radius)

- Split suites by layer and cost: fast PR gate vs targeted reruns vs slow deploy-gate or scheduled suites.
- Shard long-running suites; keep PR feedback under a fixed budget.

### Anti-Patterns (Deflake Smells)

- Adding sleeps to “stabilize” without proving the race.
- Increasing timeouts globally instead of fixing the slow step.
- Weakening assertions so failures disappear.
- Marking rerun-pass as success without tracking flake rate.
- Expanding straight to a full-suite replay before the single failing scope is understood.

## Optional: AI / Automation

Do:
- Use AI to cluster failures across builds and summarize common signatures, but require evidence links (logs/traces/stack traces).
- Use AI to propose candidate root causes; validate via targeted instrumentation and reproduction.

Avoid:
- Letting AI auto-edit tests to “heal” flakes by reducing assertions or switching to brittle selectors.

# Template: Release Coverage Audit (Feature Matrix vs Test Matrix)

## Release Context

- Release tag/branch: `________________________`
- Audit date: `YYYY-MM-DD`
- Auditor: `_______________________________`

## Coverage Table

| Feature/Backlog ID | Feature Name | Criticality | Coverage Status | Direct Evidence (path + test id) | Waiver (if any) | Owner | Due Date |
|---|---|---|---|---|---|---|---|
|  |  | High/Med/Low | direct/indirect/none |  |  |  |  |
|  |  |  |  |  |  |  |  |
|  |  |  |  |  |  |  |  |

## Summary

- Critical features total: `___`
- Directly covered: `___`
- Indirectly covered: `___`
- Uncovered: `___`

## Decision

- [ ] GO
- [ ] NO-GO

Reason:
`_____________________________________________________________`

## Required Follow-ups

1. `___________________________________________________________`
2. `___________________________________________________________`

# Test Case Design Template (Given/When/Then + Oracles)

Use this template for any test layer (unit/integration/contract/E2E) by filling only what applies.

## Core

### Metadata

- ID: __________________________
- Title: _______________________
- Owner: _______________________
- Layer: unit / component / contract / integration / E2E / exploratory
- Priority: P0 / P1 / P2 / P3
- Risk addressed: journey + failure mode(s)

### Goal (What This Test Proves)

- Hypothesis: _______________________________________________
- Why now: _________________________________________________

### Preconditions

- Environment: local / CI / staging
- Feature flags/config: _____________________________________
- Auth/user roles: __________________________________________

### Test Data

- Data setup method: fixtures / factories / seed / API setup
- Data identifiers (IDs/keys): _______________________________
- Cleanup/reset plan: _______________________________________

### Steps (Given / When / Then)

Given:
- ___________________________________________________________

When:
- ___________________________________________________________

Then:
- ___________________________________________________________

### Oracles (How You Know It’s Correct)

Functional oracles:
- Expected state/output: _____________________________________
- Contract/schema: __________________________________________

Quality oracles (if applicable):
- Security: authz/authn, sensitive data not exposed
- Accessibility: roles/labels, focus order, keyboard paths
- Performance: budget (p95/p99) and no significant regression

Negative oracles:
- What must NOT happen: _____________________________________

### Observability (Debugging Ergonomics)

- Correlation IDs captured: request ID / trace ID / build URL
- Failure artifacts expected:
  - Logs
  - Traces
  - Screenshots/video (UI)
  - Crash reports/core dumps (if relevant)

### Flake Control (Determinism)

- Time control: timezone/locale/frozen time? ________________
- Network control: mocked/stubbed boundaries? _______________
- Retries policy: ___________________________________________
- Timeout budget: ___________________________________________

### Automation Notes

- What to mock vs keep real: ________________________________
- Lowest layer alternative: can this be tested lower? ________
- CI execution: PR gate / nightly / release _________________

### Pass/Fail Criteria

- Pass criteria: ____________________________________________
- Fail criteria: ____________________________________________

## Optional: AI / Automation

Do:
- Use AI to propose edge cases and variations (boundaries, auth roles, locales).
- Use AI to draft Given/When/Then steps and candidate oracles, then validate manually.

Avoid:
- Copying AI-generated assertions without verifying the oracle and failure mode.
- Generating large combinatorial suites without a risk-based selection.

# QA Test Strategy One-Pager (Risk-Based)

Use this template to define a minimal, high-signal quality plan that balances risk coverage, CI economics, and debuggability.

## Core

### Context

- Product/area: ________________________________
- What is changing (scope): _____________________
- Release cadence: ______________________________
- Environments: local / CI / staging / prod
- Key dependencies: _____________________________

### Quality Goals (Measurable)

- Reliability: SLIs/SLOs (latency/error/availability) and error budget policy
- Performance: budgets (p95/p99), frontend budgets (if applicable)
- Security: required checks (SAST/DAST/dependency scanning)
- Accessibility: WCAG 2.2 target level and automation scope

### Risk Model (Journeys x Failure Modes)

List top user journeys and likely failure modes.

| Journey | Failure modes | Impact | Likelihood | Primary tests | Owner |
|--------|---------------|--------|------------|---------------|-------|
| Login | auth outage, session bugs | High | Med | E2E smoke + contract | ___ |
| Checkout | payment timeout, idempotency | High | Med | integration + resilience | ___ |

### Test Portfolio (Layered)

Define what runs where, and why.

- Unit: business logic, validators, pure functions
- Component: UI logic in a real browser plus accessibility smoke
- Contract: OpenAPI/AsyncAPI/JSON schema validations
- Schema fuzzing: generated valid/invalid API inputs for parser and validation drift
- Integration: API + DB + key dependencies (mock third parties)
- E2E: thin, critical user journeys only
- Exploratory: discovery and usability; convert high-ROI findings to automation
- Performance/resilience: scheduled or canary-gated, not every PR

### Shift-Left (Pre-Merge Gates)

- Required: lint, typecheck, unit tests, contract validation
- Conditional: integration smoke for affected areas
- Avoid: full E2E as a default PR gate (unless E2E-only product)

### CI/CD Stages (Economics)

- PR gate: ________________________________
- Post-merge: _____________________________
- Nightly: ________________________________
- Release: ________________________________
- Budgets (example targets):
  - PR gate p50 <= 10 min, p95 <= 20 min
  - Mainline health >= 99% green builds/day

### Flake Management

- Flake definition: fails without product change and passes on rerun.
- SLO examples (example targets):
  - Suite flake rate <= 1% weekly
  - Time-to-deflake p50 <= 2 business days, p95 <= 7 business days
- Quarantine rules: owner + ticket + expiry; never “ignore forever”.
- Runbook: `runbooks/template-flaky-test-triage-deflake-runbook.md`

### Observability for QA (Debugging Ergonomics)

- Required correlation IDs: request ID, trace ID
- Failure artifacts: logs, traces, screenshots/videos (UI), crash reports
- Where artifacts live: __________________________

### Owners and Cadence

- Suite owners: _________________________________
- Review cadence: _______________________________
- Deprecation policy for low-value tests: ________

## Optional: AI / Automation

Do:
- Use AI to draft the initial risk register and candidate test ideas; validate against domain knowledge and telemetry.
- Use AI to summarize test failures (log/trace clustering) while retaining evidence links.

Avoid:
- Accepting generated assertions/oracles without validation.
- Using AI to “heal” tests by weakening assertions.

# Unit Testing Template: Jest / Vitest

Use this template for writing unit tests with Jest or Vitest for JavaScript/TypeScript projects.

## Framework Selection

**Jest** - Best for:
- React applications (built-in React Testing Library support)
- Projects already using Jest (migration cost)
- Teams needing extensive mocking capabilities
- Zero-config setup preference

**Vitest** - Best for:
- Vite-based projects (instant compatibility)
- Projects prioritizing speed (native ESM, parallel execution)
- TypeScript/JSX without transpilation
- Modern tooling (watch mode, UI mode)

## Test File Structure

### Basic Test Structure (AAA Pattern)

```typescript
// user.service.test.ts
import { describe, it, expect, beforeEach, afterEach } from 'vitest' // or '@jest/globals'
import { UserService } from './user.service'
import { DatabaseMock } from '../__mocks__/database.mock'

describe('UserService', () => {
  let service: UserService
  let dbMock: DatabaseMock

  beforeEach(() => {
    // Arrange: Setup for each test
    dbMock = new DatabaseMock()
    service = new UserService(dbMock)
  })

  afterEach(() => {
    // Cleanup after each test
    dbMock.clear()
  })

  describe('createUser', () => {
    it('should hash password before saving', async () => {
      // Arrange
      const userData = {
        email: 'test@example.com',
        password: 'PlainPassword123',
        name: 'Test User'
      }

      // Act
      const user = await service.createUser(userData)

      // Assert
      expect(user.password).not.toBe('PlainPassword123')
      expect(user.password).toMatch(/^\$2[aby]\$.{56}$/) // bcrypt pattern
      expect(user.email).toBe('test@example.com')
    })

    it('should throw error for duplicate email', async () => {
      // Arrange
      const userData = { email: 'test@example.com', password: 'pass', name: 'Test' }
      await service.createUser(userData)

      // Act & Assert
      await expect(service.createUser(userData))
        .rejects
        .toThrow('Email already exists')
    })

    it('should validate email format', async () => {
      // Arrange
      const invalidData = { email: 'invalid-email', password: 'pass', name: 'Test' }

      // Act & Assert
      await expect(service.createUser(invalidData))
        .rejects
        .toThrow('Invalid email format')
    })
  })

  describe('findUserById', () => {
    it('should return user when exists', async () => {
      // Arrange
      const user = await service.createUser({
        email: 'test@example.com',
        password: 'pass',
        name: 'Test User'
      })

      // Act
      const found = await service.findUserById(user.id)

      // Assert
      expect(found).toBeDefined()
      expect(found?.id).toBe(user.id)
      expect(found?.email).toBe('test@example.com')
    })

    it('should return null when user not found', async () => {
      // Act
      const found = await service.findUserById('non-existent-id')

      // Assert
      expect(found).toBeNull()
    })
  })
})
```

## Testing Edge Cases

```typescript
describe('Edge Cases', () => {
  describe('boundary values', () => {
    it('should handle minimum age', () => {
      expect(service.isAdult(18)).toBe(true)
      expect(service.isAdult(17)).toBe(false)
    })

    it('should handle maximum string length', () => {
      const maxName = 'a'.repeat(100)
      const tooLong = 'a'.repeat(101)

      expect(service.validateName(maxName)).toBe(true)
      expect(() => service.validateName(tooLong)).toThrow('Name too long')
    })
  })

  describe('null and undefined handling', () => {
    it('should handle null input', () => {
      expect(() => service.processData(null)).toThrow('Invalid input')
    })

    it('should handle undefined input', () => {
      expect(() => service.processData(undefined)).toThrow('Invalid input')
    })

    it('should handle empty string', () => {
      expect(() => service.processData('')).toThrow('Invalid input')
    })
  })

  describe('special characters', () => {
    it('should escape SQL injection attempts', () => {
      const malicious = "'; DROP TABLE users; --"
      expect(() => service.searchUsers(malicious)).not.toThrow()
    })

    it('should sanitize XSS attempts', () => {
      const xss = '<script>alert("XSS")</script>'
      const sanitized = service.sanitizeInput(xss)
      expect(sanitized).not.toContain('<script>')
    })
  })
})
```

## Mocking Dependencies

### Mock Functions

```typescript
import { vi } from 'vitest' // or jest.fn()

describe('Mocking', () => {
  it('should call external service', async () => {
    // Arrange
    const emailService = {
      send: vi.fn().mockResolvedValue({ success: true })
    }
    const service = new UserService(db, emailService)

    // Act
    await service.createUser({ email: 'test@example.com', password: 'pass', name: 'Test' })

    // Assert
    expect(emailService.send).toHaveBeenCalledTimes(1)
    expect(emailService.send).toHaveBeenCalledWith({
      to: 'test@example.com',
      subject: 'Welcome',
      template: 'welcome'
    })
  })

  it('should handle service failure gracefully', async () => {
    // Arrange
    const emailService = {
      send: vi.fn().mockRejectedValue(new Error('Email service down'))
    }
    const service = new UserService(db, emailService)

    // Act & Assert
    await expect(service.createUser({ email: 'test@example.com', password: 'pass', name: 'Test' }))
      .rejects
      .toThrow('Failed to send welcome email')
  })
})
```

### Mock Modules

```typescript
// Vitest module mocking
vi.mock('../services/payment.service', () => ({
  PaymentService: vi.fn().mockImplementation(() => ({
    processPayment: vi.fn().mockResolvedValue({ transactionId: 'TX123' })
  }))
}))

// Jest module mocking
jest.mock('../services/payment.service', () => ({
  PaymentService: jest.fn().mockImplementation(() => ({
    processPayment: jest.fn().mockResolvedValue({ transactionId: 'TX123' })
  }))
}))
```

## Snapshot Testing

```typescript
describe('Snapshot Testing', () => {
  it('should match user profile snapshot', () => {
    const user = service.getUserProfile('user-123')
    expect(user).toMatchSnapshot()
  })

  it('should match inline snapshot', () => {
    const config = service.getConfig()
    expect(config).toMatchInlineSnapshot(`
      {
        "apiUrl": "https://api.example.com",
        "timeout": 5000,
        "retries": 3
      }
    `)
  })
})
```

## Test Data Factories

```typescript
// test-factories/user.factory.ts
import { faker } from '@faker-js/faker'

export class UserFactory {
  static create(overrides: Partial<User> = {}): User {
    return {
      id: faker.string.uuid(),
      email: faker.internet.email(),
      name: faker.person.fullName(),
      age: faker.number.int({ min: 18, max: 80 }),
      createdAt: faker.date.past(),
      ...overrides
    }
  }

  static createMany(count: number, overrides: Partial<User> = {}): User[] {
    return Array.from({ length: count }, () => this.create(overrides))
  }

  static createAdmin(): User {
    return this.create({ role: 'admin', permissions: ['read', 'write', 'delete'] })
  }
}

// Usage in tests
describe('UserService', () => {
  it('should process batch of users', () => {
    const users = UserFactory.createMany(10)
    const result = service.processBatch(users)
    expect(result.processed).toBe(10)
  })

  it('should grant admin access', () => {
    const admin = UserFactory.createAdmin()
    expect(service.hasPermission(admin, 'delete')).toBe(true)
  })
})
```

## Async Testing

```typescript
describe('Async Operations', () => {
  it('should resolve promise', async () => {
    const result = await service.fetchData()
    expect(result).toBeDefined()
  })

  it('should reject promise', async () => {
    await expect(service.fetchInvalidData()).rejects.toThrow('Not found')
  })

  it('should timeout after delay', async () => {
    vi.useFakeTimers()
    const promise = service.delayedOperation()
    vi.advanceTimersByTime(5000)
    await expect(promise).resolves.toBe('completed')
    vi.useRealTimers()
  })
})
```

## Coverage Configuration

### Vitest (vite.config.ts)

```typescript
import { defineConfig } from 'vitest/config'

export default defineConfig({
  test: {
    coverage: {
      provider: 'v8', // or 'istanbul'
      reporter: ['text', 'json', 'html', 'lcov'],
      exclude: [
        '**/node_modules/**',
        '**/dist/**',
        '**/*.test.ts',
        '**/*.config.ts',
        '**/types/**'
      ],
      thresholds: {
        lines: 80,
        functions: 80,
        branches: 80,
        statements: 80
      }
    }
  }
})
```

### Jest (jest.config.js)

```javascript
module.exports = {
  collectCoverageFrom: [
    'src/**/*.{ts,tsx}',
    '!src/**/*.d.ts',
    '!src/**/*.test.{ts,tsx}',
    '!src/**/__mocks__/**'
  ],
  coverageThresholds: {
    global: {
      lines: 80,
      functions: 80,
      branches: 80,
      statements: 80
    },
    './src/services/': {
      lines: 90,
      functions: 90,
      branches: 90,
      statements: 90
    }
  },
  coverageReporters: ['text', 'lcov', 'html']
}
```

## Best Practices Checklist

- [ ] Use descriptive test names (what is being tested + expected outcome)
- [ ] Follow AAA pattern (Arrange, Act, Assert)
- [ ] Test one thing per test
- [ ] Use factories for test data (avoid magic values)
- [ ] Mock external dependencies (APIs, databases)
- [ ] Test edge cases and error conditions
- [ ] Keep tests independent (no shared state)
- [ ] Use beforeEach/afterEach for setup/cleanup
- [ ] Aim for 80%+ coverage on business logic
- [ ] Run tests in parallel where possible
- [ ] Use snapshot testing sparingly (for stable output only)

## Common Pitfalls

[FAIL] **Testing implementation details**:
```typescript
// Bad
expect(service.internalHelperMethod()).toBe(true)

// Good
expect(service.publicMethod()).toBe(expectedResult)
```

[FAIL] **Shared mutable state**:
```typescript
// Bad
let sharedUser: User
beforeAll(() => {
  sharedUser = createUser() // Shared across tests
})

// Good
let user: User
beforeEach(() => {
  user = createUser() // Fresh for each test
})
```

[FAIL] **Not cleaning up after tests**:
```typescript
// Bad
afterEach(() => {
  // No cleanup
})

// Good
afterEach(() => {
  vi.clearAllMocks()
  dbMock.clear()
})
```

## Running Tests

```bash
# Vitest
npm run test              # Run once
npm run test:watch        # Watch mode
npm run test:ui           # UI mode
npm run test:coverage     # With coverage

# Jest
npm test                  # Run once
npm test -- --watch       # Watch mode
npm test -- --coverage    # With coverage
npm test -- UserService   # Run specific file
```

## Related Resources

See [../../references/comprehensive-testing-guide.md](../../references/comprehensive-testing-guide.md) for complete testing guide across all layers.

# Visual Regression Testing Template

Use this template for catching unintended visual changes in UI components, pages, and design systems.

## Framework Selection

**Playwright Visual Comparisons** - Best for:
- Full-page screenshots across browsers
- Component screenshot testing
- Built-in pixel-diff comparison
- CI/CD integration out of the box

**Chromatic (Storybook)** - Best for:
- Design system visual testing
- Component library regression
- Automated visual review workflow
- Cloud-based baseline management

**Percy (BrowserStack)** - Best for:
- Cross-browser visual testing
- Responsive design validation
- Integration with existing E2E tests
- Advanced diff algorithms

**BackstopJS** - Best for:
- Lightweight visual regression
- JSON configuration
- Headless browser testing
- Open-source, self-hosted

## Playwright Visual Testing

### Basic Screenshot Testing

```typescript
// components/Button.visual.test.ts
import { test, expect } from '@playwright/test'

test.describe('Button Visual Tests', () => {
  test('default button renders correctly', async ({ page }) => {
    await page.goto('/components/button')

    // Take screenshot of specific element
    const button = page.locator('[data-testid="default-button"]')
    await expect(button).toHaveScreenshot('button-default.png')
  })

  test('button states', async ({ page }) => {
    await page.goto('/components/button')

    // Hover state
    const button = page.locator('[data-testid="default-button"]')
    await button.hover()
    await expect(button).toHaveScreenshot('button-hover.png')

    // Focus state
    await button.focus()
    await expect(button).toHaveScreenshot('button-focus.png')

    // Disabled state
    const disabledButton = page.locator('[data-testid="disabled-button"]')
    await expect(disabledButton).toHaveScreenshot('button-disabled.png')
  })

  test('button variants', async ({ page }) => {
    await page.goto('/components/button')

    const variants = ['primary', 'secondary', 'outline', 'ghost', 'destructive']

    for (const variant of variants) {
      const button = page.locator(`[data-testid="button-${variant}"]`)
      await expect(button).toHaveScreenshot(`button-${variant}.png`)
    }
  })

  test('button sizes', async ({ page }) => {
    await page.goto('/components/button')

    const sizes = ['sm', 'md', 'lg']

    for (const size of sizes) {
      const button = page.locator(`[data-testid="button-${size}"]`)
      await expect(button).toHaveScreenshot(`button-size-${size}.png`)
    }
  })
})
```

### Full Page Screenshots

```typescript
// pages/Dashboard.visual.test.ts
import { test, expect } from '@playwright/test'

test.describe('Dashboard Visual Tests', () => {
  test.beforeEach(async ({ page }) => {
    // Setup: Login and navigate
    await page.goto('/login')
    await page.fill('[name="email"]', 'test@example.com')
    await page.fill('[name="password"]', 'password')
    await page.click('button[type="submit"]')
    await page.waitForURL('/dashboard')
  })

  test('dashboard initial state', async ({ page }) => {
    // Wait for app-owned ready state
    await expect(page.getByTestId('dashboard-root')).toBeVisible()

    // Take full page screenshot
    await expect(page).toHaveScreenshot('dashboard-initial.png', {
      fullPage: true,
      animations: 'disabled' // Disable animations for consistent screenshots
    })
  })

  test('dashboard with filters applied', async ({ page }) => {
    // Apply filters
    await page.click('[data-testid="filter-button"]')
    await page.click('[data-testid="filter-last-30-days"]')
    await page.click('[data-testid="apply-filters"]')

    // Wait for filtered data
    await page.waitForResponse(resp => resp.url().includes('/api/analytics'))
    await expect(page.getByTestId('dashboard-root')).toBeVisible()

    await expect(page).toHaveScreenshot('dashboard-filtered.png', {
      fullPage: true
    })
  })

  test('dashboard responsive layouts', async ({ page }) => {
    const viewports = [
      { name: 'mobile', width: 375, height: 667 },
      { name: 'tablet', width: 768, height: 1024 },
      { name: 'desktop', width: 1920, height: 1080 }
    ]

    for (const viewport of viewports) {
      await page.setViewportSize({ width: viewport.width, height: viewport.height })
      await expect(page.getByTestId('dashboard-root')).toBeVisible()

      await expect(page).toHaveScreenshot(`dashboard-${viewport.name}.png`, {
        fullPage: true
      })
    }
  })
})
```

### Cross-Browser Visual Testing

```typescript
// playwright.config.ts
import { defineConfig, devices } from '@playwright/test'

export default defineConfig({
  projects: [
    {
      name: 'chromium',
      use: { ...devices['Desktop Chrome'] }
    },
    {
      name: 'firefox',
      use: { ...devices['Desktop Firefox'] }
    },
    {
      name: 'webkit',
      use: { ...devices['Desktop Safari'] }
    },
    {
      name: 'mobile-chrome',
      use: { ...devices['Pixel 5'] }
    },
    {
      name: 'mobile-safari',
      use: { ...devices['iPhone 13'] }
    }
  ],

  // Visual comparison settings
  expect: {
    toHaveScreenshot: {
      maxDiffPixels: 100, // Allow up to 100 pixels difference
      threshold: 0.2, // 20% threshold for pixel color difference
      animations: 'disabled'
    }
  }
})
```

### Advanced Visual Testing Techniques

```typescript
// components/Chart.visual.test.ts
import { test, expect } from '@playwright/test'

test.describe('Chart Visual Tests', () => {
  test('chart with stable mock data', async ({ page }) => {
    // Mock API to return consistent data
    await page.route('**/api/chart-data', route => {
      route.fulfill({
        status: 200,
        body: JSON.stringify({
          labels: ['Jan', 'Feb', 'Mar', 'Apr', 'May'],
          data: [10, 20, 15, 25, 30]
        })
      })
    })

    await page.goto('/dashboard/charts')

    // Wait for chart to render
    await expect(page.locator('canvas.chart-canvas')).toBeVisible()

    // Take screenshot with mask for dynamic elements
    await expect(page).toHaveScreenshot('chart-stable.png', {
      mask: [page.locator('[data-testid="timestamp"]')] // Hide timestamp
    })
  })

  test('chart with animations complete', async ({ page }) => {
    await page.goto('/dashboard/charts')

    // Wait for animations to complete
    await page.waitForTimeout(1000) // Wait for chart animation

    await expect(page.locator('.chart-container')).toHaveScreenshot('chart-animated.png')
  })

  test('chart theme variations', async ({ page }) => {
    const themes = ['light', 'dark', 'high-contrast']

    for (const theme of themes) {
      await page.goto('/dashboard/charts')
      await page.evaluate((t) => {
        document.documentElement.setAttribute('data-theme', t)
      }, theme)

      await page.waitForTimeout(500) // Wait for theme transition

      await expect(page.locator('.chart-container')).toHaveScreenshot(`chart-${theme}.png`)
    }
  })
})
```

## Chromatic Visual Testing (Storybook)

### Story Configuration

```typescript
// Button.stories.tsx
import type { Meta, StoryObj } from '@storybook/react'
import { Button } from './Button'

const meta: Meta<typeof Button> = {
  title: 'Components/Button',
  component: Button,
  parameters: {
    chromatic: {
      viewports: [375, 768, 1200], // Test multiple viewports
      delay: 300, // Wait 300ms before screenshot
      pauseAnimationAtEnd: true
    }
  }
}

export default meta
type Story = StoryObj<typeof Button>

export const Primary: Story = {
  args: {
    variant: 'primary',
    children: 'Click me'
  }
}

export const AllVariants: Story = {
  render: () => (
    <div style={{ display: 'flex', gap: '1rem', flexDirection: 'column' }}>
      <Button variant="primary">Primary</Button>
      <Button variant="secondary">Secondary</Button>
      <Button variant="outline">Outline</Button>
      <Button variant="ghost">Ghost</Button>
      <Button variant="destructive">Destructive</Button>
    </div>
  ),
  parameters: {
    chromatic: { disableSnapshot: false }
  }
}

export const InteractiveStates: Story = {
  render: () => (
    <div style={{ display: 'flex', gap: '1rem' }}>
      <Button>Default</Button>
      <Button className="hover">Hover</Button>
      <Button className="focus">Focus</Button>
      <Button disabled>Disabled</Button>
    </div>
  ),
  parameters: {
    pseudo: { hover: ['.hover'], focus: ['.focus'] } // Simulate states
  }
}

export const DarkMode: Story = {
  args: {
    variant: 'primary',
    children: 'Dark Mode'
  },
  parameters: {
    backgrounds: { default: 'dark' },
    chromatic: { modes: { dark: { theme: 'dark' } } }
  }
}
```

### Chromatic Configuration

```javascript
// .storybook/main.js
module.exports = {
  stories: ['../src/**/*.stories.@(ts|tsx)'],
  addons: [
    '@storybook/addon-essentials',
    '@storybook/addon-interactions'
  ],
  framework: {
    name: '@storybook/react-vite',
    options: {}
  }
}
```

```javascript
// chromatic.config.json
{
  "projectToken": "your-project-token",
  "buildScriptName": "build-storybook",
  "exitZeroOnChanges": true,
  "exitOnceUploaded": true,
  "onlyChanged": true, // Only test changed components
  "skip": "dependabot/**", // Skip bot PRs
  "ignoreLastBuildOnBranch": "main"
}
```

## Percy Visual Testing

### Percy with Playwright

```typescript
// tests/visual/HomePage.percy.test.ts
import { test } from '@playwright/test'
import percySnapshot from '@percy/playwright'

test.describe('Home Page Visual Tests', () => {
  test('homepage renders correctly', async ({ page }) => {
    await page.goto('/')
    await expect(page.getByTestId('home-root')).toBeVisible()

    // Take Percy snapshot
    await percySnapshot(page, 'Homepage - Desktop')
  })

  test('homepage responsive', async ({ page }) => {
    await page.goto('/')
    await expect(page.getByTestId('home-root')).toBeVisible()

    // Percy automatically tests configured breakpoints
    await percySnapshot(page, 'Homepage - Responsive', {
      widths: [375, 768, 1280, 1920]
    })
  })

  test('homepage with user logged in', async ({ page, context }) => {
    // Set auth cookie
    await context.addCookies([{
      name: 'session',
      value: 'test-session-token',
      domain: 'localhost',
      path: '/'
    }])

    await page.goto('/')
    await expect(page.getByTestId('home-root')).toBeVisible()

    await percySnapshot(page, 'Homepage - Logged In')
  })

  test('homepage dark mode', async ({ page }) => {
    await page.goto('/')
    await page.evaluate(() => {
      document.documentElement.setAttribute('data-theme', 'dark')
    })
    await page.waitForTimeout(300) // Theme transition

    await percySnapshot(page, 'Homepage - Dark Mode')
  })
})
```

### Percy Configuration

```yaml
# .percy.yml
version: 2
static:
  cleanUrls: true
  include: '**/*.{html,htm}'
  exclude: '**/node_modules/**'

snapshot:
  widths:
    - 375  # Mobile
    - 768  # Tablet
    - 1280 # Desktop
    - 1920 # Large Desktop

  min-height: 1024

  # Enable Percy-specific features
  enable-javascript: true

  # CSS for stabilizing screenshots
  percy-css: |
    * {
      animation-duration: 0s !important;
      transition-duration: 0s !important;
    }
    [data-percy-hide] {
      visibility: hidden !important;
    }

discovery:
  allowed-hostnames:
    - localhost
    - '*.yourdomain.com'

  network-idle-timeout: 750
```

## BackstopJS Visual Regression

### BackstopJS Configuration

```javascript
// backstop.config.js
module.exports = {
  id: 'visual_regression_test',
  viewports: [
    {
      label: 'phone',
      width: 375,
      height: 667
    },
    {
      label: 'tablet',
      width: 768,
      height: 1024
    },
    {
      label: 'desktop',
      width: 1920,
      height: 1080
    }
  ],

  scenarios: [
    {
      label: 'Homepage',
      url: 'http://localhost:3000',
      delay: 1000,
      misMatchThreshold: 0.1,
      requireSameDimensions: true
    },
    {
      label: 'Button Component',
      url: 'http://localhost:3000/components/button',
      selectors: ['[data-testid="button-showcase"]'],
      delay: 500,
      hoverSelector: '[data-testid="button-primary"]',
      clickSelector: '[data-testid="button-toggle"]'
    },
    {
      label: 'Dashboard - Logged In',
      url: 'http://localhost:3000/dashboard',
      cookiePath: 'backstop_data/cookies.json',
      delay: 2000,
      removeSelectors: [
        '[data-testid="timestamp"]', // Hide dynamic timestamp
        '[data-testid="live-data"]'   // Hide live updating data
      ]
    },
    {
      label: 'Form Validation',
      url: 'http://localhost:3000/contact',
      onBeforeScript: 'puppet/onBefore.js',
      onReadyScript: 'puppet/fillForm.js',
      delay: 500
    }
  ],

  paths: {
    bitmaps_reference: 'backstop_data/bitmaps_reference',
    bitmaps_test: 'backstop_data/bitmaps_test',
    engine_scripts: 'backstop_data/engine_scripts',
    html_report: 'backstop_data/html_report',
    ci_report: 'backstop_data/ci_report'
  },

  report: ['browser', 'CI'],
  engine: 'puppeteer',
  engineOptions: {
    args: ['--no-sandbox']
  },

  asyncCaptureLimit: 5,
  asyncCompareLimit: 50,

  debug: false,
  debugWindow: false
}
```

### BackstopJS Custom Scripts

```javascript
// backstop_data/engine_scripts/puppet/fillForm.js
module.exports = async (page, scenario, viewport) => {
  console.log('Filling form for scenario:', scenario.label)

  // Fill form fields
  await page.type('[name="email"]', 'test@example.com')
  await page.type('[name="name"]', 'Test User')
  await page.type('[name="message"]', 'This is a test message')

  // Trigger validation by clicking submit
  await page.click('button[type="submit"]')

  // Wait for validation messages
  await expect(page.locator('.validation-message')).toBeVisible({ timeout: 1000 })
}
```

```javascript
// backstop_data/engine_scripts/puppet/onBefore.js
module.exports = async (page, scenario, viewport) => {
  console.log('Running onBefore for:', scenario.label)

  // Set cookies for authenticated scenarios
  if (scenario.cookiePath) {
    const cookies = require(scenario.cookiePath)
    await page.setCookie(...cookies)
  }

  // Hide dynamic elements
  await page.evaluateOnNewDocument(() => {
    window.localStorage.setItem('disable-animations', 'true')
  })
}
```

## CI/CD Integration

### GitHub Actions with Playwright

```yaml
# .github/workflows/visual-tests.yml
name: Visual Regression Tests

on:
  pull_request:
    branches: [main, develop]

jobs:
  visual-tests:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: 18
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright Browsers
        run: npx playwright install --with-deps

      - name: Run visual tests
        run: npm run test:visual

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: visual-test-results
          path: test-results/
          retention-days: 30

      - name: Upload screenshots
        if: failure()
        uses: actions/upload-artifact@v3
        with:
          name: failed-screenshots
          path: test-results/**/*-diff.png
```

### GitHub Actions with Chromatic

```yaml
# .github/workflows/chromatic.yml
name: Chromatic Visual Tests

on: push

jobs:
  chromatic:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v3
        with:
          fetch-depth: 0 # Full git history for Chromatic

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: 18

      - name: Install dependencies
        run: npm ci

      - name: Run Chromatic
        uses: chromaui/action@v1
        with:
          projectToken: ${{ secrets.CHROMATIC_PROJECT_TOKEN }}
          buildScriptName: 'build-storybook'
          exitZeroOnChanges: true
          onlyChanged: true # Only test changed stories
```

## Best Practices Checklist

- [ ] Disable animations and transitions in visual tests
- [ ] Wait for network idle before taking screenshots
- [ ] Use data-testid attributes for stable selectors
- [ ] Mask or hide dynamic content (timestamps, live data)
- [ ] Test multiple viewports (mobile, tablet, desktop)
- [ ] Test interactive states (hover, focus, disabled)
- [ ] Test theme variations (light, dark, high-contrast)
- [ ] Use consistent mock data for charts and dynamic content
- [ ] Set appropriate mismatch thresholds (0.1% - 1%)
- [ ] Store reference screenshots in version control or cloud
- [ ] Review visual diffs in CI/CD pipeline
- [ ] Test cross-browser compatibility (Chrome, Firefox, Safari)
- [ ] Isolate component testing with Storybook

## Common Pitfalls

[FAIL] **Not waiting for content to load**:
```typescript
// Bad - Screenshot taken before content loads
await page.goto('/dashboard')
await expect(page).toHaveScreenshot()

// Good - Wait for an app-owned ready marker
await page.goto('/dashboard')
await expect(page.getByTestId('dashboard-root')).toBeVisible()
await expect(page).toHaveScreenshot()
```

[FAIL] **Testing with animations enabled**:
```typescript
// Bad - Animations cause flaky tests
await expect(page).toHaveScreenshot()

// Good - Disable animations
await expect(page).toHaveScreenshot({
  animations: 'disabled'
})
```

[FAIL] **Not handling dynamic content**:
```typescript
// Bad - Timestamp causes every test to fail
await expect(page).toHaveScreenshot('dashboard.png')

// Good - Mask dynamic elements
await expect(page).toHaveScreenshot('dashboard.png', {
  mask: [page.locator('[data-testid="timestamp"]')]
})
```

[FAIL] **Overly strict thresholds**:
```typescript
// Bad - Fails on minor anti-aliasing differences
await expect(page).toHaveScreenshot({
  maxDiffPixels: 0
})

// Good - Allow minor pixel differences
await expect(page).toHaveScreenshot({
  maxDiffPixels: 100,
  threshold: 0.2
})
```

## Testing Workflow

1. **Initial baseline**: Run tests and accept all screenshots as baseline
```bash
npm run test:visual -- --update-snapshots
```

2. **Development**: Make UI changes and run tests
```bash
npm run test:visual
```

3. **Review diffs**: Check diff images for unintended changes
```bash
open test-results/*-diff.png
```

4. **Update baselines**: Accept intentional changes
```bash
npm run test:visual -- --update-snapshots
```

5. **CI/CD**: Automated visual testing on every PR

## Related Resources

See [../../references/comprehensive-testing-guide.md](../../references/comprehensive-testing-guide.md) for complete testing guide across all layers.

{
  "metadata": {
    "skill": "qa-testing-strategy",
    "updated": "2026-07-11",
    "version": "3.6",
    "total_sources": 45,
    "description": "Primary references for risk-based testing strategy, shift-left gates, flake control, CI economics, component testing, contract testing, schema fuzzing, observability-driven testing, synthetic data, mutation coverage, test impact analysis, property-based testing, and shift-right techniques. AI sources are optional.",
    "title": "QA Testing Strategy - Sources",
    "last_updated": "2026-06-09"
  },
  "categories": {
    "sre_and_reliability": [
      {
        "name": "Google SRE Book - Service Level Objectives",
        "url": "https://sre.google/sre-book/service-level-objectives/",
        "description": "SLO/error budget framing for quality gates and release decisions.",
        "add_as_web_search": true,
        "optional": false
      },
      {
        "name": "Google SRE Book - Effective Troubleshooting",
        "url": "https://sre.google/sre-book/effective-troubleshooting/",
        "description": "Evidence-based troubleshooting workflow that maps well to flaky-test and incident triage.",
        "add_as_web_search": true,
        "optional": false
      }
    ],
    "contracts_and_schemas": [
      {
        "name": "OpenAPI Specification (Latest)",
        "url": "https://spec.openapis.org/oas/latest.html",
        "description": "Canonical contract format for REST APIs; use for shift-left contract validation and test oracles.",
        "add_as_web_search": true,
        "optional": false
      },
      {
        "name": "AsyncAPI Specification (Latest)",
        "url": "https://www.asyncapi.com/docs/reference/specification/latest",
        "description": "Current contract format for event-driven APIs; use for schema and integration tests.",
        "add_as_web_search": true,
        "optional": false
      },
      {
        "name": "JSON Schema",
        "url": "https://json-schema.org/",
        "description": "Schema standard for structured data and contract testing.",
        "add_as_web_search": true,
        "optional": false
      },
      {
        "name": "Pact - Contract Testing Docs",
        "url": "https://docs.pact.io/",
        "description": "Consumer-driven contract testing patterns and tooling.",
        "add_as_web_search": true,
        "optional": false
      },
      {
        "name": "Specmatic - Contract-Driven Development",
        "url": "https://specmatic.io/",
        "description": "Contract-driven development using OpenAPI as executable contracts; alternative to Pact for API-first teams.",
        "add_as_web_search": true,
        "optional": false
      },
      {
        "name": "Karate - API Testing DSL",
        "url": "https://karatelabs.github.io/karate/",
        "description": "Unified DSL for API testing, contract testing, and performance testing in one framework.",
        "add_as_web_search": true,
        "optional": false
      }
    ],
    "e2e_and_ui_testing": [
      {
        "name": "Playwright - Best Practices",
        "url": "https://playwright.dev/docs/best-practices",
        "description": "High-signal E2E practices: locators, web-first assertions, debugging tooling, sharding.",
        "add_as_web_search": true,
        "optional": false
      },
      {
        "name": "Playwright - Locators",
        "url": "https://playwright.dev/docs/locators",
        "description": "Locator strategy (roles/labels/test IDs) and stability guidance.",
        "add_as_web_search": true,
        "optional": false
      },
      {
        "name": "Playwright - Accessibility Testing",
        "url": "https://playwright.dev/docs/accessibility-testing",
        "description": "Official guidance for integrating accessibility checks into Playwright suites.",
        "add_as_web_search": true,
        "optional": false
      },
      {
        "name": "Cypress Documentation",
        "url": "https://docs.cypress.io/",
        "description": "Alternative E2E framework with excellent DX; strong for frontend-focused teams.",
        "add_as_web_search": true,
        "optional": false
      }
    ],
    "unit_testing": [
      {
        "name": "Vitest Documentation",
        "url": "https://vitest.dev/",
        "description": "Fast Vite-native unit test runner; increasingly popular for modern JS/TS projects.",
        "add_as_web_search": true,
        "optional": false
      },
      {
        "name": "Vitest - Browser Component Testing",
        "url": "https://vitest.dev/guide/browser/component-testing",
        "description": "Component testing in a real browser using Vitest Browser Mode.",
        "add_as_web_search": true,
        "optional": false
      },
      {
        "name": "Vitest - Browser Visual Regression Testing",
        "url": "https://vitest.dev/guide/browser/visual-regression-testing",
        "description": "Real-browser visual regression workflows for component and page tests.",
        "add_as_web_search": true,
        "optional": false
      }
    ],
    "schema_fuzzing": [
      {
        "name": "Schemathesis Documentation",
        "url": "https://schemathesis.readthedocs.io/",
        "description": "Schema-aware API fuzzing and property-based checks for OpenAPI and GraphQL.",
        "add_as_web_search": true,
        "optional": false
      }
    ],
    "integration_testing": [
      {
        "name": "Testcontainers",
        "url": "https://testcontainers.com/",
        "description": "Hermetic integration testing with real dependencies (DB, queues) running in containers.",
        "add_as_web_search": true,
        "optional": false
      },
      {
        "name": "testcontainers-node",
        "url": "https://node.testcontainers.org/",
        "description": "Testcontainers for Node.js/TypeScript; common choice for API + DB integration tests.",
        "add_as_web_search": true,
        "optional": false
      }
    ],
    "performance_and_capacity": [
      {
        "name": "k6 Documentation",
        "url": "https://grafana.com/docs/k6/latest/",
        "description": "Current k6 documentation for load testing scenarios, thresholds, and CI integration.",
        "add_as_web_search": true,
        "optional": false
      },
      {
        "name": "Web Vitals",
        "url": "https://web.dev/vitals/",
        "description": "Core Web Vitals guidance for performance budgets and monitoring.",
        "add_as_web_search": true,
        "optional": false
      }
    ],
    "chaos_engineering": [
      {
        "name": "Principles of Chaos Engineering",
        "url": "https://principlesofchaos.org/",
        "description": "Foundational principles for chaos engineering practice.",
        "add_as_web_search": true,
        "optional": false
      },
      {
        "name": "LitmusChaos",
        "url": "https://litmuschaos.io/",
        "description": "Open-source Kubernetes-native chaos engineering platform.",
        "add_as_web_search": true,
        "optional": false
      },
      {
        "name": "Gremlin - Chaos Engineering",
        "url": "https://www.gremlin.com/chaos-engineering",
        "description": "Enterprise chaos engineering platform with attack library and gameday automation.",
        "add_as_web_search": true,
        "optional": false
      },
      {
        "name": "AWS Fault Injection Simulator",
        "url": "https://aws.amazon.com/fis/",
        "description": "AWS-native chaos engineering service for controlled experiments.",
        "add_as_web_search": true,
        "optional": false
      }
    ],
    "observability_and_tracing": [
      {
        "name": "OpenTelemetry Documentation",
        "url": "https://opentelemetry.io/docs/",
        "description": "Vendor-neutral observability framework for traces, metrics, and logs.",
        "add_as_web_search": true,
        "optional": false
      },
      {
        "name": "Tracetest",
        "url": "https://tracetest.io/",
        "description": "Trace-based testing tool for asserting on OpenTelemetry traces.",
        "add_as_web_search": true,
        "optional": false
      },
      {
        "name": "Jaeger Tracing",
        "url": "https://www.jaegertracing.io/",
        "description": "Open-source distributed tracing platform for monitoring microservices.",
        "add_as_web_search": true,
        "optional": false
      }
    ],
    "synthetic_data": [
      {
        "name": "K2view Test Data Management",
        "url": "https://www.k2view.com/solutions/test-data-management/",
        "description": "Enterprise test data management with subsetting, masking, and synthetic generation.",
        "add_as_web_search": true,
        "optional": false
      },
      {
        "name": "Synthesized - Synthetic Data Platform",
        "url": "https://www.synthesized.io/",
        "description": "AI-powered synthetic data generation for privacy-compliant testing.",
        "add_as_web_search": true,
        "optional": false
      }
    ],
    "accessibility": [
      {
        "name": "W3C - WCAG Overview",
        "url": "https://www.w3.org/WAI/standards-guidelines/wcag/",
        "description": "Accessibility baseline; align automated checks and manual audits with WCAG 2.2 targets.",
        "add_as_web_search": true,
        "optional": false
      },
      {
        "name": "axe-core",
        "url": "https://github.com/dequelabs/axe-core",
        "description": "Automation engine for accessibility rules; integrate into component/UI tests.",
        "add_as_web_search": true,
        "optional": false
      }
    ],
    "quality_metrics": [
      {
        "name": "Stryker Mutator Documentation",
        "url": "https://stryker-mutator.io/docs/",
        "description": "Mutation testing guidance for measuring test effectiveness beyond line coverage. Primary 2026 gate for validating AI/agent-authored tests: line coverage measures execution, mutation score measures detection. Official VS Code plugin announced 2025-11-07, StrykerJS-only at launch (v9.3.0+); other flavors on roadmap.",
        "add_as_web_search": true,
        "optional": false,
        "last_verified": "2026-07-11"
      },
      {
        "name": "PIT (Pitest) - Java Mutation Testing",
        "url": "https://pitest.org/",
        "description": "De facto mutation testing tool for Java and Kotlin; bytecode-level mutation with Maven/Gradle integration. Version 1.19.x adds scmMutationCoverage, which delegates change detection to an <scm> block via Maven SCM rather than reading Git directly, for incremental PR-scoped runs.",
        "add_as_web_search": true,
        "optional": false,
        "last_verified": "2026-07-11"
      }
    ],
    "test_impact_analysis": [
      {
        "name": "Launchable - Predictive Test Selection",
        "url": "https://www.launchableinc.com/",
        "description": "ML-powered test impact analysis; subsets CI test runs to the tests most likely to catch regressions for a given change.",
        "add_as_web_search": true,
        "optional": false
      },
      {
        "name": "Datadog Test Optimization",
        "url": "https://docs.datadoghq.com/tests/",
        "description": "CI test observability: per-test traces, flake detection, early flake detection, and test impact analysis across languages. Renamed from 'CI Visibility' to 'Test Visibility' to 'Test Optimization' across 2023-2025; confirmed current as of 2026-07-11.",
        "add_as_web_search": true,
        "optional": false,
        "last_verified": "2026-07-11"
      },
      {
        "name": "BuildPulse - Flake Trending",
        "url": "https://buildpulse.io/",
        "description": "Flake detection and trending from JUnit XML reports; quarantine recommendations and owner attribution.",
        "add_as_web_search": true,
        "optional": false
      }
    ],
    "security": [
      {
        "name": "OWASP ZAP Documentation",
        "url": "https://www.zaproxy.org/docs/",
        "description": "DAST scanning patterns and automation.",
        "add_as_web_search": true,
        "optional": false
      },
      {
        "name": "OWASP Application Security Verification Standard (ASVS)",
        "url": "https://owasp.org/www-project-application-security-verification-standard/",
        "description": "Security verification requirements for planning test coverage and gates.",
        "add_as_web_search": true,
        "optional": false
      }
    ],
    "optional_ai_automation": [
      {
        "name": "OWASP Top 10 for LLM Applications",
        "url": "https://owasp.org/www-project-top-10-for-large-language-model-applications/",
        "description": "Risk categories and mitigations for AI-assisted workflows; treat as optional extension.",
        "add_as_web_search": true,
        "optional": true
      },
      {
        "name": "NIST AI Risk Management Framework",
        "url": "https://www.nist.gov/itl/ai-risk-management-framework",
        "description": "AI governance baseline; useful when adopting AI-assisted testing and triage.",
        "add_as_web_search": true,
        "optional": true
      },
      {
        "name": "Meticulous - AI E2E Testing",
        "url": "https://meticulous.ai/",
        "description": "AI-powered automated E2E test recording and maintenance; emerging 2026 tool.",
        "add_as_web_search": true,
        "optional": true
      }
    ],
    "property_based_testing": [
      {
        "name": "fast-check Documentation",
        "url": "https://fast-check.dev/",
        "description": "Property-based testing for JavaScript and TypeScript; works with Vitest and Jest. Best-in-class shrinking; used for universal invariants, round-trip proofs, and AI-code edge-case discovery.",
        "add_as_web_search": true,
        "optional": false,
        "last_verified": "2026-06-09"
      },
      {
        "name": "Hypothesis Documentation",
        "url": "https://hypothesis.readthedocs.io/",
        "description": "Property-based testing for Python; integrates natively with pytest. Supports stateful (rule-based state machine) testing. Mature and stable as of Hypothesis 6.x.",
        "add_as_web_search": true,
        "optional": false,
        "last_verified": "2026-06-09"
      },
      {
        "name": "jqwik User Guide",
        "url": "https://jqwik.net/docs/current/user-guide.html",
        "description": "Property-based testing framework for Java built on JUnit 5. More feature-complete than QuickCheck ports for JVM teams.",
        "add_as_web_search": true,
        "optional": false,
        "last_verified": "2026-06-09"
      }
    ],
    "merge_queue_and_ci": [
      {
        "name": "GitHub Merge Queue Documentation",
        "url": "https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue",
        "description": "Official GitHub merge queue setup and configuration. Critical for understanding how flaky tests interact with queued merges and what checks to designate as required vs informational.",
        "add_as_web_search": true,
        "optional": false,
        "last_verified": "2026-06-09"
      }
    ]
  }
}

# qa-testing-strategy — Consolidated Learnings

Curated, dated, committed memory for this skill. Pruned from raw `learnings.md` via `agents-skills-feedback-loop/scripts/consolidate.py`. Human-approved.

Cap: 60 entries. When exceeded, promote durable rules to `references/`.

## Filter Override

<!-- Add 2-4 bullets that sharpen what counts as a learning for this skill. Leave empty to use the default filter from agents-skills-feedback-loop/references/learnings-format.md. -->

## Patterns That Work

## Mistakes to Avoid

## Domain Knowledge

## Open Questions

## Consolidated Principles

# qa-testing-strategy — Learnings

## Patterns That Work

## Mistakes to Avoid

- [2026-07-11] reliability-theory-applied.md's error-budget gate example confused a 720-hour SLO window with a 720-minute budget total (actual budget: 43.2 min), yielding the wrong gate tier — always re-derive worked arithmetic in examples.
## Domain Knowledge

- [2026-07-11] Coverage/pyramid-ratio guidance must be explicitly scoped to critical-path modules and paired with a mutation-score gate, or it silently contradicts the skill's own 100%-coverage-mandate anti-pattern warning.
## Open Questions

## Consolidated Principles

# Chaos Engineering & Resilience Testing

## Table of Contents

- [Contents](#contents)
- [When to Use This Reference](#when-to-use-this-reference)
- [Core Principles](#core-principles)
- [Chaos Engineering Tools](#chaos-engineering-tools)
- [Experiment Categories](#experiment-categories)
- [CI/CD Integration](#cicd-integration)
- [Compliance: DORA & SOC 2](#compliance-dora--soc-2)
- [Chaos Experiment Report](#chaos-experiment-report)
- [Steady State Metrics](#steady-state-metrics)
- [Best Practices](#best-practices)
- [Quick Start Checklist](#quick-start-checklist)
- [Related References](#related-references)
- [External Resources](#external-resources)

Proactive reliability validation through controlled failure injection. Use chaos engineering to discover weaknesses before they cause production incidents.

## Contents

- When to Use This Reference
- Core Principles
- Chaos Engineering Tools
- Experiment Categories
- CI/CD Integration
- Compliance: DORA & SOC 2
- Chaos Experiment Report
- Steady State Metrics
- Best Practices
- Quick Start Checklist
- Related References
- External Resources

---

## When to Use This Reference

- Validating system resilience before major releases
- Preparing for compliance audits (DORA, SOC 2)
- Building confidence in disaster recovery plans
- Testing failover mechanisms and circuit breakers
- Validating auto-scaling and self-healing infrastructure

---

## Core Principles

### Build-Measure-Learn Cycle

```text
1. STEADY STATE
   Define normal behavior metrics (latency, error rate, throughput)

2. HYPOTHESIS
   "The system will maintain <metric> within <threshold> when <failure>"

3. EXPERIMENT
   Inject controlled failure in staging/production

4. OBSERVE
   Measure deviation from steady state

5. LEARN
   Fix weaknesses, update runbooks, repeat
```

### Blast Radius Containment

| Environment | Blast Radius | Approval |
|-------------|--------------|----------|
| Development | Full chaos | None |
| Staging | Targeted services | Team lead |
| Production (canary) | 1-5% traffic | SRE + Engineering lead |
| Production (full) | Full system | VP Engineering + SRE |

**Rule:** Start small, expand gradually, always have a kill switch.

---

## Chaos Engineering Tools

### Tool Comparison

| Tool | Best For | Language | Kubernetes | Cloud |
|------|----------|----------|------------|-------|
| **Gremlin** | Enterprise, SaaS | Any | Yes | AWS, GCP, Azure |
| **LitmusChaos** | Kubernetes-native, OSS | Go | Yes | Any |
| **AWS FIS** | AWS workloads | Any | EKS | AWS only |
| **Chaos Monkey** | Netflix OSS ecosystem | Java | Limited | AWS |
| **Steadybit** | SRE workflows | Any | Yes | Multi-cloud |
| **Chaos Toolkit** | Extensible, CI/CD | Python | Yes | Any |

### LitmusChaos Example

```yaml
# litmus-experiment.yaml
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosExperiment
metadata:
  name: pod-delete
spec:
  definition:
    scope: Namespaced
    permissions:
      - apiGroups: [""]
        resources: ["pods"]
        verbs: ["delete", "list", "get"]
    image: litmuschaos/go-runner:latest
    args:
      - -c
      - ./experiments -name pod-delete
    env:
      - name: TOTAL_CHAOS_DURATION
        value: "30"
      - name: CHAOS_INTERVAL
        value: "10"
      - name: FORCE
        value: "false"
```

### Gremlin Attack Types

```text
Resource Attacks:
├── CPU              # Consume CPU cycles
├── Memory           # Consume memory
├── Disk             # Fill disk space
├── IO               # Slow disk I/O
└── Process Killer   # Kill specific processes

Network Attacks:
├── Latency          # Add network delay
├── Packet Loss      # Drop packets
├── Blackhole        # Drop all traffic
├── DNS              # DNS failures
└── Certificate      # TLS/SSL failures

State Attacks:
├── Shutdown         # Graceful shutdown
├── Time Travel      # Change system clock
└── Process Killer   # Kill by name/PID
```

---

## Experiment Categories

### 1. Infrastructure Failures

| Experiment | Validates | Example |
|------------|-----------|---------|
| **Instance termination** | Auto-scaling, failover | Kill 1 of 3 API servers |
| **Zone failure** | Multi-AZ deployment | Blackhole us-east-1a |
| **Disk exhaustion** | Alerting, cleanup jobs | Fill 95% disk |
| **Memory pressure** | OOM handling, graceful degradation | Consume 90% memory |

### 2. Network Failures

| Experiment | Validates | Example |
|------------|-----------|---------|
| **Latency injection** | Timeout handling, SLOs | Add 500ms to database calls |
| **Packet loss** | Retry logic, circuit breakers | 10% packet loss to cache |
| **DNS failure** | Fallback resolution | Block DNS for payment service |
| **Partition** | Split-brain handling | Isolate region from cluster |

### 3. Application Failures

| Experiment | Validates | Example |
|------------|-----------|---------|
| **Dependency failure** | Circuit breakers, fallbacks | Kill Redis |
| **Slow dependency** | Timeout configuration | Add 2s latency to auth service |
| **Error injection** | Error handling, logging | Return 500 from 10% of API calls |
| **Resource exhaustion** | Connection pooling, limits | Exhaust database connections |

---

## CI/CD Integration

### GitHub Actions Example

```yaml
name: Chaos Testing
on:
  schedule:
    - cron: '0 2 * * 1-5'  # Weekday nights
  workflow_dispatch:
    inputs:
      experiment:
        description: 'Chaos experiment to run'
        required: true
        type: choice
        options:
          - pod-delete
          - network-latency
          - cpu-stress

jobs:
  chaos-test:
    runs-on: ubuntu-latest
    environment: staging
    steps:
      - uses: actions/checkout@v4

      - name: Setup kubectl
        uses: azure/setup-kubectl@v3

      - name: Install LitmusChaos
        run: |
          kubectl apply -f https://litmuschaos.github.io/litmus/litmus-operator-v3.0.0.yaml
          kubectl wait --for=condition=Ready pods -l app=chaos-operator -n litmus

      - name: Run Chaos Experiment
        run: |
          kubectl apply -f chaos-experiments/${{ inputs.experiment }}.yaml
          kubectl wait --for=condition=ChaosResultVerdict=Pass \
            chaosresult/${{ inputs.experiment }}-result -n default --timeout=300s

      - name: Collect Results
        if: always()
        run: |
          kubectl get chaosresult -n default -o yaml > chaos-results.yaml

      - name: Upload Results
        uses: actions/upload-artifact@v4
        with:
          name: chaos-results
          path: chaos-results.yaml
```

### Game Day Automation

```bash
#!/bin/bash
# game-day-runner.sh

set -euo pipefail

EXPERIMENTS=(
  "pod-delete:api-service"
  "network-latency:database:500ms"
  "cpu-stress:worker:80%"
)

echo "Starting Game Day: $(date)"

for exp in "${EXPERIMENTS[@]}"; do
  IFS=':' read -r type target params <<< "$exp"

  echo "Running: $type on $target with $params"

  # Run experiment
  litmus run --experiment "$type" --target "$target" --params "$params"

  # Collect metrics during experiment
  prometheus-query "rate(http_requests_total{status=~'5..'}[1m])" > "metrics-$type.json"

  # Wait for recovery
  sleep 60

  # Verify steady state restored
  if ! verify-steady-state; then
    echo "ALERT: System did not recover from $type"
    exit 1
  fi
done

echo "Game Day Complete: All experiments passed"
```

---

## Compliance: DORA & SOC 2

### DORA (Digital Operational Resilience Act)

DORA requires financial entities to regularly test ICT resilience. Chaos engineering provides:

| DORA Requirement | Chaos Engineering Practice |
|------------------|---------------------------|
| ICT risk management | Proactive failure discovery |
| ICT-related incident management | Runbook validation |
| Digital operational resilience testing | Chaos experiments |
| Third-party risk management | Dependency failure testing |
| Information sharing | Post-mortem culture |

### SOC 2 Alignment

| SOC 2 Criteria | Chaos Engineering Evidence |
|----------------|---------------------------|
| Availability | Uptime during chaos experiments |
| Processing Integrity | Data consistency after failures |
| Confidentiality | Access controls during incidents |
| Security | Attack surface validation |

### Audit Documentation

```markdown
## Chaos Experiment Report

**Experiment ID:** CHX-2026-001
**Date:** `<YYYY-MM-DD>`
**Environment:** Production (5% canary)
**Conducted By:** SRE Team

### Hypothesis
The payment service will maintain <100ms p99 latency when
the primary database fails over to replica.

### Experiment Details
- **Attack Type:** Database primary termination
- **Duration:** 5 minutes
- **Blast Radius:** 5% of production traffic
- **Kill Switch:** Immediate rollback via feature flag

### Results
| Metric | Baseline | During Experiment | Pass/Fail |
|--------|----------|-------------------|-----------|
| p99 Latency | 45ms | 120ms | FAIL |
| Error Rate | 0.01% | 0.8% | FAIL |
| Failover Time | N/A | 45s | N/A |

### Findings
1. Connection pool not warming on failover
2. DNS TTL too high (300s → should be 30s)
3. Health checks not detecting stale connections

### Remediation
- [ ] Implement connection pool pre-warming
- [ ] Reduce DNS TTL to 30s
- [ ] Add active health checks to connection pool

### Sign-off
- Engineering Lead: _____________ Date: _______
- SRE Lead: _____________ Date: _______
```

---

## Steady State Metrics

### Define Before Experimenting

```yaml
steady_state:
  metrics:
    - name: error_rate
      query: "sum(rate(http_requests_total{status=~'5..'}[5m])) / sum(rate(http_requests_total[5m]))"
      threshold: "< 0.01"  # 1%

    - name: p99_latency
      query: "histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))"
      threshold: "< 0.2"  # 200ms

    - name: throughput
      query: "sum(rate(http_requests_total[5m]))"
      threshold: "> 1000"  # 1000 RPS

    - name: saturation
      query: "avg(container_memory_usage_bytes / container_spec_memory_limit_bytes)"
      threshold: "< 0.8"  # 80%
```

### SLO-Based Thresholds

| SLI | SLO | Chaos Threshold |
|-----|-----|-----------------|
| Availability | 99.9% | Error rate < 5% during experiment |
| Latency (p99) | < 200ms | < 500ms during experiment |
| Throughput | > 1000 RPS | > 800 RPS during experiment |

---

## Best Practices

### Do

- Start with read-only experiments (latency, not data corruption)
- Run experiments during business hours (team available)
- Have clear rollback procedures before starting
- Document hypotheses and results
- Share learnings across teams
- Automate recurring experiments in CI/CD

### Avoid

- Running in production without staging validation first
- Experiments without clear success criteria
- Chaos without observability (you won't know what happened)
- Skipping post-mortems after failures
- Running during incident response or high-traffic events

---

## Quick Start Checklist

- [ ] Define steady state metrics (error rate, latency, throughput)
- [ ] Choose chaos tool (LitmusChaos for K8s, Gremlin for enterprise)
- [ ] Start in development/staging environment
- [ ] Write first hypothesis: "System X will maintain Y when Z fails"
- [ ] Run experiment with minimal blast radius
- [ ] Document findings and remediation
- [ ] Schedule recurring experiments (weekly/monthly)
- [ ] Integrate with CI/CD for pre-release validation

---

## Related References

- [operational-playbook.md](operational-playbook.md) — Test pyramid and CI gates
- [../SKILL.md](../SKILL.md) — Main testing strategy overview
- [../../ops-devops-platform/SKILL.md](../../ops-devops-platform/SKILL.md) — CI/CD and infrastructure
- [../../software-security-appsec/SKILL.md](../../software-security-appsec/SKILL.md) — Security testing

---

## External Resources

- [Gremlin Chaos Engineering](https://www.gremlin.com/chaos-engineering)
- [LitmusChaos](https://litmuschaos.io/)
- [AWS Fault Injection Simulator](https://aws.amazon.com/fis/)
- [Steadybit](https://steadybit.com/)
- [Principles of Chaos Engineering](https://principlesofchaos.org/)

# Compliance Testing

## Table of Contents

- [Contents](#contents)
- [Compliance Standards Overview](#compliance-standards-overview)
- [Compliance-as-Code](#compliance-as-code)
- [Audit Evidence Automation](#audit-evidence-automation)
- [Access Control Testing](#access-control-testing)
- [Data Residency Verification](#data-residency-verification)
- [Encryption Validation](#encryption-validation)
- [PII Handling Tests](#pii-handling-tests)
- [Data Retention Policy Enforcement](#data-retention-policy-enforcement)
- [Audit Log Completeness Testing](#audit-log-completeness-testing)
- [Penetration Testing Requirements](#penetration-testing-requirements)
- [Vulnerability Scanning Cadence](#vulnerability-scanning-cadence)
- [Compliance Test Matrices](#compliance-test-matrices)
- [CI Integration for Compliance Gates](#ci-integration-for-compliance-gates)
- [Documentation and Evidence Collection](#documentation-and-evidence-collection)
- [Compliance Testing Checklist](#compliance-testing-checklist)
- [Related Resources](#related-resources)

Compliance testing patterns for regulated environments -- automating audit evidence, validating security controls, and enforcing policy-as-code for SOC 2, HIPAA, GDPR, and PCI-DSS.

## Contents

- Compliance Standards Overview
- Compliance-as-Code
- Audit Evidence Automation
- Access Control Testing
- Data Residency Verification
- Encryption Validation
- PII Handling Tests
- Data Retention Policy Enforcement
- Audit Log Completeness Testing
- Penetration Testing Requirements
- Vulnerability Scanning Cadence
- Compliance Test Matrices
- CI Integration for Compliance Gates
- Documentation and Evidence Collection
- Compliance Testing Checklist
- Related Resources

---

## Compliance Standards Overview

| Standard | Scope | Key Requirements | Applies To |
|----------|-------|-----------------|------------|
| **SOC 2** | Service organizations | Security, availability, processing integrity, confidentiality, privacy | SaaS, cloud services |
| **HIPAA** | Healthcare data | PHI protection, access controls, audit trails, encryption | Healthcare apps |
| **GDPR** | EU personal data | Consent, right to erasure, data portability, breach notification | Any app with EU users |
| **PCI-DSS** | Payment card data | Cardholder data protection, network security, access control | E-commerce, payments |

### Testing Obligations by Standard

| Testing Type | SOC 2 | HIPAA | GDPR | PCI-DSS |
|-------------|-------|-------|------|---------|
| Access control testing | Required | Required | Required | Required |
| Encryption validation | Required | Required | Required | Required |
| Audit log testing | Required | Required | Recommended | Required |
| Penetration testing | Recommended | Required | Recommended | Required (annual) |
| Vulnerability scanning | Recommended | Required | Recommended | Required (quarterly) |
| Data retention testing | Recommended | Required | Required | Required |
| Incident response testing | Recommended | Required | Required | Required |

---

## Compliance-as-Code

### Chef InSpec

InSpec defines compliance controls as testable code. Each control maps to a specific regulatory requirement.

```ruby
# controls/encryption.rb
control 'ENCRYPT-001' do
  impact 1.0
  title 'Data at rest must be encrypted'
  desc 'All database volumes and storage must use AES-256 encryption.'
  tag compliance: ['SOC2-CC6.1', 'HIPAA-164.312(a)(2)(iv)', 'PCI-DSS-3.4']

  describe aws_ebs_volumes do
    it { should exist }
    its('entries') { should all(be_encrypted) }
  end

  describe aws_rds_instances do
    it { should exist }
    its('entries') { should all(have_storage_encrypted) }
  end

  describe aws_s3_buckets do
    it { should exist }
  end
end

control 'ENCRYPT-002' do
  impact 1.0
  title 'Data in transit must use TLS 1.2+'
  desc 'All external endpoints must enforce TLS 1.2 or higher.'
  tag compliance: ['SOC2-CC6.7', 'PCI-DSS-4.1']

  describe ssl(host: 'api.example.com', port: 443) do
    it { should be_enabled }
    its('protocols') { should_not include 'ssl2' }
    its('protocols') { should_not include 'ssl3' }
    its('protocols') { should_not include 'tls1.0' }
    its('protocols') { should_not include 'tls1.1' }
  end
end
```

```bash
# Run InSpec compliance checks
inspec exec controls/ --reporter cli json:results/compliance-report.json
```

### Open Policy Agent (OPA)

OPA validates infrastructure configurations and API requests against policy.

```rego
# policy/data_residency.rego
package compliance.data_residency

# GDPR: EU data must stay in EU regions
allowed_eu_regions := {"eu-west-1", "eu-west-2", "eu-central-1", "eu-north-1"}

deny[msg] {
    resource := input.resources[_]
    resource.type == "aws_rds_instance"
    resource.tags.data_classification == "eu_personal_data"
    not allowed_eu_regions[resource.region]
    msg := sprintf(
        "RDS instance '%s' with EU personal data is in non-EU region '%s'",
        [resource.name, resource.region]
    )
}

# PCI-DSS: Cardholder data must be in PCI-scoped environments
deny[msg] {
    resource := input.resources[_]
    resource.tags.data_classification == "cardholder_data"
    not resource.tags.pci_scope == "true"
    msg := sprintf(
        "Resource '%s' contains cardholder data but is not in PCI scope",
        [resource.name]
    )
}
```

```bash
# Evaluate OPA policy
opa eval --data policy/ --input infrastructure.json "data.compliance.data_residency.deny"
```

### Terraform Compliance

```python
# features/encryption.feature (BDD for Terraform)
Feature: Encryption controls
  In order to comply with SOC2 and PCI-DSS
  As an infrastructure engineer
  I need to ensure all storage is encrypted

  Scenario: All S3 buckets must be encrypted
    Given I have aws_s3_bucket defined
    Then it must have server_side_encryption_configuration

  Scenario: All RDS instances must be encrypted
    Given I have aws_rds_cluster defined
    Then it must have storage_encrypted
    And its value must be true

  Scenario: All EBS volumes must be encrypted
    Given I have aws_ebs_volume defined
    Then it must have encrypted
    And its value must be true
```

```bash
# Run terraform-compliance
terraform plan -out=plan.out
terraform show -json plan.out > plan.json
terraform-compliance -p plan.json -f features/
```

---

## Audit Evidence Automation

### Evidence Collection Pipeline

```python
#!/usr/bin/env python3
"""Automated audit evidence collection for SOC2/HIPAA."""
import json
import subprocess
from datetime import datetime
from pathlib import Path

def collect_evidence(output_dir: str = "audit-evidence"):
    """Collect compliance evidence artifacts."""
    timestamp = datetime.utcnow().strftime("%Y%m%d_%H%M%S")
    evidence_dir = Path(output_dir) / timestamp
    evidence_dir.mkdir(parents=True, exist_ok=True)

    evidence = {}

    # 1. Infrastructure compliance scan
    result = subprocess.run(
        ["inspec", "exec", "controls/", "--reporter", "json"],
        capture_output=True, text=True
    )
    (evidence_dir / "inspec-results.json").write_text(result.stdout)
    evidence["infrastructure_scan"] = {
        "tool": "Chef InSpec",
        "timestamp": timestamp,
        "file": "inspec-results.json",
    }

    # 2. Access control audit
    iam_report = subprocess.run(
        ["aws", "iam", "generate-credential-report"],
        capture_output=True, text=True
    )
    subprocess.run(
        ["aws", "iam", "get-credential-report", "--output", "json"],
        capture_output=True, text=True,
        stdout=open(evidence_dir / "iam-credentials.json", "w")
    )
    evidence["access_control"] = {
        "tool": "AWS IAM",
        "timestamp": timestamp,
        "file": "iam-credentials.json",
    }

    # 3. Vulnerability scan results
    vuln_result = subprocess.run(
        ["trivy", "image", "--format", "json", "myapp:latest"],
        capture_output=True, text=True
    )
    (evidence_dir / "vulnerability-scan.json").write_text(vuln_result.stdout)
    evidence["vulnerability_scan"] = {
        "tool": "Trivy",
        "timestamp": timestamp,
        "file": "vulnerability-scan.json",
    }

    # Write evidence manifest
    (evidence_dir / "manifest.json").write_text(
        json.dumps(evidence, indent=2)
    )
    print(f"Evidence collected in: {evidence_dir}")

if __name__ == "__main__":
    collect_evidence()
```

### CI Evidence Collection

```yaml
# .github/workflows/compliance-evidence.yml
name: Compliance Evidence Collection
on:
  schedule:
    - cron: '0 6 * * 1'  # Weekly Monday 6am UTC
  workflow_dispatch:

jobs:
  collect-evidence:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run InSpec compliance scan
        run: |
          inspec exec controls/ \
            --reporter cli json:evidence/inspec-results.json

      - name: Run vulnerability scan
        run: |
          trivy image --format json --output evidence/vuln-scan.json myapp:latest

      - name: Run access control audit
        run: python scripts/audit-access-controls.py > evidence/access-audit.json

      - name: Upload evidence artifacts
        uses: actions/upload-artifact@v4
        with:
          name: compliance-evidence-${{ github.run_id }}
          path: evidence/
          retention-days: 365  # Keep for audit period
```

---

## Access Control Testing

### RBAC Verification

```typescript
import { test, expect } from '@playwright/test';

const roles = ['admin', 'editor', 'viewer', 'guest'] as const;

const accessMatrix = {
  '/admin/users': { admin: 200, editor: 403, viewer: 403, guest: 401 },
  '/admin/settings': { admin: 200, editor: 403, viewer: 403, guest: 401 },
  '/api/posts': { admin: 200, editor: 200, viewer: 200, guest: 401 },
  '/api/posts/create': { admin: 201, editor: 201, viewer: 403, guest: 401 },
  '/api/posts/delete': { admin: 200, editor: 403, viewer: 403, guest: 401 },
};

for (const [endpoint, expected] of Object.entries(accessMatrix)) {
  for (const role of roles) {
    test(`${role} accessing ${endpoint} returns ${expected[role]}`, async ({ request }) => {
      const token = await getTokenForRole(role);
      const response = await request.get(endpoint, {
        headers: { Authorization: `Bearer ${token}` },
      });
      expect(response.status()).toBe(expected[role]);
    });
  }
}
```

### Privilege Escalation Tests

```typescript
test.describe('Privilege escalation prevention', () => {
  test('viewer cannot modify own role to admin', async ({ request }) => {
    const viewerToken = await getTokenForRole('viewer');
    const response = await request.patch('/api/users/me', {
      headers: { Authorization: `Bearer ${viewerToken}` },
      data: { role: 'admin' },
    });
    // Should either reject or ignore the role field
    expect(response.status()).toBe(403);
  });

  test('user cannot access another user private data', async ({ request }) => {
    const userAToken = await getTokenForRole('user', 'user-a@example.com');
    const response = await request.get('/api/users/user-b-id/private', {
      headers: { Authorization: `Bearer ${userAToken}` },
    });
    expect(response.status()).toBe(403);
  });
});
```

---

## Data Residency Verification

```typescript
test.describe('Data residency compliance', () => {
  test('EU user data stored in EU region', async ({ request }) => {
    // Create EU user
    const createResponse = await request.post('/api/users', {
      data: {
        email: 'eu-user@example.de',
        country: 'DE',
        name: 'Test EU User',
      },
    });
    const userId = (await createResponse.json()).id;

    // Verify storage region via admin API
    const regionResponse = await request.get(`/api/admin/data-location/${userId}`, {
      headers: { Authorization: `Bearer ${adminToken}` },
    });
    const location = await regionResponse.json();
    expect(location.region).toMatch(/^eu-/);
    expect(location.database).toMatch(/eu-/);
  });
});
```

### Infrastructure Residency Check

```ruby
# InSpec: verify data residency
control 'GDPR-RESIDENCY-001' do
  impact 1.0
  title 'EU personal data must reside in EU regions'
  tag compliance: ['GDPR-Art.44']

  aws_rds_instances.where(tags: { data_region: 'eu' }).entries.each do |db|
    describe db do
      its('availability_zone') { should match(/^eu-/) }
    end
  end

  aws_s3_buckets.where(tags: { data_region: 'eu' }).entries.each do |bucket|
    describe bucket do
      its('region') { should match(/^eu-/) }
    end
  end
end
```

---

## Encryption Validation

### At-Rest Encryption

```ruby
# InSpec: encryption at rest
control 'ENCRYPT-AT-REST-001' do
  impact 1.0
  title 'All databases encrypted at rest with AES-256'
  tag compliance: ['SOC2-CC6.1', 'HIPAA-164.312(a)(2)(iv)', 'PCI-DSS-3.4']

  aws_rds_instances.entries.each do |db|
    describe db do
      it { should have_storage_encrypted }
    end
  end
end
```

### In-Transit Encryption

```typescript
import { test, expect } from '@playwright/test';
import https from 'https';
import tls from 'tls';

test('API endpoint enforces TLS 1.2+', async () => {
  const host = 'api.example.com';

  const result = await new Promise<tls.TLSSocket>((resolve, reject) => {
    const socket = tls.connect({ host, port: 443, servername: host }, () => {
      resolve(socket);
    });
    socket.on('error', reject);
  });

  const protocol = result.getProtocol();
  expect(['TLSv1.2', 'TLSv1.3']).toContain(protocol);

  result.destroy();
});

test('HTTP redirects to HTTPS', async ({ request }) => {
  // This test verifies HTTP to HTTPS redirect
  const response = await request.get('http://api.example.com/', {
    maxRedirects: 0,
  });
  expect(response.status()).toBe(301);
  expect(response.headers()['location']).toMatch(/^https:/);
});
```

---

## PII Handling Tests

```typescript
test.describe('PII protection', () => {
  test('PII not exposed in API responses to unauthorized roles', async ({ request }) => {
    const viewerToken = await getTokenForRole('viewer');
    const response = await request.get('/api/users', {
      headers: { Authorization: `Bearer ${viewerToken}` },
    });
    const users = await response.json();

    for (const user of users.data) {
      expect(user).not.toHaveProperty('ssn');
      expect(user).not.toHaveProperty('date_of_birth');
      expect(user.email).toMatch(/^[\w]{1,3}\*+@/); // Masked email
      expect(user.phone).toMatch(/^\*+\d{4}$/);      // Last 4 digits only
    }
  });

  test('PII not logged in application logs', async ({ request }) => {
    // Trigger an operation that processes PII
    await request.post('/api/users', {
      data: { email: 'pii-test@example.com', ssn: '123-45-6789', name: 'PII Test' },
    });

    // Check recent logs via admin API
    const logsResponse = await request.get('/api/admin/logs?last=100', {
      headers: { Authorization: `Bearer ${adminToken}` },
    });
    const logs = await logsResponse.json();
    const logText = JSON.stringify(logs);

    expect(logText).not.toContain('123-45-6789');
    expect(logText).not.toContain('pii-test@example.com');
  });

  test('GDPR right to erasure works completely', async ({ request }) => {
    // Create user with PII
    const createResp = await request.post('/api/users', {
      data: { email: 'delete-me@example.com', name: 'Delete Me', phone: '+1234567890' },
    });
    const userId = (await createResp.json()).id;

    // Request erasure
    const deleteResp = await request.delete(`/api/users/${userId}/gdpr-erase`, {
      headers: { Authorization: `Bearer ${adminToken}` },
    });
    expect(deleteResp.status()).toBe(200);

    // Verify erasure
    const verifyResp = await request.get(`/api/admin/data-audit/${userId}`, {
      headers: { Authorization: `Bearer ${adminToken}` },
    });
    const audit = await verifyResp.json();
    expect(audit.user_record).toBeNull();
    expect(audit.audit_logs_anonymized).toBe(true);
    expect(audit.backups_queued_for_purge).toBe(true);
  });
});
```

---

## Data Retention Policy Enforcement

```typescript
test.describe('Data retention policies', () => {
  test('expired data is purged according to policy', async ({ request }) => {
    const policyResponse = await request.get('/api/admin/retention-policies', {
      headers: { Authorization: `Bearer ${adminToken}` },
    });
    const policies = await policyResponse.json();

    // Verify each policy has been enforced
    for (const policy of policies) {
      const auditResp = await request.get(
        `/api/admin/retention-audit?data_type=${policy.data_type}`,
        { headers: { Authorization: `Bearer ${adminToken}` } },
      );
      const audit = await auditResp.json();

      expect(audit.oldest_record_age_days).toBeLessThanOrEqual(policy.retention_days);
      expect(audit.expired_records_count).toBe(0);
    }
  });
});
```

### Retention Policy Matrix

| Data Type | SOC 2 | HIPAA | GDPR | PCI-DSS |
|-----------|-------|-------|------|---------|
| Audit logs | 1 year | 6 years | Per purpose | 1 year |
| User accounts | Per policy | 6 years after last interaction | Until consent withdrawn | Per policy |
| Payment data | 7 years (financial) | N/A | Minimal necessary | Until no longer needed |
| Session logs | 90 days | 6 years | 30 days | 90 days |
| Backup data | 90 days | 6 years | Same as source | 90 days |

---

## Audit Log Completeness Testing

```typescript
test.describe('Audit log completeness', () => {
  const auditableActions = [
    { action: 'user.login', trigger: () => login('test@example.com') },
    { action: 'user.logout', trigger: () => logout() },
    { action: 'user.create', trigger: () => createUser({ email: 'new@example.com' }) },
    { action: 'user.delete', trigger: () => deleteUser('test-user-id') },
    { action: 'data.export', trigger: () => exportData('users') },
    { action: 'settings.change', trigger: () => updateSettings({ mfa: true }) },
    { action: 'permission.grant', trigger: () => grantPermission('user-id', 'admin') },
  ];

  for (const { action, trigger } of auditableActions) {
    test(`${action} is recorded in audit log`, async ({ request }) => {
      const beforeResp = await request.get('/api/admin/audit-logs?limit=1', {
        headers: { Authorization: `Bearer ${adminToken}` },
      });
      const before = await beforeResp.json();
      const lastId = before.data[0]?.id || 0;

      // Trigger the auditable action
      await trigger();

      // Verify audit log entry
      const afterResp = await request.get(`/api/admin/audit-logs?after=${lastId}`, {
        headers: { Authorization: `Bearer ${adminToken}` },
      });
      const after = await afterResp.json();

      const entry = after.data.find((e: any) => e.action === action);
      expect(entry).toBeDefined();
      expect(entry.timestamp).toBeDefined();
      expect(entry.actor_id).toBeDefined();
      expect(entry.ip_address).toBeDefined();
      expect(entry.user_agent).toBeDefined();
    });
  }

  test('audit logs are immutable', async ({ request }) => {
    const logsResp = await request.get('/api/admin/audit-logs?limit=1', {
      headers: { Authorization: `Bearer ${adminToken}` },
    });
    const logEntry = (await logsResp.json()).data[0];

    // Attempt to modify audit log (should fail)
    const modifyResp = await request.patch(`/api/admin/audit-logs/${logEntry.id}`, {
      headers: { Authorization: `Bearer ${adminToken}` },
      data: { action: 'tampered' },
    });
    expect([403, 404, 405]).toContain(modifyResp.status());

    // Attempt to delete audit log (should fail)
    const deleteResp = await request.delete(`/api/admin/audit-logs/${logEntry.id}`, {
      headers: { Authorization: `Bearer ${adminToken}` },
    });
    expect([403, 404, 405]).toContain(deleteResp.status());
  });
});
```

---

## Penetration Testing Requirements

| Standard | Frequency | Scope | Required By |
|----------|-----------|-------|-------------|
| SOC 2 | Annual (recommended) | External + internal | Trust services criteria |
| HIPAA | Annual (recommended) | All ePHI systems | Security rule |
| GDPR | Risk-based | Data processing systems | Art. 32 |
| PCI-DSS | Annual (external), quarterly (internal) | Cardholder data environment | Req. 11.3 |

### Pen Test Automation (Supplemental)

```bash
# OWASP ZAP baseline scan (automated)
docker run --rm -t zaproxy/zap-stable zap-baseline.py \
  -t https://staging.example.com \
  -r zap-report.html \
  -J zap-report.json \
  -l WARN

# Nuclei vulnerability scanner
nuclei -u https://staging.example.com \
  -t cves/ \
  -t vulnerabilities/ \
  -t misconfigurations/ \
  -o nuclei-results.txt \
  -severity critical,high
```

---

## Vulnerability Scanning Cadence

| Scan Type | Frequency | Tool Examples | CI Integration |
|-----------|-----------|---------------|----------------|
| Dependency scan | Every commit | Snyk, Dependabot, npm audit | PR gate |
| Container scan | Every build | Trivy, Grype, Snyk Container | Build gate |
| SAST (static) | Every commit | Semgrep, CodeQL, SonarQube | PR gate |
| DAST (dynamic) | Weekly / pre-release | OWASP ZAP, Nuclei | Scheduled CI |
| Infrastructure | Weekly | ScoutSuite, Prowler | Scheduled CI |

```yaml
# GitHub Actions: vulnerability scanning pipeline
name: Security Scans
on:
  push:
    branches: [main]
  pull_request:
  schedule:
    - cron: '0 4 * * 1'  # Weekly Monday 4am

jobs:
  dependency-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm audit --audit-level=high
      - uses: snyk/actions/node@master
        with:
          args: --severity-threshold=high

  container-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: docker build -t myapp:scan .
      - uses: aquasecurity/trivy-action@master
        with:
          image-ref: myapp:scan
          severity: CRITICAL,HIGH
          exit-code: 1

  sast-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: returntocorp/semgrep-action@v1
        with:
          config: p/owasp-top-ten
```

---

## Compliance Test Matrices

### SOC 2 Test Matrix (Excerpt)

| Control | Test | Automation | Frequency |
|---------|------|------------|-----------|
| CC6.1 - Encryption at rest | InSpec `ENCRYPT-AT-REST-001` | Fully automated | Weekly |
| CC6.7 - Encryption in transit | TLS version check | Fully automated | Daily |
| CC6.1 - Access control | RBAC matrix test | Fully automated | Every PR |
| CC7.2 - Security monitoring | Audit log completeness | Fully automated | Daily |
| CC8.1 - Change management | PR approval requirement | GitHub branch protection | Every PR |

### HIPAA Test Matrix (Excerpt)

| Safeguard | Test | Automation | Frequency |
|-----------|------|------------|-----------|
| 164.312(a)(1) - Access control | User auth + RBAC tests | Fully automated | Every PR |
| 164.312(a)(2)(iv) - Encryption | At-rest + in-transit checks | Fully automated | Weekly |
| 164.312(b) - Audit controls | Audit log completeness | Fully automated | Daily |
| 164.312(c)(1) - Integrity | Data checksums, immutable logs | Fully automated | Daily |
| 164.312(d) - Authentication | MFA enforcement test | Fully automated | Every PR |
| 164.312(e)(1) - Transmission security | TLS enforcement | Fully automated | Daily |

---

## CI Integration for Compliance Gates

```yaml
# .github/workflows/compliance-gate.yml
name: Compliance Gate
on:
  pull_request:
    branches: [main]

jobs:
  compliance:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Run compliance tests
        run: |
          npm run test:compliance
          inspec exec controls/ --reporter json:compliance-results.json

      - name: Evaluate compliance gate
        run: |
          python scripts/evaluate-compliance.py compliance-results.json
          # Exits non-zero if any critical controls fail

      - name: Upload compliance report
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: compliance-report
          path: compliance-results.json
```

---

## Documentation and Evidence Collection

### Evidence Folder Structure

```text
audit-evidence/
├── 2026-Q1/
│   ├── manifest.json              # Evidence index
│   ├── infrastructure/
│   │   ├── inspec-results.json    # Infrastructure compliance scan
│   │   ├── terraform-plan.json    # Infrastructure drift check
│   │   └── network-scan.json      # Network security scan
│   ├── access-control/
│   │   ├── iam-audit.json         # IAM credential report
│   │   ├── rbac-test-results.json # RBAC verification
│   │   └── mfa-audit.json        # MFA enrollment status
│   ├── vulnerability/
│   │   ├── dependency-scan.json   # Dependency vulnerabilities
│   │   ├── container-scan.json    # Container image scan
│   │   └── pentest-report.pdf     # Annual penetration test
│   └── data-protection/
│       ├── encryption-audit.json  # Encryption validation
│       ├── residency-check.json   # Data residency verification
│       └── retention-audit.json   # Data retention compliance
```

### Evidence Manifest

```json
{
  "audit_period": "2026-Q1",
  "generated_at": "<YYYY-MM-DDTHH:MM:SSZ>",
  "standards": ["SOC2", "HIPAA"],
  "evidence": [
    {
      "control": "CC6.1",
      "description": "Encryption at rest verification",
      "file": "infrastructure/inspec-results.json",
      "automated": true,
      "frequency": "weekly",
      "last_pass": "<YYYY-MM-DDTHH:MM:SSZ>"
    }
  ]
}
```

---

## Compliance Testing Checklist

### Initial Setup

- [ ] Map regulatory requirements to testable controls
- [ ] Write InSpec / OPA policies for infrastructure controls
- [ ] Implement RBAC verification tests
- [ ] Create encryption validation tests (at-rest + in-transit)
- [ ] Build audit log completeness tests
- [ ] Set up vulnerability scanning pipeline
- [ ] Configure evidence collection automation

### Ongoing Operations

- [ ] Weekly: automated infrastructure compliance scan
- [ ] Weekly: vulnerability scan results reviewed
- [ ] Monthly: access control audit (IAM review)
- [ ] Quarterly: compliance test matrix updated
- [ ] Quarterly: evidence folder archived
- [ ] Annually: penetration test (PCI-DSS, SOC 2)
- [ ] Annually: compliance framework mapping reviewed

### Pre-Audit Preparation

- [ ] Evidence folder complete for audit period
- [ ] All critical controls passing
- [ ] Remediation plan for any open findings
- [ ] Access provisioned for auditors
- [ ] Key personnel briefed on audit scope
- [ ] Previous audit findings addressed

---

## Related Resources

- [test-environment-management.md](./test-environment-management.md) -- secrets management and environment isolation
- [quality-metrics-dashboard.md](./quality-metrics-dashboard.md) -- compliance metrics in dashboards
- [operational-playbook.md](./operational-playbook.md) -- CI gates for compliance enforcement
- [SKILL.md](../SKILL.md) -- parent testing strategy skill
- [Chef InSpec Documentation](https://docs.chef.io/inspec/)
- [Open Policy Agent](https://www.openpolicyagent.org/docs/)
- [OWASP Testing Guide](https://owasp.org/www-project-web-security-testing-guide/)
- [SOC 2 Trust Services Criteria](https://www.aicpa.org/interestareas/frc/assuranceadvisoryservices/trustservices.html)
- [HIPAA Security Rule](https://www.hhs.gov/hipaa/for-professionals/security/)

# Component Testing in Browser Mode

Use this reference when UI logic is too rich for pure unit tests but full E2E would be too slow or expensive.

## Why this layer exists

- Runs components in a real browser, not a DOM shim
- Catches rendering, focus, accessibility, and interaction issues earlier than E2E
- Keeps scope narrow, so failures stay cheaper to diagnose than full end-to-end tests

## Default recommendation

- JS/TS web apps: prefer Vitest Browser Mode for component tests
- Keep component tests below E2E in the strategy stack
- Use them for state transitions, validation messages, loading states, keyboard navigation, and a11y smoke

## What belongs here

- Form validation and error rendering
- Loading, empty, and error states
- Keyboard and focus behavior
- Design-system components with meaningful interaction
- Visual and accessibility smoke on stable components

## What does not belong here

- Pure business rules with no rendering risk
- Full cross-page workflows
- Third-party integration semantics better covered by contract or integration tests

## Decision rules

```text
Need confidence in a UI behavior?
    │
    ├─ No browser semantics involved
    │   └─ Unit test
    │
    ├─ Single component or narrow UI composition
    │   └─ Browser-mode component test
    │
    └─ Multi-page user journey or auth/payment flow
        └─ E2E test
```

## Good assertions

- User-visible text and error states
- ARIA roles, names, and focus order
- Screenshot diffs for stable components only
- Network or callback outcomes that are observable from the component boundary

## Avoid

- Recreating a full app flow inside component tests
- Heavy mocking that hides broken contracts
- Snapshot-only assertions with no behavioral checks

# Comprehensive Software Testing Guide

This file was previously a full implementation guide covering Jest, pytest, Playwright, k6, and security tooling. It has been retired to prevent drift with dedicated sibling skills that now own each test layer.

## Where to Go

| Test Layer | Dedicated Skill |
|---|---|
| Unit testing | `qa-testing-strategy` decision tree + language-specific guidance in repos |
| Component testing | `qa-testing-strategy/references/component-testing-browser-mode.md` |
| Contract testing | `qa-api-testing-contracts` |
| Integration | `qa-testing-strategy` decision tree |
| E2E (web) | `qa-testing-playwright` |
| E2E (mobile) | `qa-testing-mobile`, `qa-testing-ios`, `qa-testing-android` |
| Performance | `qa-testing-performance` |
| Accessibility | `qa-testing-accessibility` |
| Security | `qa-security-testing` |
| Resilience/chaos | `qa-resilience` |
| Observability-driven | `qa-observability` |
| Debugging failed tests | `qa-debugging` |
| Refactoring tests | `qa-refactoring` |
| Docs coverage | `qa-docs-coverage` |
| Agent/LLM testing | `qa-agent-testing` |

If you arrived here from a plan or PRD link, update the reference to the appropriate sibling skill above.

# Contract Testing — Expanded

## Table of Contents

- [Contents](#contents)
- [Approaches Comparison](#approaches-comparison)
- [Specmatic (Contract-Driven)](#specmatic-contract-driven)
- [Pact (Consumer-Driven)](#pact-consumer-driven)
- [Pact vs Specmatic Decision Tree](#pact-vs-specmatic-decision-tree)
- [Karate (Unified DSL)](#karate-unified-dsl)
- [Bi-Directional Contract Testing (BDCT)](#bi-directional-contract-testing-bdct)
- [Contract Testing in CI/CD](#contract-testing-in-cicd)
- [Common Pitfalls](#common-pitfalls)

Contract testing validates API compatibility between services before integration, catching issues early in development.

## Contents

- Approaches Comparison
- Specmatic (Contract-Driven)
- Pact (Consumer-Driven)
- Pact vs Specmatic Decision Tree
- Karate (Unified DSL)
- Bi-Directional Contract Testing (BDCT)
- Contract Testing in CI/CD
- Common Pitfalls

## Approaches Comparison

| Approach | Tool | When to Use |
| -------- | ---- | ----------- |
| **Consumer-Driven (CDC)** | Pact | Consumer knows what it needs |
| **Contract-Driven (CDD)** | Specmatic | OpenAPI as single source of truth |
| **Bi-Directional (BDCT)** | Pactflow | Both sides define expectations |
| **Unified API Testing** | Karate | API, contract, and performance in one |

## Specmatic (Contract-Driven)

OpenAPI spec becomes the executable contract—no separate contract files.

```bash
# Validate API implementation against OpenAPI spec
specmatic test --contract openapi.yaml --host localhost:8080

# Generate stubs from OpenAPI for consumer testing
specmatic stub --contract openapi.yaml --port 9000
```

### Specmatic CI Integration

```yaml
# GitHub Actions
- name: Contract Test
  run: |
    specmatic test \
      --contract ./api/openapi.yaml \
      --host localhost:8080 \
      --report junit

- name: Upload Results
  uses: actions/upload-artifact@v4
  with:
    name: contract-test-results
    path: build/reports/specmatic/
```

## Pact (Consumer-Driven)

Consumer defines expectations, provider verifies.

```typescript
// Consumer test (generates contract)
const provider = new PactV4({
  consumer: 'OrderService',
  provider: 'InventoryService',
});

await provider.executeTest(async (mockServer) => {
  const response = await fetch(`${mockServer.url}/inventory/item-1`);
  expect(response.status).toBe(200);
});
```

```typescript
// Provider verification
const verifier = new Verifier({
  providerBaseUrl: 'http://localhost:3000',
  pactUrls: ['./pacts/orderservice-inventoryservice.json'],
});

await verifier.verifyProvider();
```

## Pact vs Specmatic Decision Tree

```text
Use Pact when:
├── Consumer team owns contract definition
├── Multiple consumers with different needs
├── Gradual migration from no contracts
└── Need Pact Broker for contract sharing

Use Specmatic when:
├── OpenAPI is already the source of truth
├── Strict contract-first development
├── Both provider and consumer use same spec
└── Want to avoid dual maintenance (OpenAPI + Pact JSON)
```

## Karate (Unified DSL)

Single DSL for API, contract, and performance testing.

```gherkin
Feature: Order API

Scenario: Create order
  Given url 'http://localhost:8080/orders'
  And request { userId: 'user-1', productId: 'prod-1' }
  When method POST
  Then status 201
  And match response contains { orderId: '#string' }

Scenario: Get order
  Given url 'http://localhost:8080/orders/order-1'
  When method GET
  Then status 200
  And match response == { orderId: 'order-1', status: '#string' }
```

### Karate Performance Testing

```gherkin
Feature: Order API Performance

Scenario: Load test create order
  * configure driver = { type: 'chrome' }
  * def result = karate.callSingle('create-order.feature')
  * print 'Response time:', result.responseTime

Background:
  * configure readTimeout = 30000
```

## Bi-Directional Contract Testing (BDCT)

Pactflow enables both consumer and provider to contribute to contract definition.

```text
┌─────────────┐     ┌─────────────┐     ┌─────────────┐
│  Consumer   │────▶│  Pactflow   │◀────│  Provider   │
│  (Pact)     │     │  Broker     │     │  (OpenAPI)  │
└─────────────┘     └─────────────┘     └─────────────┘
        │                  │                    │
        └──────────────────┼────────────────────┘
                           ▼
                   Contract Merge +
                   Compatibility Check
```

## Contract Testing in CI/CD

```yaml
# Recommended pipeline stages
stages:
  - unit-tests
  - contract-tests  # Before integration
  - integration-tests
  - e2e-tests

contract-tests:
  stage: contract-tests
  script:
    # Consumer: Generate contracts
    - npm run test:contract:consumer
    # Publish to broker
    - pact-broker publish ./pacts --broker-base-url $PACT_BROKER_URL
    # Provider: Verify contracts
    - npm run test:contract:provider
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
```

## Common Pitfalls

| Pitfall | Solution |
| ------- | -------- |
| Testing implementation details | Test behavior (inputs/outputs), not internals |
| Overly specific contracts | Use loose matchers (`#string`, `#number`) |
| Ignoring breaking changes | Use can-i-deploy check before release |
| Missing edge cases | Include error responses in contracts |
| Stale contracts | Automate contract generation in CI |

# Feature Matrix vs Test Matrix Gate

Use this gate before release to ensure implemented features have direct, auditable test evidence.

## Objective

Prevent release drift where backlog/features are marked complete but test coverage is missing or indirect.

## Gate Steps

1. Enumerate release-scoped features/backlog IDs.
2. Map each feature to direct test evidence:
   - unit/integration/contract/e2e
   - file path + case identifier
3. Classify status:
   - `direct` (explicit test)
   - `indirect` (covered as side effect)
   - `none`
4. Assign risk and owner for each non-direct item.
5. Block release if critical item is `none` without approved waiver.

## Evidence Rules

- Evidence must be machine-locatable (`path`, test name, grep-able ID).
- "We manually tested it" is supplemental, not replacement for direct test evidence.
- Waivers must include expiry date and follow-up owner.

## Suggested Query Pattern

```bash
# Example: find tests mentioning feature IDs or endpoint paths
rg -n "BL-023|BL-024|/api/feature-x|feature_x" e2e tests src --glob "**/*.{spec,test}.{ts,tsx,js}"
```

## Release Decision Rule

- `GO`: all critical features have direct evidence or approved waiver.
- `NO-GO`: any critical feature lacks direct evidence and no waiver.

# Observability-Driven Testing

## Table of Contents

- [Contents](#contents)
- [When to Use This Reference](#when-to-use-this-reference)
- [Core Concept: Observability-Driven Development (ODD)](#core-concept-observability-driven-development-odd)
- [OpenTelemetry Integration](#opentelemetry-integration)
- [Trace-Based Testing Paths](#trace-based-testing-paths)
- [Production Trace → Test Case Conversion](#production-trace--test-case-conversion)
- [Observability in CI/CD](#observability-in-cicd)
- [Debugging with Traces](#debugging-with-traces)
- [Metrics-Based Test Validation](#metrics-based-test-validation)
- [Best Practices](#best-practices)
- [Quick Start Checklist](#quick-start-checklist)
- [Related References](#related-references)
- [External Resources](#external-resources)

Use production telemetry (traces, metrics, logs) as the foundation for test design, validation, and debugging. OpenTelemetry is the default foundation for instrumentation and evidence-rich failure analysis.

## Contents

- When to Use This Reference
- Core Concept: Observability-Driven Development (ODD)
- OpenTelemetry Integration
- Trace-Based Testing Paths
- Production Trace → Test Case Conversion
- Observability in CI/CD
- Debugging with Traces
- Metrics-Based Test Validation
- Best Practices
- Quick Start Checklist
- Related References
- External Resources

---

## When to Use This Reference

- Designing tests for distributed systems and microservices
- Debugging flaky or intermittent test failures
- Converting production incidents into regression tests
- Validating instrumentation coverage
- Building trace-based test assertions

---

## Core Concept: Observability-Driven Development (ODD)

> "The best engineers do a form of observability-driven development — they understand their software as they write it, include instrumentation when they ship it, then check it regularly to make sure it looks as expected." — Charity Majors

### ODD Workflow

```text
1. WRITE CODE
   Include instrumentation (spans, metrics, logs) as you code

2. SHIP WITH TELEMETRY
   Deploy with traces, metrics, structured logs enabled

3. OBSERVE IN PRODUCTION
   Check telemetry matches expectations

4. CONVERT TO TESTS
   Production traces become test assertions
   Real failure patterns become test cases

5. ITERATE
   Use production insights to improve test coverage
```

### Traditional vs Observability-Driven Testing

| Traditional Testing | Observability-Driven Testing |
|---------------------|------------------------------|
| Write tests, hope they catch bugs | Observe production, write tests for real issues |
| Mock everything | Use real traces for validation |
| Test in isolation | Test distributed behavior |
| Debug with logs | Debug with traces |
| Coverage = lines executed | Coverage = behaviors observed |

---

## OpenTelemetry Integration

### The Three Pillars

```text
┌─────────────┐  ┌─────────────┐  ┌─────────────┐
│   TRACES    │  │   METRICS   │  │    LOGS     │
│             │  │             │  │             │
│ Request flow│  │ Aggregates  │  │ Events      │
│ across      │  │ over time   │  │ with        │
│ services    │  │             │  │ context     │
└─────────────┘  └─────────────┘  └─────────────┘
        │                │                │
        └────────────────┼────────────────┘
                         ▼
              ┌─────────────────────┐
              │   OpenTelemetry     │
              │   Unified Protocol  │
              └─────────────────────┘
```

### Instrumenting Test Code

```typescript
// test-instrumentation.ts
import { trace, SpanStatusCode } from '@opentelemetry/api';
import { NodeTracerProvider } from '@opentelemetry/sdk-trace-node';
import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { OTLPTraceExporter } from '@opentelemetry/exporter-trace-otlp-http';

// Initialize tracer for tests
const provider = new NodeTracerProvider();
provider.addSpanProcessor(
  new SimpleSpanProcessor(
    new OTLPTraceExporter({ url: 'http://localhost:4318/v1/traces' })
  )
);
provider.register();

const tracer = trace.getTracer('test-suite');

// Instrumented test
describe('Order Service', () => {
  it('should create order with payment', async () => {
    await tracer.startActiveSpan('test:create-order-with-payment', async (span) => {
      try {
        // Arrange
        span.setAttribute('test.phase', 'arrange');
        const user = await createTestUser();
        const product = await createTestProduct();

        // Act
        span.setAttribute('test.phase', 'act');
        const order = await orderService.create({
          userId: user.id,
          productId: product.id,
          quantity: 1
        });

        // Assert
        span.setAttribute('test.phase', 'assert');
        expect(order.status).toBe('paid');
        expect(order.paymentId).toBeDefined();

        span.setStatus({ code: SpanStatusCode.OK });
      } catch (error) {
        span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
        span.recordException(error);
        throw error;
      } finally {
        span.end();
      }
    });
  });
});
```

---

## Trace-Based Testing Paths

Trace-based assertions are one way to validate distributed behavior. Tracetest is a common option, but the core recommendation is broader: instrument first with OpenTelemetry, retain traces for failures, and assert on distributed behavior with the toolchain your team can operate reliably.

### Example Path: Tracetest

Tracetest enables assertions on OpenTelemetry traces, validating distributed behavior.

### Installation

```bash
# Install Tracetest CLI
curl -L https://raw.githubusercontent.com/kubeshop/tracetest/main/install-cli.sh | bash

# Configure backend
tracetest configure --server-url http://localhost:11633
```

### Test Definition

```yaml
# tracetest-order-flow.yaml
type: Test
spec:
  name: Order Creation Flow
  description: Validate order creation spans and attributes
  trigger:
    type: http
    httpRequest:
      url: http://api.example.com/orders
      method: POST
      headers:
        - key: Content-Type
          value: application/json
      body: |
        {
          "userId": "user-123",
          "productId": "prod-456",
          "quantity": 2
        }

  specs:
    # Validate order service span
    - selector: span[name="POST /orders"]
      assertions:
        - attr:http.status_code = 201
        - attr:http.response.body contains "orderId"

    # Validate payment service was called
    - selector: span[name="payment.process"]
      assertions:
        - attr:payment.status = "success"
        - attr:payment.amount > 0

    # Validate database write
    - selector: span[name="db.orders.insert"]
      assertions:
        - attr:db.system = "postgresql"
        - attr:db.operation = "INSERT"

    # Validate total duration
    - selector: span[name="POST /orders"]
      assertions:
        - attr:tracetest.span.duration < 2000ms
```

### Running Trace Tests

```bash
# Run single test
tracetest run test --file tracetest-order-flow.yaml

# Run in CI with JUnit output
tracetest run test --file tracetest-order-flow.yaml --output junit > test-results.xml

# Run test suite
tracetest run test --file tests/*.yaml --parallel 4
```

---

## Production Trace → Test Case Conversion

### Workflow

```text
1. CAPTURE
   Export production traces from observability backend

2. FILTER
   Select traces representing critical user journeys

3. SANITIZE
   Remove PII, replace IDs with test fixtures

4. CONVERT
   Transform trace into executable test case

5. VALIDATE
   Run test, verify it reproduces expected behavior
```

### Automated Conversion Script

```python
# trace_to_test.py
import json
from opentelemetry.proto.trace.v1 import trace_pb2

def trace_to_tracetest(trace_data: dict, test_name: str) -> dict:
    """Convert production trace to Tracetest format."""

    root_span = find_root_span(trace_data['spans'])

    test = {
        'type': 'Test',
        'spec': {
            'name': test_name,
            'description': f'Generated from production trace {trace_data["traceId"]}',
            'trigger': {
                'type': 'http',
                'httpRequest': extract_http_request(root_span)
            },
            'specs': []
        }
    }

    # Generate assertions from span attributes
    for span in trace_data['spans']:
        assertions = generate_assertions(span)
        if assertions:
            test['spec']['specs'].append({
                'selector': f'span[name="{span["name"]}"]',
                'assertions': assertions
            })

    return test

def generate_assertions(span: dict) -> list:
    """Generate assertions from span attributes."""
    assertions = []

    attrs = span.get('attributes', {})

    # HTTP assertions
    if 'http.status_code' in attrs:
        assertions.append(f"attr:http.status_code = {attrs['http.status_code']}")

    # Database assertions
    if 'db.operation' in attrs:
        assertions.append(f"attr:db.operation = \"{attrs['db.operation']}\"")

    # Duration assertion (allow 2x production latency)
    if 'duration_ms' in span:
        assertions.append(f"attr:tracetest.span.duration < {span['duration_ms'] * 2}ms")

    return assertions
```

---

## Observability in CI/CD

### GitHub Actions Integration

```yaml
name: Observability-Driven Tests
on: [push, pull_request]

jobs:
  trace-tests:
    runs-on: ubuntu-latest
    services:
      jaeger:
        image: jaegertracing/all-in-one:latest
        ports:
          - 16686:16686
          - 4317:4317

      tracetest:
        image: kubeshop/tracetest:latest
        ports:
          - 11633:11633
        env:
          TRACETEST_DEV: true

    steps:
      - uses: actions/checkout@v4

      - name: Start Application with Instrumentation
        run: |
          docker-compose -f docker-compose.test.yml up -d
          sleep 10

      - name: Run Trace-Based Tests
        run: |
          tracetest configure --server-url http://localhost:11633
          tracetest run test --file tests/trace-tests/*.yaml --output junit > trace-test-results.xml

      - name: Upload Test Results
        uses: actions/upload-artifact@v4
        with:
          name: trace-test-results
          path: trace-test-results.xml

      - name: Publish Test Results
        uses: EnricoMi/publish-unit-test-result-action@v2
        if: always()
        with:
          files: trace-test-results.xml
```

### Telemetry Validation Gate

```yaml
# Validate instrumentation coverage before deploy
- name: Validate Telemetry Coverage
  run: |
    # Query spans from test run
    SPAN_COUNT=$(curl -s "$JAEGER_URL/api/traces?service=order-service&limit=100" | jq '.data | length')

    # Ensure minimum span coverage
    if [ "$SPAN_COUNT" -lt 10 ]; then
      echo "ERROR: Insufficient instrumentation coverage"
      exit 1
    fi

    # Validate required spans exist
    REQUIRED_SPANS=("POST /orders" "payment.process" "db.orders.insert")
    for span in "${REQUIRED_SPANS[@]}"; do
      if ! curl -s "$JAEGER_URL/api/traces?service=order-service" | grep -q "$span"; then
        echo "ERROR: Missing required span: $span"
        exit 1
      fi
    done
```

---

## Debugging with Traces

### Flaky Test Investigation

```text
Test: Order creation intermittently fails

STEP 1: Collect traces from passing and failing runs
        tracetest run test --file order.yaml --output json > passing.json
        # Wait for failure
        tracetest run test --file order.yaml --output json > failing.json

STEP 2: Compare trace structure
        diff <(jq '.spans[].name' passing.json | sort) \
             <(jq '.spans[].name' failing.json | sort)

STEP 3: Identify timing differences
        jq '.spans[] | {name, duration: .endTime - .startTime}' passing.json
        jq '.spans[] | {name, duration: .endTime - .startTime}' failing.json

STEP 4: Check for missing spans or errors
        jq '.spans[] | select(.status.code == 2)' failing.json
```

### Root Cause Analysis

```typescript
// Use trace context in test assertions
it('should handle concurrent orders', async () => {
  const traceId = generateTraceId();

  // Inject trace context
  const result = await orderService.create(order, {
    traceId,
    spanId: generateSpanId()
  });

  // On failure, print trace link
  if (result.status !== 'success') {
    console.log(`Trace: ${JAEGER_URL}/trace/${traceId}`);
    console.log(`Failing span: ${result.spanId}`);
  }

  expect(result.status).toBe('success');
});
```

---

## Metrics-Based Test Validation

### SLO Validation in Tests

```typescript
// slo-validation.test.ts
describe('SLO Validation', () => {
  it('should meet latency SLO under load', async () => {
    const results = await runLoadTest({
      duration: '5m',
      vus: 100,
      rps: 1000
    });

    // Query actual metrics
    const p99Latency = await prometheus.query(
      'histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))'
    );

    const errorRate = await prometheus.query(
      'sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m]))'
    );

    // Assert SLOs
    expect(p99Latency).toBeLessThan(0.2);  // 200ms
    expect(errorRate).toBeLessThan(0.01);   // 1%
  });
});
```

### Automated SLO Checks

```yaml
# slo-gate.yaml
name: SLO Validation Gate
on:
  schedule:
    - cron: '0 * * * *'  # Hourly

jobs:
  validate-slos:
    runs-on: ubuntu-latest
    steps:
      - name: Check Latency SLO
        run: |
          P99=$(curl -s "$PROMETHEUS_URL/api/v1/query?query=histogram_quantile(0.99,rate(http_request_duration_seconds_bucket[1h]))" | jq '.data.result[0].value[1]')
          if (( $(echo "$P99 > 0.2" | bc -l) )); then
            echo "SLO BREACH: p99 latency $P99 > 200ms"
            exit 1
          fi

      - name: Check Error Rate SLO
        run: |
          ERROR_RATE=$(curl -s "$PROMETHEUS_URL/api/v1/query?query=sum(rate(http_requests_total{status=~'5..'}[1h]))/sum(rate(http_requests_total[1h]))" | jq '.data.result[0].value[1]')
          if (( $(echo "$ERROR_RATE > 0.001" | bc -l) )); then
            echo "SLO BREACH: error rate $ERROR_RATE > 0.1%"
            exit 1
          fi
```

---

## Best Practices

### Do

- Instrument code as you write it, not after
- Use trace IDs to correlate test failures with production behavior
- Convert production incidents into automated regression tests
- Validate instrumentation coverage in CI/CD gates
- Use semantic conventions (OpenTelemetry standards)

### Avoid

- Adding observability only after production issues
- Testing mocked services when traces can validate real behavior
- Ignoring flaky tests without trace investigation
- Shipping code without required spans

---

## Quick Start Checklist

- [ ] Install OpenTelemetry SDK in application
- [ ] Configure trace exporter (Jaeger, Tempo, Honeycomb)
- [ ] Add spans to critical code paths
- [ ] Install Tracetest for trace-based testing
- [ ] Write first trace-based test for happy path
- [ ] Add telemetry coverage validation to CI
- [ ] Create production trace → test conversion workflow

---

## Related References

- [operational-playbook.md](operational-playbook.md) — Test pyramid and CI gates
- [chaos-resilience-testing.md](chaos-resilience-testing.md) — Resilience testing with observability
- [../SKILL.md](../SKILL.md) — Main testing strategy overview
- [../../ops-devops-platform/SKILL.md](../../ops-devops-platform/SKILL.md) — Observability infrastructure

---

## External Resources

- [OpenTelemetry Documentation](https://opentelemetry.io/docs/)
- [Tracetest](https://tracetest.io/)
- [Observability-Driven Development (Charity Majors)](https://charity.wtf/tag/observability/)
- [Honeycomb Guide to Observability](https://www.honeycomb.io/what-is-observability)
- [Jaeger Tracing](https://www.jaegertracing.io/)

# Operational Testing Playbook

## Table of Contents

- [Contents](#contents)
- [Core Testing Principles](#core-testing-principles)
- [Navigation](#navigation)
- [Pattern: Test Pyramid](#pattern-test-pyramid)
- [Pattern: Given–When–Then (BDD)](#pattern-givenwhenthen-bdd)
- [Pattern: Test Data Management](#pattern-test-data-management)
- [Pattern: CI Test Gates](#pattern-ci-test-gates)
- [Quick Reference: Framework Selection](#quick-reference-framework-selection-2026)
- [Coverage Goals](#coverage-goals)
- [Common Anti-Patterns to Avoid](#common-anti-patterns-to-avoid)
- [Testing Decision Tree](#testing-decision-tree)
- [External Resources](#external-resources)
- [Best Practices Checklist](#best-practices-checklist)
- [Getting Started](#getting-started)

Compact navigation hub for layered testing, CI gates, and ready-to-use templates.

## Contents

- Core Testing Principles
- Navigation
- Pattern: Test Pyramid
- Pattern: Test Shape Selection
- Pattern: Given–When–Then (BDD)
- Pattern: Test Data Management
- Pattern: CI Test Gates
- Quick Reference: Framework Selection
- Coverage Goals
- Common Anti-Patterns to Avoid
- Testing Decision Tree
- External Resources
- Best Practices Checklist

## Core Testing Principles

### Test Pyramid Distribution

```
       /\
      /E2E\         5-10% - End-to-end tests (critical paths)
     /------\
    /  API  \       15-25% - API/Integration tests
   /----------\
  / Component \     20-30% - Component tests
 /--------------\
/      Unit       \ 40-60% - Unit tests (fast, isolated)
```

These ratios describe a logic-heavy monolith and are a starting illustration, not a target to hit by adding tests. Treat them as one shape among several — see [Pattern: Test Shape Selection](#pattern-test-shape-selection) below, which is the actual decision rule: allocate coverage to where defects originate, then let the ratio fall out of that decision. A frontend-heavy app or a microservice mesh should land on a visibly different distribution, and that is correct, not a deviation to fix.

**Rationale:**
- **Unit tests**: Fast (ms), isolated, easy to debug, cheap to maintain
- **Component tests**: Balance speed + integration, good for business logic
- **Integration tests**: Test service interactions, catch integration issues
- **E2E tests**: Expensive but validate critical user journeys

### Core Themes

- Clarify what to test vs what to assume
- Use fast, deterministic unit tests for core logic
- Use integration tests for cross-service flows
- Use E2E tests only for critical user paths
- Keep flaky tests out of required gates; fix or quarantine them
- Mock external boundaries, use real implementations internally
- Test behavior, not implementation details

---

## Navigation

### Resources (Detailed Guides)

- [references/comprehensive-testing-guide.md](comprehensive-testing-guide.md) — Complete testing playbook across unit, integration, E2E, performance, and security layers with modern practices
- [references/component-testing-browser-mode.md](component-testing-browser-mode.md) — Real-browser component testing strategy for modern JS/TS UI apps
- [references/shift-left-testing.md](shift-left-testing.md) — Shift-left tactics, BDD in requirements, TDD workflow, preview environments, and continuous testing
- [references/schema-aware-api-fuzzing.md](schema-aware-api-fuzzing.md) — Schema-aware fuzzing for OpenAPI-driven APIs
- [references/test-automation-patterns.md](test-automation-patterns.md) — Patterns and anti-patterns: Page Object Model, test doubles, fixtures, contract testing, retry logic, and common pitfalls
- [data/sources.json](../data/sources.json) — Curated external references for frameworks (Jest, Vitest, Playwright, k6, Cucumber), tools, and best practices

### Templates by Testing Type

**Unit Testing:**
- [assets/unit/template-jest-vitest.md](../assets/unit/template-jest-vitest.md) — Jest/Vitest unit tests with AAA pattern, mocking, snapshot testing, test factories, async testing, and coverage

**Component Testing:**
- [assets/component/template-vitest-browser.md](../assets/component/template-vitest-browser.md) — Vitest Browser Mode component tests for real-browser UI behavior and accessibility smoke

**E2E Testing:**
- [assets/e2e/template-playwright.md](../assets/e2e/template-playwright.md) — Playwright cross-browser E2E tests with setup projects, traces on retry, accessibility smoke, and parallel execution

**Performance Testing:**
- [assets/performance/template-k6-load-testing.md](../assets/performance/template-k6-load-testing.md) — k6 load testing with realistic scenarios, spike testing, stress testing, soak testing, custom metrics, and CI/CD integration

**BDD (Behavior-Driven Development):**
- [assets/bdd/template-cucumber-gherkin.md](../assets/bdd/template-cucumber-gherkin.md) — Cucumber BDD with Gherkin syntax, scenario outlines, data tables, tags, step definitions, and best practices for declarative testing

**Strategy & Pipeline:**
- [assets/test-strategy-template.md](../assets/test-strategy-template.md) — Test strategy one-pager with quality goals, scope by layer, data handling, and ownership
- [assets/automation-pipeline-template.md](../assets/automation-pipeline-template.md) — CI/CD pipeline blueprint with stages, gates, parallelization, and rollback rules

### Related Skills

- [../software-backend/SKILL.md](../../software-backend/SKILL.md) — Backend testing with Node.js, Python, Java (language-specific unit/integration patterns)
- [../software-frontend/SKILL.md](../../software-frontend/SKILL.md) — Frontend component testing, React Testing Library, accessibility, and visual testing
- [../software-mobile/SKILL.md](../../software-mobile/SKILL.md) — Mobile testing with XCTest, Espresso, Detox, and Appium
- [../qa-resilience/SKILL.md](../../qa-resilience/SKILL.md) — Chaos engineering, resilience testing, and reliability validation
- [../ops-devops-platform/SKILL.md](../../ops-devops-platform/SKILL.md) — CI/CD pipelines, observability, and incident response integration
- [../software-security-appsec/SKILL.md](../../software-security-appsec/SKILL.md) — Security testing, OWASP ZAP, vulnerability scanning, and penetration testing

---

## Pattern: Test Pyramid

Use this pattern to balance test types for optimal speed, coverage, and maintainability.

**Structure:**

**Base: Unit tests (40-60%)**
- Many, fast, close to the code
- No network, filesystem, or external services
- AAA pattern (Arrange, Act, Assert)
- Test business logic in isolation

**Middle: Integration tests (30-40%)**
- Fewer, slower, validate interactions
- Test with real databases, queues, external services (or Docker containers)
- Verify cross-component contracts

**Top: E2E/system tests (5-10%)**
- Small number, slowest, cover critical user journeys
- Test complete workflows through UI
- Focus on happy paths and critical edge cases

**Checklist:**

- [ ] Most new logic has unit tests
- [ ] Cross-service flows have integration or E2E coverage
- [ ] Avoid over-relying on UI-only tests for backend behavior
- [ ] Flaky tests are quarantined and fixed, not ignored
- [ ] Tests run in parallel where possible

---

---

## Pattern: Test Shape Selection

The "test pyramid" is one model. Others exist and are better suited to specific architectures. Pick based on where bugs actually live in your system.

| Shape | Description | Best For |
|-------|-------------|----------|
| **Pyramid** | Many unit → fewer integration → few E2E | Monoliths, logic-heavy backends, algorithmic code |
| **Trophy** (Kent C. Dodds) | Static analysis base, integration-heavy middle, few unit and few E2E | Frontend/full-stack JS/TS apps, API-driven services |
| **Honeycomb** (Spotify) | Integration at the center; unit tests minimized | Microservice architectures where bugs live at service boundaries |
| **Risk-based** (default recommendation) | Coverage allocated by defect probability and impact, not shape | Any architecture — start here when unsure |

**Decision rule**: ask "where do our production bugs actually come from?" and allocate coverage there. For microservices, integration tests are typically the highest-ROI layer. For domain-heavy monoliths, unit tests dominate. For frontend apps, integration tests on user-facing behavior (React Testing Library / component tests) outperform unit tests on implementation details.

Do not debate shapes as ideology. Pick the layer that can prove the behavior with the least cost. The shape emerges from that decision, not the other way around.

---

## Pattern: Given–When–Then (BDD)

Use for tests that encode requirements clearly in natural language.

**Structure:**

- **Given**: Initial state and inputs (setup)
- **When**: Action under test (execution)
- **Then**: Expected observable outcomes (assertions)

**Example (Gherkin):**

```gherkin
Scenario: Successful login with valid credentials
  Given I am on the login page
  When I enter email "user@example.com"
  And I enter password "SecurePass123"
  And I click the "Login" button
  Then I should see my dashboard
  And I should see "Welcome back, John"
```

**Guidelines:**

- Use descriptive test names that capture Given/When/Then in plain language
- Keep each test focused on a single behavior
- Use fixtures or builders to set up complex state without hiding important details

See [assets/bdd/template-cucumber-gherkin.md](../assets/bdd/template-cucumber-gherkin.md) for full BDD implementation.

---

## Pattern: Test Data Management

Use when tests rely on non-trivial data.

**Strategies:**

**In-memory data:**
- Prefer for unit tests (fast, isolated)
- Use factories to generate test data
- Avoid global shared state

**Database tests:**
- Use transactions and rollbacks per test where possible
- Reset state between tests to avoid cross-test coupling
- Use Docker containers (Testcontainers) for integration tests

**Large datasets:**
- Use factories/builders to construct minimal required data
- Keep golden files small and understandable
- Regenerate intentionally, not automatically

**Example (Factory Pattern):**

```typescript
import { faker } from '@faker-js/faker'

export class UserFactory {
  static create(overrides = {}) {
    return {
      email: faker.internet.email(),
      name: faker.person.fullName(),
      age: faker.number.int({ min: 18, max: 80 }),
      role: 'user',
      ...overrides
    }
  }

  static createMany(count: number, overrides = {}) {
    return Array.from({ length: count }, () => this.create(overrides))
  }
}
```

See [references/test-automation-patterns.md](test-automation-patterns.md) for more data management patterns.

---

## Pattern: CI Test Gates

Use when wiring tests into CI/CD pipelines.

**Stages:**

**Fast linting and unit tests:**
- Run on every push and PR
- Fail fast on style or obvious logic errors
- Target: < 5 minutes

**Integration and E2E tests:**
- Run on main branch and release branches
- Gate deployments for critical services
- Target: < 15 minutes

**Performance and security tests:**
- Run nightly or on release branches
- Track trends over time
- Target: < 30 minutes

**Flaky tests:**
- Track flakiness explicitly (retry count, failure rate)
- Quarantine or stabilize them instead of ignoring failures
- Use tags (@flaky) to separate from required gates

**Merge queues (GitHub, GitLab, Trunk, Aviator):**
- Flaky tests that fail 5% per run can block the entire queue on every cycle — impact is amplified.
- Split checks into required (unit, lint, type-check, security scan) and informational (E2E, visual regression, performance); only required checks block the queue.
- Enable automatic quarantine in your queue tool: quarantined tests still run and log output but do not eject PRs.
- Use Nx `affected` or Jest `--findRelatedTests` to subset tests per queue batch and cut CI time.
- Never rely solely on retry counts to absorb flake — instrument flake rate trends and fix root causes.

**Flaky-test economics (worked example, illustrative assumptions):**

The reason a 5% flake rate is not "just noise" is that it compounds with queue volume. Model it explicitly rather than eyeballing it:

```text
Assumptions (replace with your own measured numbers):
  PRs merged per day via queue (M)         = 40
  Flaky test's failure rate per run (f)    = 5%   (0.05)
  CI minutes burned per requeue-and-rerun  = 12 min

Expected ejections/day  = M x f            = 40 x 0.05 = 2 ejections/day
Wasted CI minutes/day   = ejections x 12   = 2 x 12    = 24 CI-minutes/day
Wasted CI minutes/month = 24 x ~21 workdays = ~504 CI-minutes/month
```

At 2 ejections/day, every merge behind that flaky test in the queue also requeues — the cost is not the 24 CI-minutes alone, it is 2 unplanned interruptions per day for whichever engineers happen to be queued behind it. That is the argument for a hard rule: **a test above the flake-SLO threshold (>1% weekly, see Core Targets) gets quarantined with an owner and expiry within one business day, rather than left to keep taxing the queue while "someone gets to it."** Recompute this model with your own `M` and `f` before deciding whether a specific flaky test justifies emergency quarantine or can wait for the next sprint — the decision should follow the number, not a blanket policy.

**Checklist:**

- [ ] Unit tests run on every commit
- [ ] Integration tests run on PR and main branch
- [ ] E2E tests run on staging before production deploy
- [ ] Performance tests run nightly with trend analysis
- [ ] Security scans run on every PR (OWASP ZAP, Snyk)
- [ ] Flaky tests are tracked and fixed, not ignored
- [ ] Merge queue required checks separated from informational checks

See [assets/automation-pipeline-template.md](../assets/automation-pipeline-template.md) for CI/CD pipeline blueprint.

---

## Quick Reference: Framework Selection

### Unit Testing

**Jest** - Best for:
- React applications (built-in React Testing Library support)
- Zero-config setup preference
- Extensive mocking capabilities

**Vitest** - Best for:
- Vite-based projects (instant compatibility)
- Speed priority (native ESM, parallel execution)
- Modern tooling (watch mode, UI mode)

**Verdict:** Vitest for new Vite projects, Jest for React/established codebases.

### E2E Testing

**Playwright** - Best for:
- Cross-browser testing (Chromium, Firefox, WebKit)
- Parallel execution by default
- Network interception and API mocking
- Mobile device emulation

**Cypress** - Best for:
- Real-time reloading and time-travel debugging
- Easier learning curve
- Excellent developer experience

**Verdict:** Playwright for comprehensive cross-browser coverage, Cypress for developer ergonomics.

### Performance Testing

**k6** - Best for:
- Developer-centric (JavaScript DSL)
- Modern CI/CD integration
- Grafana Cloud integration
- Protocol Buffers/gRPC support

**JMeter** - Best for:
- Legacy systems
- GUI-based test creation
- Java ecosystem

**Verdict:** k6 for modern applications, JMeter for legacy/Java ecosystems.

See [data/sources.json](../data/sources.json) for complete framework references and official documentation.

---

## Coverage Goals

**Critical paths**: 100%
- Authentication, payment processing, data persistence
- Security-sensitive operations

**Business logic**: 90%+
- Service layer, domain models
- Validation, calculations, workflows

**Overall**: 80%+
- Repository-wide average

**UI components**: 70%+
- Component rendering, user interactions

**Note:** Coverage is a metric, not a goal. Quality > quantity. Test behavior, not lines.

**Reconciling with the coverage anti-pattern:** [references/quality-metrics-dashboard.md](quality-metrics-dashboard.md) flags "100% coverage mandate" as an anti-pattern because a repo-wide blanket target invites Goodhart's-Law gaming (tests that execute a line without asserting anything). The targets above are the opposite of that: they are risk-scoped to a small set of critical-path modules (auth, payments, persistence), not a repo-wide rule. Even on those modules, line coverage is necessary but not sufficient — pair the 100% target with a mutation-score gate (see [Operationalising Mutation Coverage](quality-metrics-dashboard.md#operationalising-mutation-coverage)) so the tests are proven to detect regressions, not just execute code.

---

## Common Anti-Patterns to Avoid

### BAD: Testing Implementation Details

```typescript
// Bad - Tests internal method
expect(service.internalHelper()).toBe(true)

// Good - Tests public behavior
expect(service.publicMethod()).toBe(expectedResult)
```

### BAD: Flaky Tests (Race Conditions)

```typescript
// Bad - Sleep (flaky)
await sleep(1000) // Hope data loads

// Good - Explicit wait
await expect(page.getByText('Loaded')).toBeVisible()
```

### BAD: Shared Mutable State

```typescript
// Bad - Shared across tests
let user: User
beforeAll(() => { user = createUser() })

// Good - Fresh for each test
beforeEach(() => { user = createUser() })
```

### BAD: Excessive Mocking

```typescript
// Bad - Mock everything
const db = { save: jest.fn(), find: jest.fn() }
const cache = { get: jest.fn(), set: jest.fn() }

// Good - Use real implementations for internal code
const db = new InMemoryDatabase() // Real logic
const emailService = mockEmailService() // Mock external
```

### BAD: Brittle Selectors

```typescript
// Bad - Implementation-coupled
await page.locator('.btn.btn-primary.submit-v2').click()

// Good - Semantic
await page.getByRole('button', { name: 'Submit' }).click()
await page.getByTestId('submit-button').click()
```

See [references/test-automation-patterns.md](test-automation-patterns.md) for complete anti-patterns guide.

---

## Testing Decision Tree

**What should I test?**

```
Is it a UI interaction?
├─ YES → E2E test (Playwright/Cypress)
└─ NO
   ├─ Is it business logic?
   │  └─ YES → Unit test (Jest/Vitest)
   └─ Is it API contract?
      └─ YES → Contract test (Pact) + Integration test
```

**Should I mock this?**

```
Is it an external service (API, payment gateway)?
├─ YES → Mock it
└─ NO
   ├─ Is it a database?
   │  ├─ Unit test → Use in-memory/mock
   │  └─ Integration test → Use real DB (Docker)
   └─ Is it internal code?
      └─ Use real implementation
```

---

## External Resources

See [data/sources.json](../data/sources.json) for curated references across 13 categories:

- Unit testing frameworks (Jest, Vitest, Pytest, JUnit, RSpec)
- E2E testing (Playwright, Cypress, Selenium, Puppeteer)
- API testing (Supertest, REST Assured, Pact, Postman)
- Performance testing (k6, JMeter, Gatling, Locust)
- BDD frameworks (Cucumber, SpecFlow, Behave)
- Mobile testing (Appium, XCTest, Espresso, Detox)
- Visual regression (Percy, Chromatic, BackstopJS)
- Test data (Faker.js, Factory Bot, Testcontainers)
- Security testing (OWASP ZAP, Snyk, Burp Suite)
- Accessibility (Axe Core, Pa11y, Lighthouse CI)
- CI/CD integration (GitHub Actions, GitLab CI, Jenkins)
- Coverage & quality (Istanbul, Codecov, SonarQube)
- Property/mutation testing (fast-check, Stryker)

---

## Best Practices Checklist

**Test Design:**
- [ ] Use AAA pattern (Arrange, Act, Assert)
- [ ] One assertion per test (or related group)
- [ ] Test behavior, not implementation details
- [ ] Keep tests independent (no shared state)
- [ ] Use descriptive test names

**Test Data:**
- [ ] Use factories for test data generation
- [ ] Avoid magic values (use constants or factories)
- [ ] Clean up after tests (beforeEach/afterEach)

**Test Coverage:**
- [ ] 100% coverage on critical paths
- [ ] 90%+ coverage on business logic
- [ ] 80%+ overall coverage
- [ ] Track coverage trends in CI

**Test Maintenance:**
- [ ] Run tests in parallel
- [ ] Fix or quarantine flaky tests immediately
- [ ] Refactor tests alongside code
- [ ] Review test failures in CI before merging

**CI/CD Integration:**
- [ ] Unit tests on every commit (< 5 min)
- [ ] Integration tests on PR (< 15 min)
- [ ] E2E tests on staging (< 30 min)
- [ ] Performance tests nightly
- [ ] Security scans on every PR

---

## Getting Started

1. **Choose your testing stack** based on project type:
   - **JavaScript/TypeScript**: Jest/Vitest + Playwright + k6
   - **Python**: Pytest + Playwright + Locust
   - **Java**: JUnit 5 + REST Assured + Gatling
   - **Ruby**: RSpec + Capybara + JMeter

2. **Pick a test shape from where defects originate** (see [Pattern: Test Shape Selection](#pattern-test-shape-selection)) rather than targeting a fixed ratio; the pyramid distribution above is a reasonable starting illustration for logic-heavy monoliths only

3. **Configure CI/CD** with test gates at each stage

4. **Implement test data factories** for consistent, reusable test data

5. **Add code coverage tracking** with thresholds (80%+ overall)

6. **Monitor test flakiness** and fix root causes

7. **Run tests in parallel** to reduce feedback time

See [references/comprehensive-testing-guide.md](comprehensive-testing-guide.md) for complete testing playbook and [references/shift-left-testing.md](shift-left-testing.md) for early testing practices.

# Playwright Webapp Testing — Moved

Use `frameworks/shared-skills/skills/qa-testing-playwright/` for all Playwright guidance.

This stub remains for backward-compatible links.

# Production Testing and Shift-Right

## Table of Contents

- [Why Shift-Right?](#why-shift-right)
- [Synthetic Monitoring](#synthetic-monitoring)
- [Dark Launches and Gradual Rollouts](#dark-launches-and-gradual-rollouts)
- [Feature Flag-Gated Rollouts](#feature-flag-gated-rollouts)
- [MTTR-Flake SLO](#mttr-flake-slo)
- [Production Replay Testing](#production-replay-testing)
- [Observability-Driven Gates](#observability-driven-gates)
- [Shift-Right Anti-Patterns](#shift-right-anti-patterns)
- [Related Resources](#related-resources)

Shift-right testing validates behaviour in or near production. It complements shift-left gates with continuous verification, gradual exposure, and observability signals that no pre-merge suite can replicate.

---

## Why Shift-Right?

Pre-merge gates (unit, contract, integration, E2E) catch defects before deployment. Shift-right techniques catch the remainder: configuration drift, environment-specific failures, capacity surprises, and subtle regressions that only appear under real traffic.

The four primary shift-right signals are:

| Signal | What it catches |
|--------|-----------------|
| Synthetic monitors | Availability and latency regressions between deployments |
| Dark launches / canary | Real-traffic behaviour before full exposure |
| Feature flags | Decoupled deploy and release; instant rollback without redeployment |
| Production replay | Regressions invisible to synthetic or low-volume canary traffic |

---

## Synthetic Monitoring

Synthetic monitoring runs scripted checks against production (or a production-like environment) on a schedule, independent of CI. It answers: "Is the service healthy right now for an external caller?"

### Tools

| Tool | Strengths | Typical Use |
|------|-----------|-------------|
| **Datadog Synthetics** | Deep APM integration, multi-step API and browser tests, alert-to-trace correlation | Teams already on Datadog |
| **Checkly** | Code-first monitors (Playwright/fetch), Monitoring as Code via CLI/Terraform, GitHub integration | Engineering-owned monitors in source control |
| **Grafana Synthetic Monitoring** | Open-source Blackbox Exporter + Grafana Cloud, Prometheus-native alerting | Grafana/Prometheus stacks |

### What to Monitor Synthetically

1. **Critical journey smoke** – sign-in, core API endpoint, payment ping – at 1-minute intervals.
2. **API contract probes** – POST/GET against key endpoints; assert on status code, response schema, and latency p95.
3. **SSL/TLS expiry** – certificate validity with 30-day and 7-day warning thresholds.
4. **Third-party dependency health** – CDN, auth provider, payment gateway reachability.

### Checkly Monitor as Code (example)

```typescript
// checks/api-health.check.ts
import { ApiCheck, AssertionBuilder } from '@checkly/cli/constructs';

new ApiCheck('api-health', {
  name: 'POST /orders health',
  request: {
    url: 'https://api.example.com/orders',
    method: 'POST',
    body: JSON.stringify({ items: [] }),
    headers: [{ key: 'Content-Type', value: 'application/json' }],
  },
  assertions: [
    AssertionBuilder.statusCode().equals(200),
    AssertionBuilder.jsonBody('$.status').equals('ok'),
    AssertionBuilder.responseTime().lessThan(800),
  ],
  frequency: 1, // minutes
  locations: ['eu-west-1', 'us-east-1'],
});
```

### Alerting Thresholds (defaults)

```yaml
synthetic_slo:
  availability: 99.9%          # alert if 3 consecutive checks fail
  latency_p95_ms: 800          # alert if p95 > 800 ms in a 5-min window
  error_rate_window: 5m
  paging_severity: critical    # page on-call if availability drops below 99%
```

---

## Dark Launches and Gradual Rollouts

A dark launch exposes new code to real traffic without exposing it to users. Three patterns apply at different risk levels:

### Canary Deployment

Route a small percentage of real traffic (1–5%) to the new version. Observe error rates, latency, and business metrics before widening.

```text
Traffic split example (nginx / Envoy):
  prod-v2: 5%   ← canary
  prod-v1: 95%  ← stable baseline

Promotion gates:
  - Error rate delta < 0.1% vs baseline over 15 min
  - Latency p99 regression < 10%
  - No new error classes in logs/traces
```

Tooling: Argo Rollouts, Flagger, AWS CodeDeploy linear/canary, Kubernetes traffic splitting via Gateway API or Istio.

### Ring Deployment

Ordered concentric rings of exposure: internal employees → beta users → 10% of production → full rollout. Each ring is a gate; promotion requires passing observability checks.

```text
Ring 0 (dogfood):    internal users only
Ring 1 (beta):       opted-in users (~1%)
Ring 2 (limited GA): ~10% of production
Ring 3 (full GA):    100%

Gate criteria per ring:
  - Error rate < threshold
  - Latency p95 < budget
  - Support ticket spike absent
  - Business metric (conversion, click-through) not degraded
```

### Traffic Shadowing / Mirroring

Clone production requests and send them to the shadow service in parallel. The shadow response is discarded; only errors and latency are observed. No user impact.

```yaml
# Istio VirtualService mirror example
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: orders-vs
spec:
  hosts: [orders]
  http:
    - route:
        - destination:
            host: orders
            subset: v1
          weight: 100
      mirror:
        host: orders
        subset: v2
      mirrorPercentage:
        value: 100.0
```

Use shadowing to validate a new implementation against a stable one without any user-facing risk.

---

## Feature Flag-Gated Rollouts

Feature flags decouple deployment (code in production) from release (users see the feature). The flag is the rollout mechanism.

### Rollout Progression

```text
1. Deploy with flag OFF (dark code)
2. Enable for internal / QA users (dogfood)
3. Enable for percentage rollout (1% → 10% → 50% → 100%)
4. Promote to permanent-on or remove flag
```

### CI Gate Pattern

Block promotion if the feature flag's targeting rule has drifted from the expected baseline:

```yaml
# GitHub Actions step: verify flag state before promoting
- name: Verify feature flag state
  run: |
    FLAG_STATE=$(curl -s "$LAUNCHDARKLY_API/flags/$FLAG_KEY" \
      -H "Authorization: $LD_API_KEY" | jq -r '.on')
    if [ "$FLAG_STATE" = "true" ] && [ "$ENVIRONMENT" = "production" ]; then
      echo "Flag is live — checking error rate before widening"
      ./scripts/check-canary-health.sh
    fi
```

Tooling: LaunchDarkly, Unleash, Flagsmith, AWS AppConfig, GrowthBook.

### Observability Hooks

Emit a structured log event on every flag evaluation for correlation:

```json
{
  "event": "flag_evaluation",
  "flag": "new-checkout-flow",
  "variant": "treatment",
  "user_segment": "beta",
  "session_id": "abc123",
  "timestamp": "2026-04-27T10:00:00Z"
}
```

This lets you segment error rates and latency dashboards by flag variant without relying on user-attribute joins at query time.

---

## MTTR-Flake SLO

**MTTR-Flake** (Mean Time to Resolve Flake) is the median duration from the moment a test is quarantined to the moment its fix is merged and the quarantine is lifted.

### Formula

```
MTTR-Flake = median(fix_merged_at − quarantine_opened_at)
             for all flake incidents closed in the measurement window
```

### SLO Targets

| Severity | Target MTTR-Flake |
|----------|-------------------|
| Critical path (blocks deploys) | ≤ 2 business days |
| Standard (quarantined, non-blocking) | ≤ 5 business days |
| Low (informational/intermittent) | ≤ 10 business days |

### Why MTTR-Flake Belongs in the SLO Register

A flake quarantine is technical debt with a time cost. Without an SLO, quarantined tests accumulate indefinitely, the effective coverage of CI shrinks, and flaky tests provide cover for real regressions. Tracking MTTR-Flake as a named SLO:

- Makes flake debt visible on the same dashboard as availability SLOs.
- Creates ownership pressure: the team that introduced the flake owns the SLO.
- Provides a leading indicator for suite health before the flake rate metric spikes.

### Collection Script

```python
from datetime import datetime
from statistics import median

def mttr_flake(incidents: list[dict]) -> float:
    """
    incidents: list of dicts with:
      quarantine_opened_at: ISO datetime string
      fix_merged_at: ISO datetime string (or None if still open)
    Returns median MTTR in hours for closed incidents.
    """
    durations = []
    for i in incidents:
        if i.get("fix_merged_at"):
            opened = datetime.fromisoformat(i["quarantine_opened_at"])
            fixed = datetime.fromisoformat(i["fix_merged_at"])
            durations.append((fixed - opened).total_seconds() / 3600)

    return round(median(durations), 1) if durations else float("inf")
```

---

## Production Replay Testing

Production replay captures real request/response pairs from live traffic and replays them against a new version to detect regressions that synthetic tests miss.

### Approaches

| Approach | How it works | Risk |
|----------|-------------|------|
| **Request recording + replay** | Record HTTP requests via a proxy (GoReplay, Gor); replay against shadow | PII in payloads — must scrub before storing |
| **Captured fixture upgrade** | Export a recent slice of production calls; use as integration test fixtures | Fixtures go stale; add a rotation policy |
| **Differential replay** | Replay against old and new simultaneously; diff responses | Diff noise from timestamps, IDs — normalise before diff |

### GoReplay Example

```bash
# Record production traffic to file (sample 10%)
sudo gor --input-raw :8080 \
         --output-file requests.gor \
         --output-file-append \
         --http-pprof :6060 \
         --split-output true \
         --output-file-size-limit 100mb \
         --verbose 1 &

# Replay against staging
gor --input-file requests.gor \
    --output-http http://staging.example.com \
    --stats \
    --output-http-stats
```

### Safety Contract for Replay

1. Strip PII and auth tokens from recorded payloads before storing.
2. Replay against an isolated environment — never against a second production instance.
3. Gate replay results on response-code parity and latency budget, not byte-for-byte body equality.

---

## Observability-Driven Gates

An observability-driven gate uses live telemetry — traces, metrics, error logs — as the promotion criterion instead of (or in addition to) synthetic check pass/fail.

### Gate Signal Hierarchy

```text
Tier 1 (hard block):
  - Error rate > 0.5% (new error classes)
  - p99 latency > 2× baseline
  - Health check endpoint returning non-2xx

Tier 2 (soft block — requires manual approval):
  - p95 latency regression 10–50% vs baseline
  - Increase in warn-level log events > 20% vs baseline
  - Database query plan regressions detected

Tier 3 (informational):
  - Memory / CPU usage increase < 15%
  - New dependency calls (not previously seen in traces)
```

### OpenTelemetry-Based Automated Gate

```yaml
# .github/workflows/deploy.yml (post-deploy check step)
- name: Observability gate
  run: |
    BASELINE_ERROR_RATE=$(./scripts/get-metric.sh error_rate baseline 5m)
    CANARY_ERROR_RATE=$(./scripts/get-metric.sh error_rate canary 5m)

    python3 - <<'EOF'
import sys
baseline = float("$BASELINE_ERROR_RATE")
canary = float("$CANARY_ERROR_RATE")
delta = canary - baseline
if delta > 0.005:
    print(f"BLOCK: error rate delta {delta:.3%} exceeds 0.5% threshold")
    sys.exit(1)
print(f"PASS: error rate delta {delta:.3%}")
EOF
```

### Trace-Based Assertion

Use Tracetest or a custom span query to assert span-level contracts after a canary deployment:

```yaml
# tracetest test: new-checkout-canary.yaml
type: Test
spec:
  name: Checkout canary span assertions
  trigger:
    type: http
    httpRequest:
      url: https://api.example.com/checkout
      method: POST
  specs:
    - selector: span[name="payment.charge"]
      assertions:
        - attr:http.status_code = 200
        - attr:duration < 500ms
    - selector: span[name="inventory.reserve"]
      assertions:
        - attr:db.statement notContains "FULL SCAN"
```

---

## Shift-Right Anti-Patterns

| Anti-Pattern | Problem | Better Approach |
|-------------|---------|-----------------|
| Synthetic monitors that duplicate CI E2E | Redundant; adds noise without production signal | Synthetic monitors test availability and latency, not feature correctness |
| 100% canary traffic immediately | Defeats the purpose of gradual rollout | Start at 1–5%, promote on observability gate pass |
| Feature flags never removed | Flag debt accumulates; runtime branches stay forever | Track flag age; mandate cleanup within 30 days of full rollout |
| Replay without PII scrubbing | Regulatory and privacy risk | Scrub at capture time, not replay time |
| Observability gates checked manually | Humans miss windows; rollbacks are slow | Automate tier-1 gates; require manual approval only for tier-2 |
| MTTR-Flake not tracked | Quarantine lists grow silently | Add MTTR-Flake to team SLO dashboard alongside availability |

---

## Related Resources

- [observability-driven-testing.md](./observability-driven-testing.md) -- OpenTelemetry-first debugging and trace-based validation
- [operational-playbook.md](./operational-playbook.md) -- CI/CD pipeline quality gates hub
- [chaos-resilience-testing.md](./chaos-resilience-testing.md) -- failure injection and resilience testing
- [quality-metrics-dashboard.md](./quality-metrics-dashboard.md) -- metrics collection and dashboards
- [Checkly Monitoring as Code](https://www.checklyhq.com/docs/monitoring-as-code/)
- [Datadog Synthetics](https://docs.datadoghq.com/synthetics/)
- [Argo Rollouts](https://argoproj.github.io/rollouts/)
- [GoReplay](https://goreplay.org/)

# Property-Based Testing

## Table of Contents

- [Concept](#concept)
- [When to Use](#when-to-use)
- [Tool Landscape](#tool-landscape)
- [fast-check (JavaScript / TypeScript)](#fast-check-javascript--typescript)
- [Hypothesis (Python)](#hypothesis-python)
- [JQwik / QuickTheories (Java)](#jqwik--quicktheories-java)
- [Generating Domain-Valid Inputs](#generating-domain-valid-inputs)
- [CI Integration](#ci-integration)
- [Property-Based Testing for AI-Generated Code](#property-based-testing-for-ai-generated-code)
- [Anti-Patterns](#anti-patterns)
- [Related Resources](#related-resources)

Property-based testing (PBT) replaces hand-crafted example inputs with a generator that produces many random inputs satisfying declared constraints. When a failure is found, the framework shrinks the failing case to the minimal reproducible counterexample. PBT is a high-signal complement to example-based tests: it exercises edge cases and boundary conditions that humans routinely miss.

---

## Concept

```text
Example-based test:
  Given price = 9.99, quantity = 3
  Then total = 29.97

Property-based test:
  For all price in [0.01, 999.99] and quantity in [1, 100]
  total == price * quantity (within floating-point tolerance)
  AND total >= price
  AND total >= quantity
```

The generator runs hundreds of inputs automatically. On failure, shrinking produces the smallest failing case.

---

## When to Use

| Scenario | Benefit |
|----------|---------|
| Pure functions with numeric or string inputs | Discover boundary, overflow, and encoding edge cases |
| Serialization / deserialization round-trips | Verify `deserialize(serialize(x)) == x` for all valid `x` |
| State machine / workflow invariants | Verify invariants hold across all reachable states |
| API input validation | Discover parser edge cases that hand-crafted fuzz inputs miss |
| Algebraic properties (commutativity, associativity, idempotence) | Encode mathematical contracts as tests |
| AI-generated code review | Blind-spot detection: PBT finds the edge cases LLMs routinely skip |

PBT is **not** a replacement for example-based tests. Keep example tests for readability and regression coverage; add PBT for properties that should hold universally.

---

## Tool Landscape

| Tool | Languages | Notes |
|------|-----------|-------|
| **fast-check** | JavaScript, TypeScript | Most complete JS PBT library; excellent shrinking; Vitest and Jest compatible |
| **Hypothesis** | Python | Mature; integrates with pytest; stateful testing via `RuleBasedStateMachine` |
| **jqwik** | Java | JUnit 5-native; richer than QuickCheck ports; property-level annotations |
| **QuickTheories** | Java | Simpler than jqwik; good for teams already on JUnit 5 |
| **PropEr / Eqwalizer** | Erlang/Elixir | Strong for protocol and state-machine testing |
| **FsCheck** | F# / C# | Well-integrated with xUnit and NUnit |

---

## fast-check (JavaScript / TypeScript)

```bash
npm install --save-dev fast-check
```

### Round-trip property

```typescript
import fc from 'fast-check';

test('JSON round-trip: all serializable values survive serialize/deserialize', () => {
  fc.assert(
    fc.property(fc.jsonValue(), (value) => {
      expect(JSON.parse(JSON.stringify(value))).toEqual(value);
    })
  );
});
```

### Numeric invariant

```typescript
test('total is always >= unit price and >= quantity', () => {
  fc.assert(
    fc.property(
      fc.float({ min: 0.01, max: 999.99, noNaN: true }),
      fc.integer({ min: 1, max: 100 }),
      (price, quantity) => {
        const total = computeTotal(price, quantity);
        expect(total).toBeGreaterThanOrEqual(price);
        expect(total).toBeGreaterThanOrEqual(quantity);
      }
    )
  );
});
```

### State machine property (user session)

```typescript
test('user session: authenticated state never reached from initial without valid login', () => {
  fc.assert(
    fc.property(
      fc.array(fc.oneof(
        fc.record({ type: fc.constant('login'), password: fc.string() }),
        fc.record({ type: fc.constant('logout') }),
        fc.record({ type: fc.constant('access'), resource: fc.string() }),
      )),
      (commands) => {
        const session = new UserSession();
        for (const cmd of commands) {
          session.apply(cmd);
          if (session.isAuthenticated()) {
            // Authenticated state only reachable via valid login
            expect(session.hasValidLogin()).toBe(true);
          }
        }
      }
    )
  );
});
```

### Vitest configuration

```typescript
// vitest.config.ts — no special config needed; fast-check works in any test runner
// Increase default runs for nightly / pre-release jobs:
fc.configureGlobal({ numRuns: 1000 });  // default 100; increase for thorough sweeps
```

---

## Hypothesis (Python)

```bash
pip install hypothesis pytest
```

### Basic property

```python
from hypothesis import given, settings
from hypothesis import strategies as st

@given(price=st.floats(min_value=0.01, max_value=999.99, allow_nan=False),
       quantity=st.integers(min_value=1, max_value=100))
def test_total_non_negative(price: float, quantity: int) -> None:
    total = compute_total(price, quantity)
    assert total >= 0
    assert total >= price
```

### Stateful testing (rule-based)

```python
from hypothesis.stateful import RuleBasedStateMachine, rule, initialize

class CartMachine(RuleBasedStateMachine):
    @initialize()
    def setup(self) -> None:
        self.cart = Cart()

    @rule(item=st.from_regex(r'[A-Z]{3}-\d{4}'))
    def add_item(self, item: str) -> None:
        self.cart.add(item)
        assert item in self.cart.items()

    @rule()
    def checkout(self) -> None:
        count = len(self.cart.items())
        self.cart.checkout()
        assert self.cart.total() >= 0

TestCart = CartMachine.TestCase
```

### CI settings for Hypothesis

```python
# conftest.py
from hypothesis import settings, HealthCheck

settings.register_profile("ci", max_examples=200, suppress_health_check=[HealthCheck.too_slow])
settings.register_profile("nightly", max_examples=2000)
settings.load_profile("ci")  # override with HY_PROFILE=nightly for thorough runs
```

---

## JQwik / QuickTheories (Java)

```java
// jqwik
@Property
void totalAlwaysGtePrice(@ForAll @Positive @FloatRange(max = 999.99f) float price,
                          @ForAll @IntRange(min = 1, max = 100) int quantity) {
    float total = computeTotal(price, quantity);
    Assertions.assertThat(total).isGreaterThanOrEqualTo(price);
}
```

---

## Generating Domain-Valid Inputs

Use constrained generators to avoid "unrealistic data" failures that waste debugging time.

```typescript
// Constrained: only valid email-like strings
const emailArb = fc.emailAddress();

// Custom: product SKU matching format ABC-1234
const skuArb = fc.stringMatching(/^[A-Z]{3}-\d{4}$/);

// Composing domain objects
const orderArb = fc.record({
  sku: skuArb,
  quantity: fc.integer({ min: 1, max: 50 }),
  price: fc.float({ min: 0.01, max: 999.99, noNaN: true }),
});
```

Avoid overly permissive generators (e.g., `fc.string()` for email fields). They produce inputs your code will never receive in practice, wasting test cycles on irrelevant failures.

---

## CI Integration

PBT runs are deterministic when a failing seed is logged. fast-check and Hypothesis both print the seed on failure; re-run with that seed to reproduce.

**Default CI strategy**: keep `numRuns` / `max_examples` low (100-200) in the standard PR gate. Run high-count sweeps (1000+) nightly or pre-release.

```yaml
# GitHub Actions: nightly deep PBT run
- name: Property-based tests (thorough)
  env:
    HY_PROFILE: nightly         # Hypothesis: 2000 examples
    FC_NUM_RUNS: "1000"         # fast-check: read in conftest or test setup
  run: npx vitest run --reporter=verbose tests/property/
```

**Reproducing failures**: fast-check prints the failing seed in the error message. Pass it explicitly:

```typescript
fc.assert(fc.property(...), { seed: 1234567890, path: '0' });
```

---

## Property-Based Testing for AI-Generated Code

AI-generated code tends to pass example-based tests while failing on edge cases the prompt never specified. Common failure modes:

- Off-by-one errors in bounds checks
- Missing null/undefined guards
- Incorrect handling of empty collections
- Floating-point edge cases (NaN, Infinity, negative zero)
- String encoding edge cases (Unicode, empty, whitespace-only)

PBT is the highest-ROI complement to mutation testing for AI-authored code:

1. Write PBT for any function where the AI authored the implementation.
2. Run with at least 500 examples before merging.
3. If PBT finds a failure, do not simply fix the example — update the generator to reliably produce that class of input, then fix the code.

Pair with mutation testing (see [quality-metrics-dashboard.md](./quality-metrics-dashboard.md)): mutation score measures assertion depth, PBT measures input coverage.

---

## Anti-Patterns

| Anti-Pattern | Problem | Better Approach |
|-------------|---------|-----------------|
| Over-permissive generators | Tests fail on inputs your code will never see | Constrain generators to domain-valid inputs |
| PBT replacing all example tests | Hard to read; harder to debug specific known regressions | Keep examples for known cases; PBT for universal properties |
| Not logging failing seeds | Failures not reproducible | fast-check and Hypothesis log seeds automatically; capture in CI artifacts |
| `numRuns = 10000` in every PR gate | Slow feedback loop | Use 100-200 in PR gates; run 1000+ nightly |
| Testing multiple independent properties in one `fc.assert` | Hard to diagnose failures | One property per `fc.assert` call |

---

## Related Resources

- [quality-metrics-dashboard.md](./quality-metrics-dashboard.md) -- mutation testing to pair with PBT
- [schema-aware-api-fuzzing.md](./schema-aware-api-fuzzing.md) -- schema-driven fuzzing for API contracts
- [shift-left-testing.md](./shift-left-testing.md) -- shifting quality checks earlier
- [fast-check documentation](https://fast-check.dev/)
- [Hypothesis documentation](https://hypothesis.readthedocs.io/)
- [jqwik user guide](https://jqwik.net/docs/current/user-guide.html)

# Quality Metrics and Dashboards

## Table of Contents

- [Contents](#contents)
- [Core Quality Metrics](#core-quality-metrics)
- [Test Suite Health Metrics](#test-suite-health-metrics)
- [Metric Collection Pipelines](#metric-collection-pipelines)
- [Dashboard Tools and Setup](#dashboard-tools-and-setup)
- [Dashboard Views by Audience](#dashboard-views-by-audience)
- [Trend Analysis and Forecasting](#trend-analysis-and-forecasting)
- [Quality Gates as Metrics](#quality-gates-as-metrics)
- [Release Readiness Scoring](#release-readiness-scoring)
- [Operationalising Mutation Coverage](#operationalising-mutation-coverage)
- [Metric Anti-Patterns](#metric-anti-patterns)
- [Alerting on Quality Regressions](#alerting-on-quality-regressions)
- [Implementation Checklist](#implementation-checklist)
- [Related Resources](#related-resources)

Quality metrics collection, reporting, trend analysis, and team dashboards -- from core quality indicators through executive reporting and anti-pattern avoidance.

## Contents

- Core Quality Metrics
- Test Suite Health Metrics
- Metric Collection Pipelines
- Dashboard Tools and Setup
- Dashboard Views by Audience
- Trend Analysis and Forecasting
- Quality Gates as Metrics
- Release Readiness Scoring
- Operationalising Mutation Coverage
- Metric Anti-Patterns
- Alerting on Quality Regressions
- Implementation Checklist
- Related Resources

---

## Core Quality Metrics

### Primary Quality Indicators

| Metric | Formula | Target | Collection Source |
|--------|---------|--------|-------------------|
| **Defect Escape Rate** | Prod bugs / total bugs found | <10% | Defect tracker (Jira, Linear) |
| **Mean Time to Detect (MTTD)** | Avg time from defect introduction to detection | <24 hours | Git blame + bug report timestamps |
| **Test Pass Rate** | Passing tests / total tests | >98% | CI test reporters |
| **Flake Rate** | Flaky runs / total runs | <=1% weekly | CI analytics |
| **Code Coverage (delta)** | Coverage change on PR | +/- 0% (no decrease) | Coverage tools (Istanbul, JaCoCo) |
| **Coverage Trend** | Coverage over time | Increasing or stable | Coverage history |

### Defect Escape Rate Calculation

```python
def defect_escape_rate(
    bugs_in_prod: int,
    bugs_in_staging: int,
    bugs_in_dev: int,
    bugs_in_code_review: int
) -> dict:
    """Calculate defect escape rate and detection distribution."""
    total = bugs_in_prod + bugs_in_staging + bugs_in_dev + bugs_in_code_review
    if total == 0:
        return {"escape_rate": 0, "distribution": {}}

    return {
        "escape_rate": f"{(bugs_in_prod / total) * 100:.1f}%",
        "distribution": {
            "code_review": f"{(bugs_in_code_review / total) * 100:.1f}%",
            "development": f"{(bugs_in_dev / total) * 100:.1f}%",
            "staging": f"{(bugs_in_staging / total) * 100:.1f}%",
            "production": f"{(bugs_in_prod / total) * 100:.1f}%",
        },
        "total_bugs": total,
        "assessment": "GOOD" if bugs_in_prod / total < 0.10 else "NEEDS_IMPROVEMENT",
    }

# Example
result = defect_escape_rate(
    bugs_in_prod=3,
    bugs_in_staging=12,
    bugs_in_dev=25,
    bugs_in_code_review=10
)
# escape_rate: 6.0%, assessment: GOOD
```

### Mean Time to Detect

```python
from datetime import datetime, timedelta

def calculate_mttd(defects: list[dict]) -> timedelta:
    """Calculate mean time to detect from defect records.

    Each defect has:
      - introduced_at: datetime (commit timestamp)
      - detected_at: datetime (bug report / test failure timestamp)
    """
    detection_times = []
    for defect in defects:
        introduced = datetime.fromisoformat(defect["introduced_at"])
        detected = datetime.fromisoformat(defect["detected_at"])
        detection_times.append(detected - introduced)

    if not detection_times:
        return timedelta(0)

    total_seconds = sum(dt.total_seconds() for dt in detection_times)
    avg_seconds = total_seconds / len(detection_times)
    return timedelta(seconds=avg_seconds)
```

---

## Test Suite Health Metrics

| Metric | Formula | Target | Why It Matters |
|--------|---------|--------|----------------|
| **Suite Execution Time** | Wall-clock time for full suite | <15 min (E2E), <5 min (unit) | Developer feedback speed |
| **Suite Stability** | Runs with 0 flakes / total runs | >95% | Trust in CI signal |
| **Test Count Trend** | Tests added vs removed per sprint | Net positive | Coverage growth |
| **Slowest Tests (P95)** | 95th percentile test duration | <30s (E2E), <1s (unit) | CI pipeline bottlenecks |
| **Quarantined Test Count** | Tests in quarantine | Decreasing trend | Tech debt indicator |
| **Disabled Test Count** | Skipped / disabled tests | <5% of total | Hidden coverage gaps |

### Suite Health Report Script

```typescript
// scripts/suite-health-report.ts
import { execSync } from 'child_process';

interface TestResult {
  name: string;
  duration: number;
  status: 'passed' | 'failed' | 'skipped' | 'flaky';
}

function generateReport(results: TestResult[]) {
  const total = results.length;
  const passed = results.filter(r => r.status === 'passed').length;
  const failed = results.filter(r => r.status === 'failed').length;
  const flaky = results.filter(r => r.status === 'flaky').length;
  const skipped = results.filter(r => r.status === 'skipped').length;

  const durations = results.map(r => r.duration).sort((a, b) => a - b);
  const p50 = durations[Math.floor(durations.length * 0.5)];
  const p95 = durations[Math.floor(durations.length * 0.95)];
  const totalDuration = durations.reduce((sum, d) => sum + d, 0);

  return {
    summary: {
      total,
      passed,
      failed,
      flaky,
      skipped,
      passRate: `${((passed / total) * 100).toFixed(1)}%`,
      flakeRate: `${((flaky / total) * 100).toFixed(1)}%`,
    },
    timing: {
      totalDuration: `${(totalDuration / 1000).toFixed(1)}s`,
      p50: `${(p50 / 1000).toFixed(2)}s`,
      p95: `${(p95 / 1000).toFixed(2)}s`,
    },
    slowest: results
      .sort((a, b) => b.duration - a.duration)
      .slice(0, 10)
      .map(r => ({ name: r.name, duration: `${(r.duration / 1000).toFixed(2)}s` })),
  };
}
```

---

## Metric Collection Pipelines

### Architecture

```text
┌──────────────┐    ┌──────────────┐    ┌──────────────┐    ┌──────────────┐
│  CI Pipeline │    │ Test Reporter │    │  Data Store  │    │  Dashboard   │
│  (GitHub/GL) │───>│  (JUnit XML) │───>│ (Postgres /  │───>│  (Grafana /  │
│              │    │  (JSON/CSV)  │    │  InfluxDB)   │    │  Datadog)    │
└──────────────┘    └──────────────┘    └──────────────┘    └──────────────┘
                           │
                    ┌──────┴──────┐
                    │ Bug Tracker │
                    │ (Jira API)  │
                    └─────────────┘
```

### JUnit XML Reporter (CI Standard)

```typescript
// playwright.config.ts
export default defineConfig({
  reporter: [
    ['junit', { outputFile: 'results/junit.xml' }],
    ['json', { outputFile: 'results/results.json' }],
    ['html', { open: 'never' }],
  ],
});
```

### Custom Metrics Collector

```python
#!/usr/bin/env python3
"""Collect test metrics from CI and push to metrics store."""
import json
import xml.etree.ElementTree as ET
from datetime import datetime
import requests

def parse_junit(xml_path: str) -> dict:
    """Parse JUnit XML into metrics."""
    tree = ET.parse(xml_path)
    root = tree.getroot()

    suites = root.findall('.//testsuite')
    total = sum(int(s.get('tests', 0)) for s in suites)
    failures = sum(int(s.get('failures', 0)) for s in suites)
    errors = sum(int(s.get('errors', 0)) for s in suites)
    skipped = sum(int(s.get('skipped', 0)) for s in suites)
    time_s = sum(float(s.get('time', 0)) for s in suites)

    return {
        "timestamp": datetime.utcnow().isoformat(),
        "total_tests": total,
        "passed": total - failures - errors - skipped,
        "failed": failures + errors,
        "skipped": skipped,
        "duration_seconds": round(time_s, 2),
        "pass_rate": round((total - failures - errors - skipped) / total * 100, 1) if total > 0 else 0,
    }

def push_to_datadog(metrics: dict):
    """Push metrics to Datadog."""
    series = []
    for key, value in metrics.items():
        if isinstance(value, (int, float)):
            series.append({
                "metric": f"qa.test_suite.{key}",
                "type": "gauge",
                "points": [[int(datetime.utcnow().timestamp()), value]],
                "tags": ["env:ci", f"branch:{os.getenv('BRANCH', 'main')}"],
            })

    requests.post(
        "https://api.datadoghq.com/api/v1/series",
        headers={"DD-API-KEY": os.getenv("DD_API_KEY")},
        json={"series": series},
    )

if __name__ == "__main__":
    metrics = parse_junit("results/junit.xml")
    push_to_datadog(metrics)
    print(json.dumps(metrics, indent=2))
```

### GitHub Actions: Metrics Collection Step

```yaml
- name: Collect and push test metrics
  if: always()
  env:
    DD_API_KEY: ${{ secrets.DD_API_KEY }}
    BRANCH: ${{ github.ref_name }}
  run: python scripts/collect-metrics.py results/junit.xml
```

---

## Dashboard Tools and Setup

### Grafana + PostgreSQL

```sql
-- Grafana query: test pass rate over time
SELECT
  date_trunc('day', created_at) AS time,
  AVG(pass_rate) AS avg_pass_rate,
  AVG(flake_rate) AS avg_flake_rate,
  AVG(duration_seconds) AS avg_duration
FROM test_runs
WHERE created_at > NOW() - INTERVAL '30 days'
  AND branch = 'main'
GROUP BY 1
ORDER BY 1;
```

### Datadog Dashboard

```python
# Datadog dashboard definition (Terraform)
resource "datadog_dashboard" "quality_metrics" {
  title       = "QA Quality Metrics"
  description = "Test suite health, defect metrics, and release readiness"
  layout_type = "ordered"

  widget {
    timeseries_definition {
      title = "Test Pass Rate"
      request {
        q          = "avg:qa.test_suite.pass_rate{branch:main}"
        display_type = "line"
      }
      yaxis { min = "90", max = "100" }
    }
  }

  widget {
    query_value_definition {
      title = "Current Flake Rate"
      request {
        q          = "avg:qa.test_suite.flake_rate{branch:main}.rollup(avg, 86400)"
        aggregator = "last"
      }
      precision = 1
    }
  }
}
```

### Lightweight: Markdown Report (No Infrastructure)

```python
def generate_markdown_report(metrics_history: list[dict]) -> str:
    """Generate a markdown quality report for PR comments or Slack."""
    latest = metrics_history[-1]
    previous = metrics_history[-2] if len(metrics_history) > 1 else latest

    def trend(current, prev):
        diff = current - prev
        if diff > 0: return f"+{diff:.1f} :arrow_up:"
        if diff < 0: return f"{diff:.1f} :arrow_down:"
        return "0 :left_right_arrow:"

    return f"""## Quality Metrics Report
| Metric | Current | Trend |
|--------|---------|-------|
| Pass Rate | {latest['pass_rate']}% | {trend(latest['pass_rate'], previous['pass_rate'])} |
| Flake Rate | {latest['flake_rate']}% | {trend(latest['flake_rate'], previous['flake_rate'])} |
| Suite Duration | {latest['duration_seconds']}s | {trend(latest['duration_seconds'], previous['duration_seconds'])} |
| Total Tests | {latest['total_tests']} | {trend(latest['total_tests'], previous['total_tests'])} |
| Defect Escape Rate | {latest.get('escape_rate', 'N/A')} | -- |
"""
```

---

## Dashboard Views by Audience

### Executive View

Focus on outcomes and trends, not technical details.

| Metric | Visualization | Update Frequency |
|--------|---------------|------------------|
| Defect Escape Rate (monthly) | Single number + trend line | Weekly |
| Release Cadence | Bar chart (releases/month) | Weekly |
| Deployment Success Rate | Percentage gauge | Daily |
| Mean Time to Recovery (MTTR) | Single number | Weekly |
| Customer-Reported Bugs | Trend line | Weekly |

### Team Lead View

Focus on team health and process effectiveness.

| Metric | Visualization | Update Frequency |
|--------|---------------|------------------|
| Test Pass Rate by suite | Stacked bar chart | Daily |
| Flake Rate trend | Line chart (7-day rolling) | Daily |
| CI Pipeline Duration | Line chart | Daily |
| Quarantined Test Count | Single number + trend | Daily |
| Coverage by module | Heatmap | Weekly |
| PR Review-to-Merge Time | Histogram | Weekly |

### Individual Contributor View

Focus on actionable signals.

| Metric | Visualization | Update Frequency |
|--------|---------------|------------------|
| My recent test failures | List with links | Real-time |
| My flaky tests | Table with flake % | Daily |
| My PR coverage delta | Inline in PR | Per-PR |
| Slowest tests I own | Ranked list | Weekly |

---

## Trend Analysis and Forecasting

### Rolling Averages

```python
import pandas as pd

def calculate_trends(metrics_df: pd.DataFrame) -> pd.DataFrame:
    """Calculate 7-day and 30-day rolling averages."""
    df = metrics_df.sort_values('date')

    df['pass_rate_7d'] = df['pass_rate'].rolling(window=7).mean()
    df['pass_rate_30d'] = df['pass_rate'].rolling(window=30).mean()
    df['flake_rate_7d'] = df['flake_rate'].rolling(window=7).mean()
    df['duration_7d'] = df['duration_seconds'].rolling(window=7).mean()

    return df

def detect_regression(df: pd.DataFrame, metric: str, window: int = 7, threshold: float = 0.1) -> list:
    """Detect metric regressions using rolling window comparison."""
    rolling = df[metric].rolling(window=window)
    mean = rolling.mean()
    std = rolling.std()

    regressions = []
    for i in range(window, len(df)):
        current = df[metric].iloc[i]
        expected_mean = mean.iloc[i - 1]
        expected_std = std.iloc[i - 1]

        if expected_std > 0:
            z_score = (current - expected_mean) / expected_std
            if abs(z_score) > 2:  # 2 sigma = significant change
                regressions.append({
                    "date": df['date'].iloc[i],
                    "metric": metric,
                    "value": current,
                    "expected": round(expected_mean, 2),
                    "z_score": round(z_score, 2),
                })

    return regressions
```

---

## Quality Gates as Metrics

### Gate Definition

```yaml
# quality-gates.yml
gates:
  merge:
    - metric: test_pass_rate
      operator: ">="
      threshold: 100
      description: "All tests must pass"
    - metric: coverage_delta
      operator: ">="
      threshold: 0
      description: "Coverage must not decrease"
    - metric: lint_errors
      operator: "=="
      threshold: 0
      description: "No lint errors"

  deploy_staging:
    - metric: e2e_pass_rate
      operator: ">="
      threshold: 98
      description: "E2E suite pass rate"
    - metric: smoke_tests
      operator: "=="
      threshold: "all_passed"
      description: "Smoke tests pass"

  deploy_production:
    - metric: staging_soak_hours
      operator: ">="
      threshold: 4
      description: "4 hours soak time in staging"
    - metric: performance_regression
      operator: "=="
      threshold: false
      description: "No performance regressions"
    - metric: security_scan
      operator: "=="
      threshold: "clean"
      description: "No critical vulnerabilities"
```

### Gate Evaluation

```python
def evaluate_gates(gate_name: str, metrics: dict, gates_config: dict) -> dict:
    """Evaluate quality gates and return pass/fail with details."""
    gates = gates_config["gates"][gate_name]
    results = []

    for gate in gates:
        actual = metrics.get(gate["metric"])
        threshold = gate["threshold"]
        op = gate["operator"]

        passed = {
            ">=": actual >= threshold,
            "<=": actual <= threshold,
            "==": actual == threshold,
            ">": actual > threshold,
            "<": actual < threshold,
        }.get(op, False) if actual is not None else False

        results.append({
            "metric": gate["metric"],
            "description": gate["description"],
            "threshold": threshold,
            "actual": actual,
            "passed": passed,
        })

    all_passed = all(r["passed"] for r in results)
    return {"gate": gate_name, "passed": all_passed, "results": results}
```

---

## Release Readiness Scoring

### Weighted Readiness Score

```python
def release_readiness_score(metrics: dict) -> dict:
    """Calculate weighted release readiness score (0-100)."""
    weights = {
        "test_pass_rate": 0.25,
        "e2e_pass_rate": 0.20,
        "flake_rate_inverse": 0.10,  # 100 - flake_rate
        "coverage": 0.10,
        "security_clean": 0.15,
        "performance_pass": 0.10,
        "staging_soak": 0.10,
    }

    scores = {
        "test_pass_rate": min(metrics.get("test_pass_rate", 0), 100),
        "e2e_pass_rate": min(metrics.get("e2e_pass_rate", 0), 100),
        "flake_rate_inverse": max(100 - metrics.get("flake_rate", 100), 0),
        "coverage": min(metrics.get("coverage", 0), 100),
        "security_clean": 100 if metrics.get("security_clean", False) else 0,
        "performance_pass": 100 if metrics.get("performance_pass", False) else 0,
        "staging_soak": min(metrics.get("staging_soak_hours", 0) / 4 * 100, 100),
    }

    total = sum(scores[k] * weights[k] for k in weights)

    return {
        "overall_score": round(total, 1),
        "ready": total >= 85,
        "component_scores": {k: round(v, 1) for k, v in scores.items()},
        "recommendation": "SHIP" if total >= 85 else "HOLD" if total >= 70 else "BLOCK",
    }
```

### Readiness Thresholds

| Score | Recommendation | Action |
|-------|---------------|--------|
| 85-100 | SHIP | Clear to deploy |
| 70-84 | HOLD | Review failing components, decide |
| <70 | BLOCK | Do not deploy; fix blocking issues |

---

## Operationalising Mutation Coverage

Mutation testing injects deliberate faults (mutations) into source code — e.g., flipping `>` to `>=`, removing a return value, negating a condition — and checks whether the test suite kills (detects) each mutant. Mutation score measures what percentage of mutants the suite kills.

**Why mutation score outperforms line coverage as a quality signal:**

Line coverage tells you which lines were _executed_; mutation score tells you which behaviours were _asserted_. A test that executes a branch without asserting the output contributes to 100% line coverage but zero mutation kill rate. Mutation score is a direct measure of test effectiveness, not test presence.

### Tools

| Tool | Language(s) | Notes |
|------|-------------|-------|
| **Stryker Mutator** | JavaScript, TypeScript, C#, Scala | Most widely adopted; supports Vitest, Jest, Jasmine, xUnit, NUnit. Official VS Code plugin released 2025-11-07 — StrykerJS only at launch (v9.3.0+), with Stryker.NET/Stryker4s support on the roadmap; verify current coverage at the plugin page before relying on it for non-JS stacks. Stryker.NET 4.13+ adds Microsoft Testing Platform (MTP) support. |
| **PIT (Pitest)** | Java, Kotlin | De facto standard for JVM; fast bytecode-level mutation. Version 1.19.x adds `scmMutationCoverage`, which delegates the changed-file diff to Maven SCM (requires a valid `<scm>` block) rather than reading Git directly, for incremental PR-scoped runs — commonly reported as 10-50x faster than a full run, though the ratio is workload-dependent; benchmark on your own codebase. |
| **mutmut** | Python | Simple CLI; integrates with pytest; good for greenfield projects |

### Mutation Score Thresholds

| Score | Assessment | Action |
|-------|-----------|--------|
| ≥ 70% | Strong | Mutation coverage is healthy; maintain on new code |
| 50–69% | Review | Identify surviving mutant clusters; add targeted assertions |
| < 50% | Needs work | Test suite has structural gaps; block new feature work in affected modules |

These thresholds apply to critical-path modules (auth, payments, domain rules). Lower thresholds (≥ 50% strong) may be acceptable for stable infrastructure or glue code.

### CI Integration

Run Stryker on changed modules only to keep CI time manageable. Full mutation runs are expensive; scope them to the diff.

**Stryker (TypeScript) — PR-scoped run:**

```json
// stryker.config.json
{
  "mutate": ["src/**/*.ts", "!src/**/*.spec.ts"],
  "testRunner": "vitest",
  "thresholds": {
    "high": 80,
    "low": 70,
    "break": 50
  },
  "incremental": true,
  "incrementalFile": ".stryker-tmp/incremental.json",
  "reporters": ["html", "json", "progress"]
}
```

```yaml
# GitHub Actions step
- name: Mutation testing (changed files)
  run: npx stryker run
  env:
    STRYKER_DASHBOARD_API_KEY: ${{ secrets.STRYKER_DASHBOARD_KEY }}
```

**PIT (Java/Kotlin) — Maven:**

PIT 1.19.x (latest as of 2026) adds `scmMutationCoverage`, which mutates only files changed since a baseline commit. This is 10-50x faster than a full run on large codebases and is the recommended approach for PR-scoped CI gates.

```xml
<!-- pom.xml -->
<plugin>
  <groupId>org.pitest</groupId>
  <artifactId>pitest-maven</artifactId>
  <version>1.19.1</version>
  <configuration>
    <mutationThreshold>70</mutationThreshold>
    <coverageThreshold>80</coverageThreshold>
    <targetClasses>com.example.domain.*</targetClasses>
    <outputFormats><param>HTML</param><param>XML</param></outputFormats>
  </configuration>
</plugin>
```

```bash
# Full mutation run
mvn test-compile org.pitest:pitest-maven:mutationCoverage

# Incremental: only changed files since last commit (1.19+)
mvn org.pitest:pitest-maven:scmMutationCoverage
```

**mutmut (Python):**

```bash
mutmut run --paths-to-mutate src/domain/
mutmut results
# Gate: fail CI if mutation score drops below threshold
python - <<'EOF'
import subprocess, sys, re
out = subprocess.check_output(["mutmut", "results"]).decode()
killed = int(re.search(r'(\d+) killed', out).group(1))
total = int(re.search(r'(\d+) out of', out).group(1))
score = killed / total * 100
print(f"Mutation score: {score:.1f}%")
sys.exit(0 if score >= 70 else 1)
EOF
```

### Integration with Coverage Dashboards

Upload mutation results alongside line coverage so the dashboard shows both signals side by side:

```python
# Emit mutation score as a custom metric alongside line coverage
def push_mutation_score(score: float, module: str):
    """Push mutation score to Datadog as a custom gauge."""
    requests.post(
        "https://api.datadoghq.com/api/v1/series",
        headers={"DD-API-KEY": os.getenv("DD_API_KEY")},
        json={"series": [{
            "metric": "qa.mutation_score",
            "type": "gauge",
            "points": [[int(datetime.utcnow().timestamp()), score]],
            "tags": [f"module:{module}", "env:ci"],
        }]},
    )
```

Alert when mutation score drops below the `low` threshold (70%); page when it drops below `break` (50%).

---

## Metric Anti-Patterns

| Anti-Pattern | Problem | Better Approach |
|-------------|---------|-----------------|
| **Vanity metrics** (total test count) | More tests does not equal more quality | Track defect escape rate, not test count |
| **Goodhart's Law** (gaming coverage) | Writing tests to hit % target, not to find bugs | Measure mutation score or defect escape rate |
| **Averaging flake rate** | Hides badly flaky individual tests | Track per-test flake rate, fix top offenders |
| **100% coverage mandate** | Diminishing returns past 80-85% | Risk-weighted coverage targets by module |
| **Test count as productivity** | Incentivizes trivial tests | Track bugs found per test, not tests written |
| **Monthly reporting only** | Too slow for actionable feedback | Daily automated dashboards + weekly review |
| **Ignoring test duration** | Slow feedback loops reduce developer velocity | Track and budget suite execution time |

### Goodhart's Law in Practice

```text
BAD: "Our coverage is 95%!"
  → But 30% of tests assert nothing meaningful
  → Mutation testing reveals only 60% mutation kill rate
  → Defects still escape to production

GOOD: "Our mutation kill rate is 78% on critical paths"
  → Tests actually catch real bugs
  → Coverage is a secondary indicator
  → Defect escape rate is primary measure
```

---

## Alerting on Quality Regressions

### Alert Rules

```yaml
# alerting-rules.yml
alerts:
  - name: flake_rate_spike
    metric: qa.test_suite.flake_rate
    condition: "> 5%"
    window: "24h"
    severity: warning
    channel: "#qa-alerts"
    message: "Flake rate exceeded 5% in the last 24 hours"

  - name: test_pass_rate_drop
    metric: qa.test_suite.pass_rate
    condition: "< 95%"
    window: "1h"
    severity: critical
    channel: "#engineering-alerts"
    message: "Test pass rate dropped below 95%"

  - name: suite_duration_increase
    metric: qa.test_suite.duration_seconds
    condition: "> 120% of 7-day average"
    window: "24h"
    severity: warning
    channel: "#qa-alerts"
    message: "Suite duration increased >20% vs 7-day average"

  - name: defect_escape
    metric: qa.defects.production_new
    condition: "> 0"
    window: "24h"
    severity: info
    channel: "#qa-alerts"
    message: "New production defect reported -- review for test gap"
```

### Slack/Teams Integration

```python
def send_quality_alert(alert: dict, webhook_url: str):
    """Send quality regression alert to Slack."""
    color = {"critical": "#FF0000", "warning": "#FFA500", "info": "#0000FF"}
    requests.post(webhook_url, json={
        "attachments": [{
            "color": color.get(alert["severity"], "#808080"),
            "title": f"Quality Alert: {alert['name']}",
            "text": alert["message"],
            "fields": [
                {"title": "Metric", "value": alert["metric"], "short": True},
                {"title": "Current Value", "value": str(alert["current_value"]), "short": True},
                {"title": "Threshold", "value": alert["condition"], "short": True},
                {"title": "Severity", "value": alert["severity"].upper(), "short": True},
            ],
        }],
    })
```

---

## Implementation Checklist

### Phase 1: Foundation (Week 1-2)

- [ ] Configure JUnit/JSON test reporters in CI
- [ ] Store test results in database or metrics service
- [ ] Track: pass rate, flake rate, suite duration
- [ ] Create basic Grafana/Datadog dashboard

### Phase 2: Enrichment (Week 3-4)

- [ ] Add defect tracking integration (Jira/Linear API)
- [ ] Calculate defect escape rate
- [ ] Add coverage trend tracking
- [ ] Set up quality gate automation

### Phase 3: Actionability (Week 5-6)

- [ ] Configure regression alerts (Slack/Teams)
- [ ] Build release readiness scorecard
- [ ] Create audience-specific dashboard views
- [ ] Establish weekly quality review cadence

---

## Related Resources

- [operational-playbook.md](./operational-playbook.md) -- CI/CD pipeline quality gates
- [shift-left-testing.md](./shift-left-testing.md) -- metrics for shift-left effectiveness
- [test-environment-management.md](./test-environment-management.md) -- environment health monitoring
- [SKILL.md](../SKILL.md) -- parent testing strategy skill
- [DORA Metrics](https://dora.dev/guides/dora-metrics-four-keys/)
- [Grafana Dashboards](https://grafana.com/docs/grafana/latest/dashboards/)
- [Datadog APM](https://docs.datadoghq.com/tracing/)

---
description: Reliability-theory patterns for QA strategy — FMEA-driven risk-based test selection, FTA-guided fault injection coverage, hazard-function test prioritization across the release lifecycle, test-pyramid reliability allocation, Weibull-based regression cadence, and error-budget-aware test gating.
last_verified: 2026-05-02
status: stable
primitives:
  - foundations-reliability-theory/assets/templates/reliability-theory/01-mtbf-mttr.md
  - foundations-reliability-theory/assets/templates/reliability-theory/02-availability-formulas.md
  - foundations-reliability-theory/assets/templates/reliability-theory/03-hazard-functions.md
  - foundations-reliability-theory/assets/templates/reliability-theory/04-bathtub-curve.md
  - foundations-reliability-theory/assets/templates/reliability-theory/05-fault-tree-analysis.md
  - foundations-reliability-theory/assets/templates/reliability-theory/06-fmea.md
  - foundations-reliability-theory/assets/templates/reliability-theory/07-redundancy-math.md
  - foundations-reliability-theory/assets/templates/reliability-theory/08-error-budgets.md
  - foundations-reliability-theory/assets/templates/reliability-theory/09-weibull-analysis.md
  - foundations-reliability-theory/assets/templates/reliability-theory/10-system-reliability.md
  - foundations-reliability-theory/assets/templates/reliability-theory/11-reliability-allocation.md
---

# Reliability Theory Applied — QA Testing Strategy

> **Gate before invoking:** Check [`foundations-reliability-theory` § When to Apply](../../foundations-reliability-theory/SKILL.md#when-to-apply) first. The recipes below assume the foundation is the right tool for the situation; the foundation's skip-conditions route you to a different foundation if not.


## Table of Contents

- [Why Reliability Theory for QA Strategy](#why-reliability-theory-for-qa-strategy)
- [Pattern Catalog](#pattern-catalog)
  - [P1 — FMEA-Driven Risk-Based Test Selection](#p1--fmea-driven-risk-based-test-selection)
  - [P2 — FTA-Guided Fault Injection Coverage](#p2--fta-guided-fault-injection-coverage)
  - [P3 — Hazard-Function Test Prioritization Across the Release Lifecycle](#p3--hazard-function-test-prioritization-across-the-release-lifecycle)
  - [P4 — Test-Pyramid Reliability Allocation](#p4--test-pyramid-reliability-allocation)
  - [P5 — Weibull-Based Regression Test Cadence After a Fix](#p5--weibull-based-regression-test-cadence-after-a-fix)
  - [P6 — Error-Budget-Aware Test Gating](#p6--error-budget-aware-test-gating)
- [Anti-Pattern Catalog](#anti-pattern-catalog)
  - [A1 — Uniform Coverage Ignoring Severity Distribution](#a1--uniform-coverage-ignoring-severity-distribution)
  - [A2 — Fault Injection Without MCS Prioritization](#a2--fault-injection-without-mcs-prioritization)
  - [A3 — Static Suite Composition Across the Bathtub Curve](#a3--static-suite-composition-across-the-bathtub-curve)
  - [A4 — RPN-Gated Release Without Residual-Risk Check](#a4--rpn-gated-release-without-residual-risk-check)
- [Recipe Catalog](#recipe-catalog)
  - [R1 — Pre-Release FMEA-to-Test-Plan Translation](#r1--pre-release-fmea-to-test-plan-translation)
  - [R2 — FTA Minimal-Cut-Set Fault Injection Sprint](#r2--fta-minimal-cut-set-fault-injection-sprint)
  - [R3 — Error-Budget Gate with Weibull Regression Cadence](#r3--error-budget-gate-with-weibull-regression-cadence)
- [Cross-References](#cross-references)

---

## Why Reliability Theory for QA Strategy

Test strategy decisions are made under resource constraints: finite CI budget, finite engineering hours, finite risk appetite per release. Without a formal model of failure, the decisions default to intuition — "test what changed," "add E2E for anything critical," "rerun until green." These heuristics accumulate waste and leave the highest-impact failure modes uncovered.

Reliability theory provides the quantitative foundation that converts intuition into defensible decisions:

1. **FMEA** (Primitive 06) turns a component inventory into a ranked failure-mode list with RPN scores. That ranking directly maps to a prioritized test selection — the highest-RPN failure modes own the largest test investment before any line of test code is written.

2. **FTA** (Primitive 05) reveals the minimal cut sets — the smallest combinations of component failures that produce a top-level system failure. A fault injection suite built from MCS targets the exact failure combinations that matter, rather than injecting arbitrary chaos.

3. **Hazard functions** (Primitive 03) and the **bathtub curve** (Primitive 04) show that failure rates are not constant across a component's life. Tests that are correctly weighted at release time (infant mortality zone) need different emphasis from tests in the steady-state zone or wear-out zone. A static suite ignores this structure.

4. **Reliability allocation** (Primitive 11) makes the test-pyramid layer budget explicit: when the system-level reliability target is known, the required reliability of each layer (unit, integration, E2E) can be derived, not guessed. This prevents over-investment in E2E while the unit layer leaks.

5. **Weibull analysis** (Primitive 09) models time-to-failure distributions after a fix is deployed. The shape parameter β identifies whether a fix produced a reliability improvement, degradation, or introduced early-life failures — information that drives the post-fix regression cadence rather than a fixed "run the suite once."

6. **Error budgets** (Primitive 08) convert SLO targets into a concrete release gate: how much of the budget has this release cycle consumed, and does the remaining budget permit a deploy? The gate is quantitative and audit-traceable, not a team vote.

The goal is not to make QA mathematical for its own sake. It is to prevent the most expensive failure modes in test strategy: spending test budget on low-severity, high-detectability paths while the single-point-of-failure MCS goes untested; deploying into an exhausted error budget; and running the same regression suite regardless of where the system sits on the bathtub curve.

---

## Pattern Catalog

### P1 — FMEA-Driven Risk-Based Test Selection

**Problem.** Before a release, the team has more potential test scenarios than CI budget can cover. The selection is made informally — the author of the PR nominates their own test cases, or tests are selected by coverage percentage with no weighting by impact.

**Reliability framing.** FMEA (Primitive 06: `RPN = Severity × Occurrence × Detection`) produces a ranked failure-mode table. Risk-based test selection maps test cases to FMEA rows and allocates test effort proportionally to RPN. Failure modes with S = 9–10 receive test coverage regardless of overall RPN — a low-occurrence, easily-detected catastrophic failure mode can have RPN = 90 yet still demands a test because the severity score alone warrants coverage.

**Operationalization.**

Before the release sprint, run a scoped FMEA over the components touched by the release. Produce a worksheet with one row per failure mode: component, failure mode, effect on the system SLO, S/O/D scores, and RPN.

Group rows into three tiers:

- Tier 1 (RPN ≥ 150 or S ≥ 9): mandatory test coverage. These are the scenarios that block merge or deploy.
- Tier 2 (RPN 60–149 and S ≤ 8): included in the targeted batch suite run before the deploy gate.
- Tier 3 (RPN < 60 and S ≤ 6): tracked but not actively tested in this release cycle. Reviewed if the component's failure rate changes.

For each Tier-1 row, write a corresponding test case title, assign it to the smallest effective layer (unit → component → integration → E2E in that preference order), and mark it as a gate-blocking scenario.

**Test selection output.** A test-selection manifest linking each gate-blocking scenario to its FMEA row, the chosen layer, and the RPN that justified it. This manifest survives as release evidence.

**Derives from Primitive 06: FMEA.** The RPN formula and Tier thresholds are direct applications of the FMEA scoring model. Severity-gated coverage regardless of RPN implements the FMEA canonical guidance: "always review S = 9–10 items regardless of RPN."

---

### P2 — FTA-Guided Fault Injection Coverage

**Problem.** The team runs chaos/fault injection tests but the injected faults are chosen by what is easy to inject (single-node kills, network drops) rather than by what combinations actually produce the top-level failure. The suite misses common-cause failures and multi-component MCS while over-investing in already-redundant single-node faults.

**Reliability framing.** FTA (Primitive 05) produces minimal cut sets — the smallest sets of basic events whose joint occurrence produces the top event. An MCS of size 1 is a single point of failure: injecting it alone brings down the system. An MCS of size 2 requires two simultaneous or sequential events. An MCS of size 3 or more is typically low-risk and can be deprioritized.

**Operationalization.**

Define the top event precisely: not "system failure" but "checkout service returns 5xx for > 30 s." Build the fault tree from the checkout service dependency diagram, decomposing to basic events (individual service failure, database failure, network path failure, third-party API timeout).

Enumerate MCS using the MOCUS algorithm or a BDD tool. Rank MCS by probability using basic event failure rates from incident history or MTBF data (Primitive 01).

Map each MCS to a fault injection scenario:

- MCS size 1: single-fault injection. The system must survive or degrade gracefully. If it does not, this is a P0 reliability gap.
- MCS size 2: paired fault injection (inject both faults simultaneously). The system may fail at the top event — this is acceptable provided recovery time meets the MTTR target (Primitive 01).
- MCS size 3+: covered only in scheduled resilience tests, not in pre-release gates.

Assign fault injection scenarios to CI stages:

- MCS size-1 scenarios in the pre-merge smoke gate (must pass before merge).
- MCS size-2 scenarios in the deploy gate resilience suite.
- MCS size 3+ in the quarterly resilience sprint.

**Derives from Primitive 05: Fault Tree Analysis.** MCS enumeration and size-based prioritization are core FTA outputs. Importance measures (Birnbaum, Fussell-Vesely) from Primitive 05 rank which basic events to inject first when budget is constrained.

---

### P3 — Hazard-Function Test Prioritization Across the Release Lifecycle

**Problem.** The regression suite runs at the same depth and cadence regardless of whether the component is newly deployed (infant mortality zone), has been stable for months (constant-hazard zone), or is approaching end of support (wear-out zone). This wastes CI budget on stable components and underinvests in newly deployed ones.

**Reliability framing.** The hazard function h(t) (Primitive 03) is the instantaneous failure rate at time t given survival to time t. The bathtub curve (Primitive 04) divides a component's lifecycle into three zones:

- Early-life (infant mortality): h(t) decreasing. Defects from manufacturing or integration are present and cause high early failure rates.
- Useful life (constant hazard): h(t) ≈ λ (constant). Failures are random and memoryless.
- Wear-out: h(t) increasing. Accumulated degradation, technical debt, or dependency decay drives rising failure rates.

Each zone has a different optimal test strategy.

**Operationalization.**

Tag each service and component with its lifecycle zone using a deployment age heuristic:

- Early-life: deployed < 30 days ago, or a major rewrite deployed < 14 days ago.
- Useful life: deployed 30–365 days ago with a stable incident rate.
- Wear-out: deprecated component still in path, end-of-support dependency, or component with an increasing defect density trend over the last 90 days.

Apply zone-specific test strategies:

- Early-life: run the full targeted regression batch on every merge, include integration smoke on all dependency paths, and add observability assertions (Primitive 01 MTTR-derived: alert threshold at 2× historical p99). This is the highest test density zone.
- Useful life: run smoke plus changed-path tests on merge (using TIA or `jest --findRelatedTests`). Run full regression weekly. This is the steady-state density zone.
- Wear-out: add explicit degradation tests: does the component exceed its memory budget, does its latency trend upward under steady load, does its error rate exceed the historical mean by > 1 σ? These tests are not pass/fail on functionality but on drift metrics.

**Derives from Primitive 03: Hazard Functions and Primitive 04: Bathtub Curve.** Zone classification maps to h(t) shape: decreasing (early-life), flat (useful life), increasing (wear-out). Test density follows h(t) — highest investment where failure rate is highest.

---

### P4 — Test-Pyramid Reliability Allocation

**Problem.** The team chooses test-pyramid proportions (unit vs. integration vs. E2E) by convention ("lots of unit, some integration, few E2E") without connecting the proportions to the system's reliability target. The resulting pyramid may satisfy code coverage metrics while failing to deliver the required system-level availability.

**Reliability framing.** Reliability allocation (Primitive 11) solves the inverse problem: given a system-level reliability target R_system, what reliability R_i must each subsystem i achieve? The allocation can use equal apportionment, AGREE allocation (weighted by complexity and usage), or ARINC allocation (weighted by failure rate history).

In the test-pyramid context, each layer contributes to the probability of detecting failures before they reach production. The combined detection probability across layers must meet the release quality gate: P(defect escapes all layers) ≤ escape_rate_budget.

**Operationalization.**

State the system reliability target explicitly: for example, "production error rate ≤ 0.1% of requests per week" (derived from the SLO and error budget — Primitive 08).

Compute the required defect escape rate. If the expected defect injection rate from the release is D defects per deployment, the test pyramid must achieve:

```text
P(escape) ≤ escape_budget / D
```

Allocate detection responsibility across layers. As a starting point, use AGREE-style allocation weighted by layer efficiency (unit tests catch logic defects cheaply; E2E tests catch integration failures expensively but exhaustively at the system level):

- Unit layer: targets logic and invariant defects. Allocate detection share proportional to the fraction of defects that are pure logic failures (typically 40–60% of the defect taxonomy from FMEA Tier-1 rows with low integration-surface scores).
- Integration layer: targets boundary and dependency defects. Allocate detection share proportional to defects with an integration-surface FMEA score.
- E2E layer: targets cross-service journey defects. Allocate only the residual escape budget. If unit + integration allocation already meets the budget, E2E scope can be reduced to critical-journey smoke only.

Track the allocation in the test-strategy manifest. When a production defect escapes the pyramid, audit which layer failed to detect it and adjust that layer's allocation, not the total coverage number.

**Derives from Primitive 11: Reliability Allocation and Primitive 10: System Reliability.** The series-system reliability model (Primitive 10: R_system = ∏ R_i) maps directly to the multi-layer detection model. Each layer is a component in the detection chain, and the product of detection probabilities must meet the system-level escape budget.

---

### P5 — Weibull-Based Regression Test Cadence After a Fix

**Problem.** After a production defect is fixed and deployed, the team runs the regression suite once, sees green, and returns to normal cadence. But some fixes introduce early-life failures (the fix changed adjacent code, a new code path is exercised for the first time under production load) or fail to actually improve the failure rate.

**Reliability framing.** Weibull analysis (Primitive 09) fits a two-parameter distribution to time-to-failure data: the shape parameter β determines whether the failure rate is decreasing (β < 1: early-life), constant (β ≈ 1: random), or increasing (β > 1: wear-out). After a fix is deployed, tracking the post-fix failure rate allows β estimation — and β < 1 in the post-fix window indicates the fix introduced an early-life failure pattern that warrants intensified short-term regression.

**Operationalization.**

Collect post-fix failure events. Use CI test failures, production error-rate spikes on the repaired component, and support signals. Track events in the window [deploy + 0, deploy + 72 h] at hourly resolution.

Fit a Weibull distribution to the inter-failure times or to the event count per hour using maximum likelihood estimation:

```bash
# Using scipy in Python
from scipy.stats import weibull_min
import numpy as np

# hours_to_failure: array of observed hours-to-failure events
params = weibull_min.fit(hours_to_failure, floc=0)
beta, loc, eta = params  # shape, location, scale
```

Interpret β:

- β < 0.9: early-life pattern. Run the full regression batch at 4 h, 24 h, and 72 h post-deploy. Do not reduce to smoke-only until β stabilizes above 1.0 in an updated fit at the 72 h mark.
- 0.9 ≤ β ≤ 1.1: constant-hazard. Return to normal weekly cadence.
- β > 1.1: wear-out signal on the fixed component. Escalate: the fix may have introduced technical debt that increases failure rate under accumulating load. Add a degradation test (latency trend assertion) to the nightly suite.

Document the β estimate in the post-fix retrospective alongside the MTBF change (Primitive 01): `MTBF_after / MTBF_before`. A ratio > 1.2 with β in the constant-hazard range confirms a successful, stable fix.

**Derives from Primitive 09: Weibull Analysis.** β estimation and the three-regime interpretation are direct applications of the Weibull shape-parameter model. The cadence thresholds (4 h, 24 h, 72 h) are derived from the infant mortality window implied by β < 1 distributions, where the hazard rate declines rapidly in the first few characteristic life fractions.

---

### P6 — Error-Budget-Aware Test Gating

**Problem.** Release gates are binary: all required tests pass → deploy. But a release that passes all tests may still consume the remaining error budget, leaving no headroom for the next deploy or for an unplanned incident. The gate does not account for the reliability state of the system at deploy time.

**Reliability framing.** Error budgets (Primitive 08) convert an SLO into a quantitative allowance: `error_budget = 1 − SLO_target`. Consumed budget is the observed error rate minus the SLO target, integrated over time. Budget remaining at deploy time constrains how much reliability risk the release is permitted to carry.

**Operationalization.**

At deploy decision time, compute the current error budget state:

```text
budget_remaining = (1 − SLO_target) × window_hours − cumulative_downtime_minutes / 60
```

For example, with a 99.9% SLO over a 30-day window (720 h) and 38 min of downtime consumed:

```text
budget_remaining = (0.001 × 720 h) − (38/60 h)
               = 0.72 h − 0.633 h
               = 0.087 h ≈ 5.2 minutes remaining
```

Apply a tiered gate based on budget remaining:

- Budget > 50%: standard gate. Required tests passing is sufficient to deploy.
- Budget 20–50%: elevated gate. All Tier-1 FMEA scenarios from P1 must pass, plus the MCS size-2 fault injection scenarios from P2. A FMEA re-score of changed components is required if the last FMEA is > 14 days old.
- Budget 10–20%: restricted deploy. Requires explicit sign-off from an on-call engineer, full E2E deploy-gate suite passing, and a documented risk statement citing current budget and projected consumption of the release.
- Budget < 10%: deploy freeze. Only reliability-improvement patches may deploy. Each exception requires a written justification that the patch is expected to restore budget, with a rollback plan and a post-deploy MTTR target (Primitive 01).

Publish budget state in the CI pipeline as a named check: "Error Budget Gate: 87 min / 432 min remaining (20%). Elevated gate active." This makes the gate visible in the PR timeline alongside functional test results.

**Derives from Primitive 08: Error Budgets.** The budget calculation formula and the concept of using budget depletion to trigger gate escalation are direct applications of the error budget model. The tiered gate thresholds (50%, 20%, 10%) mirror the SRE error budget policy patterns from the reliability literature.

---

## Anti-Pattern Catalog

### A1 — Uniform Coverage Ignoring Severity Distribution

**Description.** The team measures test coverage by percentage of lines, branches, or scenarios without weighting by failure mode severity. A 90% statement coverage metric is treated as evidence of adequate risk management.

**Reliability diagnosis.** Coverage metrics aggregate all code paths with equal weight. FMEA (Primitive 06) shows that failure modes with S = 1 (minor inconvenience) and S = 9 (data loss) are not equivalent. A test suite that achieves 90% coverage by exercising many low-severity paths while missing two high-severity paths (because they are harder to exercise) has the wrong shape of coverage entirely.

**Primitive misapplied.** Primitive 06 (FMEA) is either not run, or run but its severity dimension is discarded. The RPN rank is used to claim all failures are equal, violating FMEA's own canonical guidance: always cover S ≥ 9 rows regardless of RPN.

**How it manifests.** A critical authentication bypass (S = 10, O = 2, D = 7, RPN = 140) sits below the coverage threshold because the code path requires a specific JWT edge case to trigger. Meanwhile, 200 low-severity format-validation tests push the coverage percentage above target. The bypass ships to production.

**Fix.** Separate coverage tracking by severity tier. Report: "Tier-1 failure modes covered: 14/14 (100%). Tier-2 covered: 28/35 (80%). Overall line coverage: 88%." Gate on Tier-1 coverage = 100%, not on overall percentage. See P1 for the FMEA-to-test-selection mapping.

**Derives from Primitive 06: FMEA misapplied** — the severity dimension is ignored, reducing FMEA to a checkbox activity rather than a risk-ranking tool.

---

### A2 — Fault Injection Without MCS Prioritization

**Description.** The chaos engineering suite injects faults by rotating through a catalog of single-node kills and network partitions without reference to the fault tree. The injected faults are chosen by what the chaos tool makes easy, not by what the FTA shows is likely to produce the top event.

**Reliability diagnosis.** FTA (Primitive 05) identifies that many single-node failures are already handled by redundancy — their MCS has size 2 or more. Injecting them in isolation tests the redundancy mechanism but not the actual top-event risk. Meanwhile, the true SPOF (MCS of size 1) or the common-cause failure (two nodes sharing a single power domain) may never be tested because it requires a non-trivial simultaneous injection.

**Primitive misapplied.** Primitive 05 (FTA) is either skipped, or used only to draw a diagram and not to enumerate MCS and assign injection priorities.

**How it manifests.** The team demonstrates "we chaos test every week" but the quarterly failure involves a network partition on both read replicas simultaneously (MCS size 2 with a shared network path). This MCS was in the fault tree but never injected because the tool's default test list only kills one replica at a time.

**Fix.** Build the fault injection catalog directly from the MCS enumeration. For each MCS, write an explicit injection script that activates all events in the MCS simultaneously or in sequence within the failure propagation window. See P2 and R2 for the full procedure.

**Derives from Primitive 05: FTA misapplied** — MCS enumeration is skipped, leaving the top-down analytical output unused and reverting to ad hoc injection.

---

### A3 — Static Suite Composition Across the Bathtub Curve

**Description.** The regression suite depth and cadence are fixed at project inception and never adjusted for component lifecycle phase. A newly deployed service and a two-year-old stable service run the same suite at the same frequency.

**Reliability diagnosis.** The bathtub curve (Primitive 04) and hazard functions (Primitive 03) show that failure rates are not constant across a component's life. Early-life components have a decreasing hazard function — the first weeks of operation surface latent integration defects that a fixed-cadence suite may miss between runs.

**Primitive misapplied.** Primitives 03 and 04 are used to describe system behavior in documentation but not operationalized in the test schedule. The implication — that test intensity should track the hazard function — is never drawn.

**How it manifests.** A service rewrite is deployed on Monday. The regression suite runs on its normal weekly cadence. An early-life integration defect (a subtle JWT claim handling difference between old and new implementations) surfaces in production on Thursday — between cadence windows. The defect would have been caught by a 24 h post-deploy regression run.

**Fix.** Tag components by lifecycle zone and apply zone-specific cadence as described in P3. The early-life zone warrants daily regression; the useful-life zone warrants weekly; the wear-out zone warrants drift monitoring. Review zone tags at each sprint planning cycle.

**Derives from Primitives 03 and 04 misapplied** — the hazard rate model is acknowledged but not translated into a test scheduling rule, leaving cadence decisions at constant frequency regardless of h(t) shape.

---

### A4 — RPN-Gated Release Without Residual-Risk Check

**Description.** The team uses FMEA RPN thresholds as release gates: "no open Tier-1 items" means the release is cleared. But the gate is checked only against pre-mitigation RPN, and residual RPN (after mitigations are applied) is never recomputed. Alternatively, mitigations are marked "completed" when the action is taken rather than when the residual risk is measured.

**Reliability diagnosis.** FMEA (Primitive 06) produces both an initial RPN and a post-mitigation residual RPN. The checkout example in Primitive 06 shows a Payment API timeout moving from RPN 270 to projected residual RPN 90 after adding a circuit breaker and idempotency key. A gate that clears on initial RPN 270 being "resolved" without verifying the residual does not know whether the mitigation actually worked.

**Primitive misapplied.** Primitive 06 (FMEA) is used only for initial ranking and mitigation planning. The residual-RPN column of the FMEA worksheet remains unfilled, and the gate condition is "mitigation action assigned" rather than "residual RPN measured and below threshold."

**How it manifests.** A circuit breaker is added to the payment path (mitigation action: complete). The FMEA item is marked cleared. The circuit breaker is misconfigured with a timeout shorter than the payment provider's p95 response time, causing false-positive opens. Residual O is higher than projected, and residual D is lower because the misconfiguration is not monitored. The actual residual RPN is higher than the initial RPN.

**Fix.** Require a measured residual-RPN recompute before closing any Tier-1 FMEA row. The measurement must come from a test result, not from the engineer's judgment. For a circuit breaker mitigation, the test is: inject a payment timeout at the known p99 latency and verify the breaker opens only on the configured threshold, not earlier. See R1 step 5 for the residual-risk verification pattern.

**Derives from Primitive 06: FMEA misapplied** — the residual-risk column is the critical output of FMEA iteration but is treated as optional bookkeeping rather than a gate condition.

---

## Recipe Catalog

### R1 — Pre-Release FMEA-to-Test-Plan Translation

**When to use.** Planning the test scope for a release that touches a high-risk component or introduces a new integration boundary. Use when the team needs a defensible, traceable test plan rather than coverage-percentage targets.

**Steps.**

**Step 1: Scope the FMEA to the release changeset.**

List every component touched by the release (from the PR diff or ticket scope). For each component, list its functions and draft failure modes. Time-box this to 90 minutes for a typical sprint release.

```bash
# Pull changed files for the release branch
git diff main...HEAD --name-only | sort -u

# Cross-reference with service ownership map to identify component scope
```

**Step 2: Score each failure mode.**

Fill in the FMEA worksheet for each failure mode. Score S, O, D on 1–10. Anchor O scores to observed failure rates from incident history where available (Primitive 01 MTBF data). Do not score O by intuition alone.

| Component | Failure Mode | Effect | S | O | D | RPN |
|---|---|---|---|---|---|---|
| Auth service | KMS timeout | All API auth fails | 9 | 2 | 5 | 90 |
| Order DB | Replication lag > 500ms | Stale reads, duplicate orders | 8 | 3 | 4 | 96 |
| Payment API | Timeout under load | Silent order failure | 9 | 6 | 5 | 270 |

**Step 3: Tier and assign test cases.**

For each row, classify the tier (Tier 1: RPN ≥ 150 or S ≥ 9; Tier 2: RPN 60–149; Tier 3: < 60) and assign a test case to the smallest effective layer. Write the test case title in the worksheet.

```text
Payment API timeout (RPN 270, Tier 1):
  → Integration test: inject 5 s timeout on payment provider mock;
    assert circuit breaker opens; assert idempotency key prevents duplicate charge.
  → Gate: merge-blocking.

Order DB replication lag (RPN 96, Tier 2):
  → Integration test: simulate lag > 500 ms on read replica;
    assert application-layer idempotency check fires.
  → Gate: deploy-gate batch.
```

**Step 4: Map gate assignments to CI pipeline stages.**

- Tier-1 tests: added to the PR smoke gate. Failing any Tier-1 test blocks merge.
- Tier-2 tests: added to the deploy gate targeted batch. Failing blocks the deploy.
- Tier-3 tests: tracked in the weekly regression run. Failures raise a ticket but do not block.

**Step 5: Verify residual RPN after mitigations.**

After each Tier-1 mitigation action is implemented, recompute S, O, D and recalculate residual RPN. Close the FMEA row only when measured residual RPN < 100 and S ≤ 7, or when S ≥ 9 items have explicit sign-off from the release owner.

```bash
# In your CI pipeline, assert gate tiers pass before proceed
# Example: tag tests with tier in pytest markers
pytest -m "tier1" --junitxml=tier1-results.xml
# Gate: exit code 0 required for merge
pytest -m "tier2" --junitxml=tier2-results.xml
# Gate: exit code 0 required for deploy
```

**Output.** FMEA worksheet with test-case column populated, tier tags, gate assignments, and residual RPN after mitigations. This worksheet is the release evidence for risk-based test selection.

**Verify.** Every S ≥ 9 row has a corresponding test case. Every Tier-1 test is tagged as gate-blocking in CI. Residual RPN column is filled for all Tier-1 rows before deploy.

---

### R2 — FTA Minimal-Cut-Set Fault Injection Sprint

**When to use.** Before a major release, a new dependency integration, or after a production incident involving a multi-component failure. Run as a focused 1–2 day sprint with the platform and QA engineers.

**Steps.**

**Step 1: Define the top event precisely.**

Write the top event as a measurable system state with a threshold. Do not use "service failure."

```text
Top event: "Checkout service returns HTTP 5xx on > 1% of requests
            for a sustained window of > 60 seconds."
```

**Step 2: Build the fault tree.**

From the checkout service dependency diagram, decompose to basic events. Use AND gates for failure modes that require joint occurrence of independent components, and OR gates where any single failure produces the intermediate event.

```text
Top event: Checkout 5xx > 60 s
└─ OR
   ├─ Payment provider unavailable (MCS size 1 if no fallback)
   ├─ Order DB primary AND replica both unavailable (MCS size 2)
   │  ├─ Primary DB failure  [λ = 0.003/day]
   │  └─ Replica DB failure  [λ = 0.003/day]
   └─ Auth service AND JWT cache both unavailable (MCS size 2)
      ├─ Auth service failure [λ = 0.002/day]
      └─ JWT cache failure    [λ = 0.01/day]
```

**Step 3: Enumerate MCS and rank by probability.**

```python
# Compute MCS probabilities (rare-event approximation)
lambda_primary_db = 0.003   # failures/day
lambda_replica_db = 0.003
lambda_auth       = 0.002
lambda_jwt_cache  = 0.010
lambda_payment    = 0.0005  # single ISP/provider path

p_mcs_db      = lambda_primary_db * lambda_replica_db  # 9e-6/day
p_mcs_auth    = lambda_auth * lambda_jwt_cache          # 2e-5/day
p_mcs_payment = lambda_payment                          # 5e-4/day (SPOF)

# Payment provider SPOF dominates by 25×
```

**Step 4: Write a fault injection test per MCS.**

For each MCS, create a test that activates all basic events in the MCS and verifies the system response against the top event definition.

```python
# MCS size-1: payment provider unavailable
def test_payment_provider_spof(checkout_service, payment_mock):
    payment_mock.set_failure(mode="timeout", duration_seconds=90)
    response = checkout_service.post("/checkout", order_payload)
    # System should degrade gracefully — not hit top event
    assert response.status_code != 500
    # OR: if degradation is acceptable, assert it is logged and surfaced
    assert checkout_service.metrics.error_rate_pct < 1.0

# MCS size-2: both DB replicas unavailable simultaneously
def test_db_primary_and_replica_failure(checkout_service, db_primary, db_replica):
    with db_primary.pause(), db_replica.pause():
        time.sleep(5)  # allow health checks to detect
        response = checkout_service.post("/checkout", order_payload)
    # Top event allowed — verify MTTR recovery starts within SLA
    assert checkout_service.recovery_time_seconds <= 120
```

**Step 5: Assign injection tests to CI stages.**

- MCS size-1 injections → pre-merge smoke gate (must pass; any SPOF that causes the top event is a P0 finding).
- MCS size-2 injections → deploy gate resilience suite.
- Document findings as reliability gaps in the FMEA worksheet (feeds back to P1).

**Output.** Fault injection test suite with one test per MCS, CI stage assignments, and a findings report listing any MCS that produced the top event. For each finding: the MCS, the observed top-event duration, and the recommended mitigation (redundancy, circuit breaker, fallback path).

**Verify.** Every MCS size-1 has a passing test (or an open P0 ticket). Every MCS size-2 has a test that validates MTTR ≤ the target. FTA probability rank is reflected in CI stage assignments.

---

### R3 — Error-Budget Gate with Weibull Regression Cadence

**When to use.** At every release decision point. This recipe combines the P6 error-budget gate with the P5 Weibull-based post-deploy regression cadence into a single deploy-and-monitor workflow.

**Steps.**

**Step 1: Compute current error budget state before the deploy.**

```bash
# Fetch SLO window metrics from monitoring system
# Example using Prometheus / Grafana query pattern

SLO_TARGET=0.999          # 99.9%
WINDOW_HOURS=720          # 30-day rolling window
DOWNTIME_MINUTES=$(curl -s "$PROMETHEUS_URL/api/v1/query" \
  --data-urlencode 'query=sum_over_time(slo_downtime_minutes[30d])' \
  | jq '.data.result[0].value[1]' | tr -d '"')

BUDGET_REMAINING_HOURS=$(echo "scale=3; (1 - $SLO_TARGET) * $WINDOW_HOURS \
  - $DOWNTIME_MINUTES / 60" | bc)

echo "Error budget remaining: ${BUDGET_REMAINING_HOURS} hours"

# Determine gate tier
if (( $(echo "$BUDGET_REMAINING_HOURS > 0.5 * (1 - $SLO_TARGET) * $WINDOW_HOURS" | bc -l) )); then
  echo "Gate: STANDARD — functional tests required"
elif (( $(echo "$BUDGET_REMAINING_HOURS > 0.1 * (1 - $SLO_TARGET) * $WINDOW_HOURS" | bc -l) )); then
  echo "Gate: ELEVATED — Tier-1 FMEA + MCS-2 fault injection required"
elif (( $(echo "$BUDGET_REMAINING_HOURS > 0" | bc -l) )); then
  echo "Gate: RESTRICTED — explicit sign-off required"
else
  echo "Gate: FREEZE — reliability patches only"
  exit 1
fi
```

**Step 2: Run the gate-appropriate test suite.**

Under elevated gate, execute Tier-1 FMEA tests and MCS size-2 fault injections from R1 and R2 in addition to the standard suite. Emit a gate summary artifact:

```text
Error Budget Gate Summary
  SLO target:        99.9%
  Window:            30 days (720 h)
  Total budget:      43.2 min  ((1 - 0.999) x 720 h x 60 min/h)
  Budget consumed:   38 min    (88.0% of total budget)
  Budget remaining:  5.2 min   (12.0% of total budget)
  Gate tier:         RESTRICTED (12.0% falls in the 10-20% band)
  Required suites:   smoke, tier1-fmea, contract, full E2E deploy-gate suite
  Sign-off:          on-call engineer required, risk statement attached
  All suites:        PASS
  Deploy:            APPROVED WITH RESTRICTED GATE
```

Note the units: the 720-hour window is the *measurement period*, not the *budget size*. The budget size is `(1 - SLO_target) x window_hours`, converted to minutes here for readability. Confusing the window length with the budget total is a common gate-math mistake — always compute the budget from the SLO gap, never from the window size directly.

**Step 3: Deploy and start the Weibull monitoring window.**

Record the deploy timestamp. For the next 72 hours, collect hourly failure counts for the changed components from production telemetry and CI post-deploy runs.

**Step 4: Fit Weibull β at the 4 h, 24 h, and 72 h marks.**

```python
from scipy.stats import weibull_min
import numpy as np

# hours_to_failure: list of hours elapsed at each failure event
# (fill from PagerDuty/monitoring alert timestamps minus deploy time)
def estimate_weibull_beta(hours_to_failure):
    if len(hours_to_failure) < 3:
        return None, "insufficient data"
    params = weibull_min.fit(hours_to_failure, floc=0)
    beta, _, _ = params
    return beta, params

beta_4h,  _ = estimate_weibull_beta(failures_by_4h)
beta_24h, _ = estimate_weibull_beta(failures_by_24h)
beta_72h, _ = estimate_weibull_beta(failures_by_72h)

# Interpret and schedule cadence
if beta_4h is not None and beta_4h < 0.9:
    print("Early-life signal detected (β={:.2f}). Run full regression at 24h.".format(beta_4h))
elif beta_72h is not None and 0.9 <= beta_72h <= 1.1:
    print("Stable fix confirmed (β={:.2f}). Return to weekly cadence.".format(beta_72h))
elif beta_72h is not None and beta_72h > 1.1:
    print("Wear-out signal (β={:.2f}). Add drift tests to nightly suite.".format(beta_72h))
```

**Step 5: Report β and budget update to the deploy record.**

After the 72 h window closes, append to the deploy record:

```text
Post-deploy Weibull Report (72 h window)
  β estimate:        0.94 (constant-hazard zone)
  MTBF change:       +34% vs pre-fix baseline
  Error budget delta: −2 min consumed in window
  Cadence decision:  return to standard weekly regression
  Fix status:        CONFIRMED STABLE
```

**Output.** A deploy record combining: pre-deploy error budget state and gate tier, test suite results, post-deploy β estimate, MTBF delta, and cadence decision. This record is the traceability artifact connecting reliability theory to QA execution.

**Verify.** Budget remaining is computed and the correct gate tier is applied. β is estimated at least once before returning to normal cadence. Drift tests are added to the nightly suite if β > 1.1.

---

## Cross-References

**Foundation skill:**
[`foundations-reliability-theory/SKILL.md`](../../foundations-reliability-theory/SKILL.md) — canonical primitive definitions for all 11 reliability primitives referenced in this file.

**Individual primitives:**

| Primitive | File |
|---|---|
| 01 MTBF/MTTR | [`01-mtbf-mttr.md`](../../foundations-reliability-theory/assets/templates/reliability-theory/01-mtbf-mttr.md) |
| 02 Availability Formulas | [`02-availability-formulas.md`](../../foundations-reliability-theory/assets/templates/reliability-theory/02-availability-formulas.md) |
| 03 Hazard Functions | [`03-hazard-functions.md`](../../foundations-reliability-theory/assets/templates/reliability-theory/03-hazard-functions.md) |
| 04 Bathtub Curve | [`04-bathtub-curve.md`](../../foundations-reliability-theory/assets/templates/reliability-theory/04-bathtub-curve.md) |
| 05 Fault Tree Analysis | [`05-fault-tree-analysis.md`](../../foundations-reliability-theory/assets/templates/reliability-theory/05-fault-tree-analysis.md) |
| 06 FMEA | [`06-fmea.md`](../../foundations-reliability-theory/assets/templates/reliability-theory/06-fmea.md) |
| 07 Redundancy Math | [`07-redundancy-math.md`](../../foundations-reliability-theory/assets/templates/reliability-theory/07-redundancy-math.md) |
| 08 Error Budgets | [`08-error-budgets.md`](../../foundations-reliability-theory/assets/templates/reliability-theory/08-error-budgets.md) |
| 09 Weibull Analysis | [`09-weibull-analysis.md`](../../foundations-reliability-theory/assets/templates/reliability-theory/09-weibull-analysis.md) |
| 10 System Reliability | [`10-system-reliability.md`](../../foundations-reliability-theory/assets/templates/reliability-theory/10-system-reliability.md) |
| 11 Reliability Allocation | [`11-reliability-allocation.md`](../../foundations-reliability-theory/assets/templates/reliability-theory/11-reliability-allocation.md) |

**Sibling applied recipes in qa-testing-strategy:**

- [`causal-inference-applied.md`](../../qa-debugging/references/causal-inference-applied.md) — Counterfactual RCA, DiD, synthetic control, and mediation for post-mortems. Complements this file: use reliability theory to prioritize what to test before production; use causal inference to attribute what failed after production.
- [`chaos-resilience-testing.md`](chaos-resilience-testing.md) — Chaos engineering execution patterns. R2 in this file generates the MCS-ranked fault list; `chaos-resilience-testing.md` covers the tooling (Chaos Monkey, Litmus, Gremlin) for executing that list.
- [`production-testing-and-shift-right.md`](production-testing-and-shift-right.md) — Synthetic monitoring, dark launches, feature flag rollouts, and MTTR SLO. The error budget state from P6/R3 feeds directly into the release gate patterns in this reference.
- [`quality-metrics-dashboard.md`](quality-metrics-dashboard.md) — Metrics, dashboards, and mutation coverage. β estimates from P5 and FMEA RPN trends from P1 are metrics candidates for the quality dashboard.

# Schema-Aware API Fuzzing

Use this reference when contract validation is necessary but not sufficient. Schema-aware fuzzing explores valid and invalid inputs derived from the API schema, which helps catch parser, coercion, validation, and error-handling drift.

## When to add fuzzing

- Public or partner-facing APIs with many parameters
- High-risk validation logic: auth, payments, pricing, search, filtering
- Teams already using OpenAPI and wanting stronger edge-case coverage
- Bugs keep escaping because hand-written examples miss weird combinations

## Suggested stack

- Contract tests: compatibility and breaking-change detection
- Schema fuzzing: generated edge cases from the contract
- Integration smoke: real persistence, auth, and dependency behavior

## Decision rules

```text
Need to prove API quality?
    │
    ├─ Only compatibility between producer and consumer
    │   └─ Contract test
    │
    ├─ Validation, coercion, parser, and error-shape risk
    │   └─ Contract test + schema-aware fuzzing
    │
    └─ Real side effects, persistence, idempotency, or auth flow
        └─ Add integration tests
```

## Good targets

- Required vs optional fields
- Boundary values and format violations
- Enum drift and unknown values
- Nested objects and array constraints
- Error status codes and error-body stability

## Avoid

- Running unbounded fuzzing in every PR gate
- Treating fuzzing as a replacement for contract or integration tests
- Using random data without reproducible seeds or saved failing cases

# Shift-Left Testing Strategy

## Table of Contents

- [Contents](#contents)
- [Why Shift-Left?](#why-shift-left)
- [Shift-Left Practices](#shift-left-practices)
- [Decision](#decision)
- [Test Strategy](#test-strategy)
- [Test Doubles](#test-doubles)
- [Coverage Targets](#coverage-targets)
- [Shift-Left Testing Metrics](#shift-left-testing-metrics)
- [Shift-Left Anti-Patterns](#shift-left-anti-patterns)
- [Shift-Left Checklist](#shift-left-checklist)
- [Tools for Shift-Left Testing](#tools-for-shift-left-testing)
- [ROI of Shift-Left Testing](#roi-of-shift-left-testing)
- [Resources](#resources)

Shift-left testing means starting testing early in the development lifecycle—during planning, design, and development phases rather than after code is written.

## Contents

- Why Shift-Left?
- Shift-Left Practices
- Decision
- Test Strategy
- Test Doubles
- Coverage Targets
- Shift-Left Testing Metrics
- Shift-Left Anti-Patterns
- Shift-Left Checklist
- Tools for Shift-Left Testing
- ROI of Shift-Left Testing
- Resources

## Why Shift-Left?

**Traditional approach** (Shift-Right):
```
Requirements → Design → Development → Testing → Deployment
                                         ↑
                                    Testing starts here
                                    (bugs are expensive to fix)
```

**Shift-Left approach**:
```
Requirements → Design → Development → Deployment
    ↓            ↓           ↓
  Testing    Testing     Testing
  (bugs are cheap to fix at every stage)
```

**Benefits**:
- Often much cheaper to fix bugs in design/development than after release
- **Faster feedback** loops (minutes vs days)
- **Better quality** built-in, not inspected-in
- **Reduced rework** and technical debt
- **Higher confidence** in releases

## Shift-Left Practices

### 1. Testing in Requirements Phase

**Behavior-Driven Development (BDD)**:
```gherkin
# Feature file written BEFORE implementation
Feature: User Login
  As a user
  I want to log in to my account
  So that I can access my personalized dashboard

  Scenario: Successful login with valid credentials
    Given I am on the login page
    When I enter email "user@example.com"
    And I enter password "SecurePass123"
    And I click the "Login" button
    Then I should see my dashboard
    And I should see "Welcome, John"

  Scenario: Failed login with invalid password
    Given I am on the login page
    When I enter email "user@example.com"
    And I enter password "WrongPassword"
    And I click the "Login" button
    Then I should see an error "Invalid credentials"
    And I should remain on the login page
```

**Implementation with Cucumber**:
```typescript
// steps/login.steps.ts
import { Given, When, Then } from '@cucumber/cucumber';

Given('I am on the login page', async function () {
  await this.page.goto('/login');
});

When('I enter email {string}', async function (email: string) {
  await this.page.fill('input[name="email"]', email);
});

When('I enter password {string}', async function (password: string) {
  await this.page.fill('input[name="password"]', password);
});

When('I click the {string} button', async function (buttonText: string) {
  await this.page.click(`button:has-text("${buttonText}")`);
});

Then('I should see my dashboard', async function () {
  await expect(this.page.locator('.dashboard')).toBeVisible();
});

Then('I should see {string}', async function (text: string) {
  await expect(this.page.locator('body')).toContainText(text);
});
```

**Acceptance Criteria as Tests**:
```typescript
// Write tests from acceptance criteria BEFORE implementation
describe('Shopping Cart', () => {
  // AC1: Users can add items to cart
  it('should add item to cart when "Add to Cart" clicked', async () => {
    await page.click('[data-testid="add-to-cart"]');
    expect(await getCartCount()).toBe(1);
  });

  // AC2: Cart shows total price
  it('should display correct total price', async () => {
    await addItemToCart({ price: 29.99 });
    await addItemToCart({ price: 19.99 });
    expect(await getCartTotal()).toBe(49.98);
  });

  // AC3: Users can remove items
  it('should remove item when remove button clicked', async () => {
    await addItemToCart({ id: '123' });
    await page.click('[data-testid="remove-item-123"]');
    expect(await getCartCount()).toBe(0);
  });
});
```

### 2. Testing in Design Phase

**API Design Testing (Contract-First)**:
```yaml
# openapi.yml - Written BEFORE implementation
openapi: 3.0.0
info:
  title: User API
  version: 1.0.0

paths:
  /users:
    post:
      summary: Create user
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              required: [email, name]
              properties:
                email:
                  type: string
                  format: email
                name:
                  type: string
                  minLength: 2
                  maxLength: 100
      responses:
        '201':
          description: User created
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/User'
        '400':
          description: Invalid input
```

**Generate tests from OpenAPI**:
```typescript
// Auto-generated test from OpenAPI spec
import { validateAgainstSchema } from 'openapi-validator';

describe('POST /users', () => {
  it('should match OpenAPI schema', async () => {
    const response = await request(app)
      .post('/users')
      .send({ email: 'test@example.com', name: 'Test User' });

    // Validate response against OpenAPI schema
    const validation = validateAgainstSchema(response, openApiSpec, '/users', 'post');
    expect(validation.valid).toBe(true);
  });
});
```

**Architecture Decision Records (ADRs) with Test Implications**:
```markdown
# ADR-005: Use Event-Driven Architecture for Order Processing

## Decision
We will use event-driven architecture with message queues for order processing.

## Test Strategy
- **Unit tests**: Event handlers in isolation
- **Integration tests**: Message queue interactions
- **E2E tests**: Complete order flow with events
- **Idempotency tests**: Duplicate event handling
- **Ordering tests**: Event sequence correctness

## Test Doubles
- Mock message queue for unit tests
- In-memory queue for integration tests
- Real queue (RabbitMQ) for E2E tests

## Coverage Targets
- Event handlers: 100% (critical path)
- Queue integration: 90%
- Retry logic: 100%
```

### 3. Testing During Development (TDD)

**Red-Green-Refactor Cycle**:

**Step 1: Red (Write failing test)**:
```typescript
// Test written FIRST
describe('UserService', () => {
  it('should hash password before saving', async () => {
    const service = new UserService();
    const user = await service.createUser({
      email: 'test@example.com',
      password: 'PlainTextPassword'
    });

    // Password should be hashed, not stored as plaintext
    expect(user.password).not.toBe('PlainTextPassword');
    expect(user.password).toMatch(/^\$2[aby]\$.{56}$/); // bcrypt format
  });
});

// Test FAILS (implementation doesn't exist yet)
```

**Step 2: Green (Make it pass)**:
```typescript
// Minimal implementation to pass test
class UserService {
  async createUser(data: { email: string; password: string }) {
    const hashedPassword = await bcrypt.hash(data.password, 10);

    return {
      email: data.email,
      password: hashedPassword
    };
  }
}

// Test PASSES
```

**Step 3: Refactor (Improve code)**:
```typescript
// Refactored for better design
class UserService {
  constructor(
    private passwordHasher: PasswordHasher,
    private userRepository: UserRepository
  ) {}

  async createUser(data: CreateUserDTO): Promise<User> {
    const hashedPassword = await this.passwordHasher.hash(data.password);

    const user = new User({
      ...data,
      password: hashedPassword
    });

    return this.userRepository.save(user);
  }
}

// Tests still PASS (refactoring didn't break functionality)
```

**TDD Benefits**:
- Forces thinking about interface before implementation
- Ensures testable code (dependency injection, small functions)
- Provides immediate feedback
- Creates regression test suite automatically
- Documents expected behavior

### 4. Preview Environments ( Best Practice)

**Ephemeral environments for every PR**:

```yaml
# .github/workflows/preview.yml
name: Deploy Preview Environment

on:
  pull_request:
    types: [opened, synchronize]

jobs:
  deploy-preview:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Deploy to preview
        uses: amondnet/vercel-action@v25
        with:
          vercel-token: ${{ secrets.VERCEL_TOKEN }}
          vercel-org-id: ${{ secrets.VERCEL_ORG_ID }}
          vercel-project-id: ${{ secrets.VERCEL_PROJECT_ID }}
          scope: preview

      - name: Run E2E tests against preview
        env:
          PREVIEW_URL: ${{ steps.deploy.outputs.preview-url }}
        run: |
          npm run test:e2e -- --url=$PREVIEW_URL

      - name: Comment PR with preview URL
        uses: actions/github-script@v6
        with:
          script: |
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body: `Preview deployed: ${{ steps.deploy.outputs.preview-url }}`
            })
```

**Benefits of preview environments**:
- Test changes in production-like environment
- Catch integration issues early
- Enable stakeholder review before merge
- Test database migrations safely
- Verify deployment process

**Example with AWS/Kubernetes**:
```yaml
# deploy-preview.yml
apiVersion: v1
kind: Namespace
metadata:
  name: pr-{{ PR_NUMBER }}
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-preview
  namespace: pr-{{ PR_NUMBER }}
spec:
  replicas: 1
  template:
    spec:
      containers:
      - name: app
        image: myapp:pr-{{ PR_NUMBER }}
        env:
        - name: DATABASE_URL
          value: postgres://preview-{{ PR_NUMBER }}.db:5432
```

### 5. Continuous Testing in CI/CD

**Pipeline stages**:

```yaml
# Complete testing pipeline
stages:
  - validate
  - unit-test
  - integration-test
  - security-scan
  - e2e-test
  - performance-test
  - deploy

# Stage 1: Validate (seconds)
validate:
  stage: validate
  script:
    - npm run lint
    - npm run type-check
    - npm run format:check

# Stage 2: Unit tests (1-2 minutes)
unit-test:
  stage: unit-test
  script:
    - npm run test:unit -- --coverage
  coverage: '/Lines\s*:\s*(\d+\.\d+)%/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

# Stage 3: Integration tests (2-5 minutes)
integration-test:
  stage: integration-test
  services:
    - postgres:15
    - redis:7
  script:
    - npm run test:integration

# Stage 4: Security scanning (2-3 minutes)
security-scan:
  stage: security-scan
  script:
    - npm audit --audit-level=moderate
    - snyk test --severity-threshold=high

# Stage 5: E2E tests (5-10 minutes)
e2e-test:
  stage: e2e-test
  script:
    - docker-compose up -d
    - npm run test:e2e
  artifacts:
    when: on_failure
    paths:
      - test-results/

# Stage 6: Performance tests (5-10 minutes)
performance-test:
  stage: performance-test
  script:
    - k6 run performance-tests.js
  only:
    - main
    - release/*

# Stage 7: Deploy only if all tests pass
deploy-staging:
  stage: deploy
  script:
    - npm run deploy:staging
  only:
    - main
```

**Fast feedback optimization**:
```yaml
# Parallel test execution
test:
  parallel:
    matrix:
      - TEST_SUITE: [unit, integration, e2e]

  script:
    - npm run test:$TEST_SUITE

# Result: 3x faster (run simultaneously instead of sequentially)
```

### 6. Static Analysis (Pre-Development)

**TypeScript for compile-time safety**:
```typescript
// Catch errors at compile time, not runtime
interface User {
  id: string;
  email: string;
  age: number;
}

function sendEmail(user: User) {
  // TypeScript error if wrong type passed
  mailer.send(user.email);
}

// BAD: Compile error
sendEmail({ id: '123', email: 123 });  // email should be string

// GOOD: Compile success
sendEmail({ id: '123', email: 'test@example.com', age: 30 });
```

**ESLint rules for security**:
```javascript
// .eslintrc.js
module.exports = {
  plugins: ['security'],
  extends: ['plugin:security/recommended'],
  rules: {
    'security/detect-object-injection': 'error',
    'security/detect-non-literal-regexp': 'error',
    'security/detect-unsafe-regex': 'error',
    'no-eval': 'error',
    'no-implied-eval': 'error',
  }
};

// Catches security issues during development
const userInput = getUserInput();
eval(userInput);  // BAD: ESLint error: no-eval
```

**SonarQube quality gates**:
```yaml
# sonar-project.properties
sonar.qualitygate.wait=true

# Block PR if quality gate fails
sonar.qualitygate.coverage=80
sonar.qualitygate.duplications=3
sonar.qualitygate.complexity=10
sonar.qualitygate.maintainability=A
```

## Shift-Left Testing Metrics

### Early Bug Detection Rate

```
Early Bug Detection Rate = (Bugs found in Dev/Design) / (Total Bugs) × 100

Target: >80% (80% of bugs found before QA phase)
```

### Cost of Quality

```
Cost of Quality = Prevention Cost + Appraisal Cost + Failure Cost

Shift-left reduces:
- Failure cost (fewer production bugs)
- Appraisal cost (automated testing cheaper than manual)

Shift-left increases:
- Prevention cost (more upfront testing) - but net savings overall

ROI: often positive when focused on high-risk paths; validate via the metrics above
```

### Lead Time for Changes

```
Lead Time = Time from commit to production

Shift-left reduces lead time by:
- Catching bugs earlier (less rework)
- Faster testing (automated)
- Higher confidence (fewer rollbacks)

Target: <1 day for low-risk changes
```

## Shift-Left Anti-Patterns

### BAD: "We'll add tests later"
```
Problem: Tests never get written, or written as afterthought
Solution: TDD - write tests first
```

### BAD: Manual testing in dev environment
```
Problem: Slow, not repeatable, doesn't scale
Solution: Automated tests in CI/CD
```

### BAD: Testing only happy paths
```
Problem: Edge cases and errors discovered in production
Solution: Test edge cases, errors, boundaries from day 1
```

### BAD: No test ownership
```
Problem: Developers write code, QA writes tests (handoff delay)
Solution: Developers own quality - write tests for own code
```

### BAD: Skipping tests to meet deadlines
```
Problem: Technical debt accumulates, velocity decreases over time
Solution: Tests are non-negotiable part of "done"
```

## Shift-Left Checklist

### Requirements Phase
- [ ] Acceptance criteria defined for all stories
- [ ] BDD scenarios written (Given-When-Then)
- [ ] Test data requirements identified
- [ ] Performance requirements specified
- [ ] Security requirements documented

### Design Phase
- [ ] API contracts defined (OpenAPI/GraphQL schema)
- [ ] Test strategy documented in ADRs
- [ ] Testability considered in architecture
- [ ] Test environment requirements specified
- [ ] Mock/stub strategy defined

### Development Phase
- [ ] TDD practiced (test written before code)
- [ ] Unit tests for all business logic
- [ ] Integration tests for external dependencies
- [ ] Static analysis passing (linter, type-checker)
- [ ] Security scanner passing (no high vulnerabilities)

### Pre-Commit
- [ ] All tests passing locally
- [ ] Code coverage meets threshold
- [ ] No linter errors
- [ ] Git hooks running successfully

### CI/CD Pipeline
- [ ] Automated tests run on every commit
- [ ] Preview environment deployed for PRs
- [ ] E2E tests run against preview
- [ ] Performance tests for critical paths
- [ ] Security scans completed

### Pre-Merge
- [ ] All pipeline checks passing
- [ ] Code review completed (including tests)
- [ ] No failing or flaky tests
- [ ] Documentation updated (if needed)

## Tools for Shift-Left Testing

**Requirements & Design**:
- Cucumber / Gherkin (BDD)
- OpenAPI / Swagger (API contracts)
- Figma / Storybook (UI component testing)

**Development**:
- Jest / Vitest (Unit testing)
- Playwright / Cypress (E2E testing)
- Supertest (API testing)
- Docker Compose (Local integration testing)

**Pre-Commit**:
- Husky (Git hooks)
- lint-staged (Incremental linting)
- commitlint (Commit message validation)

**CI/CD**:
- GitHub Actions / GitLab CI
- Vercel / Netlify (Preview environments)
- SonarQube (Quality gates)
- Snyk / Dependabot (Security scanning)

## ROI of Shift-Left Testing

Shift-left ROI is context-dependent. Treat it as an investment decision:

- Start where risk is highest (auth, payments, data loss, distributed workflows).
- Measure outcomes (defect escape rate, lead time, incident rate, rework time, CI duration).
- Expand what works; prune suites that add cost without catching defects.

## Resources

- "Shift Left Testing" - IBM DevOps
- "Continuous Testing in DevOps" - Atlassian
- "Testing in Production" - Charity Majors
- "Accelerate" - Nicole Forsgren (DORA metrics)
- BDD with Cucumber - official documentation

# Synthetic Test Data

Ephemeral, privacy-safe test data reduces reliance on static datasets and helps avoid using real customer data in CI and staging.

## Why Synthetic Data

| Static Data | Synthetic Data |
| ----------- | -------------- |
| Privacy risks (PII) | GDPR-compliant |
| Stale, outdated | Generated on demand |
| Storage costs | Ephemeral, disposable |
| Limited edge cases | Unlimited variations |

## Synthetic Data Tools

| Tool | Best For | Features |
| ---- | -------- | -------- |
| **K2view** | Enterprise TDM | Subsetting, masking, synthetic |
| **MOSTLY AI** | Privacy-first synthetic | GDPR compliance, ML-based |
| **Synthesized** | CI/CD integration | API-first, ephemeral |
| **YData** | Data science teams | Profiling, quality scoring |
| **Faker.js** | Simple fixtures | Deterministic, lightweight |

## CI/CD Integration Pattern

```yaml
# Generate fresh synthetic data per test run
jobs:
  test:
    steps:
      - name: Generate Test Data
        run: |
          synthesized generate \
            --schema ./schemas/users.json \
            --count 1000 \
            --output ./fixtures/users.json

      - name: Run Tests
        run: npm test

      - name: Cleanup
        run: rm -rf ./fixtures  # Ephemeral, no storage
```

## Best Practices

- Generate data per test run (not shared datasets)
- Use seeded random for reproducibility
- Match production distributions (realistic edge cases)
- Dispose after test completion (ephemeral)

## Seeded Random Example

```typescript
// Reproducible test data with seed
import { faker } from '@faker-js/faker';

faker.seed(12345); // Same seed = same data

export const createTestUser = () => ({
  id: faker.string.uuid(),
  email: faker.internet.email(),
  name: faker.person.fullName(),
  createdAt: faker.date.past(),
});

// In tests
beforeEach(() => {
  faker.seed(12345); // Reset seed for reproducibility
});
```

## Privacy Compliance

| Requirement | Solution |
| ----------- | -------- |
| GDPR Right to Erasure | Ephemeral data (auto-deleted) |
| Data Minimization | Generate only needed fields |
| Pseudonymization | Synthetic replaces real PII |
| Cross-border Transfer | No real data leaves region |

## When to Use Synthetic vs Real Data

```text
Use Synthetic when:
├── PII involved (names, emails, addresses)
├── Edge cases needed (boundary values, rare scenarios)
├── Scale testing (10K+ records)
└── CI/CD pipelines (fresh data per run)

Use Real (Anonymized) when:
├── Production bug reproduction
├── Data distribution matters (ML training)
├── Regulatory audit requirements
└── Integration with live systems
```

# Test Automation Patterns

## Table of Contents

- [Contents](#contents)
- [Pattern: Page Object Model (POM)](#pattern-page-object-model-pom)
- [Pattern: Test Data Factories](#pattern-test-data-factories)
- [Pattern: Fixture Management](#pattern-fixture-management)
- [Pattern: Test Doubles (Mocks, Stubs, Fakes)](#pattern-test-doubles-mocks-stubs-fakes)
- [Pattern: Arrange-Act-Assert (AAA)](#pattern-arrange-act-assert-aaa)
- [Pattern: Test Isolation](#pattern-test-isolation)
- [Pattern: Contract Testing](#pattern-contract-testing)
- [Pattern: Retry Logic](#pattern-retry-logic)
- [Pattern: Snapshot Testing](#pattern-snapshot-testing)
- [Pattern: Test Categorization (Tags)](#pattern-test-categorization-tags)
- [Pattern: Parameterized Tests](#pattern-parameterized-tests)
- [Anti-Pattern: Testing Implementation Details](#anti-pattern-testing-implementation-details)
- [Anti-Pattern: Flaky Tests](#anti-pattern-flaky-tests)
- [Anti-Pattern: Excessive Mocking](#anti-pattern-excessive-mocking)
- [Anti-Pattern: Brittle Selectors](#anti-pattern-brittle-selectors)
- [Anti-Pattern: Testing Multiple Things](#anti-pattern-testing-multiple-things)
- [Pattern Decision Tree](#pattern-decision-tree)
- [Related Resources](#related-resources)

Modern patterns and anti-patterns for reliable, maintainable test automation.

## Contents

- Pattern: Page Object Model (POM)
- Pattern: Test Data Factories
- Pattern: Fixture Management
- Pattern: Test Doubles (Mocks, Stubs, Fakes)
- Pattern: Arrange-Act-Assert (AAA)
- Pattern: Test Isolation
- Pattern: Contract Testing
- Pattern: Retry Logic
- Pattern: Snapshot Testing
- Pattern: Test Categorization (Tags)
- Pattern: Parameterized Tests
- Anti-Pattern: Testing Implementation Details
- Anti-Pattern: Flaky Tests
- Anti-Pattern: Excessive Mocking
- Anti-Pattern: Brittle Selectors
- Anti-Pattern: Testing Multiple Things
- Pattern Decision Tree
- Related Resources

## Pattern: Page Object Model (POM)

**Use when:** Writing E2E tests with multiple page interactions.

**Benefits:**
- Encapsulates page structure
- Reduces code duplication
- Easier maintenance when UI changes
- Improves test readability

**Structure:**

```typescript
// pages/checkout.page.ts
export class CheckoutPage {
  constructor(private page: Page) {}

  // Locators (lazy evaluation)
  get addressInput() { return this.page.getByLabel('Address') }
  get cityInput() { return this.page.getByLabel('City') }
  get submitButton() { return this.page.getByRole('button', { name: 'Submit' }) }

  // Actions
  async fillShippingInfo(address: string, city: string) {
    await this.addressInput.fill(address)
    await this.cityInput.fill(city)
    await this.submitButton.click()
  }

  // Assertions
  async expectSuccessMessage() {
    await expect(this.page.getByText('Order placed')).toBeVisible()
  }
}
```

**When NOT to use:** Simple one-page tests, component tests.

---

## Pattern: Test Data Factories

**Use when:** Creating test data with complex structures.

**Benefits:**
- Consistent test data
- Easy to create variations
- Reduces magic values
- Supports factories with defaults

**Example:**

```typescript
export class UserFactory {
  static create(overrides: Partial<User> = {}): User {
    return {
      id: faker.string.uuid(),
      email: faker.internet.email(),
      name: faker.person.fullName(),
      role: 'user',
      createdAt: new Date(),
      ...overrides
    }
  }

  static createAdmin() {
    return this.create({ role: 'admin', permissions: ['all'] })
  }

  static createMany(count: number, overrides = {}) {
    return Array.from({ length: count }, () => this.create(overrides))
  }
}
```

---

## Pattern: Fixture Management

**Use when:** Tests need consistent setup data.

**Benefits:**
- Predictable test state
- Reusable across tests
- Easier debugging
- Version-controlled test data

**Example:**

```typescript
// fixtures/products.ts
export const testProducts = {
  laptop: {
    id: 'prod-1',
    name: 'MacBook Pro',
    price: 2499,
    stock: 10
  },
  mouse: {
    id: 'prod-2',
    name: 'Logitech MX',
    price: 99,
    stock: 50
  }
}

// Usage
import { testProducts } from './fixtures/products'

test('add product to cart', async () => {
  await addToCart(testProducts.laptop)
  expect(getCartTotal()).toBe(2499)
})
```

---

## Pattern: Test Doubles (Mocks, Stubs, Fakes)

**Use when:** Isolating tests from external dependencies.

**Types:**

**Mock** - Verify interactions (how many times called, with what args)
```typescript
const emailService = {
  send: jest.fn().mockResolvedValue({ success: true })
}

await service.createUser(userData)
expect(emailService.send).toHaveBeenCalledWith({
  to: 'user@example.com',
  template: 'welcome'
})
```

**Stub** - Provide predefined responses
```typescript
const paymentGateway = {
  charge: () => Promise.resolve({ transactionId: 'TX123', status: 'success' })
}
```

**Fake** - Working implementation (in-memory DB, mock server)
```typescript
class FakeDatabase {
  private data = new Map()

  async save(key, value) {
    this.data.set(key, value)
  }

  async get(key) {
    return this.data.get(key)
  }
}
```

**Guideline:** Use mocks sparingly. Prefer real implementations for internal code, use test doubles for external services.

---

## Pattern: Arrange-Act-Assert (AAA)

**Use when:** Writing any test.

**Structure:**

```typescript
test('should calculate discount', () => {
  // Arrange - Setup test data
  const order = { total: 100, items: 5 }
  const discountService = new DiscountService()

  // Act - Execute the operation
  const finalPrice = discountService.apply(order)

  // Assert - Verify the outcome
  expect(finalPrice).toBe(90) // 10% discount
})
```

**Benefits:**
- Clear test structure
- Easy to understand
- Identifies what's being tested

---

## Pattern: Test Isolation

**Use when:** Always.

**Principles:**
- Each test is independent
- No shared mutable state
- Tests can run in any order
- Tests can run in parallel

**Example:**

```typescript
// BAD: Bad - Shared state
let user: User

beforeAll(() => {
  user = createUser() // Shared across all tests
})

test('update user', () => {
  user.name = 'Updated' // Mutates shared state
})

test('delete user', () => {
  deleteUser(user.id) // Now first test will fail if this runs first
})

// GOOD: Good - Isolated state
beforeEach(() => {
  user = createUser() // Fresh user for each test
})

afterEach(() => {
  cleanupUser(user.id) // Clean up after each test
})
```

---

## Pattern: Contract Testing

**Use when:** Testing microservice APIs.

**Benefits:**
- Catches integration issues early
- Faster than E2E tests
- Consumer-driven contracts
- Safe refactoring

**Example (Pact)**:

```typescript
// Consumer test (Frontend)
describe('User API', () => {
  it('should get user by ID', async () => {
    await provider.addInteraction({
      state: 'user with ID 1 exists',
      uponReceiving: 'a request for user 1',
      withRequest: {
        method: 'GET',
        path: '/users/1'
      },
      willRespondWith: {
        status: 200,
        body: { id: 1, email: 'user@example.com' }
      }
    })

    const user = await api.getUser(1)
    expect(user.email).toBe('user@example.com')
  })
})

// Provider verification (Backend)
new Verifier({
  provider: 'UserAPI',
  providerBaseUrl: 'http://localhost:3000',
  pactUrls: ['./pacts/frontend-userapi.json']
}).verifyProvider()
```

---

## Pattern: Retry Logic

**Use when:** Tests have flakiness due to timing issues.

**Caution:** Retries mask instability. Fix root cause when possible.

**Example:**

```typescript
// Playwright (built-in)
await expect(page.getByText('Loaded')).toBeVisible({ timeout: 5000 })

// Jest with custom retry
async function retryOperation(operation, maxRetries = 3) {
  for (let i = 0; i < maxRetries; i++) {
    try {
      return await operation()
    } catch (error) {
      if (i === maxRetries - 1) throw error
      await new Promise(resolve => setTimeout(resolve, 1000 * (i + 1)))
    }
  }
}

test('fetch data with retry', async () => {
  const data = await retryOperation(() => fetchData())
  expect(data).toBeDefined()
})
```

---

## Pattern: Snapshot Testing

**Use when:** Testing output that changes rarely (config, serialization).

**Benefits:**
- Catches unintended changes
- Quick to write
- Good for regression testing

**Caution:** Easy to blindly approve changes.

**Example:**

```typescript
test('config snapshot', () => {
  const config = getConfig()
  expect(config).toMatchSnapshot()
})

test('user profile snapshot', () => {
  const profile = getUserProfile('user-123')
  expect(profile).toMatchSnapshot({
    createdAt: expect.any(Date), // Dynamic values
    id: expect.any(String)
  })
})
```

**When to update snapshots:** Only when change is intentional. Review diffs carefully.

---

## Pattern: Test Categorization (Tags)

**Use when:** Running different test suites.

**Categories:**
- **@smoke** - Critical paths, run on every commit
- **@regression** - Full suite, run nightly
- **@slow** - Long-running tests
- **@flaky** - Known flaky tests (quarantined)
- **@wip** - Work in progress

**Example (Cucumber)**:

```gherkin
@smoke @critical
Feature: Login

@happy-path
Scenario: Successful login
  # ...

@error-handling @slow
Scenario: Account lockout
  # ...
```

```bash
# Run smoke tests only
npm test -- --tags "@smoke"

# Run all except slow tests
npm test -- --tags "not @slow"
```

---

## Pattern: Parameterized Tests

**Use when:** Testing same logic with different inputs.

**Example (Jest/Vitest)**:

```typescript
describe.each([
  { input: 'hello', expected: 'HELLO' },
  { input: 'world', expected: 'WORLD' },
  { input: '', expected: '' },
  { input: '123', expected: '123' }
])('uppercase($input)', ({ input, expected }) => {
  it(`should return ${expected}`, () => {
    expect(uppercase(input)).toBe(expected)
  })
})
```

**Example (Cucumber Scenario Outline)**:

```gherkin
Scenario Outline: Validate email
  When I enter email "<email>"
  Then validation should be "<result>"

  Examples:
    | email               | result  |
    | user@example.com    | valid   |
    | invalid-email       | invalid |
    | @example.com        | invalid |
```

---

## Anti-Pattern: Testing Implementation Details

**Problem:** Tests break when refactoring, even though behavior unchanged.

**Example:**

```typescript
// BAD: Bad - Tests internal method
test('should call internal helper', () => {
  const spy = jest.spyOn(service, 'internalHelper')
  service.publicMethod()
  expect(spy).toHaveBeenCalled()
})

// GOOD: Good - Tests public behavior
test('should return correct result', () => {
  const result = service.publicMethod()
  expect(result).toBe(expectedValue)
})
```

---

## Anti-Pattern: Flaky Tests

**Problem:** Tests pass/fail randomly, undermining trust.

**Common causes:**
- Race conditions (async timing)
- Shared mutable state
- External dependencies (network, time)
- Test order dependencies

**Solutions:**
- Use explicit waits (not sleep)
- Isolate tests (fresh state per test)
- Mock time and external services
- Run tests in random order locally

```typescript
// BAD: Bad - Sleep (flaky)
await sleep(1000) // Hope data loads in 1 second

// GOOD: Good - Web-first assertion
await expect(page.locator('[data-loaded="true"]')).toBeVisible()
await expect(page.getByText('Data loaded')).toBeVisible()
```

---

## Anti-Pattern: Excessive Mocking

**Problem:** Tests pass but integration fails.

**Example:**

```typescript
// BAD: Bad - Mocking everything
const database = { save: jest.fn(), find: jest.fn() }
const cache = { get: jest.fn(), set: jest.fn() }
const logger = { log: jest.fn() }
const emailService = { send: jest.fn() }

// Unit test passes, but real integration is untested

// GOOD: Good - Use real implementations for internal code
const database = new InMemoryDatabase() // Real logic
const cache = new InMemoryCache()       // Real logic
const emailService = mockEmailService() // Mock external service
```

**Guideline:** Mock external boundaries, use real implementations internally.

---

## Anti-Pattern: Brittle Selectors

**Problem:** E2E tests break when CSS/HTML changes.

**Example:**

```typescript
// BAD: Bad - Implementation-coupled selectors
await page.locator('.btn.btn-primary.submit-btn-v2').click()
await page.locator('div > div > div > button:nth-child(3)').click()

// GOOD: Good - Semantic selectors
await page.getByRole('button', { name: 'Submit' }).click()
await page.getByTestId('submit-button').click()
await page.getByLabel('Submit form').click()
```

**Best to worst:**
1. data-testid
2. ARIA role + accessible name
3. User-visible text
4. CSS class (avoid)
5. XPath/complex selectors (avoid)

---

## Anti-Pattern: Testing Multiple Things

**Problem:** Unclear what failed when test breaks.

**Example:**

```typescript
// BAD: Bad - Tests multiple behaviors
test('user service', async () => {
  const user = await service.create({ email: 'test@example.com' })
  expect(user.id).toBeDefined()

  const found = await service.findById(user.id)
  expect(found).toBeDefined()

  await service.delete(user.id)
  const deleted = await service.findById(user.id)
  expect(deleted).toBeNull()
})

// GOOD: Good - One behavior per test
test('should create user', async () => {
  const user = await service.create({ email: 'test@example.com' })
  expect(user.id).toBeDefined()
})

test('should find user by ID', async () => {
  const user = await service.create({ email: 'test@example.com' })
  const found = await service.findById(user.id)
  expect(found).toBeDefined()
})

test('should delete user', async () => {
  const user = await service.create({ email: 'test@example.com' })
  await service.delete(user.id)
  const deleted = await service.findById(user.id)
  expect(deleted).toBeNull()
})
```

---

## Pattern Decision Tree

**What should I test?**

```
Is it a UI interaction?
├─ YES → E2E test (Playwright/Cypress)
└─ NO
   ├─ Is it business logic?
   │  └─ YES → Unit test (Jest/Vitest)
   └─ Is it API contract?
      └─ YES → Contract test (Pact) + Integration test
```

**Should I mock this?**

```
Is it an external service (API, payment gateway)?
├─ YES → Mock it
└─ NO
   ├─ Is it a database?
   │  ├─ Unit test → Use in-memory/mock
   │  └─ Integration test → Use real DB (Docker)
   └─ Is it internal code?
      └─ NO → Use real implementation
```

## Related Resources

See [comprehensive-testing-guide.md](comprehensive-testing-guide.md) for test pyramid and strategy, [shift-left-testing.md](shift-left-testing.md) for early testing practices.

# Test Environment Management

## Table of Contents

- [Contents](#contents)
- [Environment Types](#environment-types)
- [Environment-as-Code](#environment-as-code)
- [Database Seeding and Fixtures](#database-seeding-and-fixtures)
- [Service Virtualization](#service-virtualization)
- [Environment Isolation Strategies](#environment-isolation-strategies)
- [Shared vs Dedicated Environments](#shared-vs-dedicated-environments)
- [Environment Drift Detection](#environment-drift-detection)
- [Secrets Management](#secrets-management)
- [Provisioning Automation](#provisioning-automation)
- [Teardown and Cleanup](#teardown-and-cleanup)
- [Health Monitoring](#health-monitoring)
- [Cost Optimization](#cost-optimization)
- [Environment Management Checklist](#environment-management-checklist)
- [Related Resources](#related-resources)

Test environment provisioning, configuration, lifecycle management, and cost optimization -- from local dev through pre-production.

## Contents

- Environment Types
- Environment-as-Code
- Database Seeding and Fixtures
- Service Virtualization
- Environment Isolation Strategies
- Shared vs Dedicated Environments
- Environment Drift Detection
- Secrets Management
- Provisioning Automation
- Teardown and Cleanup
- Health Monitoring
- Cost Optimization
- Environment Management Checklist
- Related Resources

---

## Environment Types

| Environment | Purpose | Data | Lifecycle | Who Uses It |
|-------------|---------|------|-----------|-------------|
| **Local** | Developer testing | Synthetic/seeded | Persistent | Individual devs |
| **CI** | Automated tests | Synthetic, ephemeral | Per-pipeline | CI system |
| **Preview/PR** | Feature review | Seeded from template | Per-PR, ephemeral | Devs + reviewers |
| **Staging** | Integration testing | Sanitized prod subset | Long-lived | QA team |
| **Pre-prod** | Release validation | Prod-like volume | Long-lived | QA + Ops |
| **Prod** | Live users | Real data | Permanent | Everyone |

### Environment Maturity Model

```text
Level 1: Manual setup       → "Works on my machine" problems
Level 2: Documented setup   → Wiki/README with manual steps
Level 3: Scripted setup     → Shell scripts, Makefiles
Level 4: Environment-as-code → Docker Compose, Terraform
Level 5: Self-service       → On-demand provisioning, auto-teardown
```

---

## Environment-as-Code

### Docker Compose (Local + CI)

```yaml
# docker-compose.test.yml
version: "3.9"

services:
  app:
    build:
      context: .
      dockerfile: Dockerfile
      target: test
    ports:
      - "3000:3000"
    environment:
      - DATABASE_URL=postgres://test:test@db:5432/testdb
      - REDIS_URL=redis://cache:6379
      - NODE_ENV=test
    depends_on:
      db:
        condition: service_healthy
      cache:
        condition: service_started

  db:
    image: postgres:16
    environment:
      POSTGRES_USER: test
      POSTGRES_PASSWORD: test
      POSTGRES_DB: testdb
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U test"]
      interval: 5s
      timeout: 5s
      retries: 5
    volumes:
      - ./scripts/seed.sql:/docker-entrypoint-initdb.d/seed.sql
    tmpfs:
      - /var/lib/postgresql/data  # RAM disk for speed

  cache:
    image: redis:7-alpine

  mailhog:
    image: mailhog/mailhog
    ports:
      - "8025:8025"  # Web UI for email testing
```

```bash
# Start environment
docker compose -f docker-compose.test.yml up -d

# Run tests against it
npm run test:e2e

# Tear down
docker compose -f docker-compose.test.yml down -v
```

### Terraform (Cloud Environments)

```hcl
# environments/staging/main.tf
module "staging_env" {
  source = "../../modules/test-environment"

  environment_name = "staging"
  app_version      = var.app_version
  instance_type    = "t3.medium"
  db_instance_class = "db.t3.medium"

  # Smaller resources than prod
  min_instances    = 1
  max_instances    = 2
  db_storage_gb    = 20

  tags = {
    Environment = "staging"
    ManagedBy   = "terraform"
    CostCenter  = "engineering-qa"
  }
}
```

### Pulumi (Infrastructure in Code)

```typescript
// infra/test-environment.ts
import * as pulumi from "@pulumi/pulumi";
import * as aws from "@pulumi/aws";

export function createTestEnvironment(name: string) {
  const db = new aws.rds.Instance(`${name}-db`, {
    engine: "postgres",
    engineVersion: "16",
    instanceClass: "db.t3.micro",
    allocatedStorage: 10,
    dbName: "testdb",
    username: "test",
    password: pulumi.secret("test-password"),
    skipFinalSnapshot: true,
    tags: { Environment: name },
  });

  const app = new aws.ecs.Service(`${name}-app`, {
    desiredCount: 1,
    taskDefinition: createTaskDef(name, db.endpoint),
  });

  return { dbEndpoint: db.endpoint, appUrl: app.id };
}
```

---

## Database Seeding and Fixtures

### Seed Script Pattern

```typescript
// scripts/seed-test-data.ts
import { PrismaClient } from '@prisma/client';

const prisma = new PrismaClient();

async function seed() {
  // Clean existing data (order matters for foreign keys)
  await prisma.orderItem.deleteMany();
  await prisma.order.deleteMany();
  await prisma.product.deleteMany();
  await prisma.user.deleteMany();

  // Seed users
  const admin = await prisma.user.create({
    data: {
      email: 'admin@test.example.com',
      name: 'Test Admin',
      role: 'ADMIN',
      password: '$2b$10$hashedpassword', // pre-hashed
    },
  });

  const user = await prisma.user.create({
    data: {
      email: 'user@test.example.com',
      name: 'Test User',
      role: 'USER',
      password: '$2b$10$hashedpassword',
    },
  });

  // Seed products
  const products = await Promise.all(
    Array.from({ length: 10 }, (_, i) =>
      prisma.product.create({
        data: {
          name: `Test Product ${i + 1}`,
          price: (i + 1) * 9.99,
          stock: 100,
        },
      })
    )
  );

  console.log(`Seeded: ${2} users, ${products.length} products`);
}

seed()
  .catch(console.error)
  .finally(() => prisma.$disconnect());
```

### Fixture Factory Pattern

```typescript
// test/factories/user.factory.ts
import { faker } from '@faker-js/faker';

export function buildUser(overrides: Partial<User> = {}): User {
  return {
    id: faker.string.uuid(),
    name: faker.person.fullName(),
    email: faker.internet.email(),
    role: 'USER',
    createdAt: new Date().toISOString(),
    ...overrides,
  };
}

export function buildUsers(count: number, overrides: Partial<User> = {}): User[] {
  return Array.from({ length: count }, () => buildUser(overrides));
}

// Usage in tests
const adminUser = buildUser({ role: 'ADMIN' });
const regularUsers = buildUsers(5);
```

### Database Snapshot Pattern

```bash
# Create a snapshot of seeded database
pg_dump -U test -d testdb -F c -f test-snapshot.dump

# Restore before each test suite (fast reset)
pg_restore -U test -d testdb --clean --no-owner test-snapshot.dump
```

---

## Service Virtualization

### WireMock (HTTP API Mocking)

```json
// wiremock/mappings/payment-gateway.json
{
  "request": {
    "method": "POST",
    "urlPattern": "/api/v1/charges",
    "headers": {
      "Authorization": { "matches": "Bearer .*" }
    }
  },
  "response": {
    "status": 200,
    "headers": { "Content-Type": "application/json" },
    "jsonBody": {
      "id": "ch_test_123",
      "status": "succeeded",
      "amount": "{{jsonPath request.body '$.amount'}}",
      "currency": "usd"
    },
    "transformers": ["response-template"]
  }
}
```

```yaml
# docker-compose.test.yml - add WireMock
services:
  wiremock:
    image: wiremock/wiremock:3.3.1
    ports:
      - "8080:8080"
    volumes:
      - ./wiremock:/home/wiremock
    command: --verbose --global-response-templating
```

### MockServer

```typescript
// test/mocks/setup-mockserver.ts
import { MockServerClient } from 'mockserver-client';

const mockServer = new MockServerClient('localhost', 1080);

export async function setupExternalMocks() {
  // Mock email service
  await mockServer.mockSimpleResponse(
    '/api/send-email',
    { success: true, messageId: 'mock-123' },
    200
  );

  // Mock geolocation API
  await mockServer.mockAnyResponse({
    httpRequest: { path: '/api/geoip/.*', method: 'GET' },
    httpResponse: {
      statusCode: 200,
      body: JSON.stringify({ country: 'US', region: 'CA', city: 'San Francisco' }),
    },
  });
}
```

### When to Virtualize

| External Service | Virtualize? | Rationale |
|-----------------|-------------|-----------|
| Payment gateway (Stripe, etc.) | Yes, always | Cost, rate limits, side effects |
| Email service (SendGrid, etc.) | Yes, always | Side effects (spam), delivery delays |
| SMS provider | Yes, always | Cost, side effects |
| Auth provider (Auth0, etc.) | Usually | Rate limits; test mode may suffice |
| Analytics (Segment, etc.) | Yes | Irrelevant to test, slows execution |
| Database | No | Use real instance (Docker) |
| Message queue | Sometimes | Use real for integration, mock for unit |

---

## Environment Isolation Strategies

### Namespace Isolation (Kubernetes)

```yaml
# Per-PR namespace
apiVersion: v1
kind: Namespace
metadata:
  name: pr-${PR_NUMBER}
  labels:
    environment: preview
    pr: "${PR_NUMBER}"
    auto-cleanup: "true"
---
# NetworkPolicy: isolate from other namespaces
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: isolate-namespace
  namespace: pr-${PR_NUMBER}
spec:
  podSelector: {}
  policyTypes: [Ingress]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              name: pr-${PR_NUMBER}
```

### Database Isolation

```text
Strategy 1: Separate databases per environment
  + Full isolation, no cross-contamination
  - Higher resource cost

Strategy 2: Schema-per-tenant
  + Lower resource cost, faster provisioning
  - Shared database risks

Strategy 3: Row-level isolation (tenant_id column)
  + Cheapest, simplest
  - Data leaks possible if filter missed

Recommendation: Separate databases for staging/pre-prod;
  schema-per-tenant for ephemeral PR environments.
```

---

## Shared vs Dedicated Environments

| Dimension | Shared | Dedicated |
|-----------|--------|-----------|
| **Cost** | Low (1 env, many teams) | High (1 env per team/feature) |
| **Isolation** | Low (data conflicts possible) | High (full independence) |
| **Stability** | Lower (broken by other teams) | Higher (self-controlled) |
| **Maintenance** | Lower (one set of infra) | Higher (many environments) |
| **Best for** | Manual QA, demo | Automated testing, CI |

### Hybrid Approach

```text
Shared environments:
  - staging: manual QA, demos, exploratory testing
  - pre-prod: release validation, performance testing

Dedicated environments:
  - CI: ephemeral per pipeline, torn down after
  - PR preview: ephemeral per pull request
  - Feature: on-demand for large features (request-based)
```

---

## Environment Drift Detection

### Configuration Comparison Script

```python
#!/usr/bin/env python3
"""Detect configuration drift between environments."""
import json
import subprocess
import sys

def get_env_config(env_name: str) -> dict:
    """Fetch running config from environment."""
    result = subprocess.run(
        ["kubectl", "get", "configmap", "app-config",
         "-n", env_name, "-o", "json"],
        capture_output=True, text=True
    )
    return json.loads(result.stdout)["data"]

def compare_configs(source: str, target: str) -> list[dict]:
    """Compare two environment configurations."""
    source_config = get_env_config(source)
    target_config = get_env_config(target)

    diffs = []
    all_keys = set(source_config) | set(target_config)

    for key in sorted(all_keys):
        src_val = source_config.get(key, "<MISSING>")
        tgt_val = target_config.get(key, "<MISSING>")
        if src_val != tgt_val:
            diffs.append({
                "key": key,
                source: src_val,
                target: tgt_val,
            })
    return diffs

if __name__ == "__main__":
    drifts = compare_configs("staging", "production")
    if drifts:
        print(f"Found {len(drifts)} config differences:")
        for d in drifts:
            print(f"  {d['key']}: staging={d['staging']} prod={d['production']}")
        sys.exit(1)
    print("No drift detected.")
```

### Infrastructure Drift Check

```bash
# Terraform drift detection
terraform plan -detailed-exitcode -var-file=staging.tfvars
# Exit code 0 = no changes, 1 = error, 2 = changes detected (drift)

# Schedule weekly drift check in CI
# .github/workflows/drift-check.yml
name: Environment Drift Check
on:
  schedule:
    - cron: '0 8 * * 1'  # Monday 8am
jobs:
  drift:
    runs-on: ubuntu-latest
    steps:
      - run: terraform plan -detailed-exitcode
```

---

## Secrets Management

### Environment Variable Patterns

```bash
# .env.test (checked into git - non-sensitive only)
NODE_ENV=test
DATABASE_HOST=localhost
DATABASE_PORT=5432
DATABASE_NAME=testdb
LOG_LEVEL=warn

# .env.test.local (git-ignored - sensitive values)
DATABASE_PASSWORD=local-test-password
API_KEY=test-api-key-12345
STRIPE_SECRET_KEY=sk_test_xxxxx
```

### CI Secrets

```yaml
# GitHub Actions: secrets from repository settings
- name: Run tests
  env:
    DATABASE_URL: ${{ secrets.TEST_DATABASE_URL }}
    API_KEY: ${{ secrets.TEST_API_KEY }}
  run: npm run test:e2e
```

### Vault Integration

```bash
# HashiCorp Vault: dynamic database credentials
vault read database/creds/test-role
# Returns: username=v-test-xxxx, password=yyyy, ttl=1h

# In CI pipeline
export DATABASE_URL=$(vault read -field=connection_url database/creds/test-role)
npm run test:e2e
```

### Secrets Checklist

- [ ] No secrets in source control (git-ignored `.env.local` files)
- [ ] CI secrets stored in platform secrets manager (GitHub/GitLab)
- [ ] Test API keys scoped to test environment only
- [ ] Database passwords rotated regularly
- [ ] Secrets audit log enabled
- [ ] Production secrets never used in test environments

---

## Provisioning Automation

### Makefile Commands

```makefile
# Makefile
.PHONY: env-up env-down env-reset env-seed env-health

env-up:
	docker compose -f docker-compose.test.yml up -d
	@echo "Waiting for services..."
	@sleep 5
	$(MAKE) env-health

env-down:
	docker compose -f docker-compose.test.yml down -v

env-reset: env-down env-up env-seed

env-seed:
	docker compose exec app npx prisma db seed

env-health:
	@curl -sf http://localhost:3000/health > /dev/null && echo "App: OK" || echo "App: DOWN"
	@docker compose exec db pg_isready -U test > /dev/null && echo "DB: OK" || echo "DB: DOWN"
	@docker compose exec cache redis-cli ping > /dev/null && echo "Cache: OK" || echo "Cache: DOWN"
```

### GitHub Actions: Reusable Workflow

```yaml
# .github/workflows/test-env.yml
name: Test with Environment
on:
  workflow_call:
    inputs:
      test-command:
        required: true
        type: string

jobs:
  test:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:16
        env: { POSTGRES_PASSWORD: test, POSTGRES_DB: testdb }
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports: ["5432:5432"]
      redis:
        image: redis:7-alpine
        ports: ["6379:6379"]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with: { node-version: 20 }
      - run: npm ci
      - run: npx prisma db push && npx prisma db seed
        env:
          DATABASE_URL: postgres://postgres:test@localhost:5432/testdb
      - run: ${{ inputs.test-command }}
        env:
          DATABASE_URL: postgres://postgres:test@localhost:5432/testdb
          REDIS_URL: redis://localhost:6379
```

---

## Teardown and Cleanup

### Automatic Cleanup for Ephemeral Environments

```yaml
# Kubernetes CronJob: clean up old PR environments
apiVersion: batch/v1
kind: CronJob
metadata:
  name: cleanup-preview-envs
spec:
  schedule: "0 */6 * * *"  # Every 6 hours
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: cleanup
            image: bitnami/kubectl
            command:
            - /bin/sh
            - -c
            - |
              # Delete namespaces older than 48 hours with auto-cleanup label
              kubectl get namespaces -l auto-cleanup=true -o json | \
                jq -r '.items[] | select(
                  (.metadata.creationTimestamp | fromdateiso8601) < (now - 172800)
                ) | .metadata.name' | \
                xargs -r kubectl delete namespace
```

### GitHub Actions: Cleanup on PR Close

```yaml
name: Cleanup Preview Environment
on:
  pull_request:
    types: [closed]

jobs:
  cleanup:
    runs-on: ubuntu-latest
    steps:
      - name: Delete preview environment
        run: |
          kubectl delete namespace pr-${{ github.event.number }} --ignore-not-found
          echo "Cleaned up preview environment for PR #${{ github.event.number }}"
```

---

## Health Monitoring

### Health Check Endpoint

```typescript
// health.ts - comprehensive health check
app.get('/health', async (req, res) => {
  const checks = {
    database: await checkDatabase(),
    redis: await checkRedis(),
    externalApi: await checkExternalApi(),
  };

  const healthy = Object.values(checks).every(c => c.status === 'ok');

  res.status(healthy ? 200 : 503).json({
    status: healthy ? 'healthy' : 'unhealthy',
    timestamp: new Date().toISOString(),
    checks,
  });
});
```

### Environment Health Dashboard

```text
Environment Health - `<YYYY-MM-DD HH:MM UTC>`
------------------------------------------
Local dev:     HEALTHY  (all services up)
CI runner 1:   HEALTHY  (pipeline running)
CI runner 2:   HEALTHY  (idle)
Staging:       DEGRADED (redis high memory - 89%)
Pre-prod:      HEALTHY  (all services up)

Alerts:
  - staging/redis: Memory usage 89% (threshold: 85%)
  - Action: Scale redis or flush test data
```

---

## Cost Optimization

| Strategy | Savings | Implementation Effort |
|----------|---------|----------------------|
| Auto-shutdown non-prod at night | 40-60% | Low (cron/lambda) |
| Right-size test instances | 20-40% | Medium (monitoring) |
| Spot/preemptible instances for CI | 60-80% | Medium (retry logic) |
| Share staging across teams | 30-50% | Low (scheduling) |
| Ephemeral PR environments | Variable | High (automation) |
| RAM disk for test databases | Speed, not $ | Low (tmpfs config) |

```bash
# Auto-shutdown staging at 8pm, start at 7am (weekdays)
# AWS Lambda + CloudWatch Events
aws events put-rule --name stop-staging --schedule-expression "cron(0 20 ? * MON-FRI *)"
aws events put-rule --name start-staging --schedule-expression "cron(0 7 ? * MON-FRI *)"
```

---

## Environment Management Checklist

### New Environment Setup

- [ ] Infrastructure defined as code (Docker Compose / Terraform / Pulumi)
- [ ] Database seeding automated
- [ ] Service mocks configured for external dependencies
- [ ] Health check endpoint available
- [ ] Secrets injected from secure source (not hardcoded)
- [ ] Cleanup / teardown automated
- [ ] Cost tracking labels applied
- [ ] Access control configured (who can access what)

### Ongoing Maintenance

- [ ] Weekly drift detection between staging and production
- [ ] Monthly cost review of test environments
- [ ] Quarterly cleanup of orphaned resources
- [ ] Seed data refreshed when schema changes
- [ ] Health monitoring alerts configured

---

## Related Resources

- [synthetic-test-data.md](./synthetic-test-data.md) -- generating test data for seeding
- [operational-playbook.md](./operational-playbook.md) -- CI/CD pipeline patterns using test environments
- [shift-left-testing.md](./shift-left-testing.md) -- preview environments for PR testing
- [SKILL.md](../SKILL.md) -- parent testing strategy skill
- [Docker Compose Documentation](https://docs.docker.com/compose/)
- [Terraform Testing](https://developer.hashicorp.com/terraform/tutorials/configuration-language/test)
- [WireMock](https://wiremock.org/docs/)
- [MockServer](https://www.mock-server.com/)

# Test Impact Analysis

## Table of Contents

- [Concept](#concept)
- [Safety Contract](#safety-contract)
- [Tool Landscape](#tool-landscape)
- [Jest: findRelatedTests](#jest-findrelatedtests)
- [Launchable](#launchable)
- [Datadog Test Visibility](#datadog-test-visibility)
- [BuildPulse (Flake Trending)](#buildpulse-flake-trending)
- [NCrunch for .NET](#ncrunch-for-net)
- [CI Integration Patterns](#ci-integration-patterns)
- [TIA Anti-Patterns](#tia-anti-patterns)
- [Related Resources](#related-resources)

Test Impact Analysis (TIA) reduces CI cycle time by running only the tests that could be affected by a given code change. Instead of executing the full suite on every PR, TIA builds a change-to-test dependency map and selects the minimal subset likely to catch regressions in the changed code.

---

## Concept

The core idea: if a change touches `src/payment/charge.ts`, only tests that directly or transitively depend on `payment/charge.ts` need to run for that PR. Tests that cover unrelated modules are skipped.

```text
Code change
    │
    ▼
Dependency graph lookup
    │
    ▼
Affected test set  ──► Run (fast feedback)
    │
Unaffected tests   ──► Skip (or defer to scheduled full run)
```

The dependency graph is built from one or more signals:

- **Static analysis** – parse imports/requires; build a module dependency tree.
- **Dynamic instrumentation** – instrument the test runner to record which source files are loaded during each test execution; store the mapping for future runs.
- **Git diff** – identify changed files; join against the stored mapping.

---

## Safety Contract

TIA MUST NEVER reduce coverage below the established baseline. Violating this contract makes TIA worse than running the full suite.

Rules:

1. **Full suite on merge to `main`** – TIA applies only to PR / feature-branch runs. The mainline always runs everything.
2. **Full suite on scheduled cadence** – Run the complete suite on a nightly or pre-release schedule to catch cross-cutting regressions the impact map missed.
3. **Fallback on graph staleness** – If the dependency map has not been updated within a configurable window (e.g., 7 days since last full run), fall back to the full suite.
4. **Fallback on structural changes** – Dependency graph invalidators (package.json changes, tsconfig.json changes, build system changes, file renames) trigger a full run.
5. **Coverage baseline check** – After each full run, assert that aggregated line/branch coverage has not decreased below the stored baseline. Block merge if it has.

```yaml
# Example CI gate: enforce safety contract
tia_safety:
  full_run_triggers:
    - paths: ["package.json", "package-lock.json", "tsconfig*.json", "jest.config.*"]
    - schedule: "0 2 * * *"   # nightly full run
    - branches: ["main", "release/*"]
  coverage_baseline:
    enforce: true
    metric: lines
    minimum_delta: 0          # coverage must not decrease
```

---

## Tool Landscape

| Tool | Languages | Signal source | Hosted / Self-hosted |
|------|-----------|--------------|----------------------|
| **Launchable** | Java, Python, Go, Ruby, JS/TS, .NET | ML model on historical results | Hosted SaaS |
| **Datadog Test Optimization** | JS/TS, Python, Java, Ruby, Go, .NET | Dynamic instrumentation + APM | Hosted (Datadog) |
| **BuildPulse** | Any (JUnit XML) | Flake trending from test reports | Hosted SaaS |
| **jest --findRelatedTests** | JavaScript / TypeScript | Static import graph | CLI (built-in) |
| **NCrunch** | .NET (C#, VB.NET) | Continuous in-IDE instrumentation | Local / CI |
| **Bazel** | Polyglot | Hermetic build graph | Self-hosted |
| **Nx affected** | JS/TS monorepos | Module dependency graph | CLI (built-in) |

Note: Datadog's product is currently marketed as "Test Optimization" (not "Test Visibility" — both names appear in documentation). Verify current product name at https://docs.datadoghq.com/tests/ before configuring.

---

## Jest: findRelatedTests

Jest's built-in flag performs static import graph traversal to find tests related to changed source files. No external service required.

```bash
# Run only tests affected by changed files (from git diff)
git diff --name-only HEAD~1 HEAD | \
  grep -E '\.(ts|tsx|js|jsx)$' | \
  xargs npx jest --findRelatedTests --passWithNoTests
```

### GitHub Actions integration

```yaml
- name: Run affected tests
  run: |
    CHANGED=$(git diff --name-only ${{ github.event.before }} ${{ github.sha }} \
      | grep -E '\.(ts|tsx|js|jsx)$' || true)
    if [ -z "$CHANGED" ]; then
      echo "No JS/TS changes — skipping jest"
    else
      echo "$CHANGED" | xargs npx jest --findRelatedTests --passWithNoTests --ci
    fi
```

**Limitations**: only traces static imports; dynamic `require()` calls and barrel re-exports can cause missed tests. Combine with a full nightly run per the safety contract.

---

## Launchable

Launchable applies an ML model trained on your historical test results and code-change patterns to rank and subset tests. It predicts which tests are most likely to fail for a given change set.

### How it works

1. **Record** – Launchable CLI instruments your CI to upload test results and git metadata after each run.
2. **Train** – The model learns which tests fail when specific files change.
3. **Subset** – On each PR, the CLI returns the predicted high-value subset; the runner executes that subset.
4. **Always record** – Full runs (nightly, merge to main) continue to feed the model.

### CLI integration (Java/Maven example)

```bash
# Record build results
launchable record build --name "$BUILD_ID" --source .

# Request a subset (target: 20 minutes of the most impactful tests)
launchable subset --target 20% --build "$BUILD_ID" maven > launchable-subset.txt

# Run the subset
mvn test -Dsurefire.includesFile=launchable-subset.txt

# Record results
launchable record tests --build "$BUILD_ID" maven target/surefire-reports/
```

Launchable guarantees a configurable confidence level (e.g., 95% confidence the subset catches any failure the full suite would catch). The safety contract is enforced by Launchable's own model confidence threshold.

---

## Datadog Test Visibility

Datadog Test Visibility (part of CI Visibility) instruments test frameworks at runtime to record per-test traces, durations, and outcomes. It powers:

- **Flaky test detection** – automatic identification of non-deterministic tests across branches.
- **Test impact analysis** – correlates code changes with historically failing tests using the stored trace data.
- **Early flake detection** – new tests are run multiple times on first appearance to establish a stability baseline before they can block CI.

### Setup (JavaScript / Vitest)

```typescript
// vitest.config.ts
import { defineConfig } from 'vitest/config';
import { DatadogCIPlugin } from 'dd-trace/ci/vitest';

export default defineConfig({
  plugins: [DatadogCIPlugin()],
  test: { reporters: ['verbose'] },
});
```

```bash
# Required env vars
export DD_API_KEY=<api-key>
export DD_ENV=ci
export DD_SERVICE=my-service
npx vitest run
```

### Key metrics surfaced

| Metric | Use |
|--------|-----|
| Flakiness rate per test | Prioritise deflake work |
| Mean duration trend | Detect slowdowns before they block CI |
| Branch vs mainline failure divergence | Catch regressions introduced on a branch |
| TIA skip ratio | Measure CI time saved |

---

## BuildPulse (Flake Trending)

BuildPulse ingests JUnit XML test reports from any CI system and provides flake trending, ownership attribution, and quarantine recommendations. It does not perform TIA itself but is the recommended complement for flake visibility when using TIA tools that do not have built-in flake detection.

### Integration (GitHub Actions)

```yaml
- name: Upload test results to BuildPulse
  if: always()
  uses: buildpulse/buildpulse-action@v0
  with:
    account: ${{ secrets.BUILDPULSE_ACCOUNT_ID }}
    repository: ${{ secrets.BUILDPULSE_REPOSITORY_ID }}
    path: test-results/**/*.xml
    key: ${{ secrets.BUILDPULSE_ACCESS_KEY_ID }}
    secret: ${{ secrets.BUILDPULSE_SECRET_ACCESS_KEY }}
```

### What BuildPulse surfaces

- **Flakiness score per test** – percentage of runs that produced an inconsistent result.
- **Trend chart** – flakiness rate over time to distinguish stable, improving, and worsening tests.
- **Owner attribution** – maps flaky tests to the last committer on that test file.
- **Quarantine recommendations** – flags tests with flakiness > configurable threshold as quarantine candidates.

Pair BuildPulse trending data with MTTR-Flake SLO tracking (see [production-testing-and-shift-right.md](./production-testing-and-shift-right.md)) for a complete flake lifecycle picture.

---

## NCrunch for .NET

NCrunch is a continuous test runner for Visual Studio and Rider that instruments .NET tests at the bytecode level and runs them automatically as you type. It provides the tightest possible feedback loop: sub-second test execution for changed code paths.

### How it implements TIA

NCrunch maintains a runtime instrumentation map linking each line of source code to the tests that executed it. When a file changes, only the mapped tests re-run — in the background, in parallel, without a manual trigger.

### CI mode

NCrunch can export its coverage and impact data for use in CI pipelines:

```xml
<!-- ncrunch.project settings for CI export -->
<NCrunchProjectSettings>
  <CoverageExportFormat>opencover</CoverageExportFormat>
  <CoverageExportPath>coverage/ncrunch.xml</CoverageExportPath>
</NCrunchProjectSettings>
```

In CI, NCrunch's coverage output feeds into SonarQube or the Datadog Test Visibility .NET agent for dashboard aggregation.

**Limitation**: NCrunch is a local-first IDE tool. For CI-only .NET TIA, prefer [dotnet-coverage](https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-coverage) with a custom impact mapping script, or Datadog Test Visibility for .NET.

---

## CI Integration Patterns

### Pattern 1: Static graph (fast, no service dependency)

```text
git diff → jest --findRelatedTests (or nx affected)
  ├── affected test set → run immediately
  └── unaffected → skip

Nightly: full suite → update coverage baseline
```

### Pattern 2: ML-assisted (highest skip ratio)

```text
PR: Launchable subset → run predicted high-value tests
    ├── results → uploaded to Launchable for model training
    └── gate: Launchable confidence ≥ 95%

Merge to main: full suite always runs
Nightly: full suite → coverage baseline check
```

### Pattern 3: Instrumented visibility + manual TIA

```text
All runs: Datadog Test Visibility instruments every run
  → flake detection, duration trends, failure attribution

PR: developer checks flake dashboard; quarantines known flaky tests
    ├── CI skips quarantined tests (with expiry enforcement)
    └── Full suite on nightly schedule

BuildPulse: receives JUnit XML from all runs → flake trending dashboard
```

---

## TIA and Merge Queues

Merge queues (GitHub, GitLab, Trunk, Aviator) serialize PR merges through a shared CI pipeline to prevent the "works on my branch" class of regression. TIA interacts with merge queues in two ways that require explicit handling.

### The amplification problem

A test that fails 5% of the time on a single isolated PR run will fail on roughly every other merge queue cycle if the queue processes 10–15 PRs per hour. The merge queue amplifies flake debt from a nuisance into a pipeline-stopping event.

### How TIA helps and where it breaks

TIA reduces queue cycle time by running only the tests affected by the batch. This reduces total flake exposure because fewer tests run. However:

- If the TIA dependency map is stale, unrelated tests may be skipped, allowing regressions to pass.
- Merge queues typically batch multiple PRs; the impact set is the union of all changes in the batch. Ensure TIA tools support batch-level impact computation, not just single-PR impact.
- GitHub merge queue requires branch protection rules that reference specific status checks. Map TIA-scoped check names consistently or use a wrapper check that always reports.

### Integration pattern for GitHub merge queue + Nx affected

```yaml
# .github/workflows/ci.yml
on:
  merge_group:
    types: [checks_requested]
  pull_request:

jobs:
  test-affected:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Compute affected projects
        id: affected
        run: |
          BASE=${{ github.event.merge_group.base_sha || github.event.pull_request.base.sha }}
          AFFECTED=$(npx nx show projects --affected --base=$BASE --head=HEAD --type=lib,app)
          echo "projects=$AFFECTED" >> $GITHUB_OUTPUT

      - name: Run tests for affected projects
        run: npx nx run-many --target=test --projects=${{ steps.affected.outputs.projects }} --parallel=4

  # Required check placeholder — always runs, always passes if test-affected passes
  required-tests:
    needs: test-affected
    runs-on: ubuntu-latest
    if: always()
    steps:
      - run: |
          if [ "${{ needs.test-affected.result }}" != "success" ]; then
            echo "Tests failed"; exit 1
          fi
```

### Quarantine in merge queue context

Most queue tools (Aviator, Trunk, Mergify) support automatic quarantine: a quarantined test runs and logs output but its failure does not eject the PR from the queue. This should be paired with:

- An MTTR-Flake SLO so quarantined tests do not accumulate indefinitely.
- Quarantine expiry (maximum 5 business days for standard; 2 for critical-path).
- Never quarantine a test that gates security, auth, or payment correctness.

---

## TIA Anti-Patterns

| Anti-Pattern | Risk | Mitigation |
|-------------|------|------------|
| TIA on mainline / merge commits | Regressions hidden from the baseline run | TIA for PRs only; full suite always runs on main |
| Stale dependency graph | Map diverges from code; wrong tests skipped | Invalidate on structural file changes; max 7-day TTL |
| No nightly full run | Coverage silently erodes | Enforce nightly via cron; fail pipeline on coverage delta |
| TIA without flake visibility | Flaky tests pollute the impact signal | Pair with BuildPulse or Datadog Test Visibility |
| Trusting 100% TIA skip | Even ML models miss cross-cutting changes | Enforce safety contract (see above) regardless of tool confidence |

---

## Related Resources

- [production-testing-and-shift-right.md](./production-testing-and-shift-right.md) -- MTTR-Flake SLO and shift-right context
- [quality-metrics-dashboard.md](./quality-metrics-dashboard.md) -- flake rate metrics and dashboards
- [operational-playbook.md](./operational-playbook.md) -- CI/CD pipeline quality gates
- [Launchable Docs](https://www.launchableinc.com/docs)
- [Datadog Test Visibility](https://docs.datadoghq.com/tests/)
- [BuildPulse](https://buildpulse.io/)
- [Jest --findRelatedTests](https://jestjs.io/docs/cli#--findrelatedtests-spaceseparatedlistofsourcefiles)
- [NCrunch](https://www.ncrunch.net/)

---
name: qa-testing-strategy
description: "Risk-based test strategy for software delivery. Use when defining coverage, setting CI gates, managing flaky tests, choosing test layers, or establishing release criteria."
compatibility: Portable core. Works on Claude Code and Codex.
version: "1.1"
last_validated: 2026-07-11
---

# QA Testing Strategy

Risk-based quality engineering guidance for modern software delivery. Use this skill to decide what to test, at which layer, with which gates, and how to keep the signal trustworthy.

Start with [references/operational-playbook.md](references/operational-playbook.md) for the navigation hub. Use current official sources from [data/sources.json](data/sources.json) when you need vendor or standards guidance.

## Scope

- Create or update a risk-based test strategy
- Choose a test shape (pyramid, trophy, honeycomb) based on architecture and defect origin
- Define merge gates, deploy gates, and release evidence — including merge queue interaction
- Choose the smallest effective layer: unit, component, contract, schema fuzzing, integration, E2E, property-based
- Make failures diagnosable with artifacts, correlation IDs, traces, and ownership
- Operationalize suite health: flake SLO, quarantine policy, execution budgets, dashboards

## Use Instead

| Need | Skill |
|------|-------|
| Implement or debug Playwright suites | [qa-testing-playwright](../qa-testing-playwright/SKILL.md) |
| Design API contract suites in depth | [qa-api-testing-contracts](../qa-api-testing-contracts/SKILL.md) |
| Debug failing tests or incidents | [qa-debugging](../qa-debugging/SKILL.md) |
| Add observability, telemetry, or tracing | [qa-observability](../qa-observability/SKILL.md) |
| Test LLM agents or evaluations | [qa-agent-testing](../qa-agent-testing/SKILL.md) |
| Mobile-specific strategy or automation | [qa-testing-mobile](../qa-testing-mobile/SKILL.md) |
| Security audit or threat-model depth | [software-security-appsec](../software-security-appsec/SKILL.md) |
| CI/CD pipeline design and infra | [ops-devops-platform](../ops-devops-platform/SKILL.md) |

## Quick Reference

| Layer | Goal | Typical Use |
|------|------|-------------|
| Unit | Prove logic and invariants fast | Pure functions, domain rules, validators |
| Component | Validate UI behavior in a real browser with narrow scope | UI components, state transitions, accessibility smoke |
| Contract | Prevent breaking changes across service boundaries | OpenAPI, AsyncAPI, JSON Schema, Protobuf |
| Schema fuzzing | Stress the API contract with generated valid and invalid inputs | Request/response edge cases, parser and validation drift |
| Property-based | Verify universal invariants across generated input spaces | Serialization round-trips, numeric contracts, state-machine invariants, AI-code edge cases |
| Integration | Validate real boundaries and dependencies | API + DB, queues, adapters, auth flows |
| E2E | Validate thin critical journeys | Sign-up, checkout, publish, payment, admin recovery |
| Performance | Enforce budgets and capacity | Load, stress, soak, latency regression |
| Visual | Catch intentional vs accidental UI changes | Stable pages, design-system components |
| Accessibility | Check for common WCAG 2.2 failures early | axe smoke + manual audit plan |
| Security | Catch common web/API vulnerabilities early | SAST, DAST smoke, auth and dependency checks |

## E2E Gate Topology (Default)

Use three distinct E2E scopes instead of one monolithic suite:

- Smoke: PR gate and fastest feedback on the highest-risk journeys.
- Targeted batch/spec: local triage and deflake work for one journey, subsystem, or dependency chain.
- Deploy-gate replay: dependency-chain or critical-journey replay used only when proving release readiness.

Rules:

- Do not use full local E2E as the first response to a single failing journey.
- Treat rerun-pass as unresolved flake debt.
- Promote scope only after the smaller scope is green.

## Default Workflow

1. Clarify scope and risk: critical journeys, failure modes, compliance constraints, and non-functional risks.
2. Define quality signals: SLOs, budgets, contract checks, accessibility target, and what blocks merge vs deploy.
3. Choose the smallest effective layer first: unit, component, contract, schema fuzzing, integration, then E2E.
4. Make failures diagnosable: logs, traces, screenshots, videos, build links, request IDs, trace IDs, and owners.
5. Operationalize the suite: explicit smoke vs targeted-batch vs deploy-gate scopes, quarantine with expiry, suite budgets, retries with evidence retention, and dashboards.

## Decision Rules

```text
Need to test: [Change or Risk]
    │
    ├─ Pure business rule or invariant?
    │   └─ Unit test
    │
    ├─ UI behavior or component state in isolation?
    │   └─ Component test in a real browser
    │
    ├─ API compatibility between teams/services?
    │   └─ Contract test
    │
    ├─ API parser/validation edge cases against the schema?
    │   └─ Schema-aware fuzzing + core integration smoke
    │
    ├─ Real dependency boundary or persistence behavior?
    │   └─ Integration test with real DB/queue/service doubles only at external edges
    │
    ├─ User-critical cross-page workflow?
    │   └─ Thin E2E test
    │
    ├─ Universal invariant or property that should hold for all valid inputs?
    │   └─ Property-based test (fast-check / Hypothesis / jqwik)
    │
    └─ Capacity, resilience, or reliability regression?
        └─ Performance, resilience, or synthetic monitoring tests
```

## Principles

- Prefer the smallest layer that can prove the behavior.
- Keep pre-merge gates fast: contracts, static checks, unit tests, selective component/integration smoke.
- Prefer targeted batch reruns locally; reserve full E2E for deploy gates or scheduled regression.
- Use full E2E only for critical journeys or risks that cannot be proven lower in the stack.
- Treat flaky tests as reliability defects, not harmless noise.
- Favor web-first assertions and stable locators over custom waits or brittle selectors.
- Treat accessibility automation as partial coverage. Pair it with manual checks and inclusive design review.
- Use telemetry as evidence. Production traces, incidents, and support signals should drive new tests.
- Use AI for brainstorming and triage only when evidence stays attached. Do not weaken assertions to “heal” tests.
- Treat AI-authored test oracle quality as a first-class risk, peer to flake debt. AI-generated tests routinely hit high line coverage while passing trivially (hardcoded returns, shallow assertions). Gate them on mutation score, not coverage: a test that cannot fail when the business logic is reverted is not a test.

## Core Targets

| Signal | Default Target |
|--------|----------------|
| PR gate | p50 <= 10 min, p95 <= 20 min |
| Mainline health | >= 99% green builds/day |
| Suite flake rate | <= 1% weekly |
| Quarantine policy | owner + ticket + expiry, never indefinite |
| AI-authored test oracle quality | mutation score gate on changed files (line coverage is not a gate); calibrate threshold to the suite, never accept AI tests on coverage alone |

## Resources

- [references/operational-playbook.md](references/operational-playbook.md): start here
- [references/component-testing-browser-mode.md](references/component-testing-browser-mode.md): real-browser component strategy with Vitest Browser Mode
- [references/playwright-webapp-testing.md](references/playwright-webapp-testing.md): current Playwright guidance
- [references/schema-aware-api-fuzzing.md](references/schema-aware-api-fuzzing.md): schema-driven API fuzzing with OpenAPI
- [references/contract-testing.md](references/contract-testing.md): Pact, Specmatic, and contract decisions
- [references/observability-driven-testing.md](references/observability-driven-testing.md): OpenTelemetry-first debugging and trace-based validation
- [references/quality-metrics-dashboard.md](references/quality-metrics-dashboard.md): metrics, dashboards, mutation coverage, and anti-patterns
- [references/production-testing-and-shift-right.md](references/production-testing-and-shift-right.md): synthetic monitoring, dark launches, feature flag rollouts, MTTR-flake SLO, production replay, observability-driven gates
- [references/test-impact-analysis.md](references/test-impact-analysis.md): TIA concept, Launchable, Datadog Test Visibility, BuildPulse flake trending, jest --findRelatedTests, NCrunch
- [references/shift-left-testing.md](references/shift-left-testing.md): shift-left practices, test doubles, and coverage targets that move quality checks earlier
- [references/test-automation-patterns.md](references/test-automation-patterns.md): Page Object Model, data factories, fixtures, test doubles, AAA, and isolation patterns
- [references/test-environment-management.md](references/test-environment-management.md): environment-as-code, seeding, service virtualization, isolation, and shared-vs-dedicated tradeoffs
- [references/synthetic-test-data.md](references/synthetic-test-data.md): ephemeral, privacy-safe test data to avoid real customer data in CI and staging
- [references/chaos-resilience-testing.md](references/chaos-resilience-testing.md): chaos experiments, fault injection, CI/CD integration, and DORA/SOC 2 resilience evidence
- [references/compliance-testing.md](references/compliance-testing.md): compliance-as-code, audit-evidence automation, access control, data residency, and encryption validation
- [references/feature-matrix-vs-test-matrix-gate.md](references/feature-matrix-vs-test-matrix-gate.md): pre-release gate mapping implemented features to auditable test evidence
- [references/property-based-testing.md](references/property-based-testing.md): property-based testing with fast-check, Hypothesis, and jqwik — universal invariants, edge-case discovery, and AI-code blind-spot detection
- [references/comprehensive-testing-guide.md](references/comprehensive-testing-guide.md): retired redirect map pointing each test layer to its dedicated sibling skill

## Templates

- [assets/test-strategy-template.md](assets/test-strategy-template.md): strategy one-pager
- [assets/automation-pipeline-template.md](assets/automation-pipeline-template.md): CI/CD pipeline blueprint
- [assets/component/template-vitest-browser.md](assets/component/template-vitest-browser.md): browser-mode component tests
- [assets/e2e/template-playwright.md](assets/e2e/template-playwright.md): Playwright E2E with traces and accessibility smoke
- [assets/integration/template-api-integration.md](assets/integration/template-api-integration.md): API + DB integration tests
- [assets/performance/template-k6-load-testing.md](assets/performance/template-k6-load-testing.md): performance budgets and scenarios
- [assets/runbooks/template-flaky-test-triage-deflake-runbook.md](assets/runbooks/template-flaky-test-triage-deflake-runbook.md): deflake runbook
- [assets/template-test-case-design.md](assets/template-test-case-design.md): Given/When/Then and oracles

## ASCII Flow

```text
Test strategy request
  -> Clarify risks, critical journeys, constraints, and release criteria
  -> Pick smallest proving layer: unit, component, contract, integration, E2E
  -> Define merge gates, deploy gates, evidence artifacts, and owners
  -> Add diagnostics: logs, traces, screenshots, request IDs, and dashboards
  -> Set suite health policy: flake SLO, quarantine expiry, runtime budgets
  -> Review production signals and incidents to evolve coverage
```

## Navigation

- `## Default Workflow`, `## Decision Rules`, and `## Principles` for the baseline strategy sequence
- `## Resources` and `## Templates` for deeper materials
- `## Related Skills` for tool-specific execution handoffs
- [references/reliability-theory-applied.md](references/reliability-theory-applied.md) — Reliability primitives (MTBF/MTTR, availability, FMEA, error budgets) applied to QA testing strategy.

## Related Skills

| Skill | Purpose |
|-------|---------|
| [qa-refactoring](../qa-refactoring/SKILL.md) | Safe refactoring with behavior preservation |
| [software-code-review](../software-code-review/SKILL.md) | Code review process and checklists |
| [software-architecture-design](../software-architecture-design/SKILL.md) | System design and architecture decisions |

## Fact-Checking

- Known bugs, regressions, framework/compiler/runtime footguns, and version-specific crash or workaround guidance must be verified against current primary web sources before being treated as current fact.
- Use web search or web fetch to verify current external facts, versions, pricing, deadlines, regulations, or platform behavior before final answers.
- Prefer primary sources; report source links and dates for volatile information.
- If web access is unavailable, state the limitation and mark guidance as unverified.

## Learnings Loop

Before applying this skill on a non-trivial task, read `learnings.consolidated.md` in this directory (and `learnings.md` if present).

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to `learnings.md` via `agents-skills-feedback-loop/scripts/append_learning.py`. Do not modify `SKILL.md` itself.