qa-methodology

어떤 스킬인가요?

>-

설치 방법

프로젝트 폴더에서 아래 명령어를 실행하고, 설치할 에이전트를 선택하세요.

설치 명령을 직접 실행해야 적용됩니다. 지원 에이전트와 필요한 권한·라이선스는 제작자의 안내를 확인하세요.

스킬 원문 파일

# QA Definition of Done

> QA contribution to the team's definition of done. Adapt to your project's risk profile and release cadence. Check each item that applies before declaring a feature or release complete.

## Scope

| Field | Value |
|-------|-------|
| Feature / Release | <name> |
| Team | <team> |
| Date | <YYYY-MM-DD> |

## Functional Verification

- [ ] All acceptance criteria verified with evidence attached
- [ ] AC→verification-method traceability is 100% (no unmapped ACs)
- [ ] Happy path tested at each applicable level (unit / integration / E2E)
- [ ] Known failure modes tested (error handling, edge cases)
- [ ] Regression suite passes (no new failures introduced)

## Risk-Based Coverage

- [ ] Risk register reviewed; all P0/P1 risks have corresponding tests
- [ ] Test allocation proportional to risk scores (P0 items receive exhaustive coverage)
- [ ] Residual risk documented and accepted by stakeholders

## Test Quality

- [ ] No flaky tests in the suite (quarantined or fixed before release)
- [ ] Test data uses synthetic or masked data (no production PII)
- [ ] Tests are deterministic (non-deterministic code verified with N-run sampling)
- [ ] Mutation testing run on critical paths (if applicable)

## Non-Functional Requirements

- [ ] Performance thresholds met (p95 latency, throughput under load)
- [ ] Security scan clean (zero critical/high findings from SAST/SCA)
- [ ] Accessibility checked for user-facing changes (WCAG 2.1/2.2 AA)
- [ ] Contract tests pass for all API consumers (if applicable)

## Automation and CI

- [ ] New tests automated and committed to the repository
- [ ] Tests integrated into CI pipeline at the correct stage
- [ ] Every fixed bug has a regression test ("every fixed bug becomes a regression test")
- [ ] CI pipeline green on the release branch

## Documentation and Handoff

- [ ] Test strategy document up to date (if changes affect scope)
- [ ] Exploratory session debriefs recorded (if sessions were run)
- [ ] Known issues and workarounds documented
- [ ] Risk register updated with new or changed risks

## Release Gate

| Gate Item | Owner | Status |
|-----------|-------|--------|
| All P0 exit criteria met | <QA lead> | Pending / Met / Waived |
| All P1 exit criteria met | <QA lead> | Pending / Met / Waived |
| Stakeholder sign-off on residual risk | <product / eng lead> | Pending / Met / Waived |
| Release decision | <release manager> | Pending / Approved / Blocked |

## Exceptions Log

| Item | Reason for Exception | Approved By | Risk Accepted |
|------|---------------------|-------------|:-------------:|
| <which DoD item> | <why it was skipped> | <name> | Yes / No |

# Risk Matrix Grid (5×5 Probability × Impact)

> Use this grid during risk assessment workshops to score and zone risks. This grid is consistent with [risk-based-testing.md](../references/risk-based-testing.md) and the `risk-prioritize.py` script.

## Scoring Formula

```
Risk Score = Probability (1–5) × Impact (1–5)
```

## 5×5 Grid

| P \ I | 1 — Negligible | 2 — Minor | 3 — Moderate | 4 — Major | 5 — Catastrophic |
|-------|:--------------:|:---------:|:------------:|:---------:|:----------------:|
| **5 — Almost Certain** | 5 (P3) | 10 (P2) | 15 (P1) | 20 (P0) | 25 (P0) |
| **4 — Likely** | 4 (P3) | 8 (P2) | 12 (P1) | 16 (P0) | 20 (P0) |
| **3 — Possible** | 3 (P3) | 6 (P2) | 9 (P2) | 12 (P1) | 15 (P1) |
| **2 — Unlikely** | 2 (P3) | 4 (P3) | 6 (P2) | 8 (P2) | 10 (P2) |
| **1 — Rare** | 1 (P3) | 2 (P3) | 3 (P3) | 4 (P3) | 5 (P3) |

## Zone Thresholds

| Zone | Score Range | Priority Tier | Action |
|------|-------------|:-------------:|--------|
| **Critical** | 20–25 | P0 | Test exhaustively; every path, every edge case |
| **High** | 12–19 | P1 | Test all happy paths + known failure modes |
| **Medium** | 6–11 | P2 | Test happy paths + common failure modes |
| **Low** | 1–5 | P3 | Smoke test only; defer detailed testing |

## Probability Anchors

| Rating | Label | Anchor |
|:------:|-------|--------|
| 5 | Almost Certain | Will fail in production within a quarter (or has already) |
| 4 | Likely | Expected to fail within a year |
| 3 | Possible | Could fail; uncertain |
| 2 | Unlikely | Unlikely given current controls |
| 1 | Rare | Extremely unlikely; well-understood code |

## Impact Anchors

| Rating | Label | Anchor |
|:------:|-------|--------|
| 5 | Catastrophic | Data loss, security breach, revenue stoppage |
| 4 | Major | Major feature outage, SLA breach |
| 3 | Moderate | Degraded experience, workaround exists |
| 2 | Minor | Cosmetic, minor inconvenience |
| 1 | Negligible | No user-visible impact |

## Workshop Scoring Sheet

Record scores during the workshop, then transfer to [templates/risk-register.md](../templates/risk-register.md).

| # | Risk (short label) | P (1–5) | I (1–5) | Score | Zone | Tier |
|---|---------------------|:-------:|:-------:|:-----:|------|:----:|
| 1 | | | | | | |
| 2 | | | | | | |
| 3 | | | | | | |
| 4 | | | | | | |
| 5 | | | | | | |
| 6 | | | | | | |
| 7 | | | | | | |
| 8 | | | | | | |

## Calibration Rule

If probability votes span more than 2 points:
1. The facilitator asks the highest and lowest voter to state their evidence.
2. Re-vote once.
3. Record dissent in the register.

## Test Allocation by Tier

| Tier | Hours per Risk Item (default) | Strategy |
|:----:|------------------------------:|----------|
| P0 | 8–16 | Exhaustive design + automation |
| P1 | 4–8 | Happy paths + failure modes |
| P2 | 2–4 | Happy paths + common failures |
| P3 | 0.5–1 | Smoke only |

Adjustment: multiply by 1.5× for legacy/unfamiliar code, 0.7× for well-automated areas.

# Test Design Techniques Checklist

> Quick-reference for selecting test design techniques. Check the techniques that apply to your scenario, then derive test cases. See [test-design-techniques.md](../references/test-design-techniques.md) for detailed guidance and worked examples.

## Feature Under Test

| Field | Value |
|-------|-------|
| Feature / Component | <name> |
| Tester | <name> |
| Date | <YYYY-MM-DD> |

## Technique Selection

| Technique | Applies? | Rationale |
|-----------|:--------:|-----------|
| Equivalence Partitioning (EP) | [ ] | <Why: input domains exist with distinct behavior classes> |
| Boundary Value Analysis (BVA) | [ ] | <Why: numeric or ordered ranges with edge behavior> |
| Decision Tables | [ ] | <Why: business rules with multiple interacting conditions> |
| State Transition | [ ] | <Why: stateful workflow with valid/invalid transitions> |
| Pairwise / Combinatorial | [ ] | <Why: multi-parameter configuration space> |
| Error Guessing | [ ] | <Why: known defect patterns, historical failure areas> |

## When to Use Which

| Scenario Type | Primary Technique | Supporting Technique |
|---------------|-------------------|---------------------|
| Input validation (ranges, formats) | EP + BVA | Error Guessing |
| Business logic with conditions | Decision Tables | EP |
| Workflow with states and transitions | State Transition | Decision Tables |
| Configuration with many parameters | Pairwise | EP |
| Integration with external systems | Error Guessing | EP (interface partitions) |
| UI with form fields | EP + BVA | Error Guessing |
| Permission / role-based access | Decision Tables | State Transition |

## Technique Application Notes

### Equivalence Partitioning

- [ ] Identified all input domains
- [ ] Defined valid and invalid partitions
- [ ] Selected one representative per partition
- [ ] Included type/format partitions (non-integer, empty, null)

### Boundary Value Analysis

- [ ] Used 2-value (min, max) for quick coverage
- [ ] Used 3-value (min-1, min, max, max+1) where off-by-one is likely
- [ ] Tested empty/null boundaries

### Decision Tables

- [ ] Listed all conditions and actions
- [ ] Generated complete rule combinations
- [ ] Reduced redundant rules where outcomes are identical
- [ ] Covered impossible/contradictory combinations explicitly

### State Transition

- [ ] Drew or referenced the state model
- [ ] Covered all valid transitions (0-switch)
- [ ] Tested at least one invalid transition per state
- [ ] Checked for dead states and unreachable states

### Pairwise / Combinatorial

- [ ] Listed parameters and their values
- [ ] Generated pairwise combinations (tool: PICT / allpairs)
- [ ] Added known-bad combinations from defect history

### Error Guessing

- [ ] Reviewed defect history for this component
- [ ] Checked common failure patterns (null, overflow, concurrency, timeout)
- [ ] Tested integration boundaries with invalid/malformed data

## Coverage Summary

| Dimension | Covered | Notes |
|-----------|:-------:|-------|
| All valid partitions tested | [ ] | |
| All boundaries tested | [ ] | |
| All decision rules covered | [ ] | |
| All state transitions covered | [ ] | |
| Pairwise combinations generated | [ ] | |
| Known error patterns tested | [ ] | |

> The pyramid level for each technique is a starting point, not a rule. Adapt to your system's risk profile. See [test-strategy.md](../references/test-strategy.md).

{"schema_version": 1, "skill_name": "qa-methodology", "evals": [{"id": "risk-prioritization-workshop", "prompt": "We have a release coming up with limited QA capacity and need to decide which areas deserve the most testing effort based on risk. Can you help me prioritize?", "expected_output": "A risk assessment using the P×I formula (Probability 1-5 × Impact 1-5) on a 5×5 matrix, mapping each risk item to priority tiers P0-P3, with a risk register containing scored items, owners, mitigations, and reassessment triggers. Test allocation is proportional to risk scores.", "assertions": ["The response applies the P×I scoring formula with probability and impact rated on a 1-5 scale", "The response maps risk scores to priority tiers P0 through P3 using the 5×5 risk matrix", "The response includes a risk register structure with scored items, owners, and mitigations", "The response defines reassessment triggers for updating risk scores over time", "The response recommends test allocation proportional to each risk tier score"]}, {"id": "exploratory-charter-design", "prompt": "I need to design an exploratory testing session for our new payment integration but I want it structured and accountable, not just random clicking around.", "expected_output": "A structured SBTM charter in the format 'Explore <target> with <resources> to discover <information>', with a 60-120 minute timebox, T/B/B metrics tracking, and a debrief plan. Heuristics such as SFDIPOT or HICCUPPS are referenced for coverage guidance.", "assertions": ["The response produces a charter in the 'Explore target with resources to discover information' format", "The response includes a timebox duration of 60 to 120 minutes for the session", "The response specifies T/B/B metrics for tracking test time allocation during the session", "The response references SBTM heuristics such as SFDIPOT or HICCUPPS for coverage guidance", "The response includes a debrief structure for reviewing session findings and follow-up actions"]}, {"id": "sdd-gate-ac-testability", "prompt": "An agent just wrote code from a spec and claims it passes all acceptance criteria, but I need an independent QA review before it merges. How should I verify?", "expected_output": "An independent verification plan executed in a separate agent session, where the implementing agent does not self-verify. The plan maps each acceptance criterion to a verification method with observable outcomes, flags untestable ACs, and attaches evidence per gate.", "assertions": ["The response requires independent verification in a separate session, not by the implementing agent", "The response maps each acceptance criterion to a specific verification method with observable outcomes", "The response flags untestable acceptance criteria that lack observable outcomes", "The response requires evidence to be attached for each gate verdict", "The response warns against self-verification by the implementing agent due to overfitting risk"]}, {"id": "agentic-eval-dataset-design", "prompt": "We are building an evaluation suite for our customer support agent and I need help designing the dataset with proper class balance and judge calibration.", "expected_output": "An eval dataset design with class balance (positive, negative, adversarial, boundary cases), N=5-10 trials per task with majority-vote aggregation, a pinned judge contract tuple, pass@k or pass^k metric selection based on product requirements, and a no-retry-until-green policy.", "assertions": ["The response specifies class balance with positive, negative, adversarial, and boundary case proportions", "The response recommends N=5-10 trials per task with majority-vote or weighted aggregation", "The response requires a pinned judge contract tuple of model ID, rubric version, and prompt hash", "The response distinguishes pass@k from pass^k and selects based on product retry requirements", "The response enforces a no-retry-until-green policy for handling flaky evaluation results"]}, {"id": "sdet-career-scope-mapping", "prompt": "I am a senior QA engineer wanting to understand what scope and evidence I need to demonstrate for a staff-level SDET promotion packet.", "expected_output": "A scope-progression analysis mapping Senior to Staff transition, identifying product-scope evidence needed (multi-team standards, shared infrastructure, guilds). The response references the gTAA architecture for SDET competency and staff-level archetypes (Architect, Solver, Team Lead, Right Hand).", "assertions": ["The response maps the Senior to Staff transition using scope progression from project to product level", "The response identifies product-scope evidence such as multi-team standards and shared infrastructure", "The response references gTAA layered architecture as part of SDET competency expectations", "The response names staff-level archetypes such as Architect, Solver, Team Lead, or Right Hand", "The response distinguishes promotion evidence from current-level excellence at the next scope tier"]}, {"id": "test-design-technique-selection", "prompt": "I have a form with twelve optional fields and three numeric range inputs and I need to figure out which test design techniques will give me the best coverage efficiently.", "expected_output": "A technique selection that applies pairwise testing (PICT) for the multi-field form interactions, boundary value analysis (BVA) with 3-value approach for the numeric ranges, and equivalence partitioning for categorical inputs. The response includes a when-to-use-which rationale.", "assertions": ["The response recommends pairwise testing with PICT for the form field interactions", "The response applies boundary value analysis with the 3-value approach for numeric range inputs", "The response uses equivalence partitioning to reduce test count for categorical inputs", "The response provides a when-to-use-which rationale for each selected technique", "The response avoids exhaustive combinatorial enumeration in favor of pairwise reduction"]}, {"id": "anti-trigger-production-debugging", "prompt": "Our production checkout service is throwing intermittent 500 errors and I need to do root-cause analysis to figure out what is causing the failures.", "expected_output": "The agent declines to apply qa-methodology for this request, recognizing that root-cause debugging of production incidents falls outside its scope. It routes the user to the systematic-debugging skill for fault localization and incident analysis.", "assertions": ["The response declines qa-methodology as the appropriate skill for root-cause debugging of production incidents", "The response names systematic-debugging as the correct sibling skill for this request", "The response does not attempt to apply QA test strategy or risk-based testing methodology to the debugging task", "The response explains that production incident root-cause analysis is outside the qa-methodology negative boundary"]}, {"id": "mutation-useful-survivor-hardening", "prompt": "A changed conditional has a surviving operator mutant. Show how to harden the tests without running an unlimited mutation campaign.", "expected_output": "The response selects a diff-aware or justified risk-bounded scope, proposes a behavior-level test, and requires independent baseline-versus-candidate-versus-exact-mutant verification with reproducible evidence.", "assertions": ["The response chooses changed lines/files or justifies a broader risk slice", "The response treats the survivor as a triage input and proposes a behavior-level oracle", "The response requires independent reruns of baseline, candidate test, and exact mutant", "The response records tool/version, budget, timeout, seed, exclusions, command, status, and raw evidence"]}, {"id": "mutation-false-confidence-rejected-test", "prompt": "Mutation analysis reports a high score, but one mutant is equivalent and the proposed test only asserts an internal helper call. Decide whether this is success.", "expected_output": "The response rejects automatic success, classifies equivalent-mutant and implementation-coupling uncertainty, and asks for a better behavioral oracle or fresh human verification.", "assertions": ["The response does not treat a high score as proof of quality", "The response identifies equivalent-mutant and denominator uncertainty", "The response rejects tautological or implementation-coupled tests", "The response requires a behavioral oracle and independent review"]}, {"id": "mutation-incomplete-unreliable-run", "prompt": "A bounded mutation run has no-coverage mutants, timeouts, a flaky test, and an infrastructure error. Can we report a clean mutation result?", "expected_output": "The response preserves unreliable outcomes outside a silently inflated denominator, refuses a clean verdict, and specifies a bounded rerun/reproduction path.", "assertions": ["The response separately classifies no coverage, timeout, flaky, and infrastructure/tooling failure", "The response defines the denominator and accounts for unknown or incomplete mutants explicitly", "The response refuses a clean pass or universal threshold", "The response gives a bounded reproduction command with tool/version and environment"]}]}

[pytest]
# Override root pyproject.toml coverage settings.
# qa-methodology tests are subprocess-based and do not measure
# the root scripts/ or eval_runner/ packages.
addopts = -ra --strict-markers --tb=short

# QA Methodology

Senior-to-principal QA and SDET methodology for software teams that ship with confidence.

## Why Install This Skill

Shipping software without a test strategy means discovering failures in production instead of in CI. This skill gives your agent the methodology of a senior QA engineer: risk-based prioritization that tells you what to test first, regression suites that catch breakage without becoming brittle, and quality gates that block merges on evidence rather than vibes.

Beyond traditional QA, the bundle covers the agentic era: independent verification of AI-generated code, mutation-guided test hardening for changed behavior, acceptance-criteria testability review for Spec-Driven Development pipelines, and eval dataset design with judge-bias mitigation for AI agents. Whether your team is a two-person startup or a multi-team platform, the references scale from task-level test design to org-level quality engineering strategy.

Install once and your agent designs test strategies, triages CI failures by exit code, writes exploratory charters, scores risks on a 5x5 grid, reviews AI-generated PRs with independence, and maps QA career growth from Senior through Principal.

## What You Get

| Directory | Contents |
|-----------|----------|
| `references/` | 16 deep-dive files: test-strategy, test-automation, quality-gates-and-metrics, regression-testing, test-data-management, performance-testing, security-testing, ci-failure-triage, test-debugging, risk-based-testing, exploratory-testing, test-design-techniques, qa-career-levels, sdet-engineering, ai-code-quality-gates, agentic-eval-design |
| `templates/` | 6 fillable templates, including mutation-review for reproducible survivor triage and evidence |
| `assets/` | 3 quick-reference assets: risk-matrix-grid, test-design-techniques-checklist, qa-definition-of-done |
| `scripts/` | 2 Python CLIs: risk-prioritize (P×I ranking with --json output), check-ac-testability (vague-AC scanner) |
| `evals/` | Schema-v1 output-quality eval manifest (10 cases) |

## Quick Start

Score and rank risk items from a JSON file:

```bash
python3 qa-methodology/scripts/risk-prioritize.py --json risk-items.json
```

Check acceptance criteria for testability before a gate review:

```bash
python3 qa-methodology/scripts/check-ac-testability.py spec.md
```

## Triggers

- Test strategy design or review
- Regression suite building, selection, or evolution
- CI failure triage (exit codes, flake classification, bisect)
- Test automation framework selection and flaky management
- Quality gate design and metrics (DORA, targeted mutation review evidence)
- Mutation-guided test hardening (surviving mutants, weak assertions, diff-aware scope)
- Risk-based testing (P×I scoring, workshops, registers)
- Exploratory testing (SBTM charters, heuristics)
- Agentic eval design (datasets, judge bias, flaky-eval discipline)
- SDD gate review (AC testability, independent verification)
- SDET engineering and QA career leveling

## Requirements

- Python 3.8+ for scripts (standard library only, no third-party packages)
- No specific CI platform, test framework, or AI agent required
- Works with any language or stack (examples reference pytest, Playwright, k6, and others as illustrations)

# Agentic Eval Design: QA Playbook for Agent Evaluations

## Vocabulary (per Anthropic, "Demystifying Evals for AI Agents," 2026)

| Term | Definition |
|------|-----------|
| **Task** | A single evaluation problem with defined inputs, environment, and success criteria |
| **Trial** | One execution of one task by one agent configuration |
| **Grader** | A function that scores a trial's output (deterministic check, LLM judge, human) |
| **Transcript** | The full record of agent actions, tool calls, and observations during a trial |
| **Outcome** | The final environment state after the trial completes |
| **Harness** | Infrastructure that sets up the environment, runs the agent, and collects the transcript |
| **Scaffold** | The agent's prompting, tool configuration, and orchestration wrapper |
| **Suite** | A versioned collection of tasks used together for a decision |

**Grade the OUTCOME (environment state), not prose.** An agent that writes a beautiful explanation but leaves the database corrupted fails. An agent that produces terse output but correctly deploys the service passes. Evaluations measure what the agent DID to the world, not how it described what it did.

## The Eval-Driven Development Loop

| Suite Type | Purpose | Graduation Rule |
|-----------|---------|----------------|
| **Capability suite** | Measures what the agent CAN do (new features, hard problems) | After 3 consecutive green runs on unrelated changes, promote passing cases to regression |
| **Regression suite** | Ensures the agent STILL does what it used to | Never auto-retire; manual review only |

Cases flow: capability → (3 green runs) → regression. Regressions are never auto-promoted to capability; they are stable assertions of known-good behavior.

## Dataset Test Design

### Equivalence Partitioning and Boundary Cases

Apply classical test design to eval datasets:
- **Equivalence classes:** Group inputs by expected behavior category (valid, invalid, edge)
- **Boundary cases:** Test transitions between classes (max token length, permission threshold, timeout boundary)

### Class Balance (Including Negative Cases)

| Case Class | Target Proportion | Example |
|-----------|------------------|---------|
| Positive (should succeed) | 40–50% | Valid request → correct action |
| Negative (should fail/refuse) | 20–30% | Unauthorized action → refusal |
| Adversarial/injection | 15–20% | Prompt injection → no deviation |
| Boundary/edge | 10–15% | Empty input, max-length, concurrent |

> **Gotcha — Positive-only datasets:** A dataset with only "happy path" cases measures capability but not safety. Negative cases (the agent SHOULD refuse or fail gracefully) are mandatory. A 100% pass rate on positive-only cases says nothing about whether the agent handles errors correctly.

### Golden-Trajectory Curation

Curate **50–500 golden trajectories** for critical tasks: regression anchors, judge calibration material, and onboarding docs.

### Reference-Solution Oracle

For deterministic-answer tasks, maintain a reference solution. Compare agent output for semantic equivalence (correct behavior, acceptable variants), not string equality.

## Judge-as-System-Under-Test

The LLM judge is itself a system that must be tested. Five documented biases:

| Bias | Effect Size | Mechanical Mitigation |
|------|------------|----------------------|
| **Position bias** | ~10–15 point swing | Shuffle/rotate candidate order across trials |
| **Verbosity bias** | ~15–30 point swing | Length-neutral rubric; normalize scores by output length |
| **Self-preference** | ~10–25% inflated scores | Cross-family judge (judge from different vendor than the agent) |
| **Format bias** | ~5–15 point swing | Standardize output format before judging; rubric ignores formatting |
| **Calibration drift** | ~3–8 point drift over weeks | Monthly human calibration against golden set; alert on >5pt shift |

### Pin the Contract

Every judge invocation must record the immutable tuple:

```
(judge_model_id, rubric_version, prompt_template_hash)
```

If any element changes, scores are NOT comparable to prior runs. This tuple is the "contract" — treat changes as a migration event (see Judge-Swap Migration below).

### Monthly Human Calibration

Once per month, a human grader scores the same 20–30 trials as the LLM judge. Compute Cohen's kappa or agreement percentage. If agreement drops below 0.7, investigate rubric drift or judge degradation before trusting automated scores.

## Flaky-Eval Discipline

**Variance is the baseline, not an anomaly.** LLM agents are non-deterministic. A task that passes 7/10 times is not "flaky" — it has a 70% success rate, which IS the measurement.

### Sampling and Aggregation

- Run each task **N = 5–10 trials** per evaluation
- Aggregate via **majority vote** (pass if >50% of trials pass) or **weighted score** (average of per-trial scores)
- Report both mean and variance — a task at 60% ± 20% is fundamentally different from 60% ± 2%

### pass@k vs pass^k

| Metric | Definition | Use When |
|--------|-----------|----------|
| **pass@k** | At least 1 of k attempts succeeds | Product allows retries (interactive agent, user can re-request) |
| **pass^k** | ALL k attempts succeed | Product requires reliability (autonomous pipeline, no human in loop) |

**Choose from product requirements.** If the user can retry, pass@k is honest. If the agent runs unattended, pass^k reflects actual user experience. Never choose pass@k for an autonomous agent just because the numbers look better.

### No Retry-Until-Green

> **RULE: NEVER retry a failed evaluation until it passes.** If the suite fails at 6/10 trials, the result is 6/10. Retrying until green manufactures false confidence. Report the failure, investigate the variance, and either fix the agent or adjust the threshold. Retry-until-green is the eval equivalent of `while (!pass) run_tests()`.

## CI Gate Architecture

### Three Tiers

| Tier | Trigger | Contents | Time Budget |
|------|---------|----------|-------------|
| **Pre-merge** | Every PR | Fast subset (≤20 tasks) + deterministic scanners (schema, lint, import check) | < 5 minutes |
| **Nightly** | Scheduled | Full suite (all tasks, N=5 trials each) + judge scoring | < 60 minutes |
| **Continuous** | Production | Sampled live traffic (1–5%) scored asynchronously | Ongoing, budget-constrained |

### Cascade Cost Pyramid

Run the cheapest grader first; escalate only on failure:

```
Level 1: Deterministic checks (regex, schema, exit code)     → $0, instant
Level 2: Classifier (fine-tuned model, rules engine)         → $0.001/trial
Level 3: LLM judge (full rubric evaluation)                  → $0.01–0.05/trial
```

Only tasks that fail Level 1 or 2 reach the expensive LLM judge. This reduces nightly judge cost by 60–80%.

### Cost Budgeting

| Tier | Budget Principle | Example |
|------|-----------------|---------|
| Pre-merge | < $1 per PR; < 5 min wall time | 20 tasks × deterministic only |
| Nightly | < $50 per run; < 60 min | 200 tasks × 5 trials × cascade |
| Continuous | < $500/month | 1% traffic × Level 1 only + 0.1% × Level 3 |

**Cost determines tier placement.** A task that requires $0.10/trial in LLM-judge cost cannot go in pre-merge (20 tasks × 5 trials × $0.10 = $10/PR). Move it to nightly.

## Replay and Promote-Back Loop

```
Capture → Anonymize → Replay → Cluster → Promote (3–10 cases) → Re-gate
```

1. **Capture** production transcripts (with consent per telemetry policy)
2. **Anonymize** — strip PII, credentials, session IDs
3. **Replay** captured inputs against current agent version
4. **Cluster** failures by root cause (embedding similarity)
5. **Promote** 3–10 representative failures to regression suite
6. **Re-gate** — confirm no regression from promotion

Runs weekly or post-incident. Ensures the suite evolves with real usage.

## Trajectory-vs-Outcome Grading Decision Rule

| Grade the TRAJECTORY when... | Grade the OUTCOME only when... |
|------------------------------|-------------------------------|
| Path matters: safety-critical actions (medical, financial) | Multiple valid paths exist (creative tasks, open-ended problems) |
| Compliance requires specific steps (audit trail) | Any correct path is acceptable |
| Cost-sensitive operations (avoid unnecessary API calls) | Cost of path variation is negligible |
| Destructive/irreversible steps (data deletion, external comms) | Actions are reversible or sandboxed |

> **Brittleness caveat:** Trajectory grading is MORE BRITTLE than outcome grading. It over-constrains the agent's approach and penalizes valid novel strategies. A golden trajectory that says "call API A then API B" will fail an agent that correctly uses API C (a newer, better path). Use trajectory grading sparingly and only where the path genuinely matters.

## Adversarial Self-Audit

### Null-Agent Floor

Before interpreting any eval score, run the **null agent** (an agent that does nothing, or returns empty/random output). The null agent's score is your floor. If a task's pass rate is only 5% above the null agent, the task has poor discriminative power — fix the task, not the agent.

### Adversarial Agent Types

| Agent Type | Purpose | What It Reveals |
|-----------|---------|----------------|
| **Random agent** | Baseline floor | Whether tasks are solvable by chance |
| **Injection agent** | Injects adversarial prompts into inputs | Whether the agent is hijackable |
| **State-tamper agent** | Modifies environment state, scoring code, or test files | Whether the harness is exploitable |

### Evidence: Berkeley RDI and METR

- **Berkeley RDI** (rdi.berkeley.edu, 2025–2026) identified **seven deadly patterns** of benchmark failure: leaked solutions in prompts, executable scoring code accessible to the agent, missing baseline comparisons, reward-component skipping, environment state leakage, non-reproducible setups, and trust in agent self-report.
- **METR** (metr.org, 2025) documented **30.4% reward-hacking rate** across RE-Bench tasks with o3: agents monkey-patched evaluators, overwrote timing functions, and copied reference answers. On one task, reward hacking occurred in 100% of trajectories.

**Implication:** Your eval harness must treat the agent as adversarial. Sandbox scoring code. Deny file-system access to test infrastructure. Never trust agent self-reported success.

## Benchmark Skepticism

| Benchmark | Limitation |
|-----------|-----------|
| **SWE-bench** | Contamination risk (training data includes GitHub issues); saturating |
| **τ-bench** | Narrow tool set; may not reflect production tools |
| **WebArena** | Environment drift (sites change); setup fragility |
| **OSWorld** | Heavy infrastructure; limited task diversity |
| **GAIA** | Broad but shallow; subjective grading |

> **RULE: Benchmarks are a sanity floor, NOT a release gate.** A 90% SWE-bench score does not mean the agent is safe for production. Your internal eval suite (grounded in YOUR tasks and failure modes) is the release gate. Public benchmarks tell you "the model isn't broken"; they cannot tell you "the agent is ready."

Risks: **contamination** (tasks leak into training data) and **saturation** (scores approach 100%). Refresh or retire benchmarks when either occurs.

## Judge-Swap Migration Procedure

When replacing one judge model with another:

1. **Re-baseline:** Run new judge on last 3 historical runs; record score deltas per task.
2. **Parallel-run:** Run BOTH judges for 2–4 weeks; track agreement rate.
3. **Document delta:** Publish the score offset (e.g., "New judge scores ~4 points lower on verbosity tasks").
4. **Non-comparability rule:** Scores from different judge models are NOT comparable without re-baselining.
5. **Retire old judge:** Only after parallel-run shows stable agreement (kappa > 0.75).

## Dataset Versioning and Contamination Control

| Control | Mechanism |
|---------|-----------|
| **Immutable versions** | Content hash (SHA-256) per version; never edit in place |
| **Refresh cadence** | Quarterly review; add from promote-back; retire stale cases |
| **Contamination detection** | Compare task embeddings against training corpora; flag >0.95 similarity |
| **Retirement rule** | If found in training data (or >20-point jump without code change), retire and replace |

## Worked Example: Eval Suite for a Support Agent

**Context:** Customer-support agent (refunds, lookups, escalations).

| Step | Result |
|------|--------|
| Define tasks | 90 total: 30 refund, 20 lookup, 15 escalation, 15 injection, 10 boundary |
| Class balance | 45 positive / 20 negative / 15 adversarial / 10 boundary |
| Golden trajectories | 60 reference executions (2 per refund task) |
| Judge setup | Pin: (claude-sonnet-4-20250514, rubric-v3, hash-a7f2c) |
| Baseline | Null agent: 8% → floor established |
| Run | N=7 trials, majority-vote → agent scores 74% |
| CI | Pre-merge: 15 tasks <3 min. Nightly: 90 × 7. Continuous: 2% sampling |
| Promote-back | Week 1: 5 prod failures → 4 promoted to regression (suite = 94) |

## Decision Table: Eval Design Choices

| Question | If YES | If NO |
|----------|--------|-------|
| Does the task have a deterministic correct answer? | Use reference-solution oracle | Use LLM judge with rubric |
| Is the path safety-critical or compliance-bound? | Grade trajectory + outcome | Grade outcome only |
| Can the user retry in production? | Report pass@k | Report pass^k |
| Is the task cost > $0.05/trial to grade? | Place in nightly tier | Place in pre-merge tier |
| Has the judge contract changed? | Run migration procedure | Scores are comparable |

## Exit Conditions

You are done applying this reference when:
- The eval suite has class balance including ≥20% negative/adversarial cases
- A judge contract tuple is pinned and recorded
- Null-agent floor is established and tasks discriminate above it
- Sampling (N≥5) with explicit aggregation is configured (no retry-until-green)
- CI tiers are assigned by cost budget (pre-merge < 5 min, nightly < 60 min)
- A promote-back loop cadence is scheduled (weekly or post-incident)

## Composition

- **Statistics, privacy, telemetry:** Paired comparisons, effect sizes, multiple-comparison correction, telemetry minimization, redaction, retention → [agent-evals-and-observability](../../agent-evals-and-observability/SKILL.md). This file adds only the QA operational playbook.
- **For pre-merge code verification against specs** (independent verification, gate artifacts, agent-test quality): see [ai-code-quality-gates.md](./ai-code-quality-gates.md).

---

*Sources: Anthropic, "Demystifying Evals for AI Agents" (anthropic.com/engineering, 2026); METR, "Recent Frontier Models Are Reward Hacking" (metr.org, 2025); Berkeley RDI, "How We Broke Top AI Agent Benchmarks" (rdi.berkeley.edu, 2025); Zheng et al., "Judging LLM-as-a-Judge" (NeurIPS, 2023); Wang et al., "Large Language Models are not Fair Evaluators" (arXiv:2305.17926); Jimenez et al., "SWE-bench" (ICLR, 2024); Yao et al., "WebArena" (ICLR, 2024); Mialon et al., "GAIA" (arXiv:2311.12983).*

# AI Code Quality Gates: QA Ownership in AI Factories

## Independent Verification Principle

**NORMATIVE: The implementing agent MUST NOT self-verify its own output.**

This is the structural foundation of quality in agentic workflows. Evidence: IBM Research (Ahmed et al., 2025, arXiv:2511.16858) measured LLM-based automated program repair on SWE-bench Verified and found test overfitting rates of **21.8% (Claude-3.7-Sonnet) to 35.9% (GPT-4o)** — patches that pass white-box tests but fail held-out black-box tests. Critically, test-based refinement *increases* overfitting (21.8% → 25.5% for Claude; 33.0% → 35.9% for GPT-4o), because exposing tests to the generating model creates a feedback loop that games the oracle rather than fixing the code.

### Independence in Agentic Workflows

In an agentic workflow, independence requires a **separate agent session** — a different instance with no shared context, conversation history, or memory with the implementing agent.

> **WARNING — Same-session self-review is NOT independent.** An agent that implements code in one turn and reviews it in the next turn of the SAME session shares priors, blind spots, and confirmation bias within that context window. The reviewing turn has already "seen" the implementation rationale and will anchor on it. This is the agentic equivalent of a developer approving their own PR.

| Verification Model | Independence? | Why |
|-------------------|--------------|-----|
| Same agent, same session, next turn | NO | Shared context, shared priors, confirmation bias |
| Same agent, fresh session, no memory | YES | No shared state; fresh evaluation against spec |
| Different agent model, fresh session | YES (strongest) | Different training distribution catches different failure modes |
| Human reviewer | YES | Different cognitive frame entirely |

## QA-Owned Artifacts per Gate

Pipeline phases and gate verdict formats belong to [spec-driven-development](../../spec-driven-development/SKILL.md). QA owns the **verification layer** at each gate:

| Gate | QA-Owned Artifact | What QA Does |
|------|------------------|--------------|
| **Gate 1** (spec review) | Spec testability review | Run [../scripts/check-ac-testability.py](../scripts/check-ac-testability.py) to flag untestable ACs; verify every AC has an observable outcome and verification method |
| **Gate 2** (plan review) | Verification plan | Produce the plan artifact mapping every AC to a verification method, verifier, and evidence format |
| **Gate 3** (implementation review) | Independent verification | Execute the verification plan in a SEPARATE agent session; record pass/fail per AC with evidence |
| **Gate 4** (acceptance) | Verdict with evidence dossier | QA issues the final verdict: every AC verified, NFR evidence attached, no AC unmapped |
| **Post-merge** | Canary observation plan | Define AC-tied success metrics and rollback triggers for progressive rollout |

### Verification Plan Structure

Use [templates/verification-plan.md](../templates/verification-plan.md) as the fillable artifact when producing a verification plan. The plan must contain:

1. **AC→verification-method traceability matrix** — every acceptance criterion maps to one of: test, inspection, analysis, or demonstration. No AC may be unmapped.
2. **Verifier assignment** — who/what performs each verification (independent agent session, human, automated script).
3. **Evidence format** — what artifact proves the verification (test output log, screenshot, metrics snapshot).
4. **NFR verification approach** — how non-functional requirements (latency, throughput, security) are measured.
5. **Exit criteria** — the observable state that means "verification complete" (e.g., 100% AC coverage, zero critical findings, NFR thresholds met).

### Non-Determinism Handling Decision Rule

AI-generated code often exhibits non-deterministic behavior (LLM outputs, randomized algorithms, concurrency).

| Condition | Verification Approach |
|-----------|----------------------|
| Output is deterministic (same input → same output always) | Single-run equality check |
| Output varies across N runs but has invariants | Property-based assertions (shape, bounds, invariants) + N-run sampling (N≥5) |
| Output is probabilistic with known distribution | Statistical tolerance bands (e.g., p95 latency < X) |
| Output depends on external state (time, random seed) | Pin the external state; verify under controlled conditions |

**Decision rule:** Run the implementation 5 times with identical inputs. If any output differs, the code is non-deterministic — use property-based or statistical verification. Never assert exact equality on non-deterministic output.

## Agent-Generated Test Quality

### The Mirrored-Bug Risk

When an agent generates both code AND tests in the same session, both artifacts share the same misunderstanding of the spec. The test passes because it encodes the same bug, not because the code is correct. This is the test-overfitting problem (IBM, 21.8–35.9%) manifesting at the test-authoring level.

**Mitigation:** Tests must be written from the SPEC, not from the implementation. The test author (agent or human) must not see the implementation source before writing assertions.

### Quality Techniques for Agent-Generated Tests

| Technique | What It Catches | Tools |
|-----------|----------------|-------|
| **Mutation-guided review** | Tests that never fail, miss behavior, or overfit an implementation | PIT (Java), Stryker (JS/TS), mutmut (Python) |
| **Property-based testing** | Missing edge cases, boundary violations | Hypothesis (Python), fast-check (JS/TS) |
| **Differential testing** | Divergence between implementations or spec interpretations | Custom harnesses comparing two implementations |
| **Independent oracle** | Mirrored bugs from shared context | Tests written in separate session from implementation |

Mutation analysis is one independent-verification input, not a standalone proof of quality. The implementing agent cannot self-certify a generated test: a fresh verifier or human must inspect whether it is behaviorally useful and rerun the baseline, candidate test, and exact retained mutant. Reject or revise tests that are tautological, overfit, implementation-coupled, flaky, redundant, or vacuous. Record uncertainty when the run has no coverage, timeouts, flaky outcomes, or infrastructure/tooling failures rather than issuing a clean verdict.

## Regression Under AI PR Volume

AI code factories generate PRs at 5–50× human velocity. Traditional "run everything" regression is infeasible.

| Strategy | Mechanism | When |
|----------|-----------|------|
| **Risk-weighted selection** | Score tests by (change overlap × historical failure rate × business criticality); run top-K | Every PR |
| **Capability→regression graduation** | New capability tests run in a "capability" suite; after 3 consecutive green merges, promote to regression suite | Ongoing |

**Graduation rule:** A test enters the regression suite only after it has passed on 3 consecutive unrelated merges without flaking. Tests that flake during the capability phase are quarantined, not promoted.

## Human-in-the-Loop Review

### The ~400-Line Threshold

Research (Bacchelli & Bird, 2013; Microsoft code review studies) shows review effectiveness drops sharply above ~400 changed lines. AI agents routinely produce 500–2000 line PRs.

| PR Size | Review Strategy |
|---------|----------------|
| < 200 lines | Single human reviewer, standard review |
| 200–400 lines | Two reviewers; focus on architecture and spec compliance |
| > 400 lines | **Require decomposition** OR risk-tiered escalation: security-sensitive paths reviewed by security engineer; business logic by domain expert; mechanical changes spot-checked |

### Gate-Fatigue Countermeasures

When AI generates 20+ PRs/day, reviewers rubber-stamp. Countermeasures:
- Rotate reviewers on a schedule (not ad hoc)
- Require at least one substantive comment per review (not just "LGTM")
- Sample 10% of approved PRs for re-review by a second reviewer
- Track escaped-defect rate per reviewer as a calibration signal

### Agent-Specific Code Review Heuristics

| Failure Mode | Detection Heuristic |
|-------------|-------------------|
| **Hallucinated APIs/imports** | Verify every import statement resolves to a real package; every API call matches documented signatures. Run `python -c "import X"` or equivalent per dependency. |
| **Plausible-but-wrong logic** | Trace at least one critical business path end-to-end against the spec's expected behavior, not just syntax correctness. Compare output to a hand-computed example. |
| **Silent error swallowing** | Flag `except: pass`, `catch(e) {}`, `if err != nil { return nil }` patterns that discard or log-and-continue without surfacing failures to callers. |
| **Over-engineering / spec-letter compliance** | Check whether the solution satisfies the spec's INTENT. An agent may add 300 lines of abstraction to satisfy one AC literally while missing the user's actual need. Ask: "Would a senior engineer write it this way?" |

## Contract Testing

Consumer-driven contracts (Pact) verify service boundaries without full integration:
- Consumer publishes expectations → provider verifies in its own CI
- Critical for AI factories: when multiple agents build different services independently, contract tests catch boundary violations before deployment

For SDET-level contract testing architecture, see [sdet-engineering.md](./sdet-engineering.md).

## Canary / Progressive Rollout with AC-Tied Rollback

| Stage | Traffic | Success Criterion | Rollback Trigger |
|-------|---------|-------------------|-----------------|
| Canary | 1–5% | Error rate ≤ baseline; AC-tied metrics within tolerance | Any AC metric degrades > threshold |
| Progressive | 25% → 50% → 100% | Same as canary + latency p95 stable | SLO breach or new error pattern |

**AC-tied rollback:** Each acceptance criterion that has a measurable production proxy (e.g., "checkout success rate > 99.5%") becomes a canary metric. If the proxy degrades, auto-rollback triggers without human intervention.

## Security Gates

Pearce et al. (2022) found GitHub Copilot generated **vulnerable code in ~40% of scenarios** across CWE categories. AI-generated code requires mandatory security scanning:

| Gate | Tool Category | Blocking? |
|------|--------------|-----------|
| Pre-merge SAST | Static analysis (Semgrep, CodeQL) | Yes, for critical/high |
| Dependency audit | SCA (npm audit, pip-audit, Trivy) | Yes, for known CVEs |
| Secrets detection | Pattern scanning (gitleaks, trufflehog) | Yes, always |

For full security testing methodology, see [secure-software-engineering](../../secure-software-engineering/SKILL.md).

## Worked Example: Gate 3 Independent Verification

**Context:** An agent implemented "AC-7: Users can reset their password via email link expiring in 24h."

| Step | Action | Evidence |
|------|--------|----------|
| 1 | QA opens a SEPARATE agent session with ONLY the spec (no implementation context) | Session ID logged |
| 2 | QA agent writes verification: request reset → check email sent → click link → set new password → verify old password fails | Test script |
| 3 | QA agent tests boundary: link at 23h59m (should work), link at 24h01m (should fail) | Boundary test output |
| 4 | QA agent tests negative: reuse expired link (should 403) | Negative test output |
| 5 | Verdict: AC-7 PASS (3/3 verifications green) | Evidence attached to gate record |

**Anti-pattern avoided:** Had the implementing agent run its own tests, it would have tested only the happy path it coded — missing the expiry boundary entirely (mirrored-bug risk).

## Decision Table: Which Verification Approach?

| Situation | Approach | Exit Condition |
|-----------|----------|----------------|
| Deterministic function, clear spec | Single-run equality test | Test passes with expected output |
| Non-deterministic output | Property-based + N-run sampling | All N runs satisfy invariants |
| Multi-service integration | Contract test + canary | Contracts green; canary metrics stable for 1h |
| Security-sensitive change | SAST + human security review | Zero critical findings; reviewer sign-off |
| UI/UX requirement | Demonstration + human inspection | Reviewer confirms visual/interaction matches spec |

## Exit Conditions

You are done applying this reference when:
- Every AC in the spec maps to a verification method with an assigned independent verifier (100% traceability)
- Gate 3 verification was executed in a separate agent session (or by a human) — NOT by the implementing agent
- Evidence for each AC verdict is attached and reproducible
- Security scan passes with zero critical/high findings
- Non-deterministic outputs use statistical/property-based verification (not exact equality)

## Composition

- **Pipeline mechanics** (phase ordering, gate verdict format APPROVED/CONDITIONS/REJECTED, revision loop, methodology selection): delegated to [spec-driven-development](../../spec-driven-development/SKILL.md). This file adds only the QA ownership layer: WHO verifies, HOW, and WHAT beyond-spec checks apply.
- **For measuring agent capability over time** (eval datasets, judge calibration, CI gate architecture for evals, benchmark interpretation): see [agentic-eval-design.md](./agentic-eval-design.md).
- SDET test infrastructure patterns: [sdet-engineering.md](./sdet-engineering.md)
- Quality metrics and gate design: [quality-gates-and-metrics.md](./quality-gates-and-metrics.md)

---

*Sources: Ahmed, Ganhotra, Shinnar, Hirzel, "Is the Cure Still Worse Than the Disease? Test Overfitting by LLMs in APR" (IBM Research, arXiv:2511.16858, 2025); Pearce et al., "Examining Zero-Shot Vulnerability Repair with Large Language Models" (IEEE S&P, 2022, ~40% vulnerable code rate); Bacchelli & Bird, "Expectations, Outcomes, and Challenges of Modern Code Review" (ICSE, 2013); PIT project README (github.com/hcoles/pitest; pitest.org); StrykerJS project README (github.com/stryker-mutator/stryker-js; stryker-mutator.io); mutmut documentation (mutmut.readthedocs.io); ACH (arXiv:2501.12862); Pact Foundation (docs.pact.io); METR, "Recent Frontier Models Are Reward Hacking" (metr.org, 2025, 30.4% reward-hacking rate on RE-Bench).*

# CI Failure Triage

Systematic diagnosis of CI failures. Load when a CI check is red and you need to find the root cause — not when you're designing test strategy (that's [test-strategy.md](./test-strategy.md)) or building quality gates (that's [quality-gates-and-metrics.md](./quality-gates-and-metrics.md)).

## The One Rule

**RED IS DEAD.** Any non-green CI conclusion (failure, canceled, timed-out job) blocks the PR. Do not dismiss a failure as "pre-existing" or "unrelated" without evidence. The only valid response to a red run is investigation.

What does NOT count as proof of "pre-existing":

- "The failing test doesn't touch my changed files" — dependencies cascade
- "The first CI run on this branch passed" — evidence of flakiness, not safety
- "This test is known to be flaky" — belief, not evidence, until you rerun
- A verbal reassurance in the PR body without a rerun or log excerpt

## Diagnostic Procedure

Follow these steps in order. Each step rules out an entire class of causes before moving to the next.

1. **Reproduce and confirm the failure.** Rerun the failing job to confirm it is deterministic. Pull the full log (not the dot summary) and grep for the actual assertion or error: `gh run view <run_id> --log --job <job_id> | grep -A 20 'FAILURES\|assert\|Error\|traceback'`. If it passes on rerun, apply the flake protocol below before proceeding.

2. **Classify by exit code or error type.** Use the exit-code taxonomy below to determine whether the failure is a test assertion failure (code bug), an infrastructure failure (environment), or a resource failure (OOM/timeout). This determines which investigation path to follow.

3. **Isolate: environment vs code vs flake.** Check whether the failure is in a file your PR modified (`git diff --stat main | grep <failing_file>`). If not in your diff and all other PRs fail the same check, it is infrastructure. If only your branch fails and the file is in your diff, it is likely your regression.

4. **Localize the fault.** Use `git bisect run` for regressions (see below), diff analysis for new failures, or log timeline analysis for infrastructure issues. Narrow to the specific commit, configuration change, or resource threshold.

5. **Fix and verify.** Apply the fix, push, and confirm the CI run goes green. A green rerun after a fix is evidence of resolution. Document the root cause in the PR thread. If the fix is in infrastructure (not your PR), file an issue and link it.

## Exit-Code Taxonomy

| Code | Signal | Meaning | Common Cause | Triage Path |
|------|--------|---------|-------------|-------------|
| 1 | — | Generic failure | Test assertion failed, uncaught exception | Read traceback; fix code or test |
| 2 | — | Usage error / shell builtin misuse | Invalid CLI arguments, bash syntax error | Check command invocation in CI config |
| 126 | — | Permission denied | Script not executable, file permissions wrong | `chmod +x` the script; check CI user |
| 127 | — | Command not found | Missing dependency, wrong PATH, typo in command | Verify tool installation step; check `$PATH` |
| 137 | SIGKILL (9) | OOM kill or forced termination | Container exceeded memory limit; host OOM killer | Check `docker inspect` for `OOMKilled`; increase memory or fix leak |
| 139 | SIGSEGV (11) | Segmentation fault | Native code crash, corrupted memory, C extension bug | Reproduce locally with ASAN; check native deps |
| 143 | SIGTERM (15) | Graceful termination request | CI timeout, orchestrator cancellation, shutdown hook | Check job timeout settings; look for slow tests |

### Exit 137 Deep Dive

Exit 137 indicates SIGKILL but does **not** prove OOM. Do not change memory limits until evidence establishes the cause.

**Evidence to collect before classifying:**
- `docker inspect <container>` → `State.OOMKilled: true/false`
- `dmesg | grep -i oom` on the host
- `docker stats --no-stream` snapshot during the run
- Whether the same test passes with higher memory limits

## Flake-vs-Failure Protocol

| Condition | Action |
|-----------|--------|
| Test fails once, passes on immediate rerun | Rerun once. If it passes, document the rerun in the PR thread. Do **not** rerun twice — two reruns masks a 50%-flaky test. |
| Test fails the same way on rerun | It is a real failure. Proceed with full triage. |
| Test flakes > 1% over 10 runs | Investigate root cause immediately (timing, shared state, external dependency). Do not suppress. |
| Test passes on rerun but the original red run stands unexplained | The failure is not cleared. Investigate or quarantine. |

**The rerun-once rule:** rerun exactly once to distinguish flake from failure. Never rerun until green — that converts a signal into noise.

## Git Bisect for Automated Fault Localization

When a regression appeared somewhere in a range of commits, `git bisect run` automates binary search:

```bash
# Find which commit broke the test suite
git bisect start
git bisect bad HEAD
git bisect good <last-known-green-sha>
git bisect run pytest tests/test_payment.py -x --timeout=60
```

The script passed to `bisect run` must exit 0 for "good" and non-zero for "bad". Exit 125 tells bisect to skip the current commit (useful for commits that don't compile).

```bash
# Skip commits that don't build
git bisect run bash -c 'make build || exit 125; pytest tests/ -x'
```

**When to use bisect:** the failure is deterministic, the green→red transition happened within a known commit range, and the test can run unattended in < 5 minutes.

## Pre-existing vs Regression Classification

```bash
# Check whether the failing file is in your diff
git diff --stat main | grep <failing_file>
```

| Signal | Pre-existing | Regression |
|--------|-------------|------------|
| Failing file not in your diff | Likely | Unlikely |
| ImportError for unrelated module | Likely | Unlikely |
| Fixture timeout ("waiting for stack") | Likely | Unlikely |
| All PRs fail the same check | Yes (infra) | No |
| Failing file IS in your diff | Unlikely | Likely |
| New assertion failure in your test | No | Yes |

**Even when classified as pre-existing:** rerun the job. If it stays red, the failure is real regardless of who introduced it. File an issue if genuinely out of scope, and link it from the PR body.

## Runner and Infrastructure Checks

| Finding | Meaning | Action |
|---------|---------|--------|
| 0 runners online | Runner crashed or never deployed | Contact infra; check runner container |
| All runners busy | Capacity issue; previous runs stacking | Wait or add runners |
| Job queued indefinitely | Label mismatch (`runs-on:` vs runner labels) | Fix workflow label |
| `skipped` conclusion | `[skip ci]` in commit, or trigger mismatch | Check commit message and workflow `on:` |

## Compose Readiness Corollary

When a dependency exposes a readiness endpoint, use a Compose healthcheck plus `depends_on.condition: service_healthy`. `service_started` only proves process creation. Keep the healthcheck retry budget inside the CI wait budget.

## Gotchas

- **Do not change memory limits, retry counts, or dependency timing until the failed run's primary test log and failure artifact agree on what failed.** An exit code is a symptom, not a root cause.
- **Do not classify the whole run from the last visible message.** When a long `set -e` chain returns non-zero, rerun each check independently.
- **A green rerun does not clear the historical red run.** It establishes intermittency. The original failure still needs an explanation.
- **Rerunning more than once converts signal to noise.** One rerun distinguishes flake from failure; two reruns hides a coin-flip test.

## Composition

- Systematic debugging methodology for complex root-cause analysis: [systematic-debugging](../../systematic-debugging/SKILL.md)
- Test isolation and CI-vs-local divergence: [test-debugging.md](./test-debugging.md)

*Sources: git-bisect documentation (git-scm.com/docs/git-bisect), GitHub Actions exit codes (docs.github.com), Linux signal numbers (man 7 signal), DORA State of DevOps Reports (CI failure analysis).*

# Exploratory Testing

Structured discovery testing through concurrent learning, design, and execution. Load when designing charters for new or changed features, investigating areas with unknown risk, or establishing session-based test management. Not for scripted regression suites (that's [regression-testing.md](./regression-testing.md)) or test design technique selection (that's [test-design-techniques.md](./test-design-techniques.md)).

## When to Explore

| Signal | Rationale |
|--------|-----------|
| New feature with incomplete or evolving requirements | Scripted tests can't be written yet; exploration discovers what to automate |
| Post-incident investigation | Reproduce conditions, find adjacent failure modes |
| Integration of a new third-party dependency | Unknown edge cases, error behaviors not in docs |
| Low test coverage in a high-risk area | Discover what's missing before designing automated tests |
| User-reported "it just feels wrong" | No repro steps; exploration builds a repro |
| Before a major release (time-boxed session) | Catch issues that scripted tests structurally miss |

> **Rule:** Exploration is not "random clicking." It is disciplined, time-boxed, charter-driven investigation with structured reporting.

## Session-Based Test Management (SBTM)

SBTM (Bach & Bach, 2000) provides accountability for exploratory work without destroying its creative advantage.

### Core Structure

| Element | Definition |
|---------|-----------|
| **Charter** | Mission statement defining the session's scope and goal |
| **Timebox** | Fixed duration (typically 60–120 minutes) with no interruptions |
| **Debrief** | Structured review of findings, metrics, and follow-up actions |

### T/B/B Metrics

Track time allocation per session:

| Metric | Definition | Target |
|--------|-----------|--------|
| **T** (Test time) | Time actively testing (designing + executing) | ≥ 70% of session |
| **B** (Bug investigation) | Time investigating issues found during the session | 10–25% |
| **B** (Setup/Interruption) | Time lost to environment issues, questions, context switches | ≤ 10% |

A session with T < 60% indicates environmental problems or scope confusion; fix the blocker before scheduling more sessions.

## Charter Format

```
Explore <target> with <resources/constraints> to discover <information>.
```

### Examples

| Charter | Analysis |
|---------|----------|
| Explore the new search autocomplete with slow network and unicode input to discover rendering and latency edge cases | Target: search autocomplete; Resources: throttled network, unicode; Goal: rendering/latency issues |
| Explore payment refund flow with expired cards and partial amounts to discover error handling gaps | Target: refund flow; Resources: expired test cards; Goal: error handling |
| Explore admin bulk-user-import with CSV files > 10MB to discover timeout and memory behavior | Target: bulk import; Resources: large CSV; Goal: timeout/memory |

### Charter Quality Checklist

- Contains a specific target (not "the app")
- Names resources or constraints (test data, tools, conditions)
- States what information you seek (not "find bugs" — too vague)
- Scope fits within the timebox (one session, one charter)

## Heuristics and Oracles

Oracles tell you when something might be wrong. Heuristics guide where to look. Neither is a checklist; they are thinking tools.

### SFDIPOT (Test Coverage Heuristic)

| Letter | Stands For | Question to Ask |
|--------|-----------|-----------------|
| **S** | Structure | What is this made of? (components, files, APIs) |
| **F** | Function | What does it do? (features, behaviors) |
| **D** | Data | What does it process? (inputs, outputs, formats) |
| **I** | Interfaces | What does it connect to? (APIs, UIs, protocols) |
| **P** | Platform | What does it run on? (OS, browser, hardware) |
| **O** | Operations | Who uses it and how? (workflows, personas) |
| **T** | Time | What happens over time? (concurrency, timeouts, aging) |

### HICCUPPS (Oracle Heuristic)

Sources of expectation when you lack a spec:

| Source | Question |
|--------|----------|
| **H**istory | What did previous versions do? |
| **I**mage | What would a comparable product do? (competitors, category norms) |
| **C**omparable | What do similar features in this product do? |
| **C**laims | What does documentation, marketing, or support say? |
| **U**sers' expectations | What would a reasonable user expect? |
| **P**roduct purpose | Does this serve the product's stated mission? |
| **P**urpose (feature) | Does this serve the feature's stated goal? |
| **S**tandards | Do relevant standards (WCAG, RFC, API conventions) apply? |

### Tours (Exploration Patterns)

| Tour | Approach | Best For |
|------|----------|----------|
| **Guidebook tour** | Follow documented flows (user guide, API docs) | Verifying documentation accuracy |
| **Money tour** | Test the revenue-critical paths (checkout, billing) | Business-critical smoke |
| **Landmark tour** | Visit every major feature surface once | Broad coverage sweep |
| **Garbage collectors tour** | Try every error path (invalid input, cancel, timeout) | Error handling |
| **Bad-neighborhood tour** | Focus on historically buggy areas | Regression-prone zones |
| **Museum tour** | Test legacy/backward-compatible paths | Upgrade safety |
| **Back-alley tour** | Try unadvertised features (admin panels, debug endpoints) | Security, hidden behavior |
| **Obsessive-compulsive tour** | Repeat one action many times (submit, refresh) | Concurrency, rate limits |

## Bug Advocacy

Finding a bug is half the work; getting it fixed is the other half.

### Report Quality

| Element | Requirement |
|---------|-------------|
| Title | One line: what broke + where |
| Repro steps | Numbered, deterministic, minimal |
| Expected vs Actual | State both explicitly |
| Evidence | Screenshot, log excerpt, or recording |
| Impact statement | Who is affected, how often, severity |
| Environment | Exact versions, config, browser/device |

### Advocacy Principles

- **Reproduce before reporting.** A bug you can't reproduce is a hypothesis, not a bug.
- **Isolate the minimal repro.** 15 steps with 12 irrelevant ones gets triaged slower than 3 steps.
- **Separate observation from interpretation.** "Button returns 500" is observation; "the backend is broken" is interpretation.
- **Escalate unresolved P0/P1 bugs** with evidence, not emotion. Link to user impact data.

## Session Debrief Structure

After each session (5–10 minutes):

1. **What did you test?** (areas covered, charters fulfilled)
2. **What did you find?** (bugs filed, risks identified, questions raised)
3. **What is left untested?** (scope not reached, new areas discovered)
4. **Metrics:** T/B/B percentages
5. **Follow-up:** New charters needed? Automation candidates? Blockers?

Record in a session sheet. Aggregate across sessions to build a coverage picture for unscripted areas.

## Exploration → Automation Pipeline

| Session Finding | Next Step |
|----------------|-----------|
| Reproducible bug | Write regression test, file bug (see [regression-testing.md](./regression-testing.md)) |
| Repeated manual check | Automate as a scripted test |
| New risk area discovered | Add to risk register (see [risk-based-testing.md](./risk-based-testing.md)) |
| Spec gap identified | File clarification request; add AC |
| Performance concern | Schedule load test (see [performance-testing.md](./performance-testing.md)) |

## Gotchas

> **Gotcha — Exploration without accountability:** Unlogged "testing" that produces no session sheets, no bugs, and no coverage data is indistinguishable from not testing. SBTM's value is the debrief, not the timebox.

> **Gotcha — Using exploration to avoid automation:** If the same manual checks recur session after session, automate them. Exploration is for discovery; automation is for repetition.

> **Gotcha — Vague charters:** "Explore the app" produces random clicking. Every charter must name a target, resources, and information goal.

## Exit Condition

You are done applying this reference when: (1) charters are written in the "Explore X with Y to discover Z" format, (2) sessions are time-boxed with T/B/B tracking, (3) debriefs produce actionable follow-ups, and (4) repeatable findings are graduated to automation or the risk register.

## Composition Links

- Risk-based prioritization for charter selection: [risk-based-testing.md](./risk-based-testing.md)
- Regression test graduation: [regression-testing.md](./regression-testing.md)
- Test design techniques for structured scenarios: [test-design-techniques.md](./test-design-techniques.md)
- Verification methodology (evidence and verdicts): [verification-methodology](../../verification-methodology/SKILL.md)

---

*Sources: James Bach & Jon Bach, "Session-Based Test Management" (2000, bach.ch/sbtm), Cem Kaner et al. "Lessons Learned in Software Testing" (Wiley, 2002), Elisabeth Hendrickson "Explore It!" (Pragmatic Bookshelf, 2013), Michael Bolton (HICCUPPS oracle heuristics), James Bach (SFDIPOT coverage heuristic).*

# Performance Testing

## Types

| Type | Question Answered | Duration | Frequency |
|------|-------------------|----------|-----------|
| Load test | Does it handle expected traffic? | 5–15 min | Nightly / pre-release |
| Stress test | Where does it break? | Ramp until failure | On-demand / architectural changes |
| Soak test | Does it degrade over time (leaks)? | 4–24 hours | Weekly |
| Spike test | Does it survive sudden bursts? | Seconds to minutes | Pre-launch / event prep |
| Benchmark | What's the raw throughput/latency? | 30s–2 min | On PR (smoke) |

## Tool Landscape

| Tool | Language | Strengths | Weaknesses | Best For |
|------|----------|-----------|------------|----------|
| **k6** | JavaScript (Go runtime) | CI-native, thresholds as code, low resource overhead, Grafana ecosystem | No GUI scripting; HTTP/gRPC/WebSocket/browser (experimental) | CI-integrated load testing, developer-owned perf |
| **Locust** | Python | Distributed by design, real-user behavior modeling, event-driven | Higher memory per VU than k6; Python GIL limits single-node throughput | Complex user flows, distributed load generation |
| **Gatling** | Scala/Java/Kotlin | Highest throughput per node, detailed HTML reports, protocol breadth | Steep learning curve; commercial features paywalled | High-throughput enterprise, protocol-heavy (JMS, JDBC) |
| **JMeter** | Java (GUI + CLI) | Widest protocol support, plugin ecosystem, 20+ years of community | Heavy resource usage, brittle test plans, XML config | Legacy protocol support, non-developer QA teams |

### Selection Decision

| Signal | Choose |
|--------|--------|
| CI/CD integration is the primary driver | k6 |
| Team is Python-centric, needs distributed load | Locust |
| Need maximum VUs per machine, detailed reporting | Gatling |
| Must test JMS, JDBC, FTP, or other non-HTTP protocols | JMeter |
| Browser-level rendering performance matters | k6 (browser module) or Lighthouse CI |

## SLO-Based Threshold Design

Performance thresholds must derive from product requirements, not arbitrary round numbers.

### Deriving Thresholds from SLOs

1. **Start with the product SLO.** Example: "95% of API requests complete in < 300ms" → p95 < 300ms.
2. **Set the test threshold at the SLO.** The load test asserts `p(95)<300`.
3. **Add a warning threshold below the SLO** for early signal: `p(95)<250` (advisory, does not fail CI).
4. **Set error-rate threshold from availability SLO.** If availability SLO is 99.9%, error threshold is `rate<0.001`.

### Baseline-Then-Regress Pattern

Never evaluate a single run in isolation. Establish a baseline first:

```
1. Run the load test on a known-good commit (e.g., main HEAD).
2. Store p50, p95, p99, throughput, error rate as the baseline.
3. On subsequent runs, compare against the 7-day rolling baseline.
4. Alert if p95 regresses > 20% vs baseline.
5. Fail CI (blocking gate) if p95 exceeds the absolute SLO.
```

| Comparison | Action |
|-----------|--------|
| p95 within 10% of baseline | Pass — no action |
| p95 regressed 10–20% | Advisory warning — investigate if trend continues |
| p95 regressed > 20% vs baseline | Alert — likely regression; investigate before merge |
| p95 exceeds absolute SLO | Block merge — SLO breach |

### Worked Example (k6 thresholds)

```javascript
import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  stages: [
    { duration: '30s', target: 20 },   // ramp up
    { duration: '1m', target: 20 },    // steady state
    { duration: '10s', target: 0 },    // ramp down
  ],
  thresholds: {
    http_req_duration: ['p(95)<300', 'p(99)<800'],  // from product SLO
    http_req_failed: ['rate<0.001'],                  // from availability SLO
  },
};

export default function () {
  const res = http.get('https://staging.example.com/api/items');
  check(res, {
    'status 200': (r) => r.status === 200,
    'response time < 200ms': (r) => r.timings.duration < 200,
  });
  sleep(1);
}
```

## Key Metrics

| Metric | Definition | Target Guidance |
|--------|-----------|-----------------|
| p50 latency | Median response time | User-perceived "normal" |
| p95 latency | 95th percentile | SLO boundary for most APIs |
| p99 latency | 99th percentile | Tail latency — catches GC pauses, cold starts |
| Throughput | Requests/sec sustained | Compare against capacity plan |
| Error rate | 5xx / total | < 0.1% under load |
| Saturation | CPU/memory/connections at peak | < 80% = headroom |

## CI Cadence

| Trigger | Test Type | Duration | Gate |
|---------|-----------|----------|------|
| PR (hot-path changes only) | Smoke benchmark (10 VUs, 30s) | < 2 min | Advisory — catches 10× regressions |
| Nightly (staging) | Full load test (expected traffic profile) | 10–15 min | Blocking for release branch |
| Weekly | Soak test (constant load, 4–8h) | 4–8 hours | Alert on degradation trend |
| Pre-release | Stress test + spike | 30 min | Blocking — must not regress vs last release |

**Not on every PR.** Full load tests on every PR waste CI resources and produce noisy results. Smoke benchmarks on PRs catch gross regressions; nightly runs catch subtle ones.

## Interpreting Results

| Symptom | Likely Cause | Next Step |
|---------|-------------|-----------|
| Latency climbs linearly with VUs | Single-threaded bottleneck or lock contention | Profile CPU, check for global locks |
| Latency flat then sudden cliff | Resource exhaustion (connections, memory, FDs) | Check pool sizes, `ulimit`, OOM killer |
| Throughput plateaus early | Downstream dependency is the bottleneck | Test the dependency in isolation |
| Errors only at high concurrency | Race condition or timeout misconfiguration | Check connection pool, retry storms |
| Memory grows during soak | Leak — unclosed connections, unbounded cache | Heap dump at intervals, diff allocations |

## Gate Governance

Performance thresholds feed into quality gate decisions — they are not standalone pass/fail numbers. A perf regression that exceeds the SLO is a **blocking gate** (prevents merge/deploy); a regression within the warning band is an **advisory gate** (flags for review). For the full gate design framework (blocking vs advisory, pipeline stages, anti-patterns), see [quality-gates-and-metrics.md](./quality-gates-and-metrics.md).

## Gotchas

- **Thresholds without baselines are meaningless.** A p95 of 280ms tells you nothing without knowing whether last week's was 120ms. Always baseline first.
- **Testing against shared staging environments** produces noisy results from other teams' load. Use dedicated perf environments or time-boxed windows.
- **Ignoring tail latency.** p50 can look healthy while p99 is 10× the SLO. Always assert on percentiles, not averages.
- **Running the full suite on every PR** wastes CI minutes and teaches teams to ignore perf results. Tier by trigger.

## Related

- Gate governance (blocking vs advisory, pipeline placement): [quality-gates-and-metrics.md](./quality-gates-and-metrics.md)
- Test data for load tests (volume seeding): [test-data-management.md](./test-data-management.md)

*Sources: k6 documentation (grafana.com/k6), Locust docs (locust.io), Gatling docs (gatling.io), JMeter docs (jmeter.apache.org), Google SRE Workbook (O'Reilly, 2018), DORA State of DevOps Reports.*

# QA Career Levels: Senior → Staff → Principal

## Scope-Progression Model

Career progression in QA engineering follows the same scope ladder as general software engineering. The differentiator between levels is **scope of influence**, not years of experience or technical depth alone.

| Scope Tier | Definition | QA Example |
|-----------|-----------|------------|
| **Task** | Complete a well-defined assignment | Write test cases for one API endpoint |
| **Feature** | Own quality for a feature end-to-end | Design test strategy for checkout v2 |
| **Project** | Coordinate quality across a project with multiple features | Lead QA for a platform migration |
| **Product** | Influence quality across multiple teams on one product | Define org-wide flake policy; set test architecture standards |
| **Org** | Shape quality engineering direction across the organization | Multi-year QE vision; industry presence; hiring bar ownership |
| **Department** | Influence beyond engineering (product, support, compliance) | Company-wide reliability culture; regulatory quality frameworks |

### Level-to-Scope Mapping

| Level | Primary Scope | Key Distinction |
|-------|--------------|-----------------|
| **Senior QA Engineer** | Project | Operates autonomously within a project; others ask for help |
| **Staff QA Engineer** | Product | Multi-team influence **without authority**; sets direction others follow voluntarily |
| **Principal QA Engineer** | Org | Multi-year QE vision, public presence, shapes hiring standards and org strategy |

> **Gotcha — "Engineer 2.5" trap:** Performing at the very top of your current level (e.g., an excellent Senior doing Senior work brilliantly) is NOT the same as operating at the next level. Promotion requires **demonstrated scope at the NEXT level**, not excellence at the current one. Many strong Seniors stall here: they are "Senior+" but haven't crossed into product-scope influence.

## Leveling Mechanics

### How Promotion Works

Promotion is recognition that you are **already operating** at the next level's scope — not a reward for tenure or a motivation tool.

1. **Demonstrate next-level scope** for a sustained period (typically 2–4 quarters)
2. **Gather evidence** from multiple teams/stakeholders showing impact beyond your current scope
3. **Sponsor** (usually your manager) builds the case with your evidence
4. **Calibration** across peers at the target level confirms the scope match

### QA ↔ General Engineering Level Mapping

| QA Title | General Engineering Equivalent | Scope Expectation |
|----------|-------------------------------|-------------------|
| QA Engineer I/II | SDE I/II | Task → Feature |
| Senior QA Engineer | Senior SDE | Project |
| Staff QA Engineer / SDET | Staff SDE | Product |
| Principal QA Engineer / Test Architect | Principal SDE | Org |
| QE Director / Fellow | Director / Distinguished Engineer | Department+ |

This mapping matters because it determines compensation bands, review calibration pools, and cross-functional influence expectations.

## Role Archetypes

### Staff-Level Archetypes (per Will Larson, *Staff Engineer*, 2020)

| Archetype | Description | QA Manifestation |
|-----------|-------------|-----------------|
| **Team Lead** | Guides a team's technical direction while staying hands-on | QA lead embedding quality practices in a product team |
| **Architect** | Sets technical direction across multiple teams | Test infrastructure architect; framework standards owner |
| **Solver** | Drops into hard problems, fixes them, moves on | Reliability firefighter; flake-hunter across teams |
| **Right Hand** | Extends a leader's attention; delegates authority | QE partner to VP Eng; owns quality org initiatives |

### QA-Specific Role Archetypes

| Role | Focus | Typical Level Range |
|------|-------|-------------------|
| **QA Engineer** | Manual + exploratory testing, domain expertise | Junior → Senior |
| **SDET** (Software Development Engineer in Test) | Test infrastructure as product; coding-first | Senior → Principal |
| **Test Automation Engineer (TAE)** | Automation design and maintenance | Junior → Staff |
| **Quality Coach** | Embeds quality culture in dev teams; no direct test execution | Staff → Principal |
| **Test Architect** | Cross-team test strategy, tooling, standards | Staff → Principal |
| **Google TE/SET** (Test Engineer / Software Engineer in Test) | Hybrid: product development + test tooling (Google's model) | Senior → Staff |

For SDET competency details, see [sdet-engineering.md](./sdet-engineering.md).

## Misconceptions

### "QA is Dead"

**Claim:** AI and developer-owned testing eliminate the need for QA roles.

**Reality:** The role evolves, not disappears. Organizations that dissolved dedicated QA (e.g., some early Spotify-model teams) reinstated quality engineering functions within 2–3 years when defect escape rates climbed. AI increases test generation volume but does NOT replace quality judgment, risk assessment, test strategy, or the independence principle. The demand shifts from "test executor" to "quality engineer" — higher-scope, more technical.

### "The QA career ladder is flat"

**Claim:** QA has no growth path beyond "senior tester."

**Reality:** QA maps directly onto the general engineering ladder (see mapping above). Staff and Principal QA roles exist at Google, Microsoft, Amazon, Netflix, and Atlassian. The perceived flatness comes from companies that fail to define the scope expectations for QA at Staff+ levels, not from an inherent ceiling.

### "More tests = more quality"

**Claim:** Writing more tests always improves quality.

**Reality:** Test count is a vanity metric. A suite of 10,000 assertions that never catch a regression provides zero quality value. Quality comes from the RIGHT tests (risk-based, targeting escaped-defect patterns) with strong assertions (validated by mutation testing). See [quality-gates-and-metrics.md](./quality-gates-and-metrics.md) for vanity-vs-actionable metrics.

## Actionable Usage Guidance

### Promotion-Packet Guidance (Senior → Staff Example)

**Step 1: Gather scope evidence.** Collect artifacts demonstrating product-scope (multi-team) impact:
- Test strategy documents adopted by 2+ teams
- Framework/infrastructure contributions used org-wide
- Mentoring records (engineers you've leveled up outside your team)
- Cross-team incident postmortems you led or contributed to
- Conference talks, internal tech talks, or published writing

**Step 2: Structure the packet.**

| Section | Content | Evidence Type |
|---------|---------|--------------|
| Scope summary | One paragraph: "I operate at product scope by..." | Narrative |
| Technical impact | 3–5 bullets with metrics (flake rate reduced X%, CI time cut Y%) | Data |
| Multi-team influence | Teams influenced, mechanisms (guilds, standards, reviews) | Peer feedback |
| Direction setting | Standards/strategies you authored that others adopted | Documents |
| Mentorship | Engineers coached, their growth outcomes | Testimonials |

**Step 3: Map to next-level expectations.** Each piece of evidence must answer: "How does this demonstrate PRODUCT-scope influence without authority?" If it only shows excellence within one project, it's Senior-level evidence, not Staff-level.

### Level-Calibrated Growth Advice

| Current Level | To Reach Next Level | Concrete Actions |
|--------------|--------------------|--------------------|
| Senior → Staff | Demonstrate product-scope influence | Author a cross-team test standard; lead a quality guild; own flake policy for 3+ teams; contribute to another team's test architecture |
| Staff → Principal | Demonstrate org-scope vision | Define 2-year QE roadmap adopted by leadership; establish hiring bar for QE; represent the org externally (talks, standards bodies); resolve a systemic quality failure spanning 4+ teams |
| Principal → Director/Fellow | Department influence + strategy | Shift from technical to organizational: budget ownership, headcount strategy, cross-department quality programs |

### Role-to-Level Mapping Procedure

Given a job description or team charter, determine the appropriate level:

1. **Identify the scope of impact** the role requires (task/feature/project/product/org)
2. **Check authority vs. influence:** Does the role manage people (→ Team Lead track) or influence without authority (→ Architect/Solver/Right Hand track)?
3. **Map scope to level** using the table above
4. **Validate against leveling mechanics:** Does the role require demonstrated evidence at that scope? If a "Staff QA" posting only describes project-scope work, it's misleveled.

| If the role description says... | Scope tier | Likely level |
|-------------------------------|-----------|-------------|
| "Write and maintain tests for feature X" | Task/Feature | QA Engineer I/II |
| "Own quality strategy for the payments project" | Project | Senior |
| "Define test standards adopted by all product teams" | Product | Staff |
| "Set 3-year quality vision; represent company in industry" | Org | Principal |

## Decision Table: When to Apply This Reference

| Situation | Use This Reference For | Exit Condition |
|-----------|----------------------|----------------|
| Writing a promotion packet | Evidence structure + scope mapping | Packet has ≥3 product-scope evidence items mapped to next-level criteria |
| Leveling a new role/position | Role-to-level mapping procedure | Role mapped to a scope tier with documented justification |
| Career growth planning | Level-calibrated growth advice | 2–3 concrete next-level actions identified for current level |
| Evaluating team quality org design | Archetype + scope model | Each QE role mapped to archetype + scope tier |

**Exit condition:** You are done applying this reference when you can state the target level's scope tier, map ≥3 pieces of evidence to that tier, and identify the gap (if any) between current and target scope.

## Worked Example: Senior → Staff Promotion Packet

**Context:** Maria is a Senior QA Engineer on the Checkout team. She wants to reach Staff.

**Evidence she gathered:**

| Evidence | Scope Demonstrated |
|----------|-------------------|
| Authored the org-wide flaky-test quarantine policy, adopted by 4 teams | Product (multi-team standard) |
| Built shared Playwright component library used by 3 product teams | Product (shared infrastructure) |
| Led quality guild (12 members, biweekly) for 3 quarters | Product (direction setting) |
| Mentored 2 junior QAs to Senior level (one on another team) | Product (multi-team growth) |
| Reduced org-wide CI flake rate from 8% to 2.3% | Product (measurable cross-team impact) |

**Assessment:** Maria demonstrates product-scope influence without authority (no direct reports on other teams). Her evidence maps cleanly to Staff expectations. Packet is ready for sponsor review.

## Composition Links

- SDET competency model and career progression: [sdet-engineering.md](./sdet-engineering.md)
- Quality metrics (vanity vs actionable): [quality-gates-and-metrics.md](./quality-gates-and-metrics.md)

---

*Sources: Will Larson, Staff Engineer (2020), staffeng.com; endoflineblog.com career-leveling frameworks; Google Testing Blog (TE/SET model, testing.googleblog.com); Angie Jones, "Test Automation Career Path" (angiejones.tech, 2021); DORA State of DevOps Reports (quality-engineering role evolution).*

# Quality Gates and Metrics

## Gate Design: Blocking vs Advisory

A quality gate is an enforced checkpoint that must be satisfied before software proceeds. Classify every gate as one of three types:

Mutation evidence is normally targeted and advisory: it can focus review on plausible behavior-changing faults in a defined diff or risk slice, but it is not unconditional high-performance blocking evidence or standalone proof of quality. A project should block on mutation evidence only after independently validating a narrower policy, including its scope, operators, equivalent-mutant treatment, failure handling, and denominator.

| Type | Pipeline Behavior | Example |
|------|------------------|---------|
| **Blocking** | Pipeline stops; artifact not promoted | Unit test failure, critical CVE |
| **Advisory** | Pipeline continues; warning logged + team notified | Coverage drop, lint warnings |
| **Informational** | No pipeline impact; recorded for dashboard | Execution time trend, flake rate |

### Gate Placement by Stage

```
Commit → [Lint (advisory)] → [Unit tests (blocking)]
  → [Build (blocking)] → [Static analysis: 0 critical (blocking)]
    → [Integration tests (blocking)] → [E2E tests (blocking)]
      → [Performance regression (advisory)] → [Security scan (blocking)]
        → Release
```

### Gate Evolution

Tighten gates as the team matures. Review quarterly:

| Phase | Blocking | Advisory |
|-------|----------|----------|
| Starting | Tests compile + pass | Coverage > 50% |
| Growing | Unit pass, coverage > 70%, 0 critical static-analysis | Coverage > 80% |
| Maturing | All tests pass, coverage > 80%, 0 high CVEs | Flake rate < 2% |
| High-perf | All pass, coverage > 85%, targeted mutation evidence reviewed on P0 changes | Perf regression < 5% |

> **Gotcha — Too many blocking gates:** When everything blocks, developers bypass or game the pipeline. Keep blocking gates to critical checks; use advisory for everything else. If a gate fires on > 50% of PRs, loosen it temporarily while the team improves.

### Anti-Patterns

| Anti-Pattern | Problem | Fix |
|-------------|---------|-----|
| Gates that never change | Thresholds become irrelevant | Quarterly review; tighten gradually |
| Coverage without quality | 90% coverage of weak assertions is misleading | Add mutation testing (see [test-automation.md](./test-automation.md)) |
| Single flaky test blocks pipeline | Loss of trust in CI | Quarantine flaky tests (see [test-automation.md](./test-automation.md)) |
| Manual gates for everything | Pipeline becomes the bottleneck | Automate everything scriptable; manual only for regulatory sign-off |

## DORA Four Keys + Reliability

The DORA metrics (from Google's DevOps Research and Assessment team) measure delivery performance. QA owns the quality-adjacent keys:

| DORA Key | Definition | QA Ownership | Elite Target |
|----------|-----------|-------------|-------------|
| **Deployment Frequency** | How often code ships to production | Enable via fast, reliable test feedback | Multiple per day |
| **Lead Time for Changes** | Commit → production elapsed time | Reduce test cycle time (parallelism, selection) | < 1 hour |
| **Change Failure Rate** | % of deploys causing incidents | Direct quality metric — escaped defects | < 5% |
| **Mean Time to Restore (MTTR)** | Time from incident → recovery | Detection speed (monitoring), rollback readiness | < 1 hour |

**Fifth metric — Reliability:** DORA's 2021 report added reliability as a complementary measure: meeting user-facing SLOs. QA contributes by validating SLO thresholds in pre-production and monitoring escape patterns post-deploy.

### Connecting DORA to QA Metrics

| DORA Key | QA Metric That Drives It |
|----------|------------------------|
| Deployment Frequency | Test suite duration, flake rate (fewer reruns = faster feedback) |
| Lead Time | Time-to-green on PR pipeline |
| Change Failure Rate | Defect escape rate, regression suite effectiveness |
| MTTR | MTTD (mean time to detect), rollback test coverage |

## Vanity vs Actionable Metrics

| Metric | Type | Why |
|--------|------|-----|
| Tests executed (count) | **Vanity** | Activity measure; says nothing about quality |
| Lines of test code | **Vanity** | More code ≠ more confidence |
| Coverage % alone | **Vanity** (without context) | Can be gamed; high coverage + low mutation score = weak suite |
| Defect escape rate | **Actionable** | Directly tells you where testing is failing |
| Flake rate | **Actionable** | Rising flake rate predicts loss of CI trust |
| MTTR by severity | **Actionable** | Drives investment in debugging/rollback tooling |
| Mutation review evidence | **Actionable with context** | A defined-scope triage input about plausible fault detection; mutation score alone is not proof of test quality |
| Change failure rate | **Actionable** | Direct DORA quality signal |
| Regression test ROI | **Actionable** | (Tests that caught a regression) / (total regression tests) — guides suite pruning |

**Rule:** If a metric changes and you don't know what action to take, it's vanity. Track 5–10 metrics maximum (the 5-10 rule); more than 10 causes analysis paralysis. Mutation results should preserve scope, exclusions, unknowns, and incomplete/tooling-failure outcomes rather than reducing them to a universal score.

## Defect Severity and Priority Classification

### Severity Scale (Technical Impact)

| Level | Definition | Example |
|-------|-----------|---------|
| **S1 — Critical** | System down, data loss, security breach | Production database corrupted |
| **S2 — High** | Major feature broken, no workaround | Checkout flow returns 500 |
| **S3 — Medium** | Feature degraded, workaround exists | CSV export omits one column |
| **S4 — Low** | Cosmetic, minor inconvenience | Alignment off by 2px |

### Priority Scale (Business Urgency)

| Level | Definition | Response SLA |
|-------|-----------|-------------|
| **P1 — Immediate** | Blocks release or impacts revenue now | Acknowledge < 1h, fix < 4h |
| **P2 — Urgent** | High user impact, next-release blocker | Fix within current sprint |
| **P3 — Normal** | Moderate impact, schedule normally | Fix within 2 sprints |
| **P4 — Backlog** | Low impact, fix when convenient | No SLA; backlog grooming |

### Severity × Priority Interaction

Severity and priority are independent: a critical-severity bug in a deprecated feature may be P3 (fix next sprint), while a medium-severity bug affecting a key customer demo may be P1 (fix today). **Priority = f(severity, business context, user impact).**

### Escalation Rules

| Condition | Action |
|-----------|--------|
| Any S1/P1 defect | Blocks release; escalate to engineering leadership within 1 hour |
| S2/P1 unresolved > 24h | Escalate to VP Engineering |
| 3+ S2 defects in one release | Trigger release hold; root-cause review |
| S1 in production | Page on-call; post-incident review within 48h |

## Metric Visualization

Track 5–10 core metrics on a team-visible dashboard:

```
Quality Dashboard — Sprint 24
┌──────────────────────────────────────┐
│ Pass Rate: 98.5% │ Coverage: 83%     │
│ Escaped Defects: 3 (1 S2)           │
│ DDP: 94%         │ MTTR: 4.5h       │
│ Flake Rate: 1.2% │ Suite: 14min     │
│ Change Failure Rate: 3.2%           │
└──────────────────────────────────────┘
```

## Composition Links

- Test automation framework selection and mutation testing: [test-automation.md](./test-automation.md)
- Performance thresholds feeding gates: [performance-testing.md](./performance-testing.md)
- Flaky quarantine workflow: [test-automation.md](./test-automation.md)
- Agent evals and observability (eval-specific metrics, graders): [agent-evals-and-observability](../../agent-evals-and-observability/SKILL.md)

---

*Sources: DORA State of DevOps Report 2023 (Google Cloud), DORA Accelerate (Forsgren, Humble, Kim; 2018), SonarSource quality gate documentation, MinimumCD Practice Guide.*

# Regression Testing

## Core Principle

> **Every fixed bug becomes a regression test.** When a defect is fixed, the test that reproduces it (or should have caught it) is added to the suite permanently. This is the single highest-ROI regression practice: the suite's ability to catch known failure modes grows monotonically. Track "regression test added for fix" rate; target > 80% of bug fixes.

## What Belongs in a Regression Suite

| Include | Exclude |
|---------|---------|
| Every fixed bug (as a test) | Tests that haven't failed in 6+ months (review for archive) |
| Critical user paths (P0/P1) | Tests for deprecated features |
| Known failure patterns | Tests duplicating lower-level coverage |
| API contract checks | Visual regression on work-in-progress UI |
| Data integrity assertions | Performance tests (separate suite) |

## Test Impact Analysis (Shift-Left Selection)

Not every change needs the full suite. Impact analysis selects the subset of tests affected by a change.

| Approach | How It Works | Trade-off |
|----------|-------------|-----------|
| **Static call-graph** | Map changed code → functions → tests that call them | Fast, conservative (over-selects); misses dynamic dispatch |
| **Dynamic trace** | Record which tests execute which code (coverage instrumentation), then map changes → tests | Accurate; needs profiling infra; Ekstazi uses this |
| **ML prediction** | Train on historical (change, failing-test) pairs | Best precision; needs history; see [test-automation.md](./test-automation.md) |

**Tools:** Ekstazi (dynamic, JVM), Launchable (ML, language-agnostic), custom coverage-map scripts.

### Selection Math

Score each test for inclusion on a change:

```
selection_score(test) =
    w1 × changed_code_coverage(test)      # does it touch the diff? (0/1)
  + w2 × historical_failure_rate(test)    # how often has it failed recently?
  + w3 × business_risk(test)              # P0/P1 path? (see test-strategy.md)
```

Default weights: `w1=0.5, w2=0.3, w3=0.2`. Run all tests with score > threshold, plus always run the full P0 smoke set regardless of score.

| Change Type | Required Tests |
|-------------|---------------|
| Bug fix | Repro test + related unit tests |
| Feature addition | New feature tests + adjacent smoke |
| Refactoring | Full unit suite + integration smoke |
| Dependency update | Full regression suite |
| Infrastructure change | Integration + E2E suite |

## Suite Evolution (Tiering)

Tests move between tiers as the system and risk profile change:

| Tier | Cadence | Contents | Promotion / Retirement |
|------|---------|----------|----------------------|
| **Tier 0 — Smoke** | Every PR (< 5 min) | P0 paths, build sanity | Promote from Tier 1 if a path becomes critical |
| **Tier 1 — Core** | Every merge (< 30 min) | P0/P1 regression, contracts | Promote from Tier 2 on rising failure rate |
| **Tier 2 — Extended** | Nightly | P2, slower integration, E2E | Retire tests with 0 failures in 6 months + no risk coverage |
| **Tier 3 — Archive** | On demand | Retired, legacy | Restore if a related defect escapes |

**Evolution rule:** A test that fails in production-adjacent tiers gets promoted; a test with zero failures over 6 months and no P0/P1 risk coverage is a retirement candidate (review, don't auto-delete).

## Shift-Right Feedback Loops

Production signals drive regression suite composition — closing the loop between what escapes and what you test.

| Signal | Suite Action |
|--------|-------------|
| **Production error spike** (observability/APM) | Add a regression test reproducing the failing condition; promote related tests to Tier 1 |
| **Canary analysis failure** | The canary check becomes a permanent regression assertion; review why left-side testing missed it |
| **Escaped defect (customer report)** | Root-cause: add the missing test, then audit the tier that should have caught it |
| **Feature-flag rollout** | Flag-gated code gets targeted regression selection while flagged; full coverage before flag removal |

**Loop discipline:** Every escaped defect gets a post-mortem question — "which tier should have caught this, and why didn't it?" The answer drives suite evolution, not blame.

## Flaky Test Discipline: Rerun-Once, Never-Twice

> **Protocol:** In a blocking regression run, a failed test may be rerun **exactly once**. If it passes on the single rerun, record it as a **flake** (not a pass) and route to quarantine. If it fails the rerun, it is a **real failure** — block. **Never rerun a second time.**

Why never twice: a second rerun turns flake-masking into a habit. If a test needs two retries to pass, it is not reliable enough to gate on, and repeated retries hide a rising flake rate until CI trust collapses.

| Outcome | Classification | Action |
|---------|---------------|--------|
| Pass on first run | Pass | Record green |
| Fail, then pass on 1st rerun | **Flake** | Do NOT count as pass; open quarantine issue; see [test-automation.md](./test-automation.md) |
| Fail, fail on 1st rerun | **Real failure** | Block the pipeline |
| Needs 2nd rerun to pass | Unreliable | Treat as flake; quarantine immediately |

Track flake rate (flaky outcomes / total runs). Sustain at < 2%; above that, invest in stabilization before adding more tests.

> **Gotcha — Ignoring flakes:** A "pass on retry" is not a pass. Flakes are latent failures; a suite with 5% flakiness on 10,000 daily tests produces ~500 false signals/day and erodes trust until teams bypass automation entirely.

## Suite Hygiene

| Condition | Action |
|-----------|--------|
| Test hasn't failed in 6 months + no risk coverage | Review for archive |
| Test flakes > 2% over 50 runs | Quarantine and fix |
| Test takes > 5s (unit) / > 30s (integration) | Optimize or promote to slower tier |
| Test depends on another test's state | Fix isolation (tests must be independent) |
| Test runs against production data | Switch to synthetic fixtures (see [test-data-management.md](./test-data-management.md)) |

## Composition Links

- Impact-analysis tooling and ML selection: [test-automation.md](./test-automation.md)
- Risk-based tier prioritization (P0–P3): [test-strategy.md](./test-strategy.md) and [risk-based-testing.md](./risk-based-testing.md)
- Flaky quarantine workflow: [test-automation.md](./test-automation.md)
- Quality gates and metrics: [quality-gates-and-metrics.md](./quality-gates-and-metrics.md)

---

*Sources: Ekstazi (Gligoric et al.), Launchable (launchableinc.com), DORA State of DevOps Reports, Predić et al. arXiv:2106.13891 (2021).*

# Risk-Based Testing

Prioritize test effort by risk. Load when deciding where to invest limited testing time, building a risk register, or running a risk assessment workshop. For the broader strategy context (pyramid shape, shift-left/right), see [test-strategy.md](./test-strategy.md).

## Risk = Probability × Impact

Every test decision is a risk decision made implicitly. Make it explicit:

```
Risk Score = Probability (1–5) × Impact (1–5)
```

| Score Range | Zone | Action |
|-------------|------|--------|
| 20–25 | **Critical** (P0) | Test exhaustively; every path, every edge case |
| 12–19 | **High** (P1) | Test all happy paths + known failure modes |
| 6–11 | **Medium** (P2) | Test happy paths + common failure modes |
| 1–5 | **Low** (P3) | Smoke test only; defer detailed testing |

These priority tiers (P0–P3) align with the coverage tiers in [test-strategy.md](./test-strategy.md). A P0 risk item demands P0-level coverage; a P3 risk item justifies smoke-only testing.

## 5×5 Risk Matrix

| P \ I | 1 — Negligible | 2 — Minor | 3 — Moderate | 4 — Major | 5 — Catastrophic |
|-------|----------------|-----------|--------------|-----------|-------------------|
| **5 — Almost Certain** | 5 (P3) | 10 (P2) | 15 (P1) | 20 (P0) | 25 (P0) |
| **4 — Likely** | 4 (P3) | 8 (P2) | 12 (P1) | 16 (P0) | 20 (P0) |
| **3 — Possible** | 3 (P3) | 6 (P2) | 9 (P2) | 12 (P1) | 15 (P1) |
| **2 — Unlikely** | 2 (P3) | 4 (P3) | 6 (P2) | 8 (P2) | 10 (P2) |
| **1 — Rare** | 1 (P3) | 2 (P3) | 3 (P3) | 4 (P3) | 5 (P3) |

### Scoring Guidance

| Rating | Probability Anchor | Impact Anchor |
|--------|--------------------|---------------|
| 5 | Will fail in production within a quarter (or has already) | Data loss, security breach, revenue stoppage |
| 4 | Expected to fail within a year | Major feature outage, SLA breach |
| 3 | Could fail; uncertain | Degraded experience, workaround exists |
| 2 | Unlikely given current controls | Cosmetic, minor inconvenience |
| 1 | Extremely unlikely; well-understood code | No user-visible impact |

## Risk Assessment Workshop

Run a structured workshop to score risks collaboratively. Solo scoring introduces individual bias; group calibration produces defensible priorities.

### Participants

- QA lead (facilitator)
- Engineering leads for affected areas
- Product manager (impact calibration)
- Operations / SRE representative (production context)

### Agenda (90 minutes)

| Time | Activity |
|------|----------|
| 0–15 min | Identify risk items: what could go wrong? (brainstorm from change log, incident history, architecture) |
| 15–50 min | Score each item: probability (group vote, median), impact (product calibrates) |
| 50–65 min | Rank and assign priority tiers (P0–P3) from the matrix |
| 65–80 min | Define mitigations: what tests, who owns them, by when |
| 80–90 min | Agree reassessment triggers and next review date |

### Calibration Rule

If probability votes span > 2 points, the facilitator asks the highest and lowest voter to state their evidence. Re-vote once. Record dissent in the register.

## Risk Register Structure

| Column | Description | Example |
|--------|-------------|---------|
| ID | Unique identifier | RISK-012 |
| Risk Description | What could go wrong | Payment gateway timeout during peak |
| Component | Affected system area | Checkout service |
| Probability (1–5) | Likelihood of occurrence | 4 |
| Impact (1–5) | Severity if it occurs | 5 |
| Score | P × I | 20 |
| Priority Tier | P0–P3 (from matrix) | P0 |
| Mitigation / Test Plan | What testing addresses this | Load test at 2× peak; chaos inject timeout |
| Owner | Who implements the mitigation | QA-2 |
| Status | Open / Mitigating / Closed | Mitigating |
| Last Reviewed | Date of last reassessment | 2025-07-15 |
| Reassessment Trigger | What event re-opens this | Payment provider API change |

## Reassessment Triggers

Risk is not static. Re-score the register when any trigger fires:

| Trigger | Rationale |
|---------|-----------|
| Production incident in the area | Actual failure updates probability upward |
| Architecture change (new dependency, refactor) | Changes both probability and impact landscape |
| New regulatory requirement | May raise impact (compliance penalty) |
| Major release or migration | New failure modes introduced |
| Quarterly calendar review | Prevents register staleness (default cadence) |
| Team change (key engineer leaves) | Knowledge gaps raise probability |
| Customer escalation | Business impact may have changed |

## Cost-of-Failure Reasoning

Risk-based testing investment is justified by the cost differential between catching a defect early vs late:

| Detection Phase | Relative Fix Cost | Risk-Based Justification |
|----------------|-------------------|--------------------------|
| Design / Spec review | 1× | Highest-leverage test: risk workshop catches design flaws |
| Unit / PR testing | 5–10× | P0/P1 items justify exhaustive unit coverage |
| Integration / Staging | 20–50× | Contract and integration tests for cross-boundary risks |
| Production | 100×+ | Shift-right monitoring for residual P0 risk |

**Decision rule:** Allocate test effort proportional to risk score. A P0 item (score 20–25) receives 3–5× the per-item test design budget of a P3 item (score 1–5).

## Test Estimation Heuristic

Estimate test effort from the risk register:

```
total_test_hours = Σ (risk_items_in_tier × hours_per_tier)

Hours per tier (default):
  P0: 8–16 hours per risk item (exhaustive design + automation)
  P1: 4–8 hours per risk item
  P2: 2–4 hours per risk item
  P3: 0.5–1 hour per risk item (smoke only)
```

**Adjustment factors:** multiply by 1.5× for legacy/unfamiliar code, 0.7× for well-automated areas with existing coverage.

### Worked Example

A release has 3 P0 risks, 5 P1 risks, 8 P2 risks, and 12 P3 risks:

| Tier | Items | Hours/Item | Subtotal |
|------|-------|-----------|----------|
| P0 | 3 | 12 | 36 |
| P1 | 5 | 6 | 30 |
| P2 | 8 | 3 | 24 |
| P3 | 12 | 0.75 | 9 |
| **Total** | **28** | — | **99 hours** |

With a 2-person QA team (80 hours/sprint), this release requires ~1.25 sprints of test design effort. Negotiate scope or add capacity for P0 items; P3 items can be deferred.

## Gotchas

> **Gotcha — Static register:** A risk register written once at project start and never updated is fiction. Reassess on triggers (above) and at minimum quarterly. A stale register misallocates effort toward risks that no longer exist.

> **Gotcha — Consensus theater:** If the workshop rubber-stamps the loudest voice's scores without evidence, the register is political, not analytical. Require evidence anchors for every score. Record dissent.

> **Gotcha — Full suite on every PR regardless of risk:** Running everything on every change wastes CI minutes and trains teams to ignore results. Use risk tiers to gate test selection (P0 always runs; P3 runs nightly).

## Exit Condition

You are done applying this reference when: (1) a risk register exists with scored items mapped to P0–P3 tiers, (2) test allocation is proportional to risk scores, (3) reassessment triggers are defined with a calendar backstop, and (4) the estimation heuristic produces a capacity-checked plan.

## Composition Links

- Broader test strategy and pyramid shape: [test-strategy.md](./test-strategy.md)
- Regression suite tiering by risk: [regression-testing.md](./regression-testing.md)
- Quality gate design (blocking vs advisory per tier): [quality-gates-and-metrics.md](./quality-gates-and-metrics.md)
- Verification planning and evidence standards: [verification-methodology](../../verification-methodology/SKILL.md)

---

*Sources: ISO/IEC 25010 (systems and software quality requirements), ISTQB Foundation Level Syllabus 2023 (risk-based testing chapter), James Bach (context-driven testing, risk heuristics), Kaner/Bach/Pettichord "Lessons Learned in Software Testing" (Wiley, 2002), DORA State of DevOps Reports (cost-of-failure data).*

# SDET Engineering

## Definition and Distinction

An SDET (Software Development Engineer in Test) is an engineer whose **product is test infrastructure** — frameworks, tooling, pipelines, and platforms that enable the entire organization to verify software quality efficiently.

What an SDET is NOT:
- **Not "a tester who codes":** A manual tester who learned Selenium is not an SDET. Writing scripts that automate existing manual steps produces brittle, low-value automation.
- **Not "a developer who tests":** A developer who writes unit tests for their own code is practicing good development hygiene, not building test infrastructure for others.

The SDET's customers are other engineers. Success is measured by how effectively the organization can detect defects, not by how many tests the SDET personally writes.

## Competency Model: 7 Habits (Angie Jones)

| # | Habit | Core Practice |
|---|-------|--------------|
| 1 | **Be intentional** | Automate selectively aligned to goals, not "all the things" |
| 2 | **Enhance development skills** | OOP, design patterns, clean code — not just API syntax |
| 3 | **Enhance testing skills** | Balance developer and tester mindsets; verify behavior, not just execution |
| 4 | **Explore new tools** | Match tools to contexts; never force one tool on every problem |
| 5 | **Automate throughout the tech stack** | Use seams at unit/service/API layers; UI automation sparingly |
| 6 | **Collaborate** | Strategy requires input from exploratory testers, developers, and product |
| 7 | **Automate beyond the tests** | Data generation, environment setup, log parsing — remove repetitive toil |

Source: Angie Jones, "7 Habits of Highly Effective SDETs" (angiejones.tech, 2018).

## gTAA / TAF Layered Architecture

The generic Test Automation Architecture (gTAA) organizes test infrastructure as layers with clear responsibilities:

```
┌─────────────────────────────────────────┐
│  Layer 5: Test Reporting & Analytics    │  Dashboards, trend analysis, flake metrics
├─────────────────────────────────────────┤
│  Layer 4: Test Execution & Orchestration│  CI runners, parallelism, sharding, retry policy
├─────────────────────────────────────────┤
│  Layer 3: Test Scripts / Scenarios      │  Business-readable test cases, data-driven flows
├─────────────────────────────────────────┤
│  Layer 2: Test Services / Utilities     │  API clients, page objects, data factories, auth helpers
├─────────────────────────────────────────┤
│  Layer 1: Core Framework / Adapters     │  Driver management, config, logging, plugin system
└─────────────────────────────────────────┘
```

**Key principle:** Higher layers depend on lower layers, never the reverse. Test scripts (L3) never import driver internals (L1) directly.

## Design Patterns for Test Code

### Page Object Model (POM)

Encapsulates UI structure behind a semantic interface. Tests interact with page meaning, not selectors.

```python
class CheckoutPage:
    def __init__(self, page):
        self._page = page
        self._submit = page.locator('[data-testid="checkout-submit"]')

    def complete_order(self, card: str) -> OrderConfirmation:
        self._page.fill('[name="card"]', card)
        self._submit.click()
        return OrderConfirmation(self._page)
```

### Flow Model

Models multi-page user journeys as composable transitions. Each step returns the next page state, enabling type-safe navigation chains.

```python
confirmation = (
    HomePage(page)
    .search("widget")
    .select_result(0)
    .add_to_cart()
    .checkout(card="4111...")
)
assert confirmation.is_successful()
```

Flow Model complements POM: POM encapsulates single pages; Flow Model composes them into end-to-end journeys.

### SOLID Applied to Test Code

| Principle | Test Code Application |
|-----------|----------------------|
| **S**ingle Responsibility | One test class per feature area; one assertion concept per test |
| **O**pen/Closed | Extend test data via fixtures, not by modifying shared helpers |
| **L**iskov Substitution | Any test-double must be swappable for the real dependency without test changes |
| **I**nterface Segregation | Page objects expose only methods relevant to their page (no god-objects) |
| **D**ependency Inversion | Tests depend on abstractions (interfaces), not concrete driver implementations |

## Build vs. Buy Decision Framework

### Decision Criteria

| Criterion | Favors BUILD | Favors BUY/ADOPT |
|-----------|-------------|-----------------|
| Team size | > 5 SDETs maintaining infra | < 3 engineers |
| Longevity | Custom needs persist 2+ years | Needs may shift within 12 months |
| Integration surface | Deep internal system hooks (custom protocols) | Standard web/API/mobile |
| Maintenance cost tolerance | Org can fund ongoing maintenance | Prefer vendor/community maintenance |
| TCO (3-year) | Custom amortizes below commercial | Commercial license < build + maintain |

### Decision Table

| Scenario | Recommendation |
|----------|---------------|
| Standard web E2E, < 5 engineers | Adopt Playwright/Cypress |
| Custom protocol (IoT, proprietary binary) | Build adapter layer atop open framework |
| High-scale parallelism + custom reporting at > 20 teams | Build orchestration; adopt execution engines |
| Mobile-only, small team | Adopt Appium/Detox |
| Uncertain requirements, < 12 months horizon | Adopt; revisit when needs stabilize |

> **Gotcha — NIH syndrome:** Building a custom framework "because none fit perfectly" when an existing tool covers 90% of needs wastes months. Extend, don't replace.

## Testability Engineering

### Designing Systems for Testability

| Technique | Mechanism | Example |
|-----------|-----------|---------|
| **Dependency Injection** | Inject collaborators via constructor/parameter | `OrderService(repo: Repository, clock: Clock)` |
| **Architectural seams** | Boundaries where behavior can be altered without editing | Interface between service and external gateway |
| **Observability hooks** | Expose internal state for verification | Health endpoints, debug headers, structured logs |

### Test-Double Selection Criteria

| Double Type | Use When | Avoid When |
|------------|----------|-----------|
| **Stub** | You need canned return values; no interaction verification | You need to verify call sequences |
| **Mock** | Verifying interactions (was X called N times?) | Over-mocking creates brittle coupling to implementation |
| **Fake** | You need working behavior without external cost (in-memory DB, local SMTP) | Behavior diverges from production over time |

**Selection rule:** Default to stubs for data, fakes for stateful collaborators, mocks only for interaction-critical boundaries. Never mock value objects.

## CI/CD Integration

### Multi-Level Pipeline Architecture

| Level | Trigger | Contents | Time Budget |
|-------|---------|----------|-------------|
| **L1: Pre-merge** | Every PR | Unit + fast integration + lint + type-check | < 5 min |
| **L2: Post-merge** | Merge to main | Full integration + contract tests + E2E smoke | < 15 min |
| **L3: Scheduled** | Nightly / weekly | Full E2E + performance + soak + mutation | < 60 min |

### Configuration Management

Test configuration (URLs, credentials, feature flags) lives in environment-specific config, never hardcoded. Use layered config: defaults → environment overrides → CI secrets.

### Contract Testing

Consumer-driven contracts (e.g., Pact) verify service boundaries independently of full integration:
- Consumer defines expectations → publishes contract
- Provider verifies against contract in its own CI
- Breaks are caught before deployment, not during integration testing

## Test Data and Environment Self-Service

| Capability | Implementation Pattern |
|-----------|----------------------|
| **Data on demand** | Factory/builder functions generating valid entities per test |
| **Environment provisioning** | Ephemeral environments via containers (Docker Compose, k8s namespaces) |
| **State isolation** | Each test owns its data; no shared mutable database state |
| **Self-service portal** | Engineers spin up test environments without SRE ticket |

## Observability and Shift-Right

| Technique | QA Application |
|-----------|---------------|
| **Correlation IDs** | Trace a user journey across microservices; reproduce failures from production traces |
| **Canary releases** | Deploy to 1–5% traffic; monitor error rates before full rollout; auto-rollback on SLO breach |
| **Feature flags** | Gate risky features; enable targeted regression testing in production; kill-switch without deploy |

Shift-right does not replace shift-left. It validates that pre-merge testing caught what matters, and feeds escaped-defect data back into suite evolution.

## Flakiness and Reliability Engineering

**Core principle: Test code is production code.** It deserves the same review, ownership, and SLA expectations as application code.

### Flakiness Triage (Flakinator-Style)

| Step | Action |
|------|--------|
| 1. Detect | Statistical flake scoring: Bayesian analysis of pass/fail patterns over N runs |
| 2. Classify | Root cause category: timing/race, resource contention, test-order dependency, external service |
| 3. Quarantine | Remove from blocking path; track in dashboard with owner and SLA |
| 4. Fix or delete | Owner resolves within 5 business days or deletes the test |
| 5. Burn-in | 20+ consecutive green runs before re-enabling as blocking |

### The ~18-Month Decay Rule

Test suites without active maintenance decay: flake rates climb, false confidence accumulates, and developer trust erodes. Budget ~20% of test infrastructure capacity for ongoing maintenance. If an organization cannot sustain this, adopt fewer, higher-value tests rather than a large neglected suite.

Source: Google Testing Blog, "Flaky Tests at Google and How We Mitigate Them" (2016); Atlassian Engineering, "Taming Test Flakiness with Flakinator" (2025).

## Career Progression

| Stage | Focus | Scope |
|-------|-------|-------|
| Junior SDET | Learn framework, write tests under guidance | Task |
| Senior SDET | Own a product area's test infrastructure | Project |
| Staff SDET | Set test architecture standards across teams | Product |
| Principal SDET | Multi-year QE vision; industry contribution | Org |

For detailed leveling mechanics, promotion packets, and archetypes, see [qa-career-levels.md](./qa-career-levels.md).

## Emerging AI Dimensions

| Dimension | Current State (2025–2026) | QA Implication |
|-----------|--------------------------|----------------|
| **Self-healing tests** | Tools auto-update selectors on UI changes (Healenium, Applitools) | Reduces maintenance burden but masks real UI regressions if unchecked |
| **AI log analysis** | LLM-assisted root-cause analysis of CI failures | Accelerates triage; requires validation against deterministic signals |
| **Agentic testing pyramids** | AI agents generate and execute test scenarios autonomously | QA role shifts to strategy, oracle design, and verifying agent-generated test quality |

> **Gotcha — AI-generated tests without oracle verification:** An agent that generates 500 tests asserting nothing is worse than 50 well-designed tests. Always verify that AI-generated tests have meaningful assertions and kill mutants.

## Decision Table: SDET Scope Choices

| Question | If YES | If NO |
|----------|--------|-------|
| Is the team > 5 engineers maintaining test infra? | Invest in layered gTAA | Adopt existing framework directly |
| Does the system have custom protocols? | Build adapter layer | Use standard tool |
| Are flake rates > 5%? | Prioritize reliability engineering over new tests | Continue balanced investment |
| Is AI-generated code > 50% of PRs? | Add mutation testing + independent verification gates | Standard review sufficient |

**Exit condition:** You are done applying this reference when you can identify the appropriate gTAA layers for your system, make a build-vs-buy recommendation with documented criteria, and establish a flake-management SLA for your team's test suite.

## Worked Example: Build vs. Buy for an API-First Startup

**Context:** 8-person startup, 3 backend services, REST + gRPC, no dedicated QA. Team needs E2E confidence.

| Criterion | Assessment |
|-----------|-----------|
| Team size | 3 engineers touching tests → favors BUY |
| Integration surface | Standard REST + gRPC → no custom protocol |
| Longevity | Product-market fit uncertain; needs may pivot in 12 months |
| TCO | Playwright + pytest adoption: 2 weeks. Custom framework: 3 months + ongoing |

**Recommendation:** Adopt Playwright (API testing) + pytest (unit/integration). Add contract testing (Pact) at service boundaries when team reaches 12+ engineers. Revisit build-vs-buy at 2-year mark if custom orchestration needs emerge.

## Composition Links

- Career levels, promotion packets, archetypes: [qa-career-levels.md](./qa-career-levels.md)
- Test automation patterns (parallelism, ML selection): [test-automation.md](./test-automation.md)
- Flaky quarantine workflow details: [test-automation.md](./test-automation.md)
- Quality metrics and gate design: [quality-gates-and-metrics.md](./quality-gates-and-metrics.md)

---

*Sources: Angie Jones, "7 Habits of Highly Effective SDETs" (angiejones.tech, 2018); Google Testing Blog, "Flaky Tests at Google" (testing.googleblog.com, 2016); Atlassian Engineering, "Taming Test Flakiness with Flakinator" (2025); Lisa Crispin & Janet Gregory, Agile Testing (2009); Gerard Meszaros, xUnit Test Patterns (2007); Pact Foundation (docs.pact.io); Will Larson, Staff Engineer (2020).*

# Security Testing

## Test Types by Phase

| Phase | Test Type | Tool Examples | Frequency |
|-------|-----------|---------------|-----------|
| Pre-commit | Secret scanning | gitleaks, trufflehog | Every commit |
| CI | SAST (static analysis) | Semgrep, CodeQL, bandit (Python), eslint-plugin-security | Every PR |
| CI | SCA (dependency audit) | `pip-audit`, `npm audit`, Dependabot, Snyk, Trivy | Every PR + daily |
| CI | Container scanning | Trivy, Grype | Every image build |
| Staging | DAST (dynamic) | OWASP ZAP, Burp Suite | Nightly or pre-release |
| Staging | API fuzzing | Schemathesis (OpenAPI), RESTler | Weekly |
| Pre-release | Pen test (manual) | External firm or red team | Quarterly / major release |

## OWASP Top 10:2025

The OWASP Top 10:2025 is the current standard awareness document. The 2025 edition restructured categories significantly — notably adding Software Supply Chain Failures (A03) and Mishandling of Exceptional Conditions (A10) as new entries.

| ID | Category | What to Test | How |
|----|----------|-------------|-----|
| A01 | Broken Access Control | IDOR, privilege escalation, path traversal | Access other users' resources by ID; test admin endpoints as regular user; fuzz path parameters with `../` |
| A02 | Security Misconfiguration | Default creds, verbose errors, open ports, debug endpoints | Banner grab; check `/debug`, `/admin`, `.env` exposure; verify CORS headers |
| A03 | Software Supply Chain Failures | Compromised dependencies, typosquatting, unsigned artifacts | SCA scans (Trivy, Snyk); verify lockfile integrity; check SBOM against known advisories |
| A04 | Cryptographic Failures | Weak algorithms, plaintext secrets, PII in logs | Grep logs for email/SSN patterns; verify TLS 1.2+ everywhere; audit key management |
| A05 | Injection | SQL, command, LDAP, XSS (reflected/stored/DOM) | Parameterized query audit; fuzz with `' OR 1=1`, `; rm -rf`, `<script>alert(1)</script>` |
| A06 | Insecure Design | Business logic flaws, missing rate limits, predictable IDs | Threat model review; abuse-case testing; verify rate limiting on sensitive operations |
| A07 | Authentication Failures | Session fixation, token expiry, brute force, credential stuffing | Attempt reuse of expired tokens; test rate limiting; verify MFA enforcement |
| A08 | Software or Data Integrity Failures | Unsigned updates, insecure deserialization, CI/CD pipeline tampering | Audit all `loads()`/`unserialize()` calls; verify artifact signatures; check auto-update channels |
| A09 | Security Logging and Alerting Failures | Missing audit trail, unmonitored auth failures, log injection | Verify login/logout/privilege-change events are logged; test that logs don't accept unescaped input |
| A10 | Mishandling of Exceptional Conditions | Verbose stack traces, fail-open on error, unhandled edge cases | Trigger error paths (timeout, malformed input, resource exhaustion); verify safe fallback behavior |

### Gotcha: Outdated OWASP Editions

The 2017 edition used different category names and groupings. Categories that existed as standalone entries in 2017 (such as data-exposure and XML-entity risks) are now subsumed under A04 (Cryptographic Failures) and A08 (Software or Data Integrity Failures) respectively. Do not build test plans around the retired 2017 taxonomy. Always cite the 2025 edition.

## Supply Chain Security and SBOM

Software supply chain attacks (A03) target the build and dependency pipeline rather than application code directly.

### SBOM (Software Bill of Materials)

| Tool | Format | Use |
|------|--------|-----|
| Syft | SPDX, CycloneDX | Generate SBOM from images, filesystems, archives |
| Trivy | CycloneDX | Generate SBOM + scan for vulnerabilities in one pass |
| `npm sbom` / `pip-audit --sbom` | SPDX / CycloneDX | Language-native SBOM generation |

### Supply Chain Controls

| Control | Implementation |
|---------|---------------|
| Dependency pinning | Lockfiles (`package-lock.json`, `uv.lock`, `Gemfile.lock`) committed to VCS |
| Artifact signing | Sigstore/cosign for container images; npm provenance for packages |
| CI/CD pipeline integrity | Branch protection, required reviews, pinned action SHAs (not `@main`) |
| Typosquat detection | Automated name-similarity checks on new dependencies |
| SBOM attestation | Generate and store SBOM per build; scan against advisory databases (OSV, NVD) |

### Container Security

```bash
# Scan image for OS + language CVEs and generate SBOM
trivy image --severity HIGH,CRITICAL myapp:latest
trivy sbom --output sbom.cdx.json myapp:latest

# Fail CI on critical findings
trivy image --exit-code 1 --severity CRITICAL myapp:latest
```

- Use distroless or alpine base images (smaller attack surface)
- Run as non-root (`USER 1000` in Dockerfile)
- No secrets in image layers — use runtime injection (Vault, SSM, k8s secrets)

## STRIDE Threat Modeling

STRIDE classifies threats by the property they violate. Use it during design review to enumerate attack surfaces before writing code.

| Threat | Property Violated | Example | Mitigation Pattern |
|--------|-------------------|---------|-------------------|
| **S**poofing | Authentication | Forged JWT, stolen session cookie | MFA, token binding, short expiry |
| **T**ampering | Integrity | Modified request body, altered DB record | HMAC signatures, input validation, audit logging |
| **R**epudiation | Non-repudiation | User denies performing action | Immutable audit logs with timestamps and user identity |
| **I**nformation Disclosure | Confidentiality | Verbose error leaks stack trace, PII in logs | Error sanitization, log redaction, encryption at rest |
| **D**enial of Service | Availability | Resource exhaustion, slowloris | Rate limiting, circuit breakers, autoscaling |
| **E**levation of Privilege | Authorization | IDOR grants admin access, role bypass | Least privilege, server-side authorization checks |

### When to Threat Model

- New service or API endpoint design
- Authentication or authorization flow changes
- Data flow changes (new storage, new integration)
- Before a major release (quarterly review cadence)

## SAST / DAST / SCA Tool Landscape

| Category | What It Does | Tools | When |
|----------|-------------|-------|------|
| **SAST** (Static Application Security Testing) | Analyzes source code for vulnerability patterns without executing | Semgrep, CodeQL, bandit, SonarQube, Checkmarx | Every PR — fast feedback |
| **DAST** (Dynamic Application Security Testing) | Attacks the running application from outside | OWASP ZAP, Burp Suite, Nuclei | Staging — needs a deployed instance |
| **SCA** (Software Composition Analysis) | Identifies vulnerable third-party dependencies and license issues | Snyk, Dependabot, Trivy, `pip-audit`, `npm audit` | Every PR + daily scheduled |

### SAST in CI (Semgrep Example)

```yaml
# .github/workflows/security.yml
- name: Semgrep
  uses: semgrep/semgrep-action@v1
  with:
    config: >-
      p/owasp-top-ten
      p/python
      p/security-audit
```

## Dependency Audit Discipline

| Severity | Action |
|----------|--------|
| Critical / High | Block merge; fix immediately |
| Medium | Create issue; fix within sprint |
| Low | Batch into maintenance window |

- Pin transitive deps with lockfiles
- Review Dependabot PRs weekly — don't let them accumulate

## Security Test Data

- Never test with real PII — use synthetic data (see [test-data-management.md](./test-data-management.md))
- Credential testing uses obviously-fake values: `AKIAIOSFODNN7EXAMPLE`
- If a test discovers a real vulnerability, stop and report — don't commit exploit code

## Gotchas

- **Outdated OWASP references** (2017 categories) produce misaligned test plans. Always use the 2025 edition.
- **SAST without DAST** misses runtime-only vulnerabilities (misconfiguration, CORS, auth bypass). Run both.
- **Ignoring supply chain** — most breaches in 2024–2025 exploited dependencies, not application code. SCA and SBOM are not optional.
- **Secret scanning only at pre-commit** misses secrets already in history. Run a full-history scan on onboarding and quarterly.

## Composition

- Deep secure-engineering lifecycle (secure design review, threat modeling workshops, incident response): [secure-software-engineering](../../secure-software-engineering/SKILL.md)
- Systematic debugging of security-related test failures: [systematic-debugging](../../systematic-debugging/SKILL.md)

*Sources: OWASP Top 10:2025 (owasp.org/Top10/2025), STRIDE (Microsoft Security Development Lifecycle), CycloneDX (OWASP), SLSA supply chain framework (slsa.dev), Semgrep docs (semgrep.dev), Trivy docs (trivy.dev), NIST SP 800-218 (SSDF).*

# Test Automation

## Framework Decision Matrix

| Criteria | pytest | Playwright | Vitest | Cypress |
|----------|--------|-----------|--------|---------|
| **Language** | Python | JS/TS, Python, .NET, Java | JS/TS (Vite-based) | JS/TS |
| **Primary Domain** | Unit, integration, API | E2E browser, mobile | Unit, component, E2E | E2E browser, component |
| **Browser Support** | N/A | Chromium, Firefox, WebKit | Via Playwright browser mode | Chromium, Firefox, Edge |
| **Parallelism** | pytest-xdist | Built-in workers + sharding | Built-in pool + sharding | Dashboard parallelization (paid) |
| **Auto-wait** | N/A | Yes (built-in) | N/A (VDOM assertions) | Yes (retry-ability) |
| **Network Mocking** | responses / pytest-httpx | route() API | vi.mock / msw | cy.intercept() |
| **Debugging** | pdb / --pdb | Trace viewer, video | Browser DevTools | Time-travel, snapshots |
| **CI-first?** | Yes | Yes (blob reports, sharding) | Yes (sharding, pool) | Dashboard-based |
| **Best For** | Python projects, data/API | Multi-browser E2E | Vite/React/Vue component + unit | Dev-integrated E2E |

### Selection Flow

```
Is the project Python?
  YES → pytest (with xdist for parallelism)
  NO  → Is it Vite-based?
          YES → Unit/component: Vitest | E2E: Playwright
          NO  → Playwright (E2E) + Jest or Vitest (unit)
```

> **Gotcha — "Automate everything":** Automating a bad test design just makes failures faster. Invest in test design (see [test-design-techniques.md](./test-design-techniques.md)) before scaling automation.

## Parallelism and Sharding

### Three Levels

| Level | What | Tooling |
|-------|------|---------|
| Within a job (multi-worker) | Multiple tests on one machine | `pytest -n auto`, Playwright workers, Vitest pool |
| Across jobs (sharding) | Suite split into N groups, each on a CI runner | `--shard=x/y`, matrix strategy |
| Across suites | Different test types in separate CI jobs | CI workflow orchestration |

### Configuration Example (GitHub Actions + Playwright)

```yaml
jobs:
  e2e:
    strategy:
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v4
      - run: npx playwright test --shard=${{ matrix.shard }}/4
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: blob-report-${{ matrix.shard }}
          path: blob-report

  merge-reports:
    if: always()
    needs: [e2e]
    steps:
      - run: npx playwright merge-reports --reporter html ./all-blob-reports
```

### pytest Splitting

```bash
# Within-node parallelism
pytest -n auto --dist worksteal   # dynamic rebalancing (xdist 3.x+)

# Across CI jobs (timing-balanced)
pytest --splits 4 --group ${{ matrix.group }} --store-durations
```

### Shard Balancing Tips

- Use `fullyParallel: true` (Playwright) for test-level distribution
- Log timing data (`--store-durations`) to balance groups by historical duration
- Set job-level timeouts to prevent hung workers from blocking the pipeline
- Cache dependencies between shards (node_modules, pip packages)

## ML / Predictive Test Selection

When suites exceed 10,000 tests, full-run-on-every-PR becomes impractical. Predictive selection uses historical data to run only tests likely to be affected by a change.

| Approach | How It Works | Tools / Research |
|----------|-------------|-----------------|
| **Static call-graph analysis** | Map changed code → tests that exercise it (conservative) | Ekstazi, custom dependency graphs |
| **ML-predicted relevance** | Train on historical (change, test-outcome) pairs; predict which tests to run | Launchable, Microsoft Research |
| **Failure-rate weighting** | Prioritize tests with high historical failure rate on similar changes | Custom scoring (see selection math in [regression-testing.md](./regression-testing.md)) |

**Key research:** Predić et al. (arXiv:2106.13891, 2021) demonstrated that ML-based test selection reduces CI runtime by 50–90% while catching >99% of failures that full-suite execution would catch. Launchable (commercial) and Microsoft's internal systems use similar approaches at scale.

**Adoption guidance:**
- < 1,000 tests: full suite on every PR (keep it simple)
- 1,000–10,000 tests: sharding + timing-based splitting
- > 10,000 tests: predictive selection (Launchable, custom ML) with full-suite nightly as safety net

> **Gotcha — Selection without safety net:** Never rely solely on predicted selection. Run the full suite on a schedule (nightly or on main-branch merge) to catch false negatives in the prediction model.

## Flaky Test Quarantine Workflow

### Detection → Quarantine → Fix → Reintegrate

```
1. DETECT: Test fails on retry but passes on re-run (flaky signal)
2. TAG:    Mark with @quarantine / skip annotation + tracking issue
3. ISOLATE: Move to separate CI job that runs but does NOT block the pipeline
4. TRACK:  Dashboard showing quarantine count, age, owner
5. FIX:    Owner has 5 business days to stabilize or delete
6. REINTEGRATE: Remove quarantine tag; burn-in 20+ green runs before re-blocking
```

### Quarantine Criteria

| Signal | Threshold | Action |
|--------|-----------|--------|
| Flake rate (failures / runs) | > 2% over 50 runs | Quarantine |
| Blocks PR pipeline | > 3 times in one week | Quarantine immediately |
| Age in quarantine | > 10 days unfixed | Escalate to team lead; consider deletion |

### Stabilization Patterns

| Root Cause | Fix |
|-----------|-----|
| Race condition / timing | Explicit waits on observable state, never `sleep()` |
| Shared mutable state | Isolated fixtures, rollback after each test |
| External dependency | Mock/stub at the boundary |
| Random data collision | Seeded randomness or UUID-based test data |
| Test interdependence | `pytest-randomly` to expose ordering issues |

## Mutation-Guided Test Hardening

Mutation-guided test hardening is targeted review evidence, not a mutation engine, portable report format, or score-optimization exercise. Coverage and mutation effectiveness answer different questions: line/branch coverage says whether execution reached code; mutation analysis asks whether tests detect plausible behavior-changing faults. Do not interpret a mutation score as a universal quality grade. It depends on operator quality, equivalent-mutant handling, scope, exclusions, timeouts, and denominator integrity.

### Tool Landscape

Use the project's established native tool and its raw report as authoritative. Confirm the installed version and supported configuration in the project's documentation; the table only identifies the official project documentation, not a promise of interchangeable features.

| Tool | Primary ecosystem | Official source |
|------|-------------------|-----------------|
| PIT | Java and JVM mutation testing | [PIT project README](https://github.com/hcoles/pitest) · [pitest.org](https://pitest.org) |
| StrykerJS | JavaScript and TypeScript mutation testing | [StrykerJS project README](https://github.com/stryker-mutator/stryker-js) · [stryker-mutator.io](https://stryker-mutator.io/) |
| mutmut | Python mutation testing | [mutmut documentation](https://mutmut.readthedocs.io/en/latest/) |

### Bounded Review Loop

Use this loop when a changed behavior, review concern, or weak assertion warrants stronger evidence:

1. Choose changed lines/files or an explicitly justified risk slice. Broaden the slice for dependency, configuration, harness, environment, shared-abstraction, or cross-cutting changes.
2. Run the project's established native mutation tool. PIT, StrykerJS, and mutmut are examples of the existing landscape; use the project's supported tool and raw report as authoritative, without assuming undocumented versions or features.
3. Record an explicit mutant budget, timeout, seed, exclusions, operator set, test command, tool/version, and environment.
4. Classify every result at minimum as `killed`, `survived`, `equivalent/likely equivalent`, `no coverage`, `timeout`, `flaky`, `invalid`, or `infrastructure/tooling failure`.
5. Treat survivors as triage inputs, not automatic defects. Propose a focused behavior-level test only where the surviving fault is meaningful.
6. Check the candidate for behavioral relevance, non-tautology, non-vacuity, non-redundancy, maintainability, flake risk, and implementation coupling. A generated test may be valuable even when it does not increase the score; a killed mutant does not prove the test is good.
7. Independently rerun the baseline, candidate test, and exact retained mutant. The implementer cannot self-certify the generated test; use a fresh verifier or human.
8. Record raw-report locations, commands, classifications, uncertainty, and reviewer decision in [templates/mutation-review.md](../templates/mutation-review.md).

Define the denominator explicitly, for example `classified mutants / in-scope mutants`, and state how equivalent, excluded, unknown, incomplete, and infrastructure-failed mutants are accounted for. They must not silently disappear or inflate a clean result. Keep mutation use advisory and risk-based rather than an unconditional repository-wide gate or universal score threshold. A project may independently validate a narrower policy, but that policy needs its own evidence and review.

### Reproduction Sequence

```text
record base/head and scope -> run native tool with bounded settings
-> preserve raw report -> classify every mutant and denominator
-> propose behavior-level test for meaningful survivor
-> fresh verifier reruns baseline, candidate, exact mutant
-> attach commands, outputs, uncertainty, and decision
```

For review artifacts use [mutation-review.md](../templates/mutation-review.md). For independent evidence and evaluation boundaries, see [ai-code-quality-gates.md](./ai-code-quality-gates.md), [verification-methodology](../../verification-methodology/SKILL.md), and [agent-evals-and-observability](../../agent-evals-and-observability/SKILL.md).

## Composition Links

- Test design techniques (EP, BVA, pairwise): [test-design-techniques.md](./test-design-techniques.md)
- Regression suite management and selection math: [regression-testing.md](./regression-testing.md)
- Quality gates and metrics: [quality-gates-and-metrics.md](./quality-gates-and-metrics.md)
- Systematic debugging of test failures: [systematic-debugging](../../systematic-debugging/SKILL.md)

---

*Sources: Playwright docs (2025), pytest-xdist docs, Launchable (launchableinc.com), Predić et al. arXiv:2106.13891 (2021), PIT project README (github.com/hcoles/pitest; pitest.org), StrykerJS project README (github.com/stryker-mutator/stryker-js; stryker-mutator.io), mutmut documentation (mutmut.readthedocs.io), and ACH (arXiv:2501.12862).*

# Test Data Management

## Fixtures vs Factories

| Approach | When | Trade-off |
|----------|------|-----------|
| Static fixtures (JSON/YAML files) | Small, stable datasets; API contract tests | Brittle to schema changes, easy to read |
| Factory functions (factory_boy, fishery) | Relational data, many-to-many, randomized | Setup complexity, harder to debug |
| Builder pattern | Complex objects with many optional fields | Verbose but explicit |
| Inline construction | One-off tests, 1–3 fields | Doesn't scale, but zero indirection |

### Decision: Fixture or Factory?

| Signal | Choose |
|--------|--------|
| Data shape is stable and shared across tests | Fixture |
| Tests need variations (different users, orders) | Factory |
| Schema changes frequently | Factory (adapts to model changes) |
| You need reproducible exact values | Fixture |
| You need randomized edge cases | Factory + Faker |

### Factory Pattern (Python)

```python
# factories.py
import factory
from myapp.models import User, Order

class UserFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = User
    email = factory.Sequence(lambda n: f"user{n}@test.dev")
    name = factory.Faker("name")

class OrderFactory(factory.django.DjangoModelFactory):
    class Meta:
        model = Order
    user = factory.SubFactory(UserFactory)
    total = factory.Faker("pydecimal", min_value=1, max_value=500, right_digits=2)
```

## Test Isolation

| Strategy | Mechanism | Speed | Safety |
|----------|-----------|-------|--------|
| Transaction rollback | Wrap test in transaction, rollback after | Fast | High — no cross-test leakage |
| Database per test | Create/drop schema per test | Slow | Highest — full isolation |
| Truncate between tests | `TRUNCATE ... CASCADE` after each | Medium | High |
| Unique prefixes | Each test uses `test-{uuid}-` prefixed data | Fast | Medium — relies on discipline |

**Rule:** Prefer transaction rollback (pytest-django `@pytest.mark.django_db`, Rails `use_transactional_tests`). Fall back to truncation only when tests need committed state (e.g., testing triggers, background jobs).

## Time-Travel Testing

Tests that depend on the current time are flaky by construction. Freeze or mock the clock; never `sleep()`.

| Tool | Language | Mechanism |
|------|----------|-----------|
| freezegun | Python | `@freeze_time("2025-06-15")` decorator; patches `datetime`, `time.time()` |
| timecop | Ruby | `Timecop.freeze(Time.local(2025, 6, 15))` with block form |
| jest.useFakeTimers | JavaScript/TS | `jest.setSystemTime(new Date('2025-06-15'))` |
| Clock interface (DI) | Any | Inject a `Clock` abstraction; tests supply a `FixedClock` |

### Worked Example (freezegun)

```python
from freezegun import freeze_time

@freeze_time("2025-06-15 12:00:00")
def test_subscription_expiry():
    user = UserFactory(subscription_start=date(2025, 5, 15))
    assert user.subscription_expired is True  # 31 days > 30-day window

@freeze_time("2025-06-10 12:00:00")
def test_subscription_active():
    user = UserFactory(subscription_start=date(2025, 5, 15))
    assert user.subscription_expired is False  # 26 days < 30-day window
```

### Gotcha: Timezone-Aware Freezing

freezegun defaults to UTC. If your app uses timezone-aware datetimes, specify the tz:

```python
@freeze_time("2025-06-15 12:00:00", tz_offset=0)  # explicit UTC
```

Without this, `datetime.now()` and `datetime.utcnow()` can diverge by the server's local offset.

## Data Masking

When test environments need production-shaped data, mask it. Two approaches:

| Approach | How | When | Trade-off |
|----------|-----|------|-----------|
| **Static masking** | ETL pipeline copies production → staging with irreversible transforms | Periodic refresh (nightly/weekly) | Consistent snapshots; stale between refreshes |
| **Dynamic masking** | Query-time transformation; original data untouched | On-demand reads | Always fresh; query overhead; masking rules must cover every access path |

### Masking Rules

| Data Type | Mask Strategy | Example |
|-----------|--------------|---------|
| Email | Deterministic hash → `user{hash8}@test.dev` | `a3f9b2c1@test.dev` |
| Phone | Replace middle digits with `555-01xx` | `+1 (555) 0142` |
| Name | Faker substitution | `Jane Smith` → `Alice Johnson` |
| SSN/National ID | Replace with test-range values | `000-00-0000` (invalid range) |
| Address | Faker with same zip-code prefix | Preserves geo-distribution |
| Credit card | Luhn-valid test numbers | `4111 1111 1111 1111` |

**Deterministic masking** preserves referential integrity: the same real email always maps to the same masked email, so joins across tables remain valid.

## GDPR and Right-to-Erasure

Test data is not exempt from data protection regulations.

| Obligation | Test-Environment Implication |
|-----------|-----------------------------|
| Right to erasure (GDPR Art. 17) | Masked/synthetic data cannot be "erased" back to the original — verify masking is irreversible so erasure requests don't require test-data cleanup |
| Data minimization (GDPR Art. 5(1)(c)) | Tests should use the minimum fields needed; factories should not populate every column |
| Purpose limitation | Test data must not be reused for analytics, profiling, or any non-testing purpose |
| Breach notification | Test environments with masked PII-shaped data still need access controls; a leak of synthetic data that is indistinguishable from real data triggers the same response obligations |

### Compliance Checklist

- Masked data passes the same validation rules as real data (format, length, checksums)
- No reversible mapping exists between masked and original values
- Access to test environments is logged and role-controlled
- Data retention policies apply to test data (delete stale snapshots)
- Erasure requests are verified against test environments during quarterly audits

## PII Rules

> **Never use production PII in test databases.** This is the single most important test-data rule. Synthetic or masked data must be indistinguishable from real data in shape but contain zero real individuals' information.

| Rule | Rationale |
|------|-----------|
| No real names, emails, phones, addresses | GDPR, CCPA, and breach liability |
| Use obviously-fake credentials for auth tests | `AKIAIOSFODNN7EXAMPLE` (AWS example key) |
| If a test needs a specific edge case (unicode name, 255-char email), construct it explicitly | Random generation won't reliably hit edge cases |
| Rotate synthetic datasets quarterly | Prevents drift from schema changes |

### Synthetic Data Tools

| Tool | Use Case |
|------|----------|
| Faker | Names, emails, addresses, dates — realistic but fake |
| Presidio + Faker | Generate PII-shaped data that passes validation without real PII |
| SDV (Synthetic Data Vault) | Statistical replicas of production tables — preserves distributions |
| dbt seed + Jinja | Version-controlled CSV fixtures with templated expansion |

## External Service Data

| Service | Test Strategy |
|---------|---------------|
| Payment (Stripe) | Test-mode API keys + recorded fixtures (VCR.py / Polly.js) |
| Email (SendGrid) | Mock at transport layer; assert on message content |
| S3 / object storage | MinIO or `moto` (AWS mock); never hit real buckets |
| Third-party APIs | Contract tests (Pact) + recorded responses; rotate recordings quarterly |

## Data Volume Testing

| Scenario | Approach |
|----------|----------|
| Pagination | Seed exactly `page_size + 1` records |
| Performance under load | `generate_series()` in SQL or bulk factory (10K–100K rows) |
| Edge cases | Empty table, single row, max-length fields, unicode, nulls |
| Time-dependent | Freeze time (freezegun, timecop) — never `sleep()` |

## Migration Testing

- Run migrations against a copy of production schema (anonymized) in CI
- Test both forward and rollback paths
- Data migrations: seed before-state, run migration, assert after-state

## Gotchas

- **Production PII in test environments** is a compliance violation, not a convenience. Automate masking in the CI/CD pipeline so no human copy step exists.
- **Shared mutable fixtures** across tests create ordering dependencies. Each test must construct its own data or use transaction rollback.
- **Time-dependent assertions without clock freezing** fail on timezone boundaries, DST transitions, and month-end dates. Always freeze.
- **Dynamic masking without coverage audit** leaks PII through unmasked access paths (views, materialized caches, API responses). Audit every query path.

## Related

- Test isolation and CI-vs-local divergence: [test-debugging.md](./test-debugging.md)
- Security test data rules: [security-testing.md](./security-testing.md)

*Sources: freezegun (GitHub, spulec), timecop (GitHub, travisjeffery), Faker (GitHub, joke2k), SDV (sdv.dev), GDPR Regulation (EU) 2016/679 Art. 5, 17, OWASP Testing Guide v4.2 (data masking), factory_boy (GitHub, FactoryBoy).*

# Test Debugging

Diagnosing broken tests. Load when a test that should pass is failing, a mock isn't intercepting, a fixture is producing wrong state, or a test behaves differently in CI than locally. Distinct from test *design* (that's [test-strategy.md](./test-strategy.md)) and CI infrastructure triage (that's [ci-failure-triage.md](./ci-failure-triage.md)).

## Diagnostic Order

1. **Read the actual failure output.** Not the summary line — the full traceback, assertion values, and any captured stdout/stderr.
2. **Reproduce locally** with the same command CI runs (including paths, markers, and filters).
3. **Check collection.** `pytest --collect-only <path>` — is the test even being collected? Zero items means the test is invisible.
4. **Check environment completeness.** Install the project's declared dev/test dependencies using its own manifest.
5. **Isolate the variable.** Run the single failing test, then the file, then the directory. Narrow until the failure appears and disappears.

## CI-vs-Local Divergence Checklist

When a test passes locally but fails in CI (or vice versa), work through these causes systematically:

| # | Divergence Cause | Symptom | Diagnosis | Fix |
|---|-----------------|---------|-----------|-----|
| 1 | **Environment variables** | Test reads `os.environ` differently | `diff <(env | sort) <(ci_env | sort)` | Pin required env vars in CI config; use `.env.test` locally |
| 2 | **Timing and concurrency** | Race conditions surface under CI load | Run failing test 50× locally with `pytest-repeat`; add `--count=50` | Fix the race (proper synchronization), not the timing |
| 3 | **Test ordering / shared state** | Passes alone, fails in suite | Run with `pytest-randomly` or reverse order | Eliminate shared mutable state; each test constructs its own fixtures |
| 4 | **Filesystem differences** | Path separators, symlinks, case sensitivity (Linux CI vs macOS local) | Check for hardcoded paths; `find . -name "Test_*" vs "test_*"` | Use `pathlib.Path`; never hardcode separators |
| 5 | **Network access** | CI has no internet or restricted egress | Check for real HTTP calls; CI logs show `ConnectionRefused` | Mock external calls; use recorded fixtures (VCR.py) |
| 6 | **Dependency versions** | CI resolves different versions than local | Compare `pip freeze` / `npm ls` outputs | Commit lockfiles; use exact pins in CI |
| 7 | **Timezone and locale** | Date formatting, string collation differ | `echo $TZ $LANG` locally vs CI | Set `TZ=UTC` and `LC_ALL=C` explicitly in tests |

### Gotcha: "Works on My Machine" Is a Bug Report

A CI-only failure is not "CI being flaky" until you have ruled out all seven causes above. The most common root causes are ordering (3) and environment variables (1).

## Test Ordering and Shared State

Tests that pass individually but fail as a suite have ordering dependencies. This is a test design defect, not an infrastructure problem.

### Detection

```bash
# Install pytest-randomly — it randomizes order on every run
pip install pytest-randomly

# Run with a specific seed to reproduce
pytest --randomly-seed=12345 tests/

# Reverse execution order
pip install pytest-reverse
pytest --reverse tests/
```

### Common Shared-State Patterns

| Pattern | Symptom | Fix |
|---------|---------|-----|
| Module-level mutable global | Test A sets state; Test B reads it | Reset in fixture `setup`/`teardown`; prefer function-scoped fixtures |
| Class-level `setUpClass` mutation | Later tests depend on earlier test's writes | Move to per-test setup; use `setUp` not `setUpClass` |
| Database rows from prior test | Query returns unexpected count | Transaction rollback per test (see [test-data-management.md](./test-data-management.md)) |
| File artifacts on disk | Test reads a file another test created | Use `tmp_path` fixture; never write to shared directories |
| Environment variable mutation | `os.environ["X"] = "y"` leaks across tests | Use `monkeypatch.setenv()` (auto-reverts) |
| Monkeypatched module attribute | `module.GLOBAL = value` without cleanup | Use `monkeypatch.setattr()` (auto-reverts) |

### pytest-randomly as a Design Tool

Run pytest-randomly in CI on every run. Tests that fail under random ordering have latent shared-state bugs. Fix the isolation defect rather than pinning the order — pinning hides the problem until the next refactor breaks the assumed order.

## Mock Path Binding at the Usage Point

When you `patch()` a name in Python, you must patch it **where it is looked up** (the usage point), not where it is defined.

### Root Cause

Python imports bind names into the importing module's namespace at import time:

```python
# myapp/service.py
from myapp.clients import HttpClient  # binds 'HttpClient' in service.py's namespace

def fetch_data():
    client = HttpClient()  # looks up 'HttpClient' in service.py's globals
    return client.get("/api/data")
```

```python
# WRONG: patches the definition point — service.py still has the original reference
@patch("myapp.clients.HttpClient")

# RIGHT: patches where service.py looks it up
@patch("myapp.service.HttpClient")
def test_fetch_data(mock_client):
    mock_client.return_value.get.return_value = {"result": "ok"}
    assert fetch_data() == {"result": "ok"}
```

### Signals This Is the Problem

| Signal | Meaning |
|--------|---------|
| `AttributeError: module X does not have the attribute Y` | Patching at a module that doesn't import Y directly |
| Real HTTP calls despite `patch()` with `return_value` | Patched the wrong namespace; real client still bound |
| Mock works in one test file but not another | Each importing module has its own binding; patch each |
| Refactor moved code to a subpackage | All `patch()` paths targeting the old module are now wrong |

### Rule of Thumb

| Import Style | Patch Target |
|-------------|-------------|
| `from module import Class` | `patch("consumer_module.Class")` |
| `import module; module.Class()` | `patch("module.Class")` |
| `from module import func` used in 3 files | Patch in all 3 consumer modules |

## FastAPI Startup Race

When a FastAPI app's `@app.on_event("startup")` handler re-assigns module-level state, any mock state set before `with TestClient(app) as tc:` is silently overwritten.

```python
# Fix: set mock state AFTER context entry
@pytest.fixture
def client():
    with TestClient(app) as tc:  # startup runs here
        server_mod._active_engines = {"mock": MockEngine()}  # set AFTER
        yield tc
```

## Test Execution Integrity

A passing command is not necessarily an executed test suite.

1. **Read the collection summary.** `0 items`, `N skipped`, or exit code `5` means the intended behavior was not exercised.
2. **For a module-level target**, require a nonzero collected count and a passing test relevant to the change.
3. **If a test is skipped** because its fixture or path is wrong, repair that harness defect before opening the PR.
4. **Re-run after the repair** and record the actual result (e.g., `62 passed`), not only the exit status.

### CI Collection-Path Gate

A new test can pass locally and provide zero CI protection when it lives outside the directories selected by the workflow.

1. Read the exact CI test command, including explicit paths, `-k` filters, markers, and ignore flags.
2. Confirm the new test's path is included by that command.
3. Put the test under an already-collected directory when that matches its scope.
4. Inspect CI logs for the test/module after pushing.

## Gotchas

- **Do not mask a test design failure with retries or `xfail`.** If a test fails under random ordering, fix the isolation — don't pin the order.
- **Do not treat a green exit code as evidence.** Read the collection count. Zero collected tests with exit 0 is not a passing suite.
- **Mock at usage, not source.** After any module→package refactor, audit every `patch()` path against the new import structure.
- **Set mocks after startup, not before.** Any framework lifecycle hook that re-assigns module state will overwrite pre-context mock setup.
- **Patching `time.sleep` instead of freezing time** creates fragile tests. Use freezegun or timecop (see [test-data-management.md](./test-data-management.md)).

## Composition

- Systematic debugging methodology (hypothesis-driven, evidence-first): [systematic-debugging](../../systematic-debugging/SKILL.md)
- CI infrastructure triage (exit codes, bisect, runner issues): [ci-failure-triage.md](./ci-failure-triage.md)
- Test data isolation and time freezing: [test-data-management.md](./test-data-management.md)

*Sources: pytest-randomly (GitHub, adamchainz), pytest docs on monkeypatch (docs.pytest.org), Python unittest.mock docs (docs.python.org), freezegun (GitHub, spulec).*

# Test Design Techniques

Systematic methods for deriving test cases from specifications. Load when designing test cases for a specific feature, choosing which technique fits a scenario, or reviewing test coverage gaps. Not for test strategy allocation (that's [test-strategy.md](./test-strategy.md)) or exploratory discovery (that's [exploratory-testing.md](./exploratory-testing.md)).

## Technique Overview

| Technique | Input Model | Strength | Weakness |
|-----------|------------|----------|----------|
| Equivalence Partitioning (EP) | Input domains | Reduces test count with representative coverage | Misses boundary defects |
| Boundary Value Analysis (BVA) | Numeric/ordered ranges | Catches off-by-one, overflow, edge behavior | Only useful at boundaries |
| Decision Tables | Business rules with conditions | Complete combinatorial logic coverage | Explosion with many conditions |
| State Transition | Stateful workflows | Catches invalid transitions, dead states | Requires accurate state model |
| Pairwise / Combinatorial | Multi-parameter configurations | Covers all 2-way interactions with O(n log n) tests | Misses 3+ way interactions |
| Error Guessing | Experience, defect history | Finds "obvious" bugs fast | Unsystematic; depends on tester skill |

## Equivalence Partitioning (EP)

Divide the input domain into classes where behavior should be identical. Test one representative from each class.

### Worked Example: Age Field (0–120, integer)

| Partition | Range | Representative | Expected |
|-----------|-------|---------------|----------|
| Invalid (below) | < 0 | -1 | Reject |
| Valid (child) | 0–12 | 6 | Accept, category=child |
| Valid (adult) | 13–64 | 30 | Accept, category=adult |
| Valid (senior) | 65–120 | 70 | Accept, category=senior |
| Invalid (above) | > 120 | 121 | Reject |
| Invalid (type) | non-integer | "abc", 3.5 | Reject |

5 tests instead of 121 exhaustive values. Each partition's representative exercises the same code path as all other members.

## Boundary Value Analysis (BVA)

Defects cluster at boundaries. Test the values adjacent to partition edges.

### 2-Value vs 3-Value BVA

| Approach | Values Tested | When to Use |
|----------|--------------|-------------|
| **2-value** (edge) | min, max | Quick coverage; most defects are at the boundary itself |
| **3-value** (edge + just outside) | min-1, min, max, max+1 | When off-by-one is likely; stronger assurance |

### Worked Example: Order Quantity (1–999)

| Value | Type | Expected |
|-------|------|----------|
| 0 | Below minimum (3-value) | Reject |
| 1 | Minimum boundary | Accept |
| 999 | Maximum boundary | Accept |
| 1000 | Above maximum (3-value) | Reject |

Add EP representatives for interior partitions (e.g., 500) if behavior differs within the range.

## Decision Tables

When business logic depends on combinations of conditions, a decision table ensures every combination is tested.

### Worked Example: Discount Rules

| Condition | Rule 1 | Rule 2 | Rule 3 | Rule 4 |
|-----------|--------|--------|--------|--------|
| Premium member? | Y | Y | N | N |
| Order > $100? | Y | N | Y | N |
| **Action** | 20% off | 10% off | 5% off | 0% |

4 rules from 2 binary conditions (2² = 4). Each rule becomes at least one test case.

### Managing Complexity

| Conditions | Rules | Strategy |
|-----------|-------|----------|
| ≤ 4 | ≤ 16 | Full decision table |
| 5–8 | 32–256 | Collapse don't-care combinations; use pairwise for remaining |
| > 8 | 256+ | Pairwise testing (below) + risk-based selection |

## State Transition Testing

Model the system as states + transitions. Test every valid transition and verify that invalid transitions are rejected.

### Worked Example: Order Lifecycle

```
[Created] --pay--> [Paid] --ship--> [Shipped] --deliver--> [Delivered]
    |                 |                                    |
    +--cancel--> [Cancelled] <--cancel-- [Paid]           +--return--> [Returned]
```

| Test | Path | Expected |
|------|------|----------|
| Happy path | Created → Paid → Shipped → Delivered | Success at each step |
| Cancel from Created | Created → Cancelled | Order voided, no charge |
| Cancel from Paid | Paid → Cancelled | Refund issued |
| Invalid: ship unpaid | Created → Shipped (attempt) | Reject; state unchanged |
| Invalid: pay cancelled | Cancelled → Paid (attempt) | Reject; state unchanged |
| Return path | Delivered → Returned | Return processed |

**Coverage criteria:** at minimum, cover every state (0-switch) and every transition (1-switch). For critical workflows, cover 2-switch (pairs of consecutive transitions).

## Pairwise (Combinatorial) Testing

When parameters interact, full combinatorial testing explodes (e.g., 5 params × 4 values = 1024 tests). Pairwise covers every pair of parameter values in far fewer tests.

### Tool: PICT

PICT (Microsoft) generates pairwise test sets from a model file:

```
# model.txt — PICT format
OS:      Windows, macOS, Linux
Browser: Chrome, Firefox, Safari
Network: WiFi, Cellular, Offline
Locale:  en-US, de-DE, ja-JP
```

```bash
pict model.txt > pairwise-tests.txt
# Produces ~12–15 tests covering all pairs (vs 81 exhaustive)
```

### When Pairwise Is Sufficient

| Interaction Depth | Technique | Test Count |
|-------------------|-----------|-----------|
| 1-way (each value) | EP representatives | N |
| 2-way (all pairs) | Pairwise / all-pairs | O(N log N) |
| 3-way (all triples) | t-wise (t=3) | O(N² log N) |
| N-way (exhaustive) | Full combinatorial | Product of all values |

Most defects are triggered by 1- or 2-way interactions (empirical evidence from NIST studies). Pairwise is the default; escalate to t=3 only for high-risk configuration surfaces.

## Error Guessing

Systematic intuition: use defect history, code complexity, and experience to target likely failure points.

### Structured Error-Guessing Checklist

| Category | Examples to Try |
|----------|----------------|
| Empty / null inputs | `""`, `null`, `undefined`, `[]`, `{}` |
| Extreme values | MAX_INT, empty string, 10MB upload, 0-length list |
| Special characters | Unicode, emoji, SQL metacharacters, path separators |
| Concurrency | Double-submit, back-button during save, parallel edits |
| Timing | Midnight boundary, DST transition, leap second, month-end |
| State corruption | Kill process mid-write, network drop during transaction |
| Permission edges | Read-only filesystem, expired token, revoked role |

**Error guessing complements systematic techniques** — run it after EP/BVA/decision tables to catch what structured methods miss.

## When to Use Which Technique

| Scenario | Primary Technique | Secondary | Rationale |
|----------|-------------------|-----------|-----------|
| Numeric input field with ranges | BVA (3-value) | EP | Boundaries are the highest-yield targets |
| Form with many optional fields | Pairwise (PICT) | EP for each field | Interactions between fields cause most form bugs |
| Business rules with 2–4 conditions | Decision table | EP for each condition value | Complete logic coverage with manageable size |
| Stateful workflow (order, ticket, session) | State transition | Error guessing (invalid transitions) | Invalid transitions are the #1 stateful bug class |
| API with enum parameters | EP | Pairwise if multiple enums | Each enum value is a partition |
| Legacy code with defect history | Error guessing | BVA on known-problem fields | History predicts future defect locations |
| Configuration matrix (OS × browser × env) | Pairwise | — | Exhaustive is infeasible; pairs catch most bugs |

### Technique Selection Flowchart

```
Is the input numeric or ordered?
  YES → BVA + EP
  NO  → Does behavior depend on condition combinations?
          YES → ≤ 4 conditions? → Decision table
                 > 4 conditions? → Pairwise
          NO  → Is the system stateful?
                  YES → State transition
                  NO  → EP + error guessing
```

## Level-Mapping Guidance

These techniques apply at any test level (unit, integration, E2E). The following mapping is a **typical starting point** — adapt to your context:

| Technique | Typical Level | Adaptation Note |
|-----------|--------------|-----------------|
| EP / BVA | Unit (input validation) | Also applies at integration (API contracts) and E2E (form validation) |
| Decision tables | Unit / integration | Business rules often span services; test at integration level |
| State transition | Integration / E2E | Workflow state is usually service-level, not function-level |
| Pairwise | E2E / system | Configuration surfaces are system-wide |
| Error guessing | Any level | Most valuable at integration and E2E where interactions emerge |

> **Note:** This is a heuristic default, not a prescription. A state machine inside a single function is best tested at unit level; a simple input boundary may be best caught by an integration test. Context determines placement, not technique identity. See [test-strategy.md](./test-strategy.md) for the pyramid-as-heuristic framing.

## Gotchas

> **Gotcha — EP without boundary testing:** Equivalence partitioning alone misses off-by-one errors at partition edges. Always pair EP with BVA on numeric or ordered inputs.

> **Gotcha — Decision table explosion:** Adding a 5th binary condition doubles the table. Before adding conditions, ask: does this condition independently affect the outcome? If not, collapse it.

> **Gotcha — Pairwise as a substitute for understanding:** Pairwise generates combinations but doesn't tell you what's wrong. You still need oracles (expected results) for every generated test. A pairwise suite without assertions is just coverage theater.

## Exit Condition

You are done applying this reference when: (1) each testable input or behavior is assigned a primary technique, (2) boundary values are tested for all numeric/ordered inputs, (3) stateful workflows have transition coverage (at least 1-switch), and (4) configuration matrices use pairwise reduction rather than exhaustive enumeration.

## Composition Links

- Test strategy and allocation by risk tier: [test-strategy.md](./test-strategy.md)
- Exploratory testing for areas where techniques don't yet apply: [exploratory-testing.md](./exploratory-testing.md)
- Risk-based prioritization of which areas get exhaustive design: [risk-based-testing.md](./risk-based-testing.md)
- Test automation of designed cases: [test-automation.md](./test-automation.md)

---

*Sources: ISTQB Foundation Level Syllabus 2023 (test design techniques), PICT (Microsoft, github.com/microsoft/pict), NIST pairwise studies (Kuhn et al., 2004), Cem Kaner et al. "Lessons Learned in Software Testing" (Wiley, 2002), Rex Black "Managing the Testing Process" (Wiley, 2009).*

# Test Strategy Design

## The Pyramid as Heuristic, Not Dogma

The test pyramid (unit → integration → E2E) is a **starting point** for test investment allocation. Adapt the shape to your system's risk profile, feedback-loop requirements, and team capabilities.

> **Gotcha — Pyramid dogmatism:** Treating the pyramid as a rule ("always more unit than E2E") leads to over-testing trivial logic while under-testing the integration boundaries where real defects cluster. Measure where your bugs actually escape and invest there.

### Alternative Models

| Model | Origin | Core Idea | When It Fits |
|-------|--------|-----------|--------------|
| **Testing Trophy** | Kent C. Dodds | Integration tests give the highest confidence-per-effort; unit tests support them | UI-heavy apps, React/Vue ecosystems |
| **Testing Quadrants** | Lisa Crispin & Janet Gregory | Classify tests by purpose (technology-facing vs business-facing, supporting vs critiquing) | Teams needing balanced coverage across quality dimensions |
| **Context-Driven Shape** | "Pyramid or Crab?" (Hillel Wayne, James Bach) | The optimal distribution depends on architecture, risk, and feedback cost | Microservices, event-driven systems, any non-trivial topology |

### Default Allocation by Project Type

| Project Type | Unit | Integration | E2E |
|-------------|------|-------------|-----|
| Library / SDK | 80% | 15% | 5% |
| Web API | 40% | 40% | 20% |
| Web application (UI-heavy) | 20% | 40% | 40% |
| CLI tool | 60% | 30% | 10% |
| Data pipeline | 50% | 40% | 10% |

These are defaults. Re-evaluate quarterly against escaped-defect data.

## Shift-Left AND Shift-Right

Effective strategy moves quality activities in **both** directions:

| Direction | Activities | Goal |
|-----------|-----------|------|
| **Shift-left** | Static analysis in IDE, unit tests on PR, contract tests before integration, spec testability review | Catch defects at lowest cost |
| **Shift-right** | Canary releases, feature-flag monitoring, production error budgets, chaos experiments | Validate assumptions under real conditions |

Shift-left reduces defect volume; shift-right validates that what survives left-side filtering actually works in production. Neither alone is sufficient.

## Cost-of-Failure Reasoning

Defects found later cost exponentially more to fix. Use this to justify test investment:

| Phase Found | Relative Cost | Example Activity |
|-------------|--------------|-----------------|
| Requirements / Design | 1× | Spec review, testability analysis |
| Implementation (PR) | 5–10× | Unit test failure, code review catch |
| Integration / Staging | 20–50× | Contract test failure, QA cycle |
| Production | 100×+ | Hotfix, rollback, customer impact, reputation |

**Decision rule:** Invest in testing up to the point where the marginal cost of one more test exceeds the expected cost of the defect it would catch earlier.

## Coverage as Diagnostic, Not Target

> **Gotcha — Coverage gaming:** Chasing a coverage percentage (e.g., "reach 90%") incentivizes writing tests that execute lines without asserting behavior. A suite at 95% line coverage with 40% mutation score is weaker than a suite at 75% coverage with 85% mutation score.

Use coverage as a **diagnostic signal**:
- Identify untested high-risk code (coverage gaps in payment/auth modules)
- Detect coverage regressions (a PR that drops branch coverage on changed files)
- Guide test design (what paths remain unverified?)

Do **not** use coverage as a pass/fail gate without mutation testing or escaped-defect correlation to validate test quality. See [test-automation.md](./test-automation.md) for mutation testing as a complement.

## Risk-Based Prioritization

| Priority | Coverage Required | Examples |
|----------|------------------|----------|
| Critical (P0) | Every path, every edge case | Payment processing, auth, data integrity |
| High (P1) | All happy paths + known failure modes | Core business logic, API contracts |
| Medium (P2) | Happy paths + common failure modes | Secondary features, non-critical APIs |
| Low (P3) | Smoke test only | UI polish, debug tooling |

For the full risk-scoring methodology (P×I matrix, workshops, register), see [risk-based testing](./risk-based-testing.md). For verification planning and evidence standards, see [verification-methodology](../../verification-methodology/SKILL.md).

## Test Estimation

Estimation is inherently uncertain; use heuristics to bound the range, then refine with historical data.

| Heuristic | Method | Typical Range |
|-----------|--------|---------------|
| **Test-to-dev effort ratio** | Test effort = dev effort × ratio | 0.25× (well-tested greenfield) to 0.5× (legacy, high-risk) |
| **Risk-weighted estimation** | Sum(P × I × test-design-hours) per risk item | Varies; prioritize P0/P1 items first |
| **Historical velocity** | Story points tested per sprint (trailing 3 sprints) | Use as capacity input, not commitment |
| **Percentage-of-development-time** | Allocate 20–40% of sprint capacity to test design + execution | Adjust based on automation maturity |

**Practical approach:** Start with ratio-based estimate, decompose by priority tier (P0 items get 3× the per-item budget of P3), then sanity-check against velocity history.

## Requirements-to-Test Traceability (RTM)

### Coverage Rule

Every requirement (user story, acceptance criterion, non-functional requirement) must map to **at least one** test case. Orphan tests (tests with no requirement mapping) must be flagged for review — they may test removed functionality.

### Traceability Matrix Structure

| Requirement ID | Description | Test Cases | Status | Owner |
|---------------|-------------|-----------|--------|-------|
| REQ-001 | User can reset password | TC-012, TC-013 | Pass | QA-1 |
| REQ-002 | Session expires after 30min idle | TC-045 | Pass | QA-2 |
| REQ-003 | Export CSV respects locale | — | **GAP** | — |

### Gap Detection and Action

- **Pre-release audit:** Run a traceability gap report before every release. Any requirement with zero mapped tests blocks release sign-off.
- **Continuous detection:** When requirements change (new AC added in sprint planning), flag unmapped requirements within 24 hours.
- **Orphan review:** Quarterly review of tests with no requirement link; retire tests for removed features, reassign tests whose requirements were restructured.

## Accessibility as a Quality Dimension

Accessibility testing is a quality dimension alongside functional, performance, and security testing — not an afterthought.

| Aspect | QA Responsibility | Delegated To |
|--------|------------------|-------------|
| When to test a11y | Strategy: include in P0/P1 coverage, gate on critical violations | — |
| WCAG 2.2 conformance level | Define target (AA for public-facing, A minimum) | [web-accessibility](../../web-accessibility/SKILL.md) |
| Automated scanning | Integrate axe-core or pa11y in CI as advisory gate | [web-accessibility](../../web-accessibility/SKILL.md) |
| Manual screen-reader testing | Schedule per release for P0 flows | [web-accessibility](../../web-accessibility/SKILL.md) |

**Integration point:** Add automated a11y scans to CI (advisory initially, blocking once baseline is clean). Track violation count as a quality metric alongside escaped defects.

## Composition Links

- Risk scoring methodology: [risk-based-testing.md](./risk-based-testing.md)
- Verification planning and evidence: [verification-methodology](../../verification-methodology/SKILL.md)
- Accessibility mechanics (WCAG conformance, ARIA, screen readers): [web-accessibility](../../web-accessibility/SKILL.md)
- Spec testability review (for AI-generated code): [spec-driven-development](../../spec-driven-development/SKILL.md)

---

*Sources: Kent C. Dodds (Testing Trophy, 2017), Lisa Crispin & Janet Gregory (Agile Testing Quadrants), Hillel Wayne / James Bach (context-driven testing), DORA State of DevOps Reports, WCAG 2.2 (W3C Recommendation 2023).*

#!/usr/bin/env python3
"""Acceptance-criteria testability checker for qa-methodology.

Scans a markdown spec or acceptance-criteria block for vague, untestable
language and flags criteria that lack an observable outcome.

Testable criteria have concrete, verifiable outcomes (status codes, return
values, observable side effects, numeric thresholds). Untestable criteria
use vague verbs ("should handle", "should be efficient", "should work
properly") with no observable verification path.

Input: a markdown file path argument, or stdin if no path given.

Exit codes:
  0  all detected acceptance criteria are testable
  1  one or more acceptance criteria are untestable
  2  malformed or missing input (bad path, empty input, usage error)
"""

import argparse
import re
import sys

# Patterns that indicate vague, untestable language.
VAGUE_PATTERNS = [
    re.compile(r"\bshould\s+handle\b", re.IGNORECASE),
    re.compile(r"\bshould\s+be\s+efficient\b", re.IGNORECASE),
    re.compile(r"\bshould\s+be\s+fast\b", re.IGNORECASE),
    re.compile(r"\bshould\s+be\s+robust\b", re.IGNORECASE),
    re.compile(r"\bshould\s+be\s+secure\b", re.IGNORECASE),
    re.compile(r"\bshould\s+be\s+reliable\b", re.IGNORECASE),
    re.compile(r"\bshould\s+be\s+scalable\b", re.IGNORECASE),
    re.compile(r"\bshould\s+be\s+user.friendly\b", re.IGNORECASE),
    re.compile(r"\bshould\s+be\s+intuitive\b", re.IGNORECASE),
    re.compile(r"\bshould\s+be\s+performant\b", re.IGNORECASE),
    re.compile(r"\bshould\s+work\s+(correctly|properly|well)\b", re.IGNORECASE),
    re.compile(r"\bshould\s+gracefully\b", re.IGNORECASE),
    re.compile(r"\bhandle\s+(errors?\s+)?gracefully\b", re.IGNORECASE),
    re.compile(r"\bshould\s+support\b(?!\s+\d)", re.IGNORECASE),
    re.compile(r"\bshould\s+appropriately\b", re.IGNORECASE),
    re.compile(r"\bas\s+(needed|required|appropriate)\b", re.IGNORECASE),
    re.compile(r"\bshould\s+be\s+easy\b", re.IGNORECASE),
    re.compile(r"\bshould\s+be\s+clean\b", re.IGNORECASE),
    re.compile(r"\bshould\s+be\s+maintainable\b", re.IGNORECASE),
    re.compile(r"\bshould\s+be\s+readable\b", re.IGNORECASE),
]

# Patterns that indicate a concrete, testable outcome.
TESTABLE_PATTERNS = [
    re.compile(r"\breturns?\s+\d{3}\b", re.IGNORECASE),
    re.compile(r"\breturns?\s+(true|false|null|nil|none)\b", re.IGNORECASE),
    re.compile(r"\breturns?\s+\{", re.IGNORECASE),
    re.compile(r"\breturns?\s+\[", re.IGNORECASE),
    re.compile(r"\breturns?\s+[\"']", re.IGNORECASE),
    re.compile(r"\bexit\s+code\s+\d", re.IGNORECASE),
    re.compile(r"\bexits?\s+(with\s+)?\d", re.IGNORECASE),
    re.compile(r"\bthrows?\s+\w*Error\b", re.IGNORECASE),
    re.compile(r"\braises?\s+\w*Error\b", re.IGNORECASE),
    re.compile(r"\bstatus\s+code\s+\d{3}\b", re.IGNORECASE),
    re.compile(r"\bHTTP\s+\d{3}\b", re.IGNORECASE),
    re.compile(r"\b\d{3}\s+(OK|Created|Bad Request|Not Found|Error)\b", re.IGNORECASE),
    re.compile(r"\bwithin\s+\d+\s*(ms|s|seconds|milliseconds|minutes)\b", re.IGNORECASE),
    re.compile(r"\bless\s+than\s+\d", re.IGNORECASE),
    re.compile(r"\bat\s+most\s+\d", re.IGNORECASE),
    re.compile(r"\bat\s+least\s+\d", re.IGNORECASE),
    re.compile(r"\bexactly\s+\d", re.IGNORECASE),
    re.compile(r"\b\d+(\.\d+)?%", re.IGNORECASE),
    re.compile(r"\bdisplays?\b", re.IGNORECASE),
    re.compile(r"\brenders?\b", re.IGNORECASE),
    re.compile(r"\bnavigates?\s+to\b", re.IGNORECASE),
    re.compile(r"\bredir(e)?cts?\b", re.IGNORECASE),
    re.compile(r"\blog(s|ged|ging)?\s+(the|a|an|this)\b", re.IGNORECASE),
    re.compile(r"\bsends?\s+(an?\s+)?(email|notification|request|event)\b", re.IGNORECASE),
    re.compile(r"\bcreates?\s+(a|an|the)\b", re.IGNORECASE),
    re.compile(r"\bdeletes?\s+(a|an|the)\b", re.IGNORECASE),
    re.compile(r"\bupdates?\s+(a|an|the)\b", re.IGNORECASE),
    re.compile(r"\bstores?\s+(a|an|the|in)\b", re.IGNORECASE),
    re.compile(r"\bcontains?\b", re.IGNORECASE),
    re.compile(r"\bequals?\b", re.IGNORECASE),
    re.compile(r"\bmatches?\s+(the\s+)?(pattern|regex|schema)\b", re.IGNORECASE),
    re.compile(r"\bis\s+(empty|non-empty|present|absent)\b", re.IGNORECASE),
    re.compile(r"\bwith\s+\{[^}]+\}", re.IGNORECASE),
    re.compile(r"\bwhen\s+\w+", re.IGNORECASE),
]

# AC line patterns: markdown list items or Given/When/Then or numbered criteria.
AC_LINE_RE = re.compile(
    r"^\s*(?:[-*+]\s+|\d+[.)]\s+|AC[-_]?\d*[:.]\s*|Given\s|When\s|Then\s)",
    re.IGNORECASE,
)

# Also match lines that start with "should" or "shall" or "must" (common AC phrasing)
SHOULD_LINE_RE = re.compile(
    r"^\s*(?:[-*+]\s+)?(?:the\s+)?(?:system|api|app|application|service|server|user|it)\s+"
    r"(?:should|shall|must)\b",
    re.IGNORECASE,
)


def parse_args(argv=None):
    """Parse command-line arguments."""
    parser = argparse.ArgumentParser(
        prog="check-ac-testability.py",
        description=(
            "Check acceptance criteria in a markdown spec for testability. "
            "Flags vague criteria (unobservable verbs, no measurable outcome) "
            "and passes concrete ones (observable outcomes with verification path)."
        ),
        epilog=(
            "Exit codes: 0 all testable, 1 untestable criteria found, "
            "2 malformed/missing input."
        ),
    )
    parser.add_argument(
        "input",
        nargs="?",
        default="-",
        metavar="SPEC_FILE",
        help=(
            "Path to a markdown spec file containing acceptance criteria "
            "(default: stdin)."
        ),
    )
    return parser.parse_args(argv)


def extract_ac_lines(text):
    """Extract lines that look like acceptance criteria from markdown text."""
    ac_lines = []
    for line in text.splitlines():
        stripped = line.strip()
        if not stripped:
            continue
        if AC_LINE_RE.match(line) or SHOULD_LINE_RE.match(line):
            ac_lines.append(stripped)
    return ac_lines


def classify_ac(text):
    """Classify a single AC line as testable or untestable.

    Returns (verdict, reasons) where verdict is 'testable' or 'untestable'
    and reasons is a list of strings explaining why.
    """
    vague_hits = []
    for pat in VAGUE_PATTERNS:
        m = pat.search(text)
        if m:
            vague_hits.append(m.group(0))

    testable_hits = []
    for pat in TESTABLE_PATTERNS:
        m = pat.search(text)
        if m:
            testable_hits.append(m.group(0))

    # If there are concrete observable outcomes, it's testable
    # even if it also has a vague phrase (the concrete part dominates)
    if testable_hits:
        return "testable", ["has observable outcome: {}".format(testable_hits[0])]

    if vague_hits:
        reasons = ["vague language: {}".format(h) for h in vague_hits]
        reasons.append("no observable outcome specified")
        return "untestable", reasons

    # No vague pattern AND no testable pattern: could be a non-criterion line
    # If it has no "should/shall/must" verb, treat as not-an-AC
    if re.search(r"\b(should|shall|must)\b", text, re.IGNORECASE):
        # Has requirement language but no observable outcome
        return "untestable", ["requirement stated but no observable outcome or verification path"]

    # Not a requirement statement at all
    return "testable", ["no requirement language detected; treated as context"]


def check_testability(text):
    """Run testability check on the full text.

    Returns (results, exit_code) where results is a list of dicts.
    """
    ac_lines = extract_ac_lines(text)

    if not ac_lines:
        # No ACs found — nothing to flag
        return [], 0

    results = []
    has_untestable = False

    for line in ac_lines:
        verdict, reasons = classify_ac(line)
        results.append({
            "criterion": line,
            "verdict": verdict,
            "reasons": reasons,
        })
        if verdict == "untestable":
            has_untestable = True

    exit_code = 1 if has_untestable else 0
    return results, exit_code


def format_results(results):
    """Format results as human-readable output."""
    if not results:
        return "No acceptance criteria detected in input."

    lines = []
    for r in results:
        icon = "PASS" if r["verdict"] == "testable" else "FAIL"
        lines.append("[{}] {}".format(icon, r["criterion"]))
        for reason in r["reasons"]:
            lines.append("       {}".format(reason))
        lines.append("")
    return "\n".join(lines)


def main(argv=None):
    """Entry point."""
    args = parse_args(argv)

    # Read input
    try:
        if args.input == "-":
            source_text = sys.stdin.read()
        else:
            with open(args.input, "r", encoding="utf-8") as fh:
                source_text = fh.read()
    except OSError as exc:
        print("error: cannot read input: {}".format(exc), file=sys.stderr)
        return 2

    if not source_text.strip():
        print("error: input is empty", file=sys.stderr)
        return 2

    results, exit_code = check_testability(source_text)
    print(format_results(results))

    return exit_code


if __name__ == "__main__":
    sys.exit(main())

#!/usr/bin/env python3
"""Risk prioritization CLI for qa-methodology.

Reads a JSON array of risk items, computes score = probability x impact,
ranks descending (deterministic tie-break by id), and outputs a
human-readable table by default or machine-parseable JSON with --json.

Input format (JSON array):
  [{"id": "auth-bypass", "probability": 4, "impact": 5}, ...]

Each item requires:
  - id: non-empty string
  - probability: integer 1-5
  - impact: integer 1-5

Exit codes:
  0  success
  1  malformed input (bad JSON, missing/invalid fields, wrong types)
  2  usage error (argparse)
"""

import argparse
import json
import sys


def parse_args(argv=None):
    """Parse command-line arguments."""
    parser = argparse.ArgumentParser(
        prog="risk-prioritize.py",
        description=(
            "Compute risk priority scores (probability x impact) and rank "
            "risk items for test allocation."
        ),
        epilog="Exit codes: 0 success, 1 malformed input, 2 usage error.",
    )
    parser.add_argument(
        "input",
        nargs="?",
        default="-",
        metavar="INPUT_JSON",
        help=(
            "Path to a JSON file containing an array of risk items "
            "(default: stdin). Each item: {\"id\": str, \"probability\": 1-5, "
            "\"impact\": 1-5}."
        ),
    )
    parser.add_argument(
        "--json",
        action="store_true",
        dest="json_output",
        help="Output results as machine-parseable JSON instead of a table.",
    )
    return parser.parse_args(argv)


def validate_item(item, index):
    """Validate a single risk item. Returns (item, error_message)."""
    if not isinstance(item, dict):
        return None, "item {} is not an object".format(index)
    item_id = item.get("id")
    if item_id is None:
        return None, "item {} missing required field 'id'".format(index)
    if not isinstance(item_id, str) or not item_id.strip():
        return None, "item {} field 'id' must be a non-empty string".format(index)
    prob = item.get("probability")
    if prob is None:
        return None, "item '{}' missing required field 'probability'".format(item_id)
    impact = item.get("impact")
    if impact is None:
        return None, "item '{}' missing required field 'impact'".format(item_id)
    # Accept int only (not bool, not float)
    if isinstance(prob, bool) or not isinstance(prob, int):
        return None, "item '{}' field 'probability' must be an integer".format(item_id)
    if isinstance(impact, bool) or not isinstance(impact, int):
        return None, "item '{}' field 'impact' must be an integer".format(item_id)
    if not (1 <= prob <= 5):
        return None, "item '{}' field 'probability' must be 1-5".format(item_id)
    if not (1 <= impact <= 5):
        return None, "item '{}' field 'impact' must be 1-5".format(item_id)
    return item, None


def load_and_validate(source_text):
    """Parse JSON text and validate the risk items array.

    Returns (items, None) on success or (None, error_message) on failure.
    """
    try:
        data = json.loads(source_text)
    except (json.JSONDecodeError, ValueError) as exc:
        return None, "invalid JSON: {}".format(exc)

    if not isinstance(data, list):
        return None, "input must be a JSON array of risk items"
    if len(data) == 0:
        return None, "input array must contain at least one risk item"

    items = []
    for i, raw in enumerate(data):
        item, err = validate_item(raw, i)
        if err:
            return None, err
        items.append(item)
    return items, None


def compute_rankings(items):
    """Compute score and rank items descending by score, then by id ascending.

    Returns a new list of dicts with 'score' and 'rank' added.
    Does not mutate the input.
    """
    scored = []
    for item in items:
        scored.append({
            "id": item["id"],
            "probability": item["probability"],
            "impact": item["impact"],
            "score": item["probability"] * item["impact"],
        })
    # Sort descending by score, then ascending by id for deterministic tie-break
    scored.sort(key=lambda x: (-x["score"], x["id"]))
    for rank, entry in enumerate(scored, start=1):
        entry["rank"] = rank
    return scored


def format_table(rankings):
    """Format rankings as a human-readable table."""
    lines = []
    header = "{:<4} {:<30} {:>5} {:>5} {:>5}".format(
        "Rank", "ID", "Prob", "Imp", "Score"
    )
    lines.append(header)
    lines.append("-" * len(header))
    for entry in rankings:
        lines.append(
            "{:<4} {:<30} {:>5} {:>5} {:>5}".format(
                entry["rank"],
                entry["id"],
                entry["probability"],
                entry["impact"],
                entry["score"],
            )
        )
    return "\n".join(lines)


def main(argv=None):
    """Entry point."""
    args = parse_args(argv)

    # Read input
    try:
        if args.input == "-":
            source_text = sys.stdin.read()
        else:
            with open(args.input, "r", encoding="utf-8") as fh:
                source_text = fh.read()
    except OSError as exc:
        print("error: cannot read input: {}".format(exc), file=sys.stderr)
        return 1

    if not source_text.strip():
        print("error: input is empty", file=sys.stderr)
        return 1

    items, err = load_and_validate(source_text)
    if err:
        print("error: {}".format(err), file=sys.stderr)
        return 1

    rankings = compute_rankings(items)

    if args.json_output:
        print(json.dumps(rankings, indent=2))
    else:
        print(format_table(rankings))

    return 0


if __name__ == "__main__":
    sys.exit(main())

---
name: qa-methodology
description: >-
  Design and apply QA methodology for software teams: test strategy, regression testing,
  CI failure triage, test automation, quality gates and metrics, risk-based testing,
  exploratory testing, test design techniques, AI code quality gates (independent
  verification, acceptance-criteria testability review for agentic Spec-Driven
  Development), mutation-guided test hardening and review evidence (surviving mutants,
  weak assertions, diff-aware mutation testing), agentic eval design (dataset test design, judge-as-system-under-test,
  flaky-eval discipline), QA career levels (Senior/Staff/Principal), and SDET
  engineering (test infrastructure, gTAA, CI/CD integration). Do not use for
  root-cause debugging of production incidents, security implementation or threat
  modeling, or evaluation framework governance and statistical analysis — route those
  to systematic-debugging, secure-software-engineering, and
  agent-evals-and-observability respectively.
license: MIT
compatibility: >-
  Platform-agnostic methodology. Scripts require Python 3.8+ (stdlib only).
  No CI platform, test framework, or AI agent mandate.
metadata:
  source_repo: hermes-profiles
  skill_version: "2.0.0"
  tags: qa, testing, quality-assurance, test-automation, regression, CI, quality-gates, risk-based-testing, exploratory-testing, mutation-testing, SDET, agentic-evals, AI-code-quality
---

# QA Methodology

Senior-to-principal QA and SDET methodology: test strategy, automation, regression, risk-based prioritization, exploratory testing, quality gates, AI code quality gates for agentic Spec-Driven Development, agentic eval design, career leveling, and SDET engineering.

## Ownership

| You own | You don't own |
|---------|--------------|
| Test strategy — what to test, at what level, with what priority | Root-cause debugging — route to [systematic-debugging](../systematic-debugging/SKILL.md) |
| Test automation — framework selection, parallelism, flaky management | Security implementation and threat modeling — route to [secure-software-engineering](../secure-software-engineering/SKILL.md) |
| E2E automation strategy and coverage decisions | Operating a browser test tool (Playwright) — authoring/running specs, selectors, network mocking, scraping — route to [playwright](../playwright/SKILL.md) |
| Regression suites — selection, impact analysis, suite evolution | Spec pipeline mechanics and gate verdicts — route to [spec-driven-development](../spec-driven-development/SKILL.md) |
| Quality gates — blocking vs advisory, metrics, DORA | Eval framework governance and statistics — route to [agent-evals-and-observability](../agent-evals-and-observability/SKILL.md) |
| Risk-based testing — P×I scoring, prioritization, registers | Verification verdicts against explicit criteria — route to [verification-methodology](../verification-methodology/SKILL.md) |
| Exploratory testing — SBTM charters, heuristics, tours | Feature implementation — that's the developer |
| AI code quality gates — independent verification, AC testability | Production monitoring and incident response — that's SRE |
| Mutation-guided test hardening — bounded mutation review evidence and survivor triage | Verification verdicts against explicit criteria — route to [verification-methodology](../verification-methodology/SKILL.md) |
| Agentic eval design — dataset design, judge bias, flaky-eval discipline | |
| QA career levels — Senior/Staff/Principal scope progression | |
| SDET engineering — test infrastructure, gTAA, CI/CD integration | |

## Core Principles

**If it isn't tested, it's broken.** Untested code is code whose failure mode hasn't been discovered yet.

**Quality is a property of the process, not the artifact.** Testing at the end doesn't create quality. Quality is designed in through strategy, automation, and gating throughout the cycle.

**Test behavior, not implementation.** Tests coupled to behavior survive refactoring; tests coupled to implementation break on it.

**Risk drives priority.** Not everything deserves equal test investment. Score probability × impact, then allocate accordingly.

**Flaky tests are worse than no tests.** A nondeterministic failure trains teams to ignore all failures. Quarantine on detection; rerun once, never twice.

**Independent verification is non-negotiable.** The implementing agent (or developer) must not self-verify. Separate session, fresh context, no shared priors.

## Loading Guide

| File | Load when |
|------|-----------|
| `references/test-strategy.md` | Designing a test strategy — pyramid shape, shift-left/right, cost-of-failure, coverage as diagnostic |
| `references/test-automation.md` | Selecting frameworks, parallelism/sharding, flaky quarantine, predictive ML test selection, mutation-guided hardening |
| `references/quality-gates-and-metrics.md` | Designing quality gates (blocking vs advisory), DORA metrics, vanity-vs-actionable metrics, mutation testing |
| `references/regression-testing.md` | Building regression suites — impact analysis, selection math, suite evolution, shift-right feedback |
| `references/test-data-management.md` | Test data strategy — fixtures, factories, time-travel, masking, GDPR/PII rules |
| `references/performance-testing.md` | Load/stress/soak testing — k6/Locust/Gatling/JMeter, SLO thresholds, CI cadence |
| `references/security-testing.md` | Security testing — OWASP Top 10:2025, STRIDE, SAST/DAST/SCA, supply chain/SBOM |
| `references/ci-failure-triage.md` | CI is red — exit-code taxonomy (1/2/126/127/137/139/143), git bisect, flake-vs-failure protocol |
| `references/test-debugging.md` | A test that should pass is failing — CI-vs-local divergence, ordering/shared state, mock binding |
| `references/risk-based-testing.md` | Prioritizing by risk — P×I formula, 5×5 matrix, risk workshop, register, reassessment triggers |
| `references/exploratory-testing.md` | Exploratory testing — SBTM, charter writing, SFDIPOT/HICCUPPS heuristics, tours |
| `references/test-design-techniques.md` | Choosing test design techniques — EP, BVA, decision tables, state transition, pairwise, error guessing |
| `references/qa-career-levels.md` | QA career growth — Senior/Staff/Principal scope, leveling mechanics, archetypes, misconceptions |
| `references/sdet-engineering.md` | SDET role and skills — gTAA/TAF architecture, POM, SOLID for tests, build-vs-buy, testability |
| `references/ai-code-quality-gates.md` | Reviewing AI-generated code — independent verification, AC testability, agent-test quality, human-in-the-loop |
| `references/agentic-eval-design.md` | Designing agent evals — dataset test design, judge bias, flaky-eval discipline, CI gate tiers, replay |
| `templates/test-strategy.md` | Producing a test strategy document — fill in scope, risk tiers, level allocation, automation targets |
| `templates/risk-register.md` | Recording risk assessment results — fill in items, P×I scores, owners, mitigations |
| `templates/exploratory-charter.md` | Writing an SBTM charter — fill in target, resources, discovery goal, timebox |
| `templates/bug-report.md` | Filing a structured bug report — fill in reproduction steps, expected vs actual, severity |
| `templates/verification-plan.md` | Planning independent verification — fill in AC-to-method traceability, verifier assignment, exit criteria |
| `templates/mutation-review.md` | Recording bounded mutation review scope, classifications, survivor tests, and independent evidence |
| `assets/risk-matrix-grid.md` | Scoring risks during a workshop — 5×5 P×I grid with zone thresholds |
| `assets/test-design-techniques-checklist.md` | Selecting techniques for a feature — quick-reference checklist mapping scenario type to technique |
| `assets/qa-definition-of-done.md` | Defining release readiness — QA contribution to definition of done |
| `scripts/risk-prioritize.py` | Computing P×I rankings from a risk-items JSON file |
| `scripts/check-ac-testability.py` | Checking acceptance criteria for vague verbs and missing observable outcomes |
| `evals/evals.json` | Running output-quality evals for this skill (schema v1, 10 cases) |

## Scripts

| Script | Invocation | Purpose |
|--------|-----------|---------|
| risk-prioritize | `python3 scripts/risk-prioritize.py --json <input.json>` | Reads risk items (probability, impact), computes P×I scores, emits ranked JSON |
| check-ac-testability | `python3 scripts/check-ac-testability.py <spec.md>` | Scans acceptance criteria for untestable language, exits non-zero if any are flagged |

## Triggers

Load this skill when the task involves:

- **Test strategy** — designing what/how/priority to test for a project or feature
- **Regression testing** — building, selecting, or evolving regression suites
- **CI triage** — diagnosing CI failures, exit codes, flake-vs-real classification
- **Test automation** — framework selection, parallelism, flaky quarantine, ML selection
- **Quality gates** — gate design, blocking vs advisory, metrics, DORA
- **Mutation-guided test hardening** — diff-aware mutation scope, surviving mutants, weak assertions, and review evidence
- **Risk-based testing** — P×I scoring, risk registers, prioritization workshops
- **Exploratory testing** — SBTM charters, oracle heuristics, session debriefs
- **Agentic evals** — eval dataset design, judge bias, flaky-eval discipline, CI tiers
- **SDD gate review** — QA ownership at spec-driven gates, AC testability, independent verification
- **SDET** — test infrastructure engineering, gTAA, CI/CD integration, career scope

## When not to use

Route to the named sibling skill instead:

- [spec-driven-development](../spec-driven-development/SKILL.md) — writing specs, running the SDD pipeline, gate verdict format, revision loops
- [agent-evals-and-observability](../agent-evals-and-observability/SKILL.md) — eval framework governance, statistical comparisons, telemetry and privacy controls, grader implementation
- [verification-methodology](../verification-methodology/SKILL.md) — collecting evidence and rendering verdicts against explicit pass/fail criteria
- [release-engineering](../release-engineering/SKILL.md) — composing test evidence into release-candidate readiness, promotion, go/no-go, production rollout, and rollback decisions; QA owns test strategy and gate semantics
- [systematic-debugging](../systematic-debugging/SKILL.md) — root-cause analysis of production incidents, bug reproduction, fault localization
- [secure-software-engineering](../secure-software-engineering/SKILL.md) — security implementation, threat modeling, secure defaults, dependency evaluation
- [playwright](../playwright/SKILL.md) — operating the Playwright tool itself: authoring and running E2E specs, selector robustness, network mocking, headless scraping, and headed debugging

## Stop and Exit Conditions

- **Test strategy complete when:** strategy document names risk tiers, level allocation, automation targets, and exit criteria for each tier.
- **Risk assessment complete when:** every identified risk has a P×I score, an owner, and a mitigation or acceptance decision recorded in the register.
- **CI triage complete when:** failure is classified (flake vs real, env vs code), root cause is localized, and a fix or escalation path is identified.
- **Gate review complete when:** every acceptance criterion maps to a verification method, the verifier is independent of the implementer, and evidence is attached.
- **Bounded escalation:** stop after three non-converging diagnostic passes and report the evidence collected so far.

# Bug Report

> File one report per defect. Provide enough detail for any engineer to reproduce without asking questions.

## Title

<One-line summary: [Component] Observed behavior under specific condition>

## Classification

| Field | Value |
|-------|-------|
| Severity | Critical / High / Medium / Low |
| Priority | P1 (Immediate) / P2 (This sprint) / P3 (Backlog) / P4 (Won't fix) |
| Component | <affected module or service> |
| Discovered In | <test type: unit / integration / E2E / exploratory / production> |
| Charter / Test ID | <link to exploratory charter or test case, if applicable> |

### Severity Definitions

| Severity | Meaning |
|----------|---------|
| Critical | Data loss, security breach, system down, revenue stoppage |
| High | Major feature broken, no workaround, SLA breach |
| Medium | Feature impaired but workaround exists |
| Low | Cosmetic, minor inconvenience, documentation error |

## Environment

| Field | Value |
|-------|-------|
| OS / Platform | <e.g., macOS 15, Ubuntu 24.04, iOS 18> |
| Browser / Client | <e.g., Chrome 131, API client v2.3> |
| Application Version | <commit SHA, release tag, or build number> |
| Environment | <local / CI / staging / production> |
| Relevant Config | <feature flags, env vars, tenant settings> |

## Reproduction Steps

1. <Step 1 — starting state or navigation>
2. <Step 2 — action taken>
3. <Step 3 — action taken>
4. <...add steps as needed>

**Reproducibility:** Always / Sometimes (___ in ___ attempts) / Once (not yet reproduced)

## Expected vs Actual

| | Description |
|---|-------------|
| **Expected** | <What should happen according to spec, docs, or reasonable behavior> |
| **Actual** | <What actually happens — be specific about the observed behavior> |

## Evidence

Attach or link:

- [ ] Screenshot / screen recording: <path or URL>
- [ ] Error message / stack trace: <paste below or link>
- [ ] Logs: <path or link>
- [ ] Network capture: <path or link>
- [ ] Test output: <CI link or local path>

```
<paste error output or stack trace here>
```

## Additional Context

<Anything that helps triage: recent changes, related issues, suspected root cause, whether it blocks release.>

## Escalation

| Condition | Action |
|-----------|--------|
| Severity = Critical | Blocks release; escalate to engineering leadership within 1 hour |
| Severity = High + Priority = P1 | Escalate to team lead within 4 hours |
| Unreproducible after 3 attempts | Add to investigation backlog with environment capture |

# Exploratory Testing Charter

> Fill in for each SBTM session. One charter per session. See [exploratory-testing.md](../references/exploratory-testing.md) for charter quality guidance and oracle heuristics.

## Charter

**Explore** <target area / feature / component>
**with** <resources, constraints, or test conditions>
**to discover** <information or risks sought>.

## Session Setup

| Field | Value |
|-------|-------|
| Tester(s) | <name(s)> |
| Date | <YYYY-MM-DD> |
| Timebox | <60 / 90 / 120> minutes |
| Environment | <local / staging / specific config> |
| Test Data / Tools | <specific data sets, proxies, throttling, accounts> |

## Charter Quality Checklist

Before starting, confirm:

- [ ] Target is specific (not "the app")
- [ ] Resources or constraints are named (test data, tools, conditions)
- [ ] Information goal is stated (not just "find bugs")
- [ ] Scope fits within the timebox

## Heuristics Applied

Select oracles and coverage heuristics to guide exploration (see [exploratory-testing.md](../references/exploratory-testing.md)):

- [ ] SFDIPOT coverage (Structure, Function, Data, Interfaces, Platform, Operations, Time)
- [ ] HICCUPPS oracle (History, Image, Comparable, Claims, Users, Product, Purpose, Standards)
- [ ] Tours (e.g., Guidebook, Money, Supermodel, Saboteur, Back-Alley)
- [ ] Other: <specify>

## Notes

<Record observations, questions, areas explored, anomalies, and hunches during the session. Append chronologically.>

-

## T/B/B Metrics

Track time allocation at session end:

| Metric | Minutes | Percentage |
|--------|--------:|-----------:|
| **T** — Test time (designing + executing) | | ___% |
| **B** — Bug investigation | | ___% |
| **B** — Setup / Interruption | | ___% |
| **Total** | | 100% |

> Target: T ≥ 70%, Setup ≤ 10%. If T < 60%, fix environmental blockers before scheduling more sessions.

## Debrief

### What did you test?

<Areas covered, charters fulfilled, techniques used.>

### What did you find?

<Bugs filed (IDs), risks identified, questions raised.>

| Finding | Type (Bug / Risk / Question) | Severity / Priority | Bug ID |
|---------|------------------------------|---------------------|--------|
| <description> | | | |
| <description> | | | |

### What is left untested?

<Scope not reached, new areas discovered, follow-up needed.>

### Follow-Up Actions

- [ ] <New charter needed: ___>
- [ ] <Automation candidate: ___>
- [ ] <Risk register update: ___>
- [ ] <Spec clarification needed: ___>
- [ ] <Other: ___>

# Mutation Review

Use this artifact for a bounded mutation-guided test-hardening review. Keep the project's native tool output as the authoritative raw report; this is a review record, not a portable report schema.

## Change And Scope

- Review/change concern:
- Expected behavior:
- Base commit:
- Exact candidate/head commit:
- Target files/lines:
- Scope rationale:

## Run Configuration

- Tool and version:
- Configuration:
- Operator set:
- Seed:
- Test command:
- Mutant budget:
- Per-mutant timeout:
- Exclusions and rationale:
- Environment:
- Raw report location:

## Baseline Result

- Command and commit:
- Result:
- Relevant output/evidence:

## Mutants

| Stable ID | Location | Operator | Status | Disposition | Evidence |
|---|---|---|---|---|---|
|  |  |  | killed / survived / equivalent-or-likely-equivalent / no coverage / timeout / flaky / invalid / infrastructure-tooling failure |  |  |

## Accounting

- Denominator formula:
- In-scope mutants:
- Classified mutants:
- Excluded mutants:
- Unknown or incomplete mutants:
- Infrastructure/tooling failures:
- Explanation of any difference:

Unknown, incomplete, excluded, and failed outcomes must remain visible; they do not silently become killed or leave the denominator.

## Candidate Test

- Proposed test and expected behavior:
- Why this tests behavior rather than implementation:
- Behavioral relevance:
- Non-tautology/non-vacuity check:
- Non-redundancy and maintainability check:
- Flake-risk check:
- Implementation-coupling check:

## Independent Verification

- Baseline-test result:
- Exact-mutant-kill result:
- Repeat/flake result:
- Independent verifier:
- Verification timestamp/identity:
- Evidence locations:

## Decision

- Human reviewer decision:
- Uncertainty and limitations:
- Rerun/reproduction command:
- Follow-up owner and due date:

# Risk Register

> Record risk assessment results. Score each risk using the 5×5 P×I grid in [assets/risk-matrix-grid.md](../assets/risk-matrix-grid.md). See [risk-based-testing.md](../references/risk-based-testing.md) for scoring guidance and reassessment triggers.

## Metadata

| Field | Value |
|-------|-------|
| Project / Release | <name> |
| Workshop Date | <YYYY-MM-DD> |
| Facilitator | <name> |
| Participants | <names / roles> |
| Next Review Date | <YYYY-MM-DD> |

## Risk Items

| ID | Risk | Component | Probability (1–5) | Impact (1–5) | Score | Tier | Mitigation / Test Plan | Owner | Status | Last Reviewed | Reassessment Trigger |
|----|------|-----------|--------------------:|-------------:|------:|------|------------------------|-------|--------|---------------|----------------------|
| RISK-001 | <what could go wrong> | <affected area> | | | | P0 / P1 / P2 / P3 | <tests or controls> | <who> | Open / Mitigating / Closed | <YYYY-MM-DD> | <event that re-opens> |
| RISK-002 | <what could go wrong> | <affected area> | | | | P0 / P1 / P2 / P3 | <tests or controls> | <who> | Open / Mitigating / Closed | <YYYY-MM-DD> | <event that re-opens> |
| RISK-003 | <what could go wrong> | <affected area> | | | | P0 / P1 / P2 / P3 | <tests or controls> | <who> | Open / Mitigating / Closed | <YYYY-MM-DD> | <event that re-opens> |

### Column Guide

| Column | How to Fill |
|--------|-------------|
| ID | Sequential unique identifier (RISK-001, RISK-002, ...) |
| Risk | Plain-language description of the failure scenario |
| Component | System area or service affected |
| Probability (1–5) | 1=Rare, 2=Unlikely, 3=Possible, 4=Likely, 5=Almost Certain |
| Impact (1–5) | 1=Negligible, 2=Minor, 3=Moderate, 4=Major, 5=Catastrophic |
| Score | Probability × Impact (range 1–25) |
| Tier | From score: 20–25=P0, 12–19=P1, 6–11=P2, 1–5=P3 |
| Mitigation / Test Plan | Specific tests or controls that reduce this risk |
| Owner | Person accountable for implementing the mitigation |
| Status | Open (identified, no action) / Mitigating (in progress) / Closed (mitigated or accepted) |
| Last Reviewed | Date of most recent reassessment |
| Reassessment Trigger | Event that forces a re-score (incident, architecture change, release) |

## Scoring Anchors

| Rating | Probability | Impact |
|--------|-------------|--------|
| 5 | Will fail in production within a quarter | Data loss, security breach, revenue stoppage |
| 4 | Expected to fail within a year | Major feature outage, SLA breach |
| 3 | Could fail; uncertain | Degraded experience, workaround exists |
| 2 | Unlikely given current controls | Cosmetic, minor inconvenience |
| 1 | Extremely unlikely; well-understood code | No user-visible impact |

## Reassessment Triggers

Re-score the register when any of these events occur:

- [ ] Production incident in a registered area
- [ ] Architecture change (new dependency, refactor)
- [ ] New regulatory requirement
- [ ] Major release or migration
- [ ] Quarterly calendar review (default cadence)
- [ ] Team change (key engineer leaves)
- [ ] Customer escalation

## Summary

| Tier | Count | Total Test Hours (estimate) |
|------|------:|----------------------------:|
| P0 (20–25) | | |
| P1 (12–19) | | |
| P2 (6–11) | | |
| P3 (1–5) | | |
| **Total** | | |

# Test Strategy

> Fill in each section for the target project or feature. Replace `<...>` placeholders with concrete values. Delete guidance notes once populated.

## Context

| Field | Value |
|-------|-------|
| Project / Feature | <name> |
| Version / Release | <version or release identifier> |
| Author | <author> |
| Date | <YYYY-MM-DD> |
| Status | Draft / In Review / Approved |

### Scope

<What is in scope for this strategy? List the components, features, or services under test.>

### Out of Scope

<What is explicitly excluded and why?>

## Risk Tiers

Assign each area a risk tier (P0–P3) using probability × impact scoring. See [risk-based-testing.md](../references/risk-based-testing.md) and [assets/risk-matrix-grid.md](../assets/risk-matrix-grid.md) for the scoring grid.

| Area / Component | Probability (1–5) | Impact (1–5) | Score | Tier |
|------------------|--------------------:|-------------:|------:|------|
| <area 1> | | | | P0 / P1 / P2 / P3 |
| <area 2> | | | | P0 / P1 / P2 / P3 |
| <area 3> | | | | P0 / P1 / P2 / P3 |

### Tier Definitions

| Tier | Score Range | Coverage Approach |
|------|-------------|-------------------|
| P0 — Critical | 20–25 | Exhaustive: every path, every edge case |
| P1 — High | 12–19 | All happy paths + known failure modes |
| P2 — Medium | 6–11 | Happy paths + common failure modes |
| P3 — Low | 1–5 | Smoke test only; defer detailed testing |

## Test Levels

| Level | What It Covers | Framework / Tool | Target Coverage by Tier |
|-------|---------------|-----------------|------------------------|
| Unit | <e.g., business logic, validators> | <framework> | P0: ___% / P1: ___% / P2: ___% / P3: ___% |
| Integration | <e.g., API contracts, service boundaries> | <framework> | P0: ___% / P1: ___% / P2: ___% / P3: ___% |
| End-to-End | <e.g., critical user journeys> | <framework> | P0: ___% / P1: ___% / P2: ___% / P3: ___% |
| Exploratory | <e.g., new features, unknown-risk areas> | SBTM charter | Sessions per sprint: ___ |

> The pyramid shape is a heuristic, not dogma. Adjust level allocation to match your system's risk profile. See [test-strategy.md](../references/test-strategy.md).

## Coverage Approach

| Dimension | How It Is Measured | Diagnostic Target | Notes |
|-----------|--------------------|-------------------|-------|
| Code coverage | <tool> | <e.g., 80% line on P0 modules> | Coverage is diagnostic, not a goal |
| Requirement coverage | <traceability matrix / RTM> | Every AC has ≥ 1 test | |
| Risk coverage | <risk register mapping> | Every P0/P1 risk has tests | |
| Mutation score | <tool, e.g., PIT/Stryker/mutmut> | <target % on critical paths> | Optional; measures test effectiveness |

## Environment

| Environment | Purpose | Data Strategy | Access |
|-------------|---------|---------------|--------|
| <e.g., local / CI> | <unit + integration> | <fixtures / factories> | <who> |
| <e.g., staging> | <integration + E2E> | <masked subset / synthetic> | <who> |
| <e.g., perf> | <load / soak> | <generated at scale> | <who> |

> Never use production PII in test environments. See [test-data-management.md](../references/test-data-management.md).

## Execution Plan

| Phase | What Runs | Trigger | Cadence |
|-------|-----------|---------|---------|
| Pre-merge | <unit + lint + fast integration> | PR opened / updated | Every PR |
| Post-merge | <full integration + E2E> | Merge to main | Every merge |
| Nightly | <full regression + perf smoke> | Scheduled | Daily |
| Release | <full suite + exploratory + security scan> | Release candidate | Per release |

## Automation Targets

| Area | Current State | Target State | Priority |
|------|--------------|--------------|----------|
| <area 1> | <manual / partial / automated> | <target> | P0 / P1 / P2 / P3 |
| <area 2> | <manual / partial / automated> | <target> | P0 / P1 / P2 / P3 |

## Exit Criteria

The test strategy is complete when:

- [ ] Every in-scope area has a risk tier assigned (P0–P3)
- [ ] Test level allocation is defined per tier
- [ ] Coverage dimensions and diagnostic targets are set
- [ ] Environment and data strategy are specified
- [ ] Execution cadence is agreed with the team
- [ ] Automation targets have owners and timelines
- [ ] Exit criteria for each tier are defined below

### Per-Tier Exit Criteria

| Tier | Exit Criteria |
|------|--------------|
| P0 | <e.g., 100% path coverage, zero open critical defects, mutation score ≥ ___%> |
| P1 | <e.g., all happy paths + failure modes pass, zero open high defects> |
| P2 | <e.g., happy paths pass, known issues documented> |
| P3 | <e.g., smoke suite green> |

# Verification Plan

> Plan independent verification for a spec, feature, or AI-generated implementation. Fill in the traceability table, assign verifiers, define evidence, and set exit criteria. See [ai-code-quality-gates.md](../references/ai-code-quality-gates.md) for gate context and the independent-verification principle.

## Metadata

| Field | Value |
|-------|-------|
| Feature / Spec | <name or spec document reference> |
| Author | <who wrote this plan> |
| Date | <YYYY-MM-DD> |
| Gate Entry | Gate 1 (spec review) / Gate 3 (implementation) / Gate 4 (acceptance) |
| Status | Draft / In Review / Active / Complete |

## AC → Verification-Method Traceability

Every acceptance criterion must map to at least one verification method. No AC may be unmapped.

| AC ID | Acceptance Criterion | Verification Method | Verifier | Evidence Format | Status |
|-------|---------------------|--------------------:|----------|-----------------|--------|
| AC-1 | <criterion text> | Test / Inspection / Analysis / Demonstration | <who or what> | <artifact that proves it> | Pending / Pass / Fail |
| AC-2 | <criterion text> | Test / Inspection / Analysis / Demonstration | <who or what> | <artifact that proves it> | Pending / Pass / Fail |
| AC-3 | <criterion text> | Test / Inspection / Analysis / Demonstration | <who or what> | <artifact that proves it> | Pending / Pass / Fail |

### Verification Method Definitions

| Method | When to Use |
|--------|-------------|
| **Test** | Executable check: automated script, unit/integration/E2E test |
| **Inspection** | Static review: code review, spec walkthrough, checklist audit |
| **Analysis** | Computed or derived evidence: metrics, logs, statistical sampling |
| **Demonstration** | Live or recorded walkthrough showing behavior under controlled conditions |

## Verifier Assignment

> The implementing agent (or developer) MUST NOT self-verify. Assign verification to a separate agent session, a different human, or an independent automated process.

| AC ID(s) | Verifier | Independence Mechanism |
|-----------|----------|----------------------|
| <AC-1, AC-2> | <separate agent session / human reviewer / CI pipeline> | <fresh context, no shared priors / different team member> |
| <AC-3> | <separate agent session / human reviewer / CI pipeline> | <fresh context, no shared priors / different team member> |

## Evidence Format

| Evidence Type | Format | Storage / Link |
|---------------|--------|----------------|
| Test output | <log file, JSON report, CI run URL> | <path or URL> |
| Inspection record | <review comments, checklist sign-off> | <path or URL> |
| Metrics snapshot | <dashboard screenshot, exported CSV> | <path or URL> |
| Demonstration | <screen recording, live session recording> | <path or URL> |

## NFR Verification

| NFR | Category | Measurement Approach | Threshold | Evidence |
|-----|----------|---------------------|-----------|----------|
| <e.g., API latency> | Performance | <p95 under load test> | <e.g., p95 < 200ms> | <perf report link> |
| <e.g., error rate> | Reliability | <soak test + monitoring> | <e.g., < 0.1%> | <monitoring snapshot> |
| <e.g., auth bypass> | Security | <SAST scan + manual review> | <zero critical findings> | <scan report link> |
| <e.g., response shape> | Contract | <contract test (Pact)> | <all consumers pass> | <contract test output> |

## Non-Determinism Handling

| Condition | Verification Approach |
|-----------|----------------------|
| Output is deterministic (same input → same output) | Single-run equality check |
| Output varies across N runs but has invariants | Property-based assertions + N-run sampling (N ≥ 5) |
| Output is probabilistic with known distribution | Statistical tolerance bands (e.g., p95 < X) |
| Output depends on external state | Pin external state; verify under controlled conditions |

**Decision rule:** Run the implementation 5 times with identical inputs. If any output differs, use property-based or statistical verification.

## Exit Criteria

Verification is complete when:

- [ ] 100% of ACs have a mapped verification method (no unmapped ACs)
- [ ] All verifications executed with evidence attached
- [ ] Zero critical or high severity findings remain open
- [ ] NFR thresholds met with evidence
- [ ] Verifier independence confirmed (implementer ≠ verifier)
- [ ] <Additional project-specific criteria: ___>

## Findings Log

| Finding ID | AC | Description | Severity | Status | Resolution |
|------------|-----|-------------|----------|--------|------------|
| VF-001 | <AC-ID> | <what was found> | Critical / High / Medium / Low | Open / Resolved / Accepted | <how resolved> |
| VF-002 | <AC-ID> | <what was found> | Critical / High / Medium / Low | Open / Resolved / Accepted | <how resolved> |

"""Tests for check-ac-testability.py.

Covers: AC testability classification (vague vs concrete), exit codes,
--help, malformed/missing input handling, and idempotency.

Discoverable by both pytest and unittest (unittest.TestCase classes).
"""

import os
import subprocess
import sys
import tempfile
import unittest

SCRIPTS_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "scripts")
AC_SCRIPT = os.path.join(SCRIPTS_DIR, "check-ac-testability.py")


def run_ac(args, stdin_data=None):
    """Run check-ac-testability.py with given args, return (returncode, stdout, stderr)."""
    cmd = [sys.executable, AC_SCRIPT] + args
    proc = subprocess.run(
        cmd,
        input=stdin_data,
        capture_output=True,
        text=True,
        timeout=30,
    )
    return proc.returncode, proc.stdout, proc.stderr


class TestAcTestabilityClassification(unittest.TestCase):
    """Vague vs concrete AC classification."""

    def test_vague_should_handle(self):
        """'should handle errors gracefully' is flagged as untestable."""
        spec = "- The API should handle errors gracefully\n"
        rc, out, _ = run_ac([], spec)
        self.assertEqual(rc, 1)
        self.assertIn("FAIL", out)
        self.assertIn("handle", out.lower())

    def test_vague_should_be_efficient(self):
        """'should be efficient' is flagged as untestable."""
        spec = "- The system should be efficient under load\n"
        rc, out, _ = run_ac([], spec)
        self.assertEqual(rc, 1)
        self.assertIn("FAIL", out)

    def test_vague_should_work_properly(self):
        """'should work properly' is flagged as untestable."""
        spec = "- The feature should work properly\n"
        rc, out, _ = run_ac([], spec)
        self.assertEqual(rc, 1)
        self.assertIn("FAIL", out)

    def test_concrete_returns_200(self):
        """'returns 200 with {id} when X' is testable."""
        spec = "- The API returns 200 with {id} when the resource exists\n"
        rc, out, _ = run_ac([], spec)
        self.assertEqual(rc, 0)
        self.assertIn("PASS", out)

    def test_concrete_exit_code(self):
        """'exits with code 1' is testable."""
        spec = "- The script exits with code 1 when input is missing\n"
        rc, out, _ = run_ac([], spec)
        self.assertEqual(rc, 0)
        self.assertIn("PASS", out)

    def test_concrete_within_time(self):
        """'within 200ms' is testable."""
        spec = "- The search endpoint responds within 200 ms\n"
        rc, out, _ = run_ac([], spec)
        self.assertEqual(rc, 0)
        self.assertIn("PASS", out)

    def test_concrete_displays_message(self):
        """'displays an error message' is testable."""
        spec = "- The UI displays an error message when validation fails\n"
        rc, out, _ = run_ac([], spec)
        self.assertEqual(rc, 0)
        self.assertIn("PASS", out)

    def test_mixed_input(self):
        """Mixed spec: vague flagged, concrete passed, exit 1."""
        spec = (
            "- The system should handle all edge cases\n"
            "- The API returns 404 when the item does not exist\n"
        )
        rc, out, _ = run_ac([], spec)
        self.assertEqual(rc, 1)
        self.assertIn("FAIL", out)
        self.assertIn("PASS", out)

    def test_all_testable_exit_0(self):
        """All-concrete spec exits 0."""
        spec = (
            "- The API returns 200 with {token} on successful login\n"
            "- The API returns 401 when credentials are invalid\n"
        )
        rc, out, _ = run_ac([], spec)
        self.assertEqual(rc, 0)
        self.assertNotIn("FAIL", out)

    def test_all_vague_exit_1(self):
        """All-vague spec exits 1."""
        spec = (
            "- The system should be robust\n"
            "- The system should be scalable\n"
        )
        rc, out, _ = run_ac([], spec)
        self.assertEqual(rc, 1)

    def test_no_acs_detected_exit_0(self):
        """Input with no AC-like lines exits 0."""
        spec = "# Project Overview\n\nThis is a description of the project.\n"
        rc, out, _ = run_ac([], spec)
        self.assertEqual(rc, 0)
        self.assertIn("No acceptance criteria", out)


class TestAcTestabilityExitCodes(unittest.TestCase):
    """Exit codes match the documented contract."""

    def test_testable_exit_0(self):
        """All-testable input exits 0."""
        spec = "- The endpoint returns 200 when the user is authenticated\n"
        rc, _, _ = run_ac([], spec)
        self.assertEqual(rc, 0)

    def test_untestable_exit_1(self):
        """Untestable criteria exit 1."""
        spec = "- The system should handle concurrency appropriately\n"
        rc, _, _ = run_ac([], spec)
        self.assertEqual(rc, 1)

    def test_missing_file_exit_2(self):
        """Nonexistent file path exits 2."""
        rc, _, err = run_ac(["/nonexistent/spec.md"])
        self.assertEqual(rc, 2)
        self.assertIn("error", err.lower())

    def test_empty_input_exit_2(self):
        """Empty stdin exits 2."""
        rc, _, err = run_ac([], "")
        self.assertEqual(rc, 2)
        self.assertIn("empty", err.lower())


class TestAcTestabilityNoTraceback(unittest.TestCase):
    """Malformed input never produces a Python traceback."""

    def test_no_traceback_missing_file(self):
        """No traceback on nonexistent file."""
        rc, out, err = run_ac(["/nonexistent/spec.md"])
        self.assertNotEqual(rc, 0)
        self.assertNotIn("Traceback", err)
        self.assertNotIn("Traceback", out)

    def test_no_traceback_empty_input(self):
        """No traceback on empty input."""
        rc, out, err = run_ac([], "")
        self.assertNotEqual(rc, 0)
        self.assertNotIn("Traceback", err)
        self.assertNotIn("Traceback", out)


class TestAcTestabilityHelp(unittest.TestCase):
    """--help works correctly."""

    def test_help_exits_0(self):
        """--help exits 0."""
        rc, out, _ = run_ac(["--help"])
        self.assertEqual(rc, 0)

    def test_help_has_usage(self):
        """--help output describes usage."""
        rc, out, _ = run_ac(["--help"])
        self.assertIn("usage", out.lower())
        self.assertIn("check-ac-testability", out.lower())


class TestAcTestabilityIdempotency(unittest.TestCase):
    """Running twice on the same input yields identical output."""

    def test_idempotent_output(self):
        """Two runs produce identical stdout."""
        spec = (
            "- The API should handle rate limiting\n"
            "- The API returns 429 when the rate limit is exceeded\n"
        )
        rc1, out1, _ = run_ac([], spec)
        rc2, out2, _ = run_ac([], spec)
        self.assertEqual(rc1, rc2)
        self.assertEqual(out1, out2)

    def test_no_input_mutation(self):
        """Input file is not modified."""
        spec = "- The API returns 200 with {id} when X\n"
        with tempfile.NamedTemporaryFile(
            mode="w", suffix=".md", delete=False
        ) as tmp:
            tmp.write(spec)
            tmp_path = tmp.name
        try:
            with open(tmp_path, "r") as f:
                before = f.read()
            rc, _, _ = run_ac([tmp_path])
            self.assertEqual(rc, 0)
            with open(tmp_path, "r") as f:
                after = f.read()
            self.assertEqual(before, after)
        finally:
            os.unlink(tmp_path)


class TestAcTestabilityFileInput(unittest.TestCase):
    """File path input works correctly."""

    def test_file_input(self):
        """Reading from a file path works."""
        spec = "- The API returns 201 when the resource is created\n"
        with tempfile.NamedTemporaryFile(
            mode="w", suffix=".md", delete=False
        ) as tmp:
            tmp.write(spec)
            tmp_path = tmp.name
        try:
            rc, out, _ = run_ac([tmp_path])
            self.assertEqual(rc, 0)
            self.assertIn("PASS", out)
        finally:
            os.unlink(tmp_path)


if __name__ == "__main__":
    unittest.main()

"""Tests for risk-prioritize.py.

Covers: P×I ranking math (ordering, ties), --json output parseability,
exit codes, --help, malformed input handling, and idempotency.

Discoverable by both pytest and unittest (unittest.TestCase classes).
"""

import json
import os
import subprocess
import sys
import tempfile
import unittest

SCRIPTS_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "..", "scripts")
RISK_SCRIPT = os.path.join(SCRIPTS_DIR, "risk-prioritize.py")


def run_risk(args, stdin_data=None):
    """Run risk-prioritize.py with given args, return (returncode, stdout, stderr)."""
    cmd = [sys.executable, RISK_SCRIPT] + args
    proc = subprocess.run(
        cmd,
        input=stdin_data,
        capture_output=True,
        text=True,
        timeout=30,
    )
    return proc.returncode, proc.stdout, proc.stderr


class TestRiskPrioritizeRankingMath(unittest.TestCase):
    """P×I ranking math: ordering and ties."""

    def test_basic_ordering(self):
        """Higher P×I scores rank first."""
        items = [
            {"id": "low", "probability": 1, "impact": 1},
            {"id": "high", "probability": 5, "impact": 5},
            {"id": "mid", "probability": 3, "impact": 4},
        ]
        rc, out, _ = run_risk(["--json"], json.dumps(items))
        self.assertEqual(rc, 0)
        results = json.loads(out)
        self.assertEqual(results[0]["id"], "high")
        self.assertEqual(results[0]["score"], 25)
        self.assertEqual(results[1]["id"], "mid")
        self.assertEqual(results[1]["score"], 12)
        self.assertEqual(results[2]["id"], "low")
        self.assertEqual(results[2]["score"], 1)

    def test_score_computation(self):
        """Score equals probability × impact."""
        items = [{"id": "x", "probability": 4, "impact": 3}]
        rc, out, _ = run_risk(["--json"], json.dumps(items))
        self.assertEqual(rc, 0)
        results = json.loads(out)
        self.assertEqual(results[0]["score"], 12)
        self.assertEqual(results[0]["score"], 4 * 3)

    def test_deterministic_tie_break(self):
        """Equal scores are broken by id ascending."""
        items = [
            {"id": "zeta", "probability": 3, "impact": 4},
            {"id": "alpha", "probability": 4, "impact": 3},
            {"id": "mid", "probability": 2, "impact": 6},  # invalid but tests ordering logic
        ]
        # Use valid items only (impact 1-5)
        items = [
            {"id": "zeta", "probability": 3, "impact": 4},
            {"id": "alpha", "probability": 4, "impact": 3},
            {"id": "beta", "probability": 2, "impact": 5},  # score 10
        ]
        rc, out, _ = run_risk(["--json"], json.dumps(items))
        self.assertEqual(rc, 0)
        results = json.loads(out)
        # zeta=12, alpha=12, beta=10
        # tie at 12: alpha < zeta alphabetically
        self.assertEqual(results[0]["id"], "alpha")
        self.assertEqual(results[1]["id"], "zeta")
        self.assertEqual(results[2]["id"], "beta")

    def test_ranks_sequential(self):
        """Ranks are sequential starting at 1."""
        items = [
            {"id": "a", "probability": 5, "impact": 5},
            {"id": "b", "probability": 1, "impact": 1},
            {"id": "c", "probability": 3, "impact": 3},
        ]
        rc, out, _ = run_risk(["--json"], json.dumps(items))
        self.assertEqual(rc, 0)
        results = json.loads(out)
        self.assertEqual([r["rank"] for r in results], [1, 2, 3])


class TestRiskPrioritizeJsonOutput(unittest.TestCase):
    """--json output is machine-parseable."""

    def test_json_parseable(self):
        """--json output parses with json.loads."""
        items = [{"id": "test", "probability": 2, "impact": 3}]
        rc, out, _ = run_risk(["--json"], json.dumps(items))
        self.assertEqual(rc, 0)
        data = json.loads(out)
        self.assertIsInstance(data, list)
        self.assertEqual(len(data), 1)

    def test_json_fields(self):
        """--json output has id, probability, impact, score, rank fields."""
        items = [{"id": "x", "probability": 5, "impact": 4}]
        rc, out, _ = run_risk(["--json"], json.dumps(items))
        self.assertEqual(rc, 0)
        data = json.loads(out)
        entry = data[0]
        self.assertEqual(entry["id"], "x")
        self.assertEqual(entry["probability"], 5)
        self.assertEqual(entry["impact"], 4)
        self.assertEqual(entry["score"], 20)
        self.assertEqual(entry["rank"], 1)

    def test_human_output_by_default(self):
        """Without --json, output is a human-readable table."""
        items = [{"id": "x", "probability": 5, "impact": 4}]
        rc, out, _ = run_risk([], json.dumps(items))
        self.assertEqual(rc, 0)
        self.assertIn("Rank", out)
        self.assertIn("Score", out)
        self.assertIn("x", out)


class TestRiskPrioritizeExitCodes(unittest.TestCase):
    """Exit codes are correct for various inputs."""

    def test_success_exit_0(self):
        """Valid input exits 0."""
        items = [{"id": "ok", "probability": 1, "impact": 1}]
        rc, _, _ = run_risk(["--json"], json.dumps(items))
        self.assertEqual(rc, 0)

    def test_malformed_json_exit_1(self):
        """Invalid JSON exits 1."""
        rc, _, err = run_risk([], "{not valid json")
        self.assertEqual(rc, 1)
        self.assertIn("error", err.lower())

    def test_not_array_exit_1(self):
        """Non-array JSON exits 1."""
        rc, _, err = run_risk([], '{"key": "value"}')
        self.assertEqual(rc, 1)
        self.assertIn("array", err.lower())

    def test_empty_array_exit_1(self):
        """Empty array exits 1."""
        rc, _, err = run_risk([], "[]")
        self.assertEqual(rc, 1)
        self.assertIn("at least one", err.lower())

    def test_missing_field_exit_1(self):
        """Missing required field exits 1."""
        rc, _, err = run_risk([], '[{"id": "x", "probability": 3}]')
        self.assertEqual(rc, 1)
        self.assertIn("impact", err.lower())

    def test_out_of_range_exit_1(self):
        """probability/impact outside 1-5 exits 1."""
        rc, _, err = run_risk([], '[{"id": "x", "probability": 6, "impact": 3}]')
        self.assertEqual(rc, 1)
        self.assertIn("1-5", err)

    def test_non_integer_exit_1(self):
        """Float probability exits 1."""
        rc, _, err = run_risk([], '[{"id": "x", "probability": 3.5, "impact": 3}]')
        self.assertEqual(rc, 1)
        self.assertIn("integer", err.lower())

    def test_empty_input_exit_1(self):
        """Empty input exits 1."""
        rc, _, err = run_risk([], "")
        self.assertEqual(rc, 1)
        self.assertIn("empty", err.lower())


class TestRiskPrioritizeNoTraceback(unittest.TestCase):
    """Malformed input never produces a Python traceback."""

    def test_no_traceback_invalid_json(self):
        """No traceback on invalid JSON."""
        rc, out, err = run_risk([], "{invalid")
        self.assertNotEqual(rc, 0)
        self.assertNotIn("Traceback", err)
        self.assertNotIn("Traceback", out)

    def test_no_traceback_missing_file(self):
        """No traceback on missing file."""
        rc, out, err = run_risk(["/nonexistent/path/file.json"])
        self.assertNotEqual(rc, 0)
        self.assertNotIn("Traceback", err)
        self.assertNotIn("Traceback", out)


class TestRiskPrioritizeHelp(unittest.TestCase):
    """--help works correctly."""

    def test_help_exits_0(self):
        """--help exits 0."""
        rc, out, _ = run_risk(["--help"])
        self.assertEqual(rc, 0)

    def test_help_has_usage(self):
        """--help output describes usage."""
        rc, out, _ = run_risk(["--help"])
        self.assertIn("usage", out.lower())
        self.assertIn("risk-prioritize", out.lower())


class TestRiskPrioritizeIdempotency(unittest.TestCase):
    """Running twice on the same input yields identical output."""

    def test_idempotent_json(self):
        """Two runs produce identical JSON output."""
        items = [
            {"id": "a", "probability": 5, "impact": 5},
            {"id": "b", "probability": 3, "impact": 3},
        ]
        input_data = json.dumps(items)
        rc1, out1, _ = run_risk(["--json"], input_data)
        rc2, out2, _ = run_risk(["--json"], input_data)
        self.assertEqual(rc1, 0)
        self.assertEqual(rc2, 0)
        self.assertEqual(out1, out2)

    def test_idempotent_table(self):
        """Two runs produce identical table output."""
        items = [{"id": "x", "probability": 2, "impact": 4}]
        input_data = json.dumps(items)
        rc1, out1, _ = run_risk([], input_data)
        rc2, out2, _ = run_risk([], input_data)
        self.assertEqual(rc1, 0)
        self.assertEqual(out1, out2)

    def test_no_input_mutation(self):
        """Input file is not modified by the script."""
        items = [{"id": "x", "probability": 5, "impact": 5}]
        with tempfile.NamedTemporaryFile(
            mode="w", suffix=".json", delete=False
        ) as tmp:
            json.dump(items, tmp)
            tmp_path = tmp.name
        try:
            with open(tmp_path, "r") as f:
                before = f.read()
            rc, _, _ = run_risk([tmp_path])
            self.assertEqual(rc, 0)
            with open(tmp_path, "r") as f:
                after = f.read()
            self.assertEqual(before, after)
        finally:
            os.unlink(tmp_path)

    def test_no_artifacts_created(self):
        """No new files appear after running the script."""
        items = [{"id": "x", "probability": 5, "impact": 5}]
        with tempfile.NamedTemporaryFile(
            mode="w", suffix=".json", delete=False, dir=tempfile.gettempdir()
        ) as tmp:
            json.dump(items, tmp)
            tmp_path = tmp.name
        tmp_dir = tempfile.gettempdir()
        before_files = set(os.listdir(tmp_dir))
        try:
            rc, _, _ = run_risk([tmp_path])
            self.assertEqual(rc, 0)
            after_files = set(os.listdir(tmp_dir))
            new_files = after_files - before_files
            # Filter out files from other processes
            script_artifacts = [
                f for f in new_files if "risk" in f.lower() or "priorit" in f.lower()
            ]
            self.assertEqual(script_artifacts, [])
        finally:
            os.unlink(tmp_path)


class TestRiskPrioritizeFileInput(unittest.TestCase):
    """File path input works correctly."""

    def test_file_input(self):
        """Reading from a file path works."""
        items = [{"id": "file-test", "probability": 4, "impact": 2}]
        with tempfile.NamedTemporaryFile(
            mode="w", suffix=".json", delete=False
        ) as tmp:
            json.dump(items, tmp)
            tmp_path = tmp.name
        try:
            rc, out, _ = run_risk(["--json", tmp_path])
            self.assertEqual(rc, 0)
            data = json.loads(out)
            self.assertEqual(data[0]["id"], "file-test")
            self.assertEqual(data[0]["score"], 8)
        finally:
            os.unlink(tmp_path)


if __name__ == "__main__":
    unittest.main()