.skillmeta.yml
id: engineering.quality-assurance
name: quality-assurance
origin:
repo: alvarovillalbaa/plugins
branch: main
path: engineering/skills/quality-assurance
install:
mode: project-managed-copy
agents:
- codex
- cursor
- openclaw
- claude-code
personalization:
policy: overlay-only
local_files:
- personalize.local.yml
- "*.local.yml"
- .overlays/**
- .generated/**
- .company/**
- .user/**
upstream_contribution:
allowed_paths:
- .skillmeta.yml
- SKILL.md
- README.md
- agents/**
- scripts/**
- references/**
- templates/**
- examples/**
- hooks/**
- personalization.schema.json
- personalize.example.yml
- evals/**
forbidden_paths:
- personalize.local.yml
- "*.local.yml"
- .overlays/**
- .generated/**
- .company/**
- .user/**
- generated/**
- private/**
quality_gates:
require_eval: false
require_diff_classification: true
require_human_review: true
examples/README.md
# Examples
- `test-layout-audit-report.md` - a captured `audit_test_layout.py` report on a mostly-conforming
repository, showing a grandfathered bucket, a stale scratch file, and a correctly gated Tier 5 test.
examples/test-layout-audit-report.md
# Example: test layout audit report
Captured output from `scripts/audit_test_layout.py` run against a repository that mostly conforms:
it has a grandfathered `contract/` bucket, a stale scratch file, and one correctly gated Tier 5 test.
The point of this example is the Tier 5 case. `tests/regression/test_balance_invariant.py` connects
to production and produces **zero violations**, because it satisfies all four gate conditions - it
reads an authorization reference from the environment, verifies a read-only role, is marked so it can
be deselected, and asserts an aggregate invariant rather than pulling rows into the test process.
```python
# tests/regression/test_balance_invariant.py
import os
import pytest
REF = os.environ["PROD_AUTHORIZATION_REF"] # fails closed when unset
ROLE = os.environ["READONLY_ROLE"] # verified, not assumed
@pytest.mark.regression
@pytest.mark.tier5
def test_no_negative_balances(readonly_conn):
assert readonly_conn.scalar("select count(*) from accounts where balance < 0") == 0
```
Remove any one of those four properties and the same file becomes a violation.
## Command
```bash
python3 scripts/audit_test_layout.py /path/to/repo
```
## Output
```text
# Test layout audit
Repo: /path/to/repo
Tests dir: tests
Stack: python
## Layout
- Present: e2e, evals, integration, regression, smoke, tmp, unit
- Absent: adversarial
- Unknown buckets: none
Absent folders are not failures. `evals/` applies to AI products; `smoke/` and
`adversarial/` are optional.
## Data tiers
- `unit/`: none declared
- `integration/`: T3
- `e2e/`: none declared
- `smoke/`: none declared
- `regression/`: T5
- `evals/`: none declared
- `tmp/`: none declared
## Violations (0)
- none
## Warnings (2)
- `tests/contract`: 'contract/' predates the contract; grandfathered. New tests of this kind belong in tests/integration/ with a 'contract' marker.
- `tests/tmp/tmp_probe_20250101.py`: scratch file is 583 days old (budget 7); promote it or delete it
## Next steps
1. Promote or delete stale scratch tests under tests/tmp/.
```
A `## JSON` block with the same payload follows the report. Use `--json` to get it alone.
## Reading the result
- **Exit code 0.** Warnings do not fail the audit. Run with `--strict` to promote them.
- **`contract/` is a warning, not a violation.** Pre-existing buckets are grandfathered; only new
top-level buckets fail. New contract tests belong in `tests/integration/` with a `contract` marker.
- **The stale scratch file is the real finding.** It should have been promoted to `tests/regression/`
or deleted before the task that created it was declared done.
- **`adversarial/` absent is fine.** So is `evals/` in a non-AI product. Absence is never a failure;
only an unknown bucket, an illegal tier, or a failed gate is.
README.md
# Quality Assurance
Router for the Engineering QA lanes, and owner of the canonical test-suite contract that binds all of them.
## What it owns
- The canonical `tests/` layout: `unit/`, `integration/`, `e2e/`, `smoke/`, `regression/`, `adversarial/`, `evals/`, and `tmp/`, plus support directories.
- The five test-data tiers, from no data through mock data, a local replica database, a staging replica with rollback, and read-only production access under a four-condition gate.
- The framework-per-test-type mapping across Python, TypeScript, browser, Go, and Ruby stacks.
- The `tests/evals/` wrapping contract for AI products: tests that invoke an eval suite the eval system already defines, without redefining it.
- Routing to the narrowest child: testing, simplification, security, and AI evals.
## Install
```bash
npx -y skills add ./engineering/skills/quality-assurance
mkdir -p ~/.codex/skills
cp -R engineering/skills/quality-assurance ~/.codex/skills/
```
Codex `$skill-installer` path:
```text
https://github.com/alvarovillalbaa/plugins/tree/main/engineering/skills/quality-assurance
```
## What is bundled
- `references/` - the test-suite contract (layout, data tiers, frameworks, eval wrapping) and cross-cutting failure diagnosis.
- `scripts/audit_test_layout.py` - audits a repository's `tests/` tree against the contract. Read-only.
- `templates/test-suite-scaffold.md` - copy-ready tree, folder README stubs, runner configuration, and the `tests/README.md` contract table.
- `examples/` - a worked audit report.
Start with [`SKILL.md`](./SKILL.md).
references/debugging.md
# Debugging Reference
Cross-cutting failure diagnosis, shared by every QA lane. Reach for a child skill once the failure
class is known: `testing` for coverage and suite design, its `flake` child for intermittency, and
`frontend-e2e` for browser-level symptoms.
## Contents
- [Debugging loop](#debugging-loop)
- [Triage the failure class first](#triage-the-failure-class-first)
- [Read failures completely](#read-failures-completely)
- [Error diagnosis workflow](#error-diagnosis-workflow)
- [CI-only failure workflow](#ci-only-failure-workflow)
- [Regression correlation](#regression-correlation)
- [Flaky test protocol](#flaky-test-protocol)
- [Performance debugging](#performance-debugging)
- [Performance diagnosis workflow](#performance-diagnosis-workflow)
- [Observability-led debugging](#observability-led-debugging)
- [Error Pattern Reference](#error-pattern-reference)
- [CI-Only Failure Patterns](#ci-only-failure-patterns)
- [Debugging Async Code](#debugging-async-code)
## Debugging loop
Never jump from symptom to fix. Use this order:
1. Reproduce the failure.
2. Reduce it to the smallest reliable case.
3. Read the full error, logs, or diff.
4. Correlate with recent code, config, or dependency changes.
5. Form one concrete hypothesis at a time.
6. Prove or falsify the hypothesis quickly.
7. Fix the root cause and verify with a regression proof.
## Triage the failure class first
| Failure class | Typical signal | First move |
|---|---|---|
| Crash or exception | stack trace, panic, unhandled rejection | read the full trace from root cause outward |
| Wrong output | no crash, wrong value or DOM | tighten assertions and isolate inputs |
| Flaky | passes sometimes | search for shared state, time, randomness, race conditions |
| CI-only | local passes, CI fails | diff env, OS, container, secrets, cached state |
| Performance | slow request, timeout, OOM | measure before changing code |
| Data or rollout issue | works on fresh data, fails on real data | inspect migrations, nullability, mixed-version compatibility |
## Read failures completely
Common mistake: reading only the last line.
- In stack traces, start at the root cause and work outward.
- In assertion failures, inspect expected vs actual values, local variables, and fixture setup.
- In CI logs, identify the first failing step, not just the final job summary.
- In browser failures, inspect console errors, network failures, screenshot or trace artifacts, and DOM timing.
## Error diagnosis workflow
For application errors, move in this order:
1. Extract the concrete context: module, line, function, failing input, environment, and first visible symptom.
2. Identify the owner layer: transport, schema or validation, domain logic, persistence, async job, external integration, or UI state.
3. Correlate with recent code, config, dependency, schema, or data-shape changes.
4. Fix the root cause and add the narrowest regression proof that would have caught it.
5. Record residual risk if the bug depends on production-only data volume, concurrency, or rollout state.
Useful prompts to answer before editing:
- What exact input or state made this path fail?
- Did the code assume a field, relation, env var, cache entry, or timing guarantee that is not actually guaranteed?
- Is the symptom happening at the true owner layer, or only surfacing there from a deeper fault?
- What smaller test or reproduction isolates the fault without the rest of the stack?
## CI-only failure workflow
When something fails only in CI:
1. Compare runtime versions: language, package manager, browser, DB, OS image.
2. Compare env and secrets presence, not just values.
3. Compare working directory, path filters, and changed-files logic.
4. Check cache keys and stale artifact reuse.
5. Re-run the exact CI command locally or in a matching container if possible.
6. Inspect whether tests rely on wall clock, locale, timezone, filesystem ordering, or network availability.
Frequent causes:
- missing env var or secret
- path-dependent file lookup
- dependency drift between local and CI
- hidden reliance on local services or seeded data
- race condition exposed by slower or faster CI machines
## Regression correlation
Use history before guessing.
```bash
git log --oneline -- path/to/failing/file
git blame path/to/file
git diff <known-good>...HEAD -- path/to/file
git bisect start
```
Patterns that often explain sudden failures:
- new required field or schema change
- renamed import or moved symbol
- stricter auth or feature-flag logic
- altered serialization shape
- dependency upgrade or test runner upgrade
- hidden change in default timezone, locale, or clock
## Flaky test protocol
Flakes have causes. Find the dimension that changes:
- order: the test depends on another test having run first
- time: midnight, DST, token expiry, debounce, retry windows
- randomness: UUIDs, Faker, seeds, unstable sorting
- concurrency: unsafely shared resources, parallel workers, missing awaits
- environment: browser, OS, path casing, CPU speed, network availability
- leakage: global state, DB rows, files, sockets, mocks not reset
Useful tactics:
```bash
pytest path::test_name -x -vv
pytest path::test_name --count=20
pytest --randomly-seed=<seed>
npx playwright test --repeat-each=10
```
Fixes usually involve:
- hermetic setup and teardown
- fake or frozen time
- deterministic data
- replacing `sleep()` with explicit wait conditions
- isolating shared mutable state
- reducing reliance on test order or global caches
## Performance debugging
Measure first. Performance bugs often masquerade as correctness bugs in CI.
Watch for:
- N+1 queries or repeated HTTP calls in loops
- repeated rendering or expensive selectors in UI
- full-table scans, missing indexes, or no pagination
- loading unbounded data into memory
- retries without backoff or cancellation
Good proof patterns:
- query count assertions for hot endpoints
- benchmark or profiling comparison before and after the change
- browser traces and performance profiles for UI regressions
- memory snapshots for leaks or runaway caches
## Performance diagnosis workflow
1. Capture a baseline: response time, query count, retry count, CPU, memory, render time, or queue latency.
2. Identify the dominant cost class: database, network, render, serialization, lock contention, retry storm, or unbounded iteration.
3. Reproduce with the smallest realistic dataset or request shape that still shows the problem.
4. Change the dominant cost driver first instead of spreading small guesses across many layers.
5. Re-measure the same scenario after the fix.
Common root causes:
- N+1 queries or repeated API calls in loops
- missing indexes, full scans, or unbounded result sets
- repeated serialization or expensive derived fields on hot paths
- loading large datasets into memory instead of streaming or chunking
- retry loops without backoff, cancellation, or deduplication
- browser work tied to every render instead of meaningful state changes
## Observability-led debugging
If logs, metrics, traces, or artifacts already exist, use them before adding new instrumentation.
Useful evidence sources:
- structured error logs with request or trace IDs
- CI artifacts: screenshots, videos, junit XML, coverage XML
- DB slow query logs
- browser console and network traces
- queue retries, dead-letter queues, worker heartbeats
Add instrumentation only after confirming existing signals are insufficient, and remove temporary noise once the issue is resolved.
If the failing path is:
- request-driven: follow request or trace IDs across gateway, app, datastore, and downstream calls
- queue-driven: inspect enqueue time, retry history, dead-letter flow, and worker logs together
- browser-driven: inspect network waterfall, console, performance trace, and UI state transitions together
- rollout-driven: compare release version, config, feature flags, migration state, and environment scope
---
## Error Pattern Reference
### Python / Django
| Error | Common Cause | Fix |
|---|---|---|
| `KeyError: 'field'` | Dict access without `.get()` | `data.get('field')` with guard |
| `AttributeError: 'NoneType'` | FK or `.get()` returned None | Null check before attribute access |
| `OperationalError: no such column` | Migration not applied | `python manage.py migrate` |
| `ValidationError: {...}` | Required field missing in request | Check required fields in serializer |
| `IntegrityError: UNIQUE constraint` | Duplicate insert | Check for existing record first |
| `DoesNotExist` | `.get()` with no match | Use `.filter().first()` or handle exception |
| `MultipleObjectsReturned` | `.get()` with multiple matches | Add more specific filter |
| `RecursionError` | Circular signal or recursive call | Check signal handlers; add recursion guard |
| `celery.exceptions.Retry` | Task explicitly retrying | Expected; check max_retries |
### JavaScript / TypeScript
| Error | Common Cause | Fix |
|---|---|---|
| `TypeError: Cannot read property 'x' of undefined` | Missing null check | Optional chaining: `obj?.x` |
| `TypeError: x is not a function` | Wrong type passed | Check type before calling |
| `UnhandledPromiseRejection` | Missing `await` or `.catch()` | Add try/catch or `.catch()` |
| `ReferenceError: x is not defined` | Incorrect import or scope | Check import paths |
| `Maximum call stack size exceeded` | Infinite recursion | Add base case; check event listeners |
| `CORS error` | Missing CORS header | Configure server CORS; check origin |
| `Hydration mismatch` | SSR/CSR output differs | Ensure same rendering logic on server and client |
---
## CI-Only Failure Patterns
When a test passes locally but fails in CI:
| Pattern | Cause | Fix |
|---|---|---|
| Missing env var | CI doesn't have the same `.env` | Add to CI secrets; check `required_env_var` guards |
| Different Python/Node version | CI uses different runtime | Pin version in CI config |
| Missing migration | Local DB has extra state; CI starts fresh | Ensure migration is committed |
| Port conflict | CI runs multiple services on same port | Use dynamic ports or wait logic |
| Timezone difference | CI runs in UTC; local is different | Freeze time in tests; don't rely on local timezone |
| DB not ready | CI starts DB in parallel with tests | Add health checks / wait-for-it scripts |
| Missing `git fetch` | Shallow clone in CI misses history | Use `fetch-depth: 0` in CI config |
---
## Debugging Async Code
```python
# Python async — use asyncio.run in tests
import asyncio
import pytest
@pytest.mark.asyncio
async def test_async_operation():
result = await my_async_function()
assert result == expected
# Django async views
from django.test import AsyncClient
@pytest.mark.asyncio
@pytest.mark.django_db
async def test_async_view():
client = AsyncClient()
resp = await client.get('/api/async-endpoint/')
assert resp.status_code == 200
```
```typescript
// JavaScript async — always await assertions
it('fetches data', async () => {
const data = await fetchData()
expect(data).toEqual({ id: 1 })
})
// React async rendering
it('shows data after loading', async () => {
render(<DataComponent />)
await waitFor(() => expect(screen.getByText('Loaded')).toBeInTheDocument())
})
```
references/README.md
# references
The canonical test-suite contract - folder layout, data tiers, framework mapping, and eval-wrapping
rules - plus cross-cutting failure diagnosis shared by every QA lane.
references/test-data-tiers.md
# Test Data Tiers Reference
Every test declares exactly one data tier. The tier says what the test is allowed to touch and what
it is allowed to change. Which tiers are legal in which folder is defined by the matrix in
[`test-suite-layout.md`](test-suite-layout.md#folder-x-tier-legality); this file defines the tiers
themselves, how each is provisioned, and the gate that governs production access.
## Contents
- [The five tiers](#the-five-tiers)
- [Choosing a tier](#choosing-a-tier)
- [Tier 1 - no data](#tier-1---no-data)
- [Tier 2 - mock data](#tier-2---mock-data)
- [Tier 3 - local replica database](#tier-3---local-replica-database)
- [Tier 4 - staging replica database](#tier-4---staging-replica-database)
- [Tier 5 - production, read-only](#tier-5---production-read-only)
- [The Tier 5 gate](#the-tier-5-gate)
- [Declaring the tier](#declaring-the-tier)
- [Anti-patterns](#anti-patterns)
## The five tiers
| Tier | Name | Data source | Writes | Runs where |
| --- | --- | --- | --- | --- |
| **T1** | No data | pure inputs in the test body | not applicable | anywhere, any lane |
| **T2** | Mock data | fakes, factories, fixtures, recorded cassettes, in-memory stores | to the fake only | anywhere, any lane |
| **T3** | Local replica database | ephemeral container or dedicated local test database built from the repo's migrations | unrestricted; destructive allowed | developer machine and CI runner |
| **T4** | Staging replica database | a shared remote non-production database | allowed, but rolled back or compensated before the test exits | protected CI lane or a sanctioned remote runner |
| **T5** | Production, read-only | the real production database | **zero**, enforced by a database role with no write grants | on demand only, never on a PR gate |
**No writes to production at any tier.** T5 is a read exception, not a write exception.
**No production service calls with side effects at any tier**, including T5. No emails, charges,
webhooks, queue publishes, or third-party API calls against live accounts.
## Choosing a tier
Pick the lowest tier that can prove the claim. If a lower tier cannot prove it, move up one — not
straight to the top.
```text
Does the behavior depend on stored state at all?
├── no ─────────────────────────────────► T1
└── yes
Does it depend on real persistence semantics —
constraints, transactions, locking, indexes, query count,
serialization of actual stored values?
├── no ──────────────────────────────► T2
└── yes
Can an ephemeral local database prove it?
├── yes ─────────────────────────► T3
└── no (needs the real engine version, real
topology, real queue semantics, or
realistic data volume)
Is there a staging replica?
├── yes ─────────────────────► T4, with rollback
└── no ─────────────────────► T3 with the closest
available container, and state the
residual risk in the test plan
```
**T5 is never the answer to "can a lower tier prove it".** T5 answers a different question: does a
claim about *production's actual current data* hold? See [below](#tier-5---production-read-only).
## Tier 1 - no data
Pure inputs constructed in the test body. No database, no network, no filesystem, no wall clock, no
environment lookups.
Provisioning: none. This is the default for `tests/unit/`.
Determinism rules: inject the clock rather than reading it; inject randomness seeds; never depend on
the timezone or the locale of the machine.
## Tier 2 - mock data
Fakes and fixtures standing in for a real dependency. Includes object factories, static fixture
files, recorded HTTP cassettes, request interceptors, and in-memory stores.
**Mock the boundary, not the internal caller.** Mock outbound HTTP, object storage, mail transport,
payment providers, and external model APIs. Do not mock your own ORM, your own service methods, your
own serializers, or your own business logic — a test that mocks the thing it is testing asserts only
that the mock was configured.
An in-memory database substitute is T2, not T3. It proves your code calls the right operations; it
does not prove your constraints, migrations, or query plans work.
Assert on outcomes — returned values, stored state, emitted events — rather than only on which mock
was called.
## Tier 3 - local replica database
A real database engine, built from the repository's own migrations, that you are free to destroy.
Either an ephemeral container started for the run, or a dedicated local test database.
Provisioning:
- Build the schema from the repo's migrations, never from a hand-maintained SQL dump that drifts.
- Seed with factories, not a giant global seed script.
- Guard the connection: refuse to run if the target does not look like a test database.
- Reset state completely between tests. Transaction rollback where the framework supports it,
truncation otherwise.
This is the default for `tests/integration/`. Destructive operations are fine here; that is the
point of an environment you can drop.
## Tier 4 - staging replica database
A shared remote non-production database. Use it only for behavior a local container genuinely cannot
prove: the real engine version and extensions, real network topology, real queue semantics, or
realistic data volume.
**T4 is optional infrastructure. Many repositories do not have a staging replica, and that is a
valid state.** When there is none, fall back to T3 with the closest available container image and
record the residual risk in the test plan rather than reaching for T5.
Because the environment is shared, T4 carries obligations T3 does not:
- **Every write is rolled back or compensated before the test exits.** Prefer a per-test transaction
that always rolls back. Where a transaction cannot span the operation — anything that depends on
post-commit behavior — compensate explicitly in teardown and assert the compensation worked.
- **Never truncate.** Truncation-based cleanup is a T3 technique. On a shared database it destroys
other people's data.
- Namespace every fixture the test creates so a leak is identifiable and attributable.
- Bound concurrency. A shared database is a shared resource.
- Assume another suite is running at the same time: no test may depend on global row counts or on
being the only writer.
- Never carry production credentials in a T4 runner's configuration, and never point a T4 runner at
a production cluster.
## Tier 5 - production, read-only
The real production database, through a credential that cannot write.
Three claims legitimately need it, and they are all "does something hold about production right
now":
1. Reproducing a defect that only manifests at production data shape or volume.
2. Standing data-integrity invariants — no orphaned record, no negative balance, no active
subscription without a plan.
3. Schema-drift detection between what the code expects and what production actually has.
Everything else is a lower tier wearing a disguise. If the claim is about *your code's* behavior
rather than about *production's data*, use T3.
T5 is permitted in any folder, but permission comes from the gate below, never from the folder. Note
that `tests/tmp/` holds unreviewed scratch code by definition, and `tests/adversarial/` exists
specifically to attempt destructive actions — a T5 test in either folder is a design smell worth
justifying explicitly, and the layout auditor flags both.
## The Tier 5 gate
All four conditions are required. Missing any one means the tier is unavailable.
**1. A database role with no write grants.** The credential the test connects with must hold no
`INSERT`, `UPDATE`, `DELETE`, `TRUNCATE`, `ALTER`, `CREATE`, or `DROP` grant on any object it can
reach. This is the primary control. Everything else in this section is secondary — documentation
does not stop a write, a revoked grant does.
Verify the role, do not assume it. Query the grants at session start and fail the test if any write
privilege is present. A test that cannot prove its own role is read-only must fail rather than
proceed.
**2. A recorded authorization reference.** The run reads an authorization identifier from the
environment — a ticket, an incident, or an approval record naming who authorized production access,
for what, and until when. **Fail closed when it is unset.** Never default it.
**3. Excluded from the default suite and from every PR gate.** Marked and deselected by default, so
running the ordinary suite can never open a production connection. Invoked deliberately, on demand
or on a nightly lane.
**4. Read-only invariants only, reading no more than the assertion needs.** Aggregate and existence
checks over the narrowest possible scope. Do not select customer data into the test process to
inspect it; assert the invariant in the query and return a count. Never write production data into a
test artifact, a log line, a failure message, or a fixture file.
**Configuration is supplied, never embedded.** The connection target, the role name, and the
authorization reference all come from the environment at runtime. Never commit a connection string,
a credential, a hostname, or a role name to the repository — not in a test, not in a fixture, not as
an example in a comment.
## Declaring the tier
Every test that touches data declares its tier where the test runner can select on it, and where a
reader sees it before the test body. Use the repository's existing marker or tag mechanism rather
than inventing a parallel one — `tier1` through `tier5` as markers, tags, or suite names, whichever
the stack already uses.
Record the tiers actually in use, the T5 authorization reference, and the T5 role name in
`tests/README.md`, using the table in
[`../templates/test-suite-scaffold.md`](../templates/test-suite-scaffold.md). That file is where
environment-specific facts belong; they do not belong in this reference or in any skill.
## Anti-patterns
- A T5 test that *could* issue a write. It is a defect regardless of whether it did — the role is
wrong, and the next edit to that file is unguarded.
- Defaulting the authorization reference so an unconfigured run proceeds instead of failing.
- Truncation-based cleanup on a shared T4 database.
- A T4 test with no teardown, on the theory that staging gets reset eventually.
- Reaching for T5 because the local database was inconvenient to set up.
- Mocking your own ORM or service layer and calling the result an integration test.
- An in-memory database substitute described as T3, so constraint and migration bugs ship.
- Production data copied into a fixture file to make a test reproducible.
- A connection string, hostname, or role name committed as an example.
- Tests that share a database and depend on execution order, so sharding is permanently unsafe.
references/test-evals-wrapping.md
# Eval Wrapping Reference
The contract for `tests/evals/`. These tests do **not** define evaluations. They are a thin wrapping
layer that invokes an eval suite the eval system already owns and asserts on the gate it returns.
`ai-evals` owns specs, datasets, graders, calibration, thresholds, and the gate cascade.
`tests/evals/` owns exactly one claim: *the build currently in this repository passes the suite the
eval system already defines.*
## Contents
- [What a file here may and may not contain](#what-a-file-here-may-and-may-not-contain)
- [Two deployment shapes](#two-deployment-shapes)
- [Configuration](#configuration)
- [Skip when unconfigured](#skip-when-unconfigured)
- [Assert the gate, not the score](#assert-the-gate-not-the-score)
- [Exit-code mapping](#exit-code-mapping)
- [Sketches](#sketches)
- [Boundary to ai-evals](#boundary-to-ai-evals)
- [Anti-patterns](#anti-patterns)
## What a file here may and may not contain
**Must not contain** any of: dataset rows, expected outputs, grader or rubric definitions, score
thresholds, pass-rate minimums, prompt versions, judge model names, or scoring arithmetic. If any of
these appears, the evaluation has been redefined inside the test suite and there are now two sources
of truth that will diverge.
**Must contain exactly four things:**
1. An **external suite reference** — an identifier and a version, never inline content.
2. An **invocation** of the eval system that already exists.
3. **Assertions on the official gate** the eval system returned.
4. **Skip semantics** for when the eval system is not configured.
## Two deployment shapes
**Shape A — offline deterministic runner.** The suite is defined by `ai-evals` as a spec and dataset
pair; the test shells out to the offline runner. No network and no model credentials are involved,
so this shape may run in CI where the repository accepts the cost.
**Shape B — hosted eval service.** The test posts a run request naming the suite identifier and
version to an eval service that already exists, then reads back the official gate. Credentials are
required, so this shape is on-demand and pre-release only.
Both shapes obey the same contract. The difference is transport, not semantics.
## Configuration
Supplied at runtime, never embedded. No variable gets a default pointing at a real endpoint, a real
suite, or a real credential.
| Variable | Shape | Meaning |
| --- | --- | --- |
| `EVAL_SUITE_ID` | A, B | the suite the eval system already owns |
| `EVAL_SUITE_VERSION` | A, B | the pinned suite version or sample manifest identifier |
| `EVAL_SPEC_PATH` | A | path to the spec owned by `ai-evals` |
| `EVAL_DATASET_PATH` | A | path to the dataset owned by `ai-evals` |
| `EVAL_RUNNER` | A | path to the offline runner |
| `EVAL_SERVICE_URL` | B | the eval service endpoint |
| `EVAL_SERVICE_TOKEN` | B | the eval service credential |
## Skip when unconfigured
When the required variables are absent, **skip — do not fail, and do not fall back to a local
approximation.** A skipped eval test is an honest "not measured here". A locally approximated one is
a fabricated pass.
This is also the primary mechanism keeping evals out of ordinary CI: with nothing configured, the
ordinary suite reports skips and moves on. Pair it with a deselected `evals` marker for defence in
depth, so an accidentally configured environment still does not pull an eval run into a PR gate.
## Assert the gate, not the score
The eval system already computed a pass or fail decision using criteria it owns. Assert on that
decision.
Two integrity assertions are also recommended, because they catch the failure mode where the run
silently evaluated a different suite than the one intended: assert the returned suite identifier
matches the requested one, and the returned manifest version matches the requested version.
**Forbidden:** asserting a raw score against a number chosen in the test file. That number is a
threshold, thresholds are part of the eval definition, and the eval definition lives in the eval
system. Writing one here is exactly the boundary violation this file exists to prevent.
## Exit-code mapping
For Shape A, the runner distinguishes three outcomes and the test must preserve all three. Collapsing
them hides the difference between "the product regressed" and "the eval itself is broken".
| Runner exit | Meaning | Test outcome |
| --- | --- | --- |
| `0` | the official run gate passed | pass |
| `1` | the run completed and the gate failed | **test failure**, surfacing the failing row identifiers from the result payload |
| `2` | invalid spec, invalid dataset, holdout leakage, or an I/O error | **test error, never a soft fail** |
Exit `2` is a contract bug in the eval definition, not a quality regression in the product. Route it
to `ai-evals`. Never "fix" it by loosening the test.
## Sketches
Illustrations of the contract, not a framework to copy wholesale.
```python
# tests/evals/test_support_quality.py
import json
import os
import subprocess
import pytest
REQUIRED = ("EVAL_RUNNER", "EVAL_SPEC_PATH", "EVAL_DATASET_PATH", "EVAL_SUITE_ID")
@pytest.mark.evals
@pytest.mark.slow
def test_build_passes_the_owned_suite(tmp_path):
if any(os.environ.get(name) is None for name in REQUIRED):
pytest.skip("eval system not configured")
out = tmp_path / "results.json"
completed = subprocess.run(
[
"python3", os.environ["EVAL_RUNNER"],
"--spec", os.environ["EVAL_SPEC_PATH"],
"--dataset", os.environ["EVAL_DATASET_PATH"],
"--out", str(out),
],
capture_output=True,
text=True,
)
# Exit 2 is a broken eval definition, not a product regression. Route it, do not absorb it.
if completed.returncode == 2:
raise RuntimeError(f"eval contract error, route to ai-evals: {completed.stderr}")
result = json.loads(out.read_text())
assert result["eval_id"] == os.environ["EVAL_SUITE_ID"]
assert result["run_gate"]["passed"] is True, result["run_gate"]
```
```typescript
// tests/evals/support-quality.eval.test.ts
import { describe, expect, it } from 'vitest';
const suiteId = process.env.EVAL_SUITE_ID;
const serviceUrl = process.env.EVAL_SERVICE_URL;
describe.skipIf(!suiteId || !serviceUrl)('owned eval suite', () => {
it('passes the official gate', async () => {
const res = await fetch(`${serviceUrl}/runs`, {
method: 'POST',
headers: { authorization: `Bearer ${process.env.EVAL_SERVICE_TOKEN}` },
body: JSON.stringify({ suite_id: suiteId, suite_version: process.env.EVAL_SUITE_VERSION }),
});
if (res.status >= 500) throw new Error('eval service error, route to ai-evals');
const result = await res.json();
expect(result.eval_id).toBe(suiteId);
expect(result.run_gate.passed).toBe(true);
});
});
```
## Boundary to ai-evals
The result payload shape, the spec fields, the operation vocabulary, and the dataset split semantics
are all defined in
[`../../ai-evals/references/offline-runner-contract.md`](../../ai-evals/references/offline-runner-contract.md).
Read them there. **Do not restate them here or in a test file** — a copy is a fork waiting to
happen, which is the same defect this contract exists to prevent.
When the suite itself must change — a new dataset row, a different grader, a moved threshold — that
is `ai-evals` work. Nothing about it belongs in `tests/evals/`.
Deterministic application logic around the model — prompt builders, parsers, schema validators,
retrieval filters, tool-call shapes — is **not** an eval. It belongs in `tests/unit/` and
`tests/integration/` like any other code.
## Anti-patterns
- A dataset inlined in the test file.
- A threshold or pass-rate minimum chosen locally.
- Asserting on model output text rather than on the returned gate.
- A retry-until-green loop around a flaky eval.
- Running the eval suite on every pull request.
- Committing eval outputs as fixtures, so the test asserts against last week's answers.
- Importing a grader, a rubric, or a scoring helper into the test.
- Opening a database connection under `tests/evals/`.
- Falling back to a local approximation when the eval system is unconfigured, instead of skipping.
- Catching exit code `2` and treating it as a normal failure, hiding a broken eval definition.
references/test-frameworks.md
# Test Frameworks Reference
Which framework belongs to which test type in which stack. Folder definitions live in
[`test-suite-layout.md`](test-suite-layout.md); tier definitions live in
[`test-data-tiers.md`](test-data-tiers.md).
The governing rule: **one runner per language per repository.** A second runner doubles the
configuration surface, splits the coverage report, and guarantees that half the suite is
misconfigured at any moment.
## Contents
- [Folder x stack](#folder-x-stack)
- [Selecting a framework](#selecting-a-framework)
- [Cross-cutting concerns](#cross-cutting-concerns)
- [Runner configuration](#runner-configuration)
- [Do not add](#do-not-add)
## Folder x stack
| Folder | Python | TypeScript / Node | Browser / UI | Go | Ruby |
| --- | --- | --- | --- | --- | --- |
| `unit/` | `pytest` | `vitest` | `vitest` with Testing Library | `go test`, table-driven | `rspec` |
| `integration/` | `pytest` with `pytest-django` or `httpx`; `testcontainers` for the database | `vitest` with `supertest`; `testcontainers` | — | `go test` with `testcontainers` | `rspec` request specs |
| `e2e/` | `pytest` for API-driven flows, or Playwright for Python | `playwright` | **Playwright** | `go test` against a deployed environment | `rspec` with `capybara` |
| `smoke/` | `pytest`, API-driven | `vitest` or `playwright` | `playwright` | `go test` | `rspec` |
| `regression/` | the same runner as the layer the defect lived in | same | same | same | same |
| `adversarial/` | `pytest` with `hypothesis`; `schemathesis` for API schema fuzzing | `vitest` with `fast-check`; `playwright` for UI abuse | `playwright` | `go test` with native fuzzing | `rspec` |
| `evals/` | `pytest`, thin client only | `vitest`, thin client only | — | — | — |
| `tmp/` | the repository's default runner | same | same | same | same |
**Playwright is the default for new browser suites.** Choose Cypress only where an established
Cypress suite already exists, or where component tests are deliberately coupled to the same tool as
the end-to-end suite. See [`../../frontend-e2e/references/e2e-cypress.md`](../../frontend-e2e/references/e2e-cypress.md).
**`regression/` has no framework of its own.** A regression test is a pinned copy of the layer that
broke, so it uses that layer's runner, that layer's fixtures, and that layer's tier.
## Selecting a framework
1. **Does the repository already have a runner for this language?** Use it. A consistent
second-choice runner beats a fragmented first-choice one.
2. **Does the test need a browser?** Playwright, unless an established Cypress suite exists.
3. **Does the test need a real database engine?** Add `testcontainers` (or the stack equivalent) to
the existing runner rather than adopting a second runner.
4. **Does the test need generated inputs?** Add a property-based library to the existing runner —
`hypothesis`, `fast-check`, or native fuzzing — not a separate harness.
5. **Does the test invoke an eval suite?** The existing runner, as a thin client. See
[`test-evals-wrapping.md`](test-evals-wrapping.md).
Language-level details underneath these choices — fixture conventions, ORM specifics, transaction
handling — live in the `testing` skill's stack references.
## Cross-cutting concerns
These are **not folders**. Each is a marker or a library used inside an existing folder.
| Concern | Tooling | Where it lives |
| --- | --- | --- |
| Consumer and provider contracts | `pact`, or schema assertions against the published spec | `tests/integration/`, marked `contract` |
| Load, stress, and soak | `k6`, `Gatling` | the performance lane, **not** `tests/adversarial/` |
| Time control | `freezegun` (Python), `vi.useFakeTimers` (Vitest) | wherever the clock matters |
| HTTP boundary mocking | `responses` or `respx` (Python), `msw` (TypeScript) | Tier 2 tests in any folder |
| Object construction | `factory_boy`, `fishery`, or hand-written builders | `tests/factories/` |
| Coverage | the runner's native plugin | CI configuration, not a test folder |
Load testing is deliberately excluded from `tests/adversarial/`. Hammering a service is a capacity
question, not a red-teaming one, and mixing them makes both harder to interpret.
## Runner configuration
Configuration lives in the repository's existing config file. Three settings matter for the layout
contract regardless of stack:
- **Register the markers** for all eight folders so a typo becomes an error rather than a silently
empty selection.
- **Exclude `tests/tmp/` from collection**, so scratch tests cannot run locally or in CI.
- **Deselect the non-default lanes** — `evals`, `adversarial`, and `tmp` — so the ordinary suite
never invokes an eval service, attacks a target, or picks up scratch code.
Copy-ready pytest, Vitest, and Playwright configuration blocks are in
[`../templates/test-suite-scaffold.md`](../templates/test-suite-scaffold.md).
## Do not add
- A second test runner for a language that already has one.
- `unittest` tests alongside `pytest`, or `jest` alongside `vitest`, in new code.
- A home-grown assertion library or a custom test base class that wraps the runner's own.
- Selenium for a new browser suite.
- A separate harness for property-based or fuzz testing when the existing runner has a plugin.
- A bespoke test reporter before the standard one has actually proven inadequate.
- A mocking library for something the standard library or the runner already mocks.
references/test-suite-layout.md
# Test Suite Layout Reference
The canonical `tests/` contract. Every Engineering QA lane defers to this file for where a test
belongs. Tier definitions and provisioning live in
[`test-data-tiers.md`](test-data-tiers.md); framework selection lives in
[`test-frameworks.md`](test-frameworks.md); the `tests/evals/` wrapper contract lives in
[`test-evals-wrapping.md`](test-evals-wrapping.md).
## Contents
- [Canonical layout](#canonical-layout)
- [The eight folders](#the-eight-folders)
- [Support directories](#support-directories)
- [Folder x tier legality](#folder-x-tier-legality)
- [Folder x marker x CI lane](#folder-x-marker-x-ci-lane)
- [tests/adversarial](#testsadversarial)
- [tests/tmp](#teststmp)
- [Adopting the layout in an existing repo](#adopting-the-layout-in-an-existing-repo)
- [Conformance checklist](#conformance-checklist)
- [Anti-patterns](#anti-patterns)
## Canonical layout
```text
tests/
├── README.md # the team's filled-in suite contract
├── conftest.py # or the stack's equivalent runner config
├── unit/
├── integration/
├── e2e/
├── smoke/
├── regression/
├── adversarial/ # red-teaming your own interfaces
├── evals/ # AI products only; wraps an existing eval suite
├── tmp/ # gitignored scratch; promoted or deleted
├── factories/ # support, not a test type
├── fixtures/
├── helpers/
├── scripts/
└── data/
```
Eight test-type folders, five support directories. Sub-directories **inside** a canonical folder
are encouraged when they mirror the application (`integration/api/`, `unit/services/`). New
**top-level** buckets are not.
Audit a repository against this layout with
[`../scripts/audit_test_layout.py`](../scripts/audit_test_layout.py). Scaffold a new suite from
[`../templates/test-suite-scaffold.md`](../templates/test-suite-scaffold.md).
## The eight folders
**`unit/`** — Pure logic, transformations, parsing, validation, and policy decisions. No database,
no network, no filesystem, no wall clock. If a test needs a database it is not a unit test; move it
to `integration/`.
**`integration/`** — Real seams: HTTP handlers, ORM behavior, serializers, auth, queues, jobs,
signals, and transaction handoff. This is where consumer and provider contract tests live, tagged
`contract`, rather than in a top-level bucket of their own.
**`e2e/`** — High-value user journeys across the assembled stack. Mock only genuinely third-party
boundaries; mocking your own services defeats the point of the layer.
**`smoke/`** — A thin set of critical-path checks run before and after a deploy to prove the
system is alive and correctly wired. Kept deliberately small and fast; it is a release gate, not a
coverage mechanism.
**`regression/`** — Pinned proofs that a specific defect class has not returned. Each test names
the defect it guards. It inherits the tier and the shape of the layer where the bug actually lived,
so a `regression/` suite is heterogeneous by design.
**`adversarial/`** — Red-teaming your own interfaces. See [below](#testsadversarial).
**`evals/`** — AI products only. A thin client that invokes an eval suite the eval system already
defines and asserts on the returned official gate. It never defines datasets, graders, or
thresholds. See [`test-evals-wrapping.md`](test-evals-wrapping.md).
**`tmp/`** — Scratch tests written to verify work mid-development. See [below](#teststmp).
## Support directories
These sit alongside the test-type folders and are **not** test types. Nothing in them is collected
as a test.
| Directory | Holds |
| --- | --- |
| `factories/` | object and payload builders, organized by domain |
| `fixtures/` | static fixture files and shared runner fixtures |
| `helpers/` | assertion helpers, clients, and test utilities |
| `scripts/` | operational and infrastructure checks invoked deliberately, not collected |
| `data/` | sample payloads and seed data referenced by tests |
## Folder x tier legality
Tiers are defined in [`test-data-tiers.md`](test-data-tiers.md). In brief: **T1** no data, **T2**
mock data, **T3** local replica database, **T4** staging replica with rollback, **T5** production
read-only.
| Folder | Default | Allowed | Not allowed | Why |
| --- | --- | --- | --- | --- |
| `unit/` | T1 | T1, T2 | T3, T4 | A test that needs a database belongs in `integration/`. |
| `integration/` | T3 | T2, T3, T4 (lane approval plus proven rollback) | — | Must be free to write destructively. |
| `e2e/` | T3 | T3, T4 | T2 for the system under test | Proves the assembled stack; T2 is fine for genuinely third-party boundaries only. |
| `smoke/` | T3 | T3, T4 | T1 | A smoke test with no data proves nothing about the deployment. |
| `regression/` | inherits the layer where the bug lived | T1, T2, T3, T4 | — | A pinned copy of the layer that broke. |
| `adversarial/` | T3 | T2, T3, T4 (gated) | — | Abuse of your own interfaces on a non-production target. |
| `evals/` | T2 | T1, T2 | T3, T4 | A thin client for the eval system. A database connection here means an eval has grown inside the wrapper. |
| `tmp/` | the author's existing clearance | T1, T2, T3, T4 | — | Scratch code; it inherits, it does not escalate. |
**T5 is permitted in any folder**, and only under the four-condition gate in
[`test-data-tiers.md`](test-data-tiers.md#the-tier-5-gate). Folder placement does not authorize T5;
the read-only database role does. Because scratch code in `tmp/` is unreviewed by definition, and
because the whole intent of `adversarial/` is to make writes happen, a T5 test in either of those
two folders is worth a second look — the auditor reports it as a warning, and `--strict` turns it
into a violation for teams that want the tighter rule.
## Folder x marker x CI lane
Markers must match the directory. A test in `unit/` marked `integration` is a placement bug in one
direction or the other.
| Folder | Marker | Default lane | Blocking |
| --- | --- | --- | --- |
| `unit/` | `unit` | PR gate | yes |
| `integration/` | `integration` | PR gate | yes |
| `e2e/` | `e2e` | merge to main, pre-release | yes on main |
| `smoke/` | `smoke` | pre-release, post-deploy | yes |
| `regression/` | `regression` plus a tier marker | T1-T3 on the PR gate; T4 and T5 nightly or on demand | T1-T3 only |
| `adversarial/` | `adversarial` | nightly or on demand; never on fork PRs | no |
| `evals/` | `evals`, `slow` | on demand and pre-release; deselected by default | pre-release only |
| `tmp/` | `tmp` | never runs in CI | n/a |
## tests/adversarial
Red-teaming **your own** business logic, through **your own** public interfaces, using **your own**
test runner, against a **non-production** target.
### In scope
- Economic abuse: negative quantities, price or quantity tampering in the request body, coupon
stacking, refund loops, currency rounding exploitation.
- Authorization abuse within your own API surface: insecure direct object references, mass
assignment of protected fields, tenant or account boundary crossing, permission downgrade paths.
- Workflow bypass: skipping a required state, replay, double submit, out-of-order webhooks.
- Quota and rate-limit evasion.
- Hostile input: property-based and schema fuzzing over parsers and the documented API surface.
- For AI products: prompt injection, jailbreak, tool abuse, exfiltration-via-tool, and unsafe-output
probes — run as regression checks against a corpus owned by `ai-governance-safety` or `ai-evals`,
never as newly invented safety criteria.
### Out of scope
| Work | Owner |
| --- | --- |
| Network or infrastructure scanning, exploit chains, CVE validation, raw traffic replay | `pentest` and its `web-vuln-validation` child |
| Passive source, secret, dependency, threat-model, or compliance review | `security` |
| Defining safety policy, refusal taxonomies, harm categories, or red-team corpora | `ai-governance-safety` and `ai-evals`; this folder only consumes them |
| Load, stress, and denial-of-service | the performance lane |
### Five rules that keep this a folder and not a second pentest skill
1. **Own code, own interfaces, own runner.** If the target is not built from this repository, it is
a pentest — route it out.
2. **No exploitation tooling.** If the check needs `sqlmap`, `nuclei`, `metasploit`, `burp`, or
`nmap`, it is a pentest.
3. **Non-production target, bounded concurrency.** State the concurrency and request volume as
explicit numbers in the test, never "as fast as possible".
4. **Assert that a defensive control fires.** The passing assertion is `403`, `422`, a rollback, an
emitted audit event, or a balance left unchanged — never "we got in".
5. **Findings graduate out.** A reproduced defect becomes a bug plus a `regression/` test. This
folder never grows a report generator, a findings taxonomy, or a severity scale; those belong to
`security` and `pentest`.
### Authorization gate
This gate defers to the existing ones in [`../../testing/SKILL.md`](../../testing/SKILL.md) and
`pentest/web-vuln-validation` rather than inventing a third standard. Before any run against a
shared environment:
- Written authorization naming the target environment, its owner, and a time window, recorded in the
PR or issue and surfaced to the run as `ADVERSARIAL_AUTHORIZATION_REF`. **Fail closed if unset.**
- `ADVERSARIAL_TARGET` set, and not matching the repository's production hostname pattern. **Fail
closed if unset.**
- Blast radius limited to fixtures the suite itself created. No third-party side effects: sandbox
keys only for payments, email, and SMS.
- A stated stop condition and a rollback plan before the first request.
## tests/tmp
Scratch tests written mid-development to verify a hypothesis, then removed. Not a staging area, not
a parking lot, not a "clean it up later" folder.
**Gitignore, self-contained.** Ship `tests/tmp/.gitignore` rather than editing the repository root
ignore file, which the scaffold cannot assume exists:
```gitignore
*
!.gitignore
!README.md
```
The directory stays in git so the convention is discoverable, while every scratch file is untracked
by construction.
**Exclude at the runner too.** Gitignore does not stop a local `pytest tests/`. Add
`norecursedirs = tests/tmp` (pytest) or `exclude: ['tests/tmp/**']` (vitest), plus a `tmp` marker
deselected in the default arguments.
**Isolation.** Nothing outside `tests/tmp/` may import from it.
**Naming.** `tmp_<slug>_<yyyymmdd>` — the date makes staleness visible to a reader and checkable by
the auditor.
**Four deletion triggers, in precedence order.**
1. The claim the file was written to prove has been settled. Promote it or delete it; there is no
third option.
2. Before any commit. Never `git add -f` anything under `tests/tmp/`.
3. Before declaring the task done or opening a PR.
4. On age. A file past the age budget is stale by definition.
**Promotion.** A scratch test that reproduced a bug becomes `tests/regression/`. One that proved new
behavior becomes `tests/unit/` or `tests/integration/`. Promotion means a real name, real
assertions, the correct folder, the correct marker, and a declared tier.
**Never** put secrets, credentials, captured production data, or database dumps in `tests/tmp/`.
Gitignore protects the repository, not the disk.
## Adopting the layout in an existing repo
New suites use the canonical layout as written. An existing suite migrates by addition, not by a
rename sweep:
1. Keep every directory that already exists. `contract/`, and any other pre-existing bucket, is
grandfathered where it already exists.
2. Route new tests to the canonical folder for their type.
3. Record any deviation in `tests/README.md` so the difference is a decision rather than drift.
4. Move tests only when you are already editing them for another reason. A migration commit that
only moves files destroys `git blame` for no test-quality gain.
## Conformance checklist
- [ ] Every test lives under `tests/` in one of the eight canonical folders.
- [ ] No top-level bucket outside the canonical eight plus the five support directories.
- [ ] Every test's marker matches its directory.
- [ ] Every test that touches data declares its tier.
- [ ] No T3, T4, or T5 signal under `tests/unit/`.
- [ ] No database connection under `tests/evals/`.
- [ ] Every T5 test satisfies all four gate conditions.
- [ ] `tests/tmp/` is gitignored, excluded from the runner, and empty at PR time.
- [ ] `tests/README.md` records the tiers in use, the T5 authorization reference, and any deviation.
## Anti-patterns
- A new top-level bucket invented for one test.
- A unit test that reaches a database, then gets a longer timeout instead of a move to
`integration/`.
- A `regression/` test that does not name the defect it guards.
- An `adversarial/` test that asserts an exploit succeeded rather than that a control fired.
- Eval datasets, graders, or thresholds defined inside `tests/evals/`.
- `tests/tmp/` files committed, force-added, or left behind at PR time.
- Markers and directories that disagree, so lane selection silently runs the wrong set.
- The same behavior asserted at three layers because no one chose a proving layer.
scripts/audit_test_layout.py
#!/usr/bin/env python3
"""Audit a repository's tests/ tree against the canonical test-suite contract.
Checks folder layout, data-tier legality, the Tier 5 gate, tests/evals/ purity,
the tests/adversarial/ authorization gate, and tests/tmp/ hygiene.
Read-only: this script never creates, edits, or deletes anything.
Exit codes:
0 conformant (warnings may be present unless --strict)
1 one or more violations found
2 usage or I/O error (missing path, missing tests directory, unreadable file)
"""
from __future__ import annotations
import argparse
import json
import os
import re
import subprocess
import sys
import time
from pathlib import Path
CANONICAL_FOLDERS = (
"unit",
"integration",
"e2e",
"smoke",
"regression",
"adversarial",
"evals",
"tmp",
)
SUPPORT_DIRS = ("factories", "fixtures", "helpers", "scripts", "data")
# Pre-existing buckets that predate the contract: report, but do not fail.
GRANDFATHERED = ("contract", "contracts")
TEST_SUFFIXES = (".py", ".ts", ".tsx", ".js", ".jsx", ".rb", ".go")
# Folders whose intent makes a production credential worth a second look. Warning
# by default so the gate stays the control; --strict promotes it to a violation.
T5_REVIEW_FOLDERS = ("tmp", "adversarial")
DEFAULT_TIER_PATTERNS = (
r"\btier(?P<n>[1-5])\b",
r"@tier\(\s*[\"']?t?(?P<n2>[1-5])[\"']?\s*\)",
)
PROD_READONLY_MARKERS = (r"prod_readonly", r"production_readonly", r"prod-readonly")
DB_CONNECTION_SIGNALS = (
r"\bDATABASE_URL\b",
r"\bpsycopg\b",
r"\bpymysql\b",
r"\bcreate_engine\s*\(",
r"\bdjango_db\b",
r"\bsessionmaker\s*\(",
r"\bnew\s+Pool\s*\(",
r"\bcreateConnection\s*\(",
r"\btestcontainers\b",
)
WRITE_VERB_SIGNALS = (
r"\bINSERT\s+INTO\b",
r"\bUPDATE\s+\w+\s+SET\b",
r"\bDELETE\s+FROM\b",
r"\bDROP\s+(TABLE|DATABASE|SCHEMA|INDEX)\b",
r"\bTRUNCATE\b",
r"\bALTER\s+(TABLE|DATABASE|SCHEMA)\b",
r"\bCREATE\s+(TABLE|DATABASE|SCHEMA|INDEX)\b",
r"\.save\s*\(",
r"\.create\s*\(",
r"\.delete\s*\(",
r"\.commit\s*\(",
r"\.bulk_create\s*\(",
)
AUTHORIZATION_SIGNALS = (
r"AUTHORIZATION_REF",
r"AUTHORISATION_REF",
r"AUTHORIZATION_REFERENCE",
r"AUTH_REF",
)
READONLY_ROLE_SIGNALS = (
r"READ_?ONLY_ROLE",
r"READ_?ONLY_USER",
r"has_no_write_grants",
r"assert_read_only",
r"READONLY_DATABASE_URL",
)
EVAL_DEFINITION_SIGNALS = (
r"\bexpected_output\b",
r"\bexpectedOutput\b",
r"\brubric\b",
r"\bgrader\b",
r"\bmin_pass_rate\b",
r"\bminPassRate\b",
r"\bthreshold\s*[=:]",
r"\bpass_rate\s*[=:]",
)
EXPLOITATION_TOOLS = (r"\bsqlmap\b", r"\bnuclei\b", r"\bmetasploit\b", r"\bmsfconsole\b",
r"\bburpsuite\b", r"\bnmap\b", r"\bhydra\b")
ADVERSARIAL_GATE_SIGNALS = (
r"ADVERSARIAL_AUTHORIZATION_REF",
r"ADVERSARIAL_TARGET",
) + AUTHORIZATION_SIGNALS
MARKER_WORDS = {
"unit": r"\bunit\b",
"integration": r"\bintegration\b",
"e2e": r"\be2e\b",
"smoke": r"\bsmoke\b",
"regression": r"\bregression\b",
"adversarial": r"\badversarial\b",
"evals": r"\bevals\b",
"tmp": r"\btmp\b",
}
MARKER_DECL = re.compile(
r"(?:@pytest\.mark\.(\w+)|@pytest\.mark\.\w+|\btag(?:s)?\s*[:=]\s*\[([^\]]*)\])"
)
class AuditError(Exception):
"""Usage or I/O failure. Maps to exit code 2."""
def read_text(path: Path) -> str:
try:
return path.read_text(encoding="utf-8", errors="replace")
except OSError as exc:
raise AuditError(f"cannot read {path}: {exc}") from exc
def any_match(patterns, text: str) -> bool:
return any(re.search(p, text, re.IGNORECASE) for p in patterns)
def matched(patterns, text: str) -> list[str]:
found = []
for pattern in patterns:
hit = re.search(pattern, text, re.IGNORECASE)
if hit:
found.append(hit.group(0))
return found
def declared_tiers(text: str, extra_patterns: tuple[str, ...]) -> set[int]:
tiers: set[int] = set()
for pattern in DEFAULT_TIER_PATTERNS + extra_patterns:
for hit in re.finditer(pattern, text, re.IGNORECASE):
for value in hit.groupdict().values():
if value and value.isdigit():
tiers.add(int(value))
if any_match(PROD_READONLY_MARKERS, text):
tiers.add(5)
return tiers
def declared_markers(text: str) -> set[str]:
found: set[str] = set()
for name, pattern in MARKER_WORDS.items():
if re.search(r"@pytest\.mark\." + name + r"\b", text):
found.add(name)
elif re.search(r"['\"]@?" + name + r"['\"]", text) and "tag" in text.lower():
found.add(name)
return found
def git_output(root: Path, args: list[str]) -> tuple[int, str]:
try:
proc = subprocess.run(
["git", "-C", str(root), *args],
capture_output=True,
text=True,
timeout=15,
)
except (OSError, subprocess.SubprocessError):
return (-1, "")
return (proc.returncode, proc.stdout)
def collect_files(folder: Path) -> list[Path]:
return [
path
for path in sorted(folder.rglob("*"))
if path.is_file() and path.suffix in TEST_SUFFIXES
]
class Audit:
def __init__(self, root: Path, tests_dir: Path, args: argparse.Namespace) -> None:
self.root = root
self.tests_dir = tests_dir
self.args = args
self.violations: list[dict] = []
self.warnings: list[dict] = []
self.info: list[str] = []
self.tier_map: dict[str, list[int]] = {}
def rel(self, path: Path) -> str:
try:
return str(path.relative_to(self.root))
except ValueError:
return str(path)
def violation(self, path, message: str) -> None:
self.violations.append({"path": self.rel(Path(path)), "message": message})
def warn(self, path, message: str) -> None:
self.warnings.append({"path": self.rel(Path(path)), "message": message})
# -- checks ---------------------------------------------------------------
def check_buckets(self) -> tuple[list[str], list[str]]:
allowed = set(CANONICAL_FOLDERS) | set(SUPPORT_DIRS)
present, unknown = [], []
for child in sorted(self.tests_dir.iterdir()):
if not child.is_dir() or child.name.startswith("."):
continue
if child.name in CANONICAL_FOLDERS:
present.append(child.name)
elif child.name in allowed:
continue
elif child.name in GRANDFATHERED:
self.warn(
child,
f"'{child.name}/' predates the contract; grandfathered. New tests of this "
"kind belong in tests/integration/ with a 'contract' marker.",
)
elif child.name in ("__pycache__", "node_modules"):
continue
else:
unknown.append(child.name)
self.violation(
child,
f"unknown top-level bucket '{child.name}/'. Canonical folders: "
+ ", ".join(CANONICAL_FOLDERS)
+ ". Support directories: "
+ ", ".join(SUPPORT_DIRS)
+ ".",
)
return present, unknown
def check_tmp_hygiene(self) -> None:
tmp = self.tests_dir / "tmp"
if not tmp.is_dir():
return
code, out = git_output(self.root, ["ls-files", str(tmp.relative_to(self.root))])
if code == 0:
tracked = [line for line in out.splitlines() if line.strip()]
tracked = [
line
for line in tracked
if Path(line).name not in (".gitignore", "README.md")
]
for line in tracked:
self.violation(line, "scratch file is tracked in git; tests/tmp/ must be ignored")
elif code == -1:
self.warn(tmp, "git unavailable; could not verify tests/tmp/ is ignored")
if not (tmp / ".gitignore").exists():
self.warn(
tmp,
"no tests/tmp/.gitignore; add one containing '*', '!.gitignore', '!README.md'",
)
budget = self.args.max_tmp_age_days * 86400
now = time.time()
for path in collect_files(tmp):
age_days = int((now - path.stat().st_mtime) / 86400)
if now - path.stat().st_mtime > budget:
self.warn(
path,
f"scratch file is {age_days} days old (budget "
f"{self.args.max_tmp_age_days}); promote it or delete it",
)
self.check_tmp_imports(tmp)
def check_tmp_imports(self, tmp: Path) -> None:
pattern = re.compile(r"(?:from|import)\s+[\w./]*tests[./]tmp\b|['\"][^'\"]*tests/tmp/")
for folder in CANONICAL_FOLDERS:
if folder == "tmp":
continue
target = self.tests_dir / folder
if not target.is_dir():
continue
for path in collect_files(target):
if pattern.search(read_text(path)):
self.violation(path, "imports from tests/tmp/; scratch code must stay isolated")
def check_folder(self, folder: str) -> None:
target = self.tests_dir / folder
if not target.is_dir():
return
tiers_seen: set[int] = set()
for path in collect_files(target):
text = read_text(path)
tiers = declared_tiers(text, tuple(self.args.tier_pattern or ()))
tiers_seen |= tiers
self.check_tier_legality(path, folder, tiers, text)
self.check_t5_gate(path, tiers, text)
self.check_marker_agreement(path, folder, text)
if folder == "evals":
self.check_evals_purity(path, text)
if folder == "adversarial":
self.check_adversarial_gate(path, text)
self.tier_map[folder] = sorted(tiers_seen)
def check_tier_legality(self, path: Path, folder: str, tiers: set[int], text: str) -> None:
db_hits = matched(DB_CONNECTION_SIGNALS, text)
if folder == "unit":
if db_hits:
self.violation(
path,
"database connection signal in tests/unit/ ("
+ ", ".join(sorted(set(db_hits)))
+ "); a test that needs a database belongs in tests/integration/",
)
for tier in sorted(t for t in tiers if t in (3, 4)):
self.violation(path, f"tier {tier} is not allowed in tests/unit/ (T1 and T2 only)")
if folder == "evals":
if db_hits:
self.violation(
path,
"database connection signal in tests/evals/ ("
+ ", ".join(sorted(set(db_hits)))
+ "); the wrapper talks to the eval system, not to a database",
)
for tier in sorted(t for t in tiers if t in (3, 4)):
self.violation(path, f"tier {tier} is not allowed in tests/evals/ (T1 and T2 only)")
if folder == "smoke" and 1 in tiers:
self.warn(path, "tier 1 in tests/smoke/; a smoke test with no data proves little")
if 5 in tiers and folder in T5_REVIEW_FOLDERS:
message = (
f"tier 5 (production, read-only) in tests/{folder}/. "
+ (
"Scratch code is unreviewed by definition and should not hold a production "
"credential."
if folder == "tmp"
else "This folder exists to attempt destructive actions; pointing it at "
"production is a design smell."
)
)
if self.args.strict:
self.violation(path, message)
else:
self.warn(path, message)
def check_t5_gate(self, path: Path, tiers: set[int], text: str) -> None:
if 5 not in tiers:
return
if not any_match(AUTHORIZATION_SIGNALS, text):
self.violation(
path,
"tier 5 test has no authorization-reference signal; the gate requires a recorded "
"reference read from the environment that fails closed when unset",
)
if not any_match(READONLY_ROLE_SIGNALS, text):
self.violation(
path,
"tier 5 test does not verify a read-only database role; the gate requires a role "
"with no write grants, checked rather than assumed",
)
writes = matched(WRITE_VERB_SIGNALS, text)
if writes:
self.violation(
path,
"tier 5 test contains write operations ("
+ ", ".join(sorted(set(writes)))
+ "); zero writes to production at any tier",
)
def check_evals_purity(self, path: Path, text: str) -> None:
hits = matched(EVAL_DEFINITION_SIGNALS, text)
if hits:
self.violation(
path,
"eval-definition signal in tests/evals/ ("
+ ", ".join(sorted(set(hits)))
+ "); evals are defined in the eval system, not in tests/evals/",
)
for sibling in path.parent.glob("*.jsonl"):
self.violation(
sibling,
"dataset file under tests/evals/; datasets belong to the eval system, and the "
"wrapper references a suite by id and version instead",
)
def check_adversarial_gate(self, path: Path, text: str) -> None:
tools = matched(EXPLOITATION_TOOLS, text)
if tools:
self.violation(
path,
"exploitation tooling in tests/adversarial/ ("
+ ", ".join(sorted(set(tools)))
+ "); route this work to the pentest skill",
)
if not any_match(ADVERSARIAL_GATE_SIGNALS, text):
self.warn(
path,
"no authorization-gate signal; an adversarial test against a shared target needs "
"ADVERSARIAL_AUTHORIZATION_REF and ADVERSARIAL_TARGET, both failing closed",
)
def check_marker_agreement(self, path: Path, folder: str, text: str) -> None:
markers = declared_markers(text)
conflicting = markers & set(MARKER_WORDS) - {folder}
if conflicting and folder not in markers:
self.warn(
path,
f"marker {sorted(conflicting)} does not match directory tests/{folder}/; "
"the marker or the placement is wrong",
)
def detect_stack(self) -> list[str]:
stack = []
if any((self.root / n).exists() for n in ("pyproject.toml", "requirements.txt", "manage.py", "setup.py")):
stack.append("python")
if (self.root / "package.json").exists():
stack.append("typescript-or-javascript")
if (self.root / "go.mod").exists():
stack.append("go")
if (self.root / "Gemfile").exists():
stack.append("ruby")
return stack
def build_report(audit: Audit, present: list[str], unknown: list[str], stack: list[str]) -> str:
missing = [f for f in CANONICAL_FOLDERS if f not in present]
lines = [
"# Test layout audit",
"",
f"Repo: {audit.root}",
f"Tests dir: {audit.rel(audit.tests_dir)}",
f"Stack: {', '.join(stack) or 'undetected'}",
"",
"## Layout",
"",
f"- Present: {', '.join(present) or 'none'}",
f"- Absent: {', '.join(missing) or 'none'}",
f"- Unknown buckets: {', '.join(unknown) or 'none'}",
"",
"Absent folders are not failures. `evals/` applies to AI products; `smoke/` and",
"`adversarial/` are optional.",
"",
"## Data tiers",
"",
]
if audit.tier_map:
for folder in CANONICAL_FOLDERS:
if folder in audit.tier_map:
tiers = audit.tier_map[folder]
shown = ", ".join(f"T{t}" for t in tiers) if tiers else "none declared"
lines.append(f"- `{folder}/`: {shown}")
else:
lines.append("- no test folders found")
lines += ["", f"## Violations ({len(audit.violations)})", ""]
lines += [f"- `{v['path']}`: {v['message']}" for v in audit.violations] or ["- none"]
lines += ["", f"## Warnings ({len(audit.warnings)})", ""]
lines += [f"- `{w['path']}`: {w['message']}" for w in audit.warnings] or ["- none"]
lines += ["", "## Next steps", ""]
steps = []
if unknown:
steps.append("Move tests out of unknown buckets into the canonical folder for their type.")
if any("tier 5" in v["message"] for v in audit.violations):
steps.append("Fix the Tier 5 gate before running anything against production.")
if any("tests/evals/" in v["message"] for v in audit.violations):
steps.append("Move eval definitions back into the eval system; keep the wrapper thin.")
if any("tests/tmp/" in w["message"] or "scratch" in w["message"] for w in audit.warnings):
steps.append("Promote or delete stale scratch tests under tests/tmp/.")
if not steps:
steps.append("No structural action required.")
lines += [f"{i}. {s}" for i, s in enumerate(steps, 1)]
return "\n".join(lines)
def scaffold_text() -> str:
stubs = "\n".join(f"tests/{folder}/README.md" for folder in CANONICAL_FOLDERS)
return f"""# Canonical tests/ scaffold
tests/
├── README.md
├── unit/
├── integration/
├── e2e/
├── smoke/
├── regression/
├── adversarial/
├── evals/
├── tmp/
│ └── .gitignore
├── factories/
├── fixtures/
├── helpers/
├── scripts/
└── data/
Create only the folders the repository needs. Omitting one is fine; inventing a new
top-level bucket is not.
# tests/tmp/.gitignore
*
!.gitignore
!README.md
# Per-folder README stubs to create
{stubs}
Full stub text, runner configuration, and the tests/README.md contract table are in
the skill's templates/test-suite-scaffold.md.
"""
def parse_args(argv: list[str]) -> argparse.Namespace:
parser = argparse.ArgumentParser(
description=__doc__.splitlines()[0],
formatter_class=argparse.RawDescriptionHelpFormatter,
epilog=(
"Exit codes: 0 conformant, 1 violations found, 2 usage or I/O error.\n"
"This script is read-only; --print-scaffold writes to stdout only."
),
)
parser.add_argument("path", nargs="?", default=".", help="repository root (default: .)")
parser.add_argument(
"--tests-dir",
action="append",
help="tests directory relative to the root (default: tests). Repeat for monorepos.",
)
parser.add_argument("--json", action="store_true", help="emit JSON only")
parser.add_argument("--strict", action="store_true", help="promote warnings to violations")
parser.add_argument(
"--max-tmp-age-days",
type=int,
default=7,
help="age budget for tests/tmp/ files (default: 7)",
)
parser.add_argument(
"--tier-pattern",
action="append",
help="extra regex for tier declarations; use a named group 'n' for the digit",
)
parser.add_argument(
"--print-scaffold",
action="store_true",
help="print the canonical tree and exit without auditing",
)
return parser.parse_args(argv)
def main(argv: list[str]) -> int:
args = parse_args(argv)
if args.print_scaffold:
print(scaffold_text())
return 0
root = Path(args.path).resolve()
if not root.is_dir():
print(f"error: {root} is not a directory", file=sys.stderr)
return 2
requested = args.tests_dir or ["tests"]
targets = [root / name for name in requested]
found = [t for t in targets if t.is_dir()]
if not found:
print(
f"error: no tests directory found at {', '.join(str(t) for t in targets)}",
file=sys.stderr,
)
return 2
payloads = []
exit_code = 0
for tests_dir in found:
audit = Audit(root, tests_dir, args)
present, unknown = audit.check_buckets()
for folder in CANONICAL_FOLDERS:
audit.check_folder(folder)
audit.check_tmp_hygiene()
if args.strict:
audit.violations.extend(audit.warnings)
audit.warnings = []
stack = audit.detect_stack()
payload = {
"root": str(root),
"tests_dir": audit.rel(tests_dir),
"stack": stack,
"present": present,
"absent": [f for f in CANONICAL_FOLDERS if f not in present],
"unknown_buckets": unknown,
"tiers_by_folder": audit.tier_map,
"violations": audit.violations,
"warnings": audit.warnings,
}
payloads.append(payload)
if audit.violations:
exit_code = 1
if not args.json:
print(build_report(audit, present, unknown, stack))
print()
if args.json:
print(json.dumps(payloads if len(payloads) > 1 else payloads[0], indent=2))
else:
print("## JSON")
print()
print("```json")
print(json.dumps(payloads if len(payloads) > 1 else payloads[0], indent=2))
print("```")
return exit_code
if __name__ == "__main__":
try:
sys.exit(main(sys.argv[1:]))
except AuditError as exc:
print(f"error: {exc}", file=sys.stderr)
sys.exit(2)
except KeyboardInterrupt:
sys.exit(2)
scripts/test_audit_test_layout.py
"""Unit tests for audit_test_layout.py.
Run from this skill directory:
python3 -m unittest discover -s scripts -p 'test_*.py' -v
"""
from __future__ import annotations
import contextlib
import importlib.util
import io
import tempfile
import unittest
from pathlib import Path
SPEC = importlib.util.spec_from_file_location(
"audit_test_layout", Path(__file__).with_name("audit_test_layout.py")
)
audit_test_layout = importlib.util.module_from_spec(SPEC)
SPEC.loader.exec_module(audit_test_layout)
class AuditCase(unittest.TestCase):
def setUp(self) -> None:
self._tmp = tempfile.TemporaryDirectory()
self.root = Path(self._tmp.name)
for folder in audit_test_layout.CANONICAL_FOLDERS:
(self.root / "tests" / folder).mkdir(parents=True)
(self.root / "tests" / "tmp" / ".gitignore").write_text("*\n!.gitignore\n!README.md\n")
def tearDown(self) -> None:
self._tmp.cleanup()
def write(self, relative: str, text: str) -> Path:
path = self.root / "tests" / relative
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(text)
return path
def run_audit(self, *argv: str) -> tuple[list[dict], list[dict]]:
args = audit_test_layout.parse_args([str(self.root), *argv])
audit = audit_test_layout.Audit(self.root, self.root / "tests", args)
audit.check_buckets()
for folder in audit_test_layout.CANONICAL_FOLDERS:
audit.check_folder(folder)
audit.check_tmp_hygiene()
if args.strict:
audit.violations.extend(audit.warnings)
audit.warnings = []
return audit.violations, audit.warnings
def messages(self, entries: list[dict]) -> str:
return " | ".join(entry["message"] for entry in entries)
class TestLayout(AuditCase):
def test_clean_tree_is_conformant(self) -> None:
self.write("unit/test_pure.py", "def test_x():\n assert True\n")
violations, _ = self.run_audit()
self.assertEqual(violations, [])
def test_unknown_bucket_is_a_violation(self) -> None:
(self.root / "tests" / "weird").mkdir()
violations, _ = self.run_audit()
self.assertIn("unknown top-level bucket", self.messages(violations))
def test_grandfathered_bucket_is_only_a_warning(self) -> None:
(self.root / "tests" / "contract").mkdir()
violations, warnings = self.run_audit()
self.assertEqual(violations, [])
self.assertIn("grandfathered", self.messages(warnings))
def test_support_directories_are_allowed(self) -> None:
for name in audit_test_layout.SUPPORT_DIRS:
(self.root / "tests" / name).mkdir(exist_ok=True)
violations, _ = self.run_audit()
self.assertEqual(violations, [])
class TestTierLegality(AuditCase):
def test_database_signal_in_unit_is_a_violation(self) -> None:
self.write("unit/test_leaky.py", "import psycopg\n")
violations, _ = self.run_audit()
self.assertIn("database connection signal in tests/unit/", self.messages(violations))
def test_tier3_marker_in_unit_is_a_violation(self) -> None:
self.write("unit/test_t3.py", "import pytest\n\n@pytest.mark.tier3\ndef test_x():\n pass\n")
violations, _ = self.run_audit()
self.assertIn("tier 3 is not allowed in tests/unit/", self.messages(violations))
def test_database_signal_in_evals_is_a_violation(self) -> None:
self.write("evals/test_wrap.py", "DATABASE_URL = 1\n")
violations, _ = self.run_audit()
self.assertIn("database connection signal in tests/evals/", self.messages(violations))
def test_integration_may_use_tier3(self) -> None:
self.write(
"integration/test_db.py",
"import pytest\n\n@pytest.mark.tier3\ndef test_x(db):\n pass\n",
)
violations, _ = self.run_audit()
self.assertEqual(violations, [])
class TestTier5Gate(AuditCase):
GATED = (
"import os\nimport pytest\n\n"
"REF = os.environ['PROD_AUTHORIZATION_REF']\n"
"ROLE = os.environ['READONLY_ROLE']\n\n"
"@pytest.mark.tier5\ndef test_invariant(conn):\n"
" assert conn.scalar('select count(*) from t') >= 0\n"
)
def test_fully_gated_tier5_passes(self) -> None:
self.write("regression/test_prod.py", self.GATED)
violations, _ = self.run_audit()
self.assertEqual(violations, [])
def test_missing_authorization_reference_is_a_violation(self) -> None:
self.write(
"regression/test_prod.py",
"import pytest\nROLE = 'READONLY_ROLE'\n\n@pytest.mark.tier5\ndef test_x():\n pass\n",
)
violations, _ = self.run_audit()
self.assertIn("no authorization-reference signal", self.messages(violations))
def test_missing_readonly_role_check_is_a_violation(self) -> None:
self.write(
"regression/test_prod.py",
"import pytest\nREF = 'AUTHORIZATION_REF'\n\n@pytest.mark.tier5\ndef test_x():\n pass\n",
)
violations, _ = self.run_audit()
self.assertIn("does not verify a read-only database role", self.messages(violations))
def test_write_verb_in_tier5_is_a_violation(self) -> None:
self.write("regression/test_prod.py", self.GATED + '\n conn.execute("DELETE FROM t")\n')
violations, _ = self.run_audit()
self.assertIn("zero writes to production at any tier", self.messages(violations))
def test_prod_readonly_marker_counts_as_tier5(self) -> None:
self.write("regression/test_prod.py", "import pytest\n\n@pytest.mark.prod_readonly\ndef test_x():\n pass\n")
violations, _ = self.run_audit()
self.assertIn("tier 5", self.messages(violations))
def test_tier5_in_tmp_warns_by_default_and_fails_under_strict(self) -> None:
self.write("tmp/tmp_probe_20260101.py", self.GATED)
violations, warnings = self.run_audit()
self.assertEqual(violations, [])
self.assertIn("tier 5 (production, read-only) in tests/tmp/", self.messages(warnings))
violations, warnings = self.run_audit("--strict")
self.assertIn("tier 5 (production, read-only) in tests/tmp/", self.messages(violations))
self.assertEqual(warnings, [])
def test_tier5_in_adversarial_warns_by_default(self) -> None:
self.write("adversarial/test_abuse.py", self.GATED + "\nADVERSARIAL_TARGET = 1\n")
violations, warnings = self.run_audit()
self.assertEqual(violations, [])
self.assertIn("tier 5 (production, read-only) in tests/adversarial/", self.messages(warnings))
class TestEvalsPurity(AuditCase):
def test_grader_definition_is_a_violation(self) -> None:
self.write("evals/test_quality.py", "rubric = 'helpfulness'\n")
violations, _ = self.run_audit()
self.assertIn("evals are defined in the eval system", self.messages(violations))
def test_local_threshold_is_a_violation(self) -> None:
self.write("evals/test_quality.py", "threshold = 0.8\n")
violations, _ = self.run_audit()
self.assertIn("eval-definition signal", self.messages(violations))
def test_dataset_file_is_a_violation(self) -> None:
self.write("evals/test_wrap.py", "import os\n")
(self.root / "tests" / "evals" / "rows.jsonl").write_text('{"a":1}\n')
violations, _ = self.run_audit()
self.assertIn("dataset file under tests/evals/", self.messages(violations))
def test_thin_wrapper_passes(self) -> None:
self.write(
"evals/test_wrap.py",
"import os\n\ndef test_gate(result):\n"
" assert result['eval_id'] == os.environ['EVAL_SUITE_ID']\n"
" assert result['run_gate']['passed'] is True\n",
)
violations, _ = self.run_audit()
self.assertEqual(violations, [])
class TestAdversarialGate(AuditCase):
def test_exploitation_tooling_is_a_violation(self) -> None:
self.write("adversarial/test_x.py", "subprocess.run(['sqlmap'])\n")
violations, _ = self.run_audit()
self.assertIn("route this work to the pentest skill", self.messages(violations))
def test_missing_gate_signal_warns(self) -> None:
self.write("adversarial/test_x.py", "def test_abuse():\n pass\n")
_, warnings = self.run_audit()
self.assertIn("no authorization-gate signal", self.messages(warnings))
def test_gated_adversarial_test_passes(self) -> None:
self.write(
"adversarial/test_x.py",
"import os\nTARGET = os.environ['ADVERSARIAL_TARGET']\n"
"REF = os.environ['ADVERSARIAL_AUTHORIZATION_REF']\n\n"
"def test_abuse(client):\n assert client.post('/orders', json={'qty': -1}).status_code == 422\n",
)
violations, warnings = self.run_audit()
self.assertEqual(violations, [])
self.assertEqual(warnings, [])
class TestTmpHygiene(AuditCase):
def test_import_from_tmp_is_a_violation(self) -> None:
self.write("integration/test_x.py", "from tests.tmp import tmp_probe_20260101\n")
violations, _ = self.run_audit()
self.assertIn("imports from tests/tmp/", self.messages(violations))
def test_missing_gitignore_warns(self) -> None:
(self.root / "tests" / "tmp" / ".gitignore").unlink()
_, warnings = self.run_audit()
self.assertIn("no tests/tmp/.gitignore", self.messages(warnings))
def test_stale_scratch_file_warns(self) -> None:
import os
path = self.write("tmp/tmp_old_20200101.py", "def test_x():\n pass\n")
os.utime(path, (0, 0))
_, warnings = self.run_audit()
self.assertIn("promote it or delete it", self.messages(warnings))
class TestMarkerAgreement(AuditCase):
def test_mismatched_marker_warns(self) -> None:
self.write("unit/test_x.py", "import pytest\n\n@pytest.mark.integration\ndef test_x():\n pass\n")
_, warnings = self.run_audit()
self.assertIn("does not match directory", self.messages(warnings))
def test_matching_marker_is_silent(self) -> None:
self.write("unit/test_x.py", "import pytest\n\n@pytest.mark.unit\ndef test_x():\n pass\n")
_, warnings = self.run_audit()
self.assertNotIn("does not match directory", self.messages(warnings))
class TestCli(unittest.TestCase):
def main(self, argv: list[str]) -> int:
"""Run the CLI with stdout and stderr captured, so the shared test runner stays readable."""
with contextlib.redirect_stdout(io.StringIO()), contextlib.redirect_stderr(io.StringIO()):
return audit_test_layout.main(argv)
def test_missing_tests_dir_exits_2(self) -> None:
with tempfile.TemporaryDirectory() as tmp:
self.assertEqual(self.main([tmp]), 2)
def test_missing_path_exits_2(self) -> None:
self.assertEqual(self.main(["/nonexistent-path-for-tests"]), 2)
def test_print_scaffold_exits_0(self) -> None:
self.assertEqual(self.main(["--print-scaffold"]), 0)
def test_scaffold_lists_every_canonical_folder(self) -> None:
text = audit_test_layout.scaffold_text()
for folder in audit_test_layout.CANONICAL_FOLDERS:
self.assertIn(folder, text)
if __name__ == "__main__":
unittest.main()
SKILL.md
---
name: quality-assurance
description: Route testing, behavior-preserving simplification, security validation, and authoritative AI evaluation work to the narrowest Engineering child, and own the canonical test-suite contract - the tests folder layout, the five test-data tiers including read-only production access, and the framework-per-test-type mapping.
---
# Quality Assurance Router
## Use When
- Choosing which QA lane owns a request: testing, simplification, security, or AI evals.
- Deciding where a test belongs in the `tests/` tree, what data it may touch, or which framework it should use. This skill owns that contract for every child.
- Authorizing or refusing production-database access for a test.
- Diagnosing a failing suite, a CI-only failure, or a performance regression before a lane is selected.
## Standards
These bind every child skill. Depth is in `## References`.
- Tests live under `tests/` in eight folders: `unit/`, `integration/`, `e2e/`, `smoke/`, `regression/`, `adversarial/`, `evals/` (AI products only), and `tmp/`. Support directories - `factories/`, `fixtures/`, `helpers/`, `scripts/`, `data/` - sit alongside them and are not test types. Sub-directories inside a canonical folder are encouraged; new top-level buckets are not.
- Every test that touches data declares one tier: T1 no data, T2 mock data, T3 local replica database, T4 staging replica with rollback, T5 production read-only.
- **Zero writes to production at any tier**, and no production service calls with side effects at any tier. T5 requires all four gate conditions: a database role with no write grants, a recorded authorization reference that fails closed, exclusion from the default suite and every PR gate, and read-only assertions only. If the role cannot be proven read-only, the tier is unavailable - fail the test rather than proceed.
- `tests/evals/` wraps an eval suite the eval system already defines. It never defines datasets, graders, or thresholds.
- `tests/adversarial/` abuses your own interfaces on a non-production target and asserts that a defensive control fires. Exploitation tooling and third-party targets route to `pentest`.
- `tests/tmp/` is gitignored, never runs in CI, and is promoted or deleted before the task is declared done.
## Children
- [`testing`](../testing/SKILL.md) - Test strategy, coverage, backend, frontend E2E, flake, and authorized race-testing work.
- [`simplify`](../simplify/SKILL.md) - Behavior-preserving reduction of unnecessary code and complexity.
- [`security`](../security/SKILL.md) - Security work.
- [`ai-evals`](../ai-evals/SKILL.md) - Provider-neutral AI eval design, datasets, graders, calibration, experiments, statistics, and official release gates.
## Route
| Request | Use |
| --- | --- |
| Test strategy, coverage, backend tests, frontend E2E, flakes, or explicitly authorized business-logic and race testing | [`testing`](../testing/SKILL.md) |
| Reduce duplication, indirection, branching, wrappers, or compatibility residue without changing behavior | [`simplify`](../simplify/SKILL.md) |
| Remove AI-generated code slop, vague naming, noisy comments, or weak generated patterns | External `deslop` skill |
| security requests | [`security`](../security/SKILL.md) |
| LLM/agent/RAG scenarios, eval datasets, graders, calibration, statistical comparison, prompt optimization, or release gates | [`ai-evals`](../ai-evals/SKILL.md) |
| Production AI traces, telemetry, drift dashboards, score monitoring, or operational debugging | [`ai-evals-observability`](../ai-evals-observability/SKILL.md) through `ai-engineering` |
| Where a test belongs, which data tier it may use, which framework fits, or whether production access is permitted | Handle directly with this skill - see `## References` |
| Wrapping an already-defined eval suite as a repository test | [`references/test-evals-wrapping.md`](references/test-evals-wrapping.md), then [`ai-evals`](../ai-evals/SKILL.md) if the suite itself must change |
| Diagnosing a failing test, CI-only failure, flaky symptom, or performance regression before a lane is chosen | [`references/debugging.md`](references/debugging.md), then [`testing`](../testing/SKILL.md) and its `flake` child for intermittency |
## References
Load only the material the request needs.
- Test-suite contract, owned here; `testing` and its children defer to these: [`references/test-suite-layout.md`](references/test-suite-layout.md) for the folder layout and the folder-by-tier matrix, [`references/test-data-tiers.md`](references/test-data-tiers.md) for tier definitions, provisioning, and the Tier 5 gate, [`references/test-frameworks.md`](references/test-frameworks.md) for framework selection, and [`references/test-evals-wrapping.md`](references/test-evals-wrapping.md) for the `tests/evals/` contract.
- Cross-cutting failure diagnosis shared by every lane: [`references/debugging.md`](references/debugging.md).
Use `scripts/audit_test_layout.py` to audit a repository's `tests/` tree against the contract, and [`templates/test-suite-scaffold.md`](templates/test-suite-scaffold.md) when establishing or migrating a suite.
## Chain Rules
Chain to these skills when the task crosses this skill's boundary:
- `pentest`
- `ai-engineering/ai-evals-observability`
- `frontend/performance`
- `frontend`
- `backend`
- `prs`
- `cloud`
## Operating Rules
- Keep this parent compact; use children for lane-specific execution depth. The one depth this parent owns is the shared test-suite contract, because it binds every child - do not push it down into `testing`.
- When a child's reference conflicts with the test-suite contract, the contract wins; fix the child rather than forking the rule.
- Prefer the child skill's bundled resources when a child owns the request.
- Preserve local skill rules, repo facts, safety gates, product/channel constraints, and explicit local exceptions over external guidance when they conflict.
## External Skill Chains
Use live external skills when they are installed. If one is missing, report the fallback command instead of copying its guidance inline. Local skill rules, repo-specific facts, safety gates, product/channel constraints, and explicit local exceptions win over external guidance when they conflict.
- `deslop`: Remove AI-generated code slop from the current diff without changing behavior. Install: `python3 scripts/install-external-skills.py --skill deslop --agent codex`.
- `thermo-nuclear-code-quality-review`: Run an unusually strict maintainability and abstraction-quality review. Install: `python3 scripts/install-external-skills.py --skill thermo-nuclear-code-quality-review --agent codex`.
- `no-mistakes`: Gate explicit ship, push, PR, or validate flows through the no-mistakes pipeline. Install: `python3 scripts/install-external-skills.py --skill no-mistakes --agent codex`.
- `improve`: Run a read-only senior codebase audit and write execution-ready plans for other agents. Install: `python3 scripts/install-external-skills.py --skill improve --agent codex`.
- `browserbase-ui-test`: Use Browserbase UI testing guidance for browser-level product verification. Install: `python3 scripts/install-external-skills.py --skill browserbase-ui-test --agent codex`.
Registry: [`../../../references/external-skills.yaml`](../../../references/external-skills.yaml).
Reference-only sources: [`../../../references/external-sources.yaml`](../../../references/external-sources.yaml).
## Shared Map
See [`../../../skills-chaining-map.md`](../../../skills-chaining-map.md) for the complete skills-chaining graph.
templates/README.md
# Templates
- `test-suite-scaffold.md` - the canonical `tests/` tree, per-folder README stubs, `tests/tmp/.gitignore`,
pytest/Vitest/Playwright runner configuration, and the `tests/README.md` contract table a team fills in
with its own environment facts.
templates/test-suite-scaffold.md
---
purpose: Establish a conforming tests/ tree in a repository that has none, or migrate one that predates the contract.
audience: Engineers and agents scaffolding or migrating a test suite.
contract: ../references/test-suite-layout.md
---
# Test Suite Scaffold
Copy-ready material for a `tests/` tree that conforms to
[`../references/test-suite-layout.md`](../references/test-suite-layout.md). Print the same content
from the command line with `../scripts/audit_test_layout.py --print-scaffold`.
Migrating an existing suite? Keep every directory that already exists, route new tests to the
canonical folder for their type, and record deviations in `tests/README.md`. Do not open a
rename-only commit.
## Canonical tree
```text
tests/
├── README.md
├── conftest.py
├── unit/
│ └── README.md
├── integration/
│ └── README.md
├── e2e/
│ └── README.md
├── smoke/
│ └── README.md
├── regression/
│ └── README.md
├── adversarial/
│ └── README.md
├── evals/
│ └── README.md
├── tmp/
│ ├── .gitignore
│ └── README.md
├── factories/
├── fixtures/
├── helpers/
├── scripts/
└── data/
```
Create only the folders the repository actually needs. `evals/` is for AI products; `adversarial/`
and `smoke/` are optional. Omitting a folder is fine; inventing a new top-level one is not.
## Folder README stubs
`tests/unit/README.md`
```markdown
Pure logic and transformations. Data tiers: T1 (no data) or T2 (mock). No database, network,
filesystem, or wall clock - a test that needs a database belongs in `integration/`.
Marker: `unit`. Lane: PR gate, blocking.
```
`tests/integration/README.md`
```markdown
Real seams: HTTP handlers, ORM, serializers, auth, queues, jobs, signals. Contract tests live here
tagged `contract`. Data tiers: T2 or T3 (local replica); T4 only with lane approval and proven
rollback. Marker: `integration`. Lane: PR gate, blocking.
```
`tests/e2e/README.md`
```markdown
High-value user journeys across the assembled stack. Mock only genuinely third-party boundaries.
Data tiers: T3 or T4. Marker: `e2e`. Lane: merge to main and pre-release.
```
`tests/smoke/README.md`
```markdown
A thin set of critical-path checks proving the deployment is alive and correctly wired. Kept small
and fast; a release gate, not a coverage mechanism. Data tiers: T3 or T4.
Marker: `smoke`. Lane: pre-release and post-deploy.
```
`tests/regression/README.md`
```markdown
Pinned proofs that a specific defect class has not returned. Each test names the defect it guards.
Inherits the tier and shape of the layer where the bug lived, so this suite is heterogeneous by
design. Marker: `regression` plus a tier marker.
Lane: T1-T3 on the PR gate; T4 and T5 on demand only.
```
`tests/adversarial/README.md`
```markdown
Red-teaming your own interfaces on a non-production target: economic abuse, authorization abuse,
workflow bypass, quota evasion, hostile input, and AI safety probes against a corpus owned
elsewhere. Assert that a defensive control fires - never that an exploit succeeded.
Data tiers: T2, T3, or T4 under the authorization gate. No exploitation tooling; route those to
`pentest`. Marker: `adversarial`. Lane: nightly or on demand, non-blocking, never on fork PRs.
```
`tests/evals/README.md`
```markdown
AI products only. A thin client that invokes an eval suite the eval system already defines and
asserts on the returned official gate. Does NOT define datasets, graders, or thresholds.
Data tiers: T1 or T2 only - no database connection belongs here.
Skips when the eval system is unconfigured. Markers: `evals`, `slow`.
Lane: on demand and pre-release.
```
`tests/tmp/README.md`
```markdown
Scratch tests written to verify work mid-development. Gitignored, never run in CI, never imported
from elsewhere. Promote (to `unit/`, `integration/`, or `regression/`) or delete before the task is
declared done - there is no third option. Name files `tmp_<slug>_<yyyymmdd>`.
Never place secrets, credentials, captured production data, or database dumps here.
```
## tests/tmp/.gitignore
```gitignore
*
!.gitignore
!README.md
```
Self-contained, so the scaffold does not have to assume a repository-root ignore file exists. The
directory stays in git and the convention stays discoverable, while every scratch file is untracked
by construction.
## Runner configuration
### pytest (`pyproject.toml`)
```toml
[tool.pytest.ini_options]
testpaths = ["tests"]
norecursedirs = ["tests/tmp"]
addopts = "-m 'not evals and not adversarial and not tmp'"
markers = [
"unit: pure logic, tiers T1-T2",
"integration: real seams, tiers T2-T4",
"e2e: assembled-stack journeys, tiers T3-T4",
"smoke: critical-path deployment checks",
"regression: pinned proof a defect has not returned",
"adversarial: red-teaming own interfaces, gated",
"evals: wraps an externally defined eval suite",
"tmp: scratch, never runs in CI",
"slow: long-running",
"tier1: no data",
"tier2: mock data",
"tier3: local replica database",
"tier4: staging replica, writes rolled back",
"tier5: production read-only, gated",
]
```
### Vitest (`vitest.config.ts`)
```typescript
export default defineConfig({
test: {
include: ['tests/**/*.{test,spec}.ts'],
exclude: ['tests/tmp/**', 'tests/e2e/**', 'tests/evals/**', 'tests/adversarial/**'],
},
});
```
### Playwright (`playwright.config.ts`)
```typescript
export default defineConfig({
testDir: 'tests/e2e',
testIgnore: ['**/tmp/**'],
});
```
## tests/README.md
The repository's own copy of the contract. This is where environment-specific facts belong; they do
not belong in a skill or a reference document. Fill in every row.
```markdown
# Test suite
Layout and tier definitions: <link to the team's copy of the contract>
## Suite contract
| Folder | Present | Tiers in use | Connection env var | CI lane |
| --- | --- | --- | --- | --- |
| unit | | | n/a | |
| integration | | | | |
| e2e | | | | |
| smoke | | | | |
| regression | | | | |
| adversarial | | | | |
| evals | | | n/a | |
## Data tiers
- Local replica (T3) provisioned by: {container, fixture command, or migration path}
- Staging replica (T4) available: {yes / no - if no, state the residual risk accepted at T3}
- Tier 5 in use: {yes / no}
- Tier 5 read-only role: {role name, or "n/a"}
- Tier 5 authorization reference: {ticket or approval record, or "n/a"}
## Commands
- Local tight loop: {command}
- PR gate: {command}
- Pre-release: {command}
- On demand (evals, adversarial, tier 4-5): {command}
## Deviations from the canonical layout
{grandfathered directories and why, or "none"}
```