references/recovery-scripts.md
# Refactor recovery — full playbook and scripts
The executable playbook behind `selector-drift-recovery`. Assumes **Playwright >= 1.50, TypeScript** (trace-viewer DOM-snapshot panel, `ariaSnapshot`, `getByRole().filter()`). Cypress adaptation note at the end — but note the trace-zip, JSON-reporter shape, and `getByRole` ladder are Playwright-specific.
---
## 0. Preconditions
- [ ] You can run the affected suite locally and reproduce the failures.
- [ ] You have the new build deployed (preview) or running locally.
- [ ] You have at least one old-DOM source: a pre-refactor CI trace artifact, a Storybook setup, or a staging env on the old build.
- [ ] You have a feature branch off main for the recovery PR.
Missing any of these → fix it first. The skill cannot generate recovery from no reference point.
---
## 1. Snapshot the old DOM
### Option A — From the last green Playwright trace
```bash
gh run list --branch main --workflow ci.yml --status success --limit 5
gh run download <RUN_ID> -n playwright-traces
# Open a trace; select an action and read the per-action DOM SNAPSHOT panel.
# (There is no "Copy HTML at this step" menu item anymore — read the snapshot panel,
# or dump it programmatically by replaying, below.)
npx playwright show-trace traces/checkout.zip
```
Programmatic dump (an aria-snapshot is the signal role-first recovery needs; raw HTML optional):
```typescript
// scripts/snapshot-dom.ts — run against any reachable build (old or new)
import { chromium } from '@playwright/test';
import fs from 'fs';
const [, , baseUrl, outDir] = process.argv; // e.g. http://localhost:6006 .drift-recovery/old
const routes = ['/checkout', '/cart', '/account'];
(async () => {
const browser = await chromium.launch();
const page = await browser.newPage();
fs.mkdirSync(outDir, { recursive: true });
for (const route of routes) {
await page.goto(baseUrl + route);
await page.waitForLoadState('networkidle'); // snapshot AFTER hydration
const aria = await page.locator('body').ariaSnapshot();
const name = route.replace(/\//g, '_') || 'root';
fs.writeFileSync(`${outDir}/${name}.aria.yml`, aria);
fs.writeFileSync(`${outDir}/${name}.html`, await page.content());
}
await browser.close();
})();
```
### Option B — Storybook at a pre-refactor commit
```bash
git checkout <PRE_REFACTOR_SHA>
npm run storybook -- --port 6006 --no-open &
sleep 5
npx tsx scripts/snapshot-dom.ts http://localhost:6006 .drift-recovery/old
git checkout -
```
### Option C — Staging on the old version
```bash
npx tsx scripts/snapshot-dom.ts https://staging-old.example.com .drift-recovery/old
```
Pick **one** option, document which, move on.
**Aria-snapshot diff** (the drift signal): `diff .drift-recovery/old/_checkout.aria.yml .drift-recovery/new/_checkout.aria.yml`. A role-tree diff highlights *which roles/names moved* and ignores class renames and wrapper divs that a raw-HTML diff drowns in.
---
## 2. Snapshot the new DOM
Run the same routes in the new build (a Vercel/Netlify preview is ideal):
```bash
PREVIEW_URL=$(gh pr view <REFACTOR_PR> --json deployments -q '.deployments[0].url')
npx tsx scripts/snapshot-dom.ts "$PREVIEW_URL" .drift-recovery/new
```
For every old snapshot there should be a matching new one. A route that 404s in the new build is a deleted flow — mark its tests for deletion in Phase 6.
---
## 3. Identify broken selectors (and capture intent + route)
```bash
PLAYWRIGHT_TEST_BASE_URL=$PREVIEW_URL \
npx playwright test --reporter=json > .drift-recovery/results.json
```
The parser captures file / line / old-locator / **errorType** and — the two fields the reporter does NOT give you — the **inferred intent** and **page route**, read from the test source. Both are required by Phase 4; without them the generator runs on `undefined`.
```typescript
// scripts/identify-drift.ts
import fs from 'fs';
import path from 'path';
interface Failure {
file: string;
line: number;
oldLocator: string;
errorType: 'timeout' | 'assertion' | 'other';
intent: string; // inferred from source — NOT in the reporter
pageRoute: string; // which new/*.aria.yml to load — NOT in the reporter
}
const results = JSON.parse(fs.readFileSync('.drift-recovery/results.json', 'utf8'));
// Map a source line to a short intent + the route under test, by reading the spec.
function inferContext(file: string, line: number): { intent: string; pageRoute: string } {
const src = fs.readFileSync(file, 'utf8').split('\n');
// intent: nearest comment, test title, or the action/assertion on this line
const around = src.slice(Math.max(0, line - 4), line + 2).join(' ');
const titleMatch = around.match(/test\(['"`](.+?)['"`]/);
const intent = (titleMatch?.[1] || around.replace(/\s+/g, ' ').trim()).slice(0, 60);
// route: nearest page.goto('...') above this line
let pageRoute = '/';
for (let i = line; i >= 0; i--) {
const m = src[i]?.match(/goto\(['"`]([^'"`]+)['"`]/);
if (m) { pageRoute = new URL(m[1], 'http://x').pathname; break; }
}
return { intent, pageRoute };
}
const failures: Failure[] = [];
for (const suite of results.suites || []) {
for (const spec of collectSpecs(suite)) {
for (const test of spec.tests || []) {
for (const r of test.results || []) {
if (r.status !== 'failed') continue;
const msg = r.error?.message || '';
// A DRIFT failure is a locator timeout; an assertion failure is a value mismatch.
const isTimeout = /TimeoutError/.test(msg) && /locator\./.test(msg);
const locatorMatch = msg.match(/(locator\([^\n]*?\)|getBy\w+\([^\n]*?\))/);
const file = r.error?.location?.file;
const line = r.error?.location?.line;
if (!file || !line || !locatorMatch) continue;
failures.push({
file, line,
oldLocator: locatorMatch[1],
errorType: isTimeout ? 'timeout' : /expect/.test(msg) ? 'assertion' : 'other',
...inferContext(file, line),
});
}
}
}
}
// Group by file for the PR; persist a flat list keyed for Phase 4.
fs.writeFileSync('.drift-recovery/failures.json', JSON.stringify(failures, null, 2));
console.log(`Identified ${failures.length} broken locators across ` +
`${new Set(failures.map(f => f.file)).size} files`);
function collectSpecs(suite: any): any[] { // suites nest; flatten
return [...(suite.specs || []), ...(suite.suites || []).flatMap(collectSpecs)];
}
```
> **POM caveat:** `r.error.location` points at the *failing line*. For inline locators that is the locator. For a locator wrapped in a Page Object, it is the POM helper, not the test — `file`/`line` will be the helper file. Re-read the locator from the trace action, or grep the POM source, before applying. The script above silently assumes inline locators.
Assertion-type rows (`errorType: 'assertion'`) are not drift — drop them from the candidate run; the locator resolved fine and failed a value check.
---
## 4. Generate replacement candidates
Loads the new-DOM HTML for each failure's `pageRoute`, generates candidates by the role-first ladder, and only awards **score 3 after region scoping makes the match unique**.
```typescript
// scripts/generate-candidates.ts
import { chromium, Page } from '@playwright/test';
import fs from 'fs';
interface Candidate { selector: string; score: number; rationale: string; }
async function generate(page: Page, intent: string): Promise<Candidate[]> {
const out: Candidate[] = [];
const rx = new RegExp(intent.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'), 'i');
// 5 — data-testid added by the refactor
for (const el of await page.locator('[data-testid]').all()) {
const id = await el.getAttribute('data-testid');
const text = (await el.textContent())?.trim().slice(0, 50) || '';
if (id && rx.test(text)) out.push({ selector: `getByTestId('${id}')`, score: 5, rationale: `testid matches "${intent}"` });
}
// 4 — getByLabel for form fields, then getByRole + name (unique)
const label = page.getByLabel(rx);
if (await label.count() === 1) out.push({ selector: `getByLabel(/${intent}/i)`, score: 4, rationale: 'unique labelled field' });
for (const role of ['button', 'link', 'heading', 'textbox', 'checkbox'] as const) {
const loc = page.getByRole(role, { name: rx });
const n = await loc.count();
if (n === 1) {
out.push({ selector: `getByRole('${role}', { name: /${intent}/i })`, score: 4, rationale: 'unique role+name' });
} else if (n > 1) {
// 3 — ONLY if region scoping resolves it to exactly one. An un-scoped multi-match is NOT a 3.
for (const region of await page.getByRole('region').all()) {
const regionName = await region.getAttribute('aria-label') || (await region.getByRole('heading').first().textContent())?.trim();
if (!regionName) continue;
const scoped = region.getByRole(role, { name: rx });
if (await scoped.count() === 1) {
out.push({
selector: `getByRole('region', { name: /${escapeName(regionName)}/i }).getByRole('${role}', { name: /${intent}/i })`,
score: 3, rationale: `region-scoped to one match in "${regionName}"`,
});
break;
}
}
}
}
// 2 — text only
if (await page.getByText(rx).count() === 1) out.push({ selector: `getByText(/${intent}/i)`, score: 2, rationale: 'text-only, fragile' });
return out.sort((a, b) => b.score - a.score);
}
const escapeName = (s: string) => s.slice(0, 40).replace(/[/\\]/g, '');
(async () => {
const failures = JSON.parse(fs.readFileSync('.drift-recovery/failures.json', 'utf8'))
.filter((f: any) => f.errorType === 'timeout'); // drift only
const browser = await chromium.launch();
const page = await browser.newPage();
fs.mkdirSync('.drift-recovery/screenshots', { recursive: true });
const result: any[] = [];
for (const f of failures) {
const route = f.pageRoute.replace(/\//g, '_') || 'root';
const html = fs.readFileSync(`.drift-recovery/new/${route}.html`, 'utf8');
await page.setContent(html);
const best = (await generate(page, f.intent))[0];
if (best && best.score >= 3) {
const shot = `.drift-recovery/screenshots/${route}-${f.line}.png`;
try { await page.locator(best.selector).first().screenshot({ path: shot }); } catch {}
result.push({ ...f, ...best, screenshotPath: shot, applied: false });
} else {
result.push({ ...f, selector: null, score: 0, rationale: 'no safe candidate', applied: false });
}
}
await browser.close();
fs.writeFileSync('.drift-recovery/candidates.json', JSON.stringify(result, null, 2));
})();
```
A `count > 1` selector that no region scoped to one match falls through with **no score-3 candidate** — exactly right. It surfaces as score 0 (flag for human), not as a false 3.
---
## 5. Apply (line-anchored), validate, iterate
Replace the specific `(file, line)` — never a content-wide `String.replace`. The reporter's `oldLocator` is a *rendered* string (`locator('.summary > h2')`) that usually does not match the source expression verbatim, `String.replace` hits only the first occurrence, and identical locators on different lines collide.
```typescript
// scripts/apply-recovery.ts
import fs from 'fs';
const candidates = JSON.parse(fs.readFileSync('.drift-recovery/candidates.json', 'utf8'));
const byFile: Record<string, any[]> = {};
for (const c of candidates) {
if (c.score < 3 || !c.selector) continue; // never auto-apply below 3
(byFile[c.file] ||= []).push(c);
}
for (const [file, changes] of Object.entries(byFile)) {
const lines = fs.readFileSync(file, 'utf8').split('\n');
// bottom-up so edits above don't shift the lines below
for (const c of changes.sort((a, b) => b.line - a.line)) {
const i = c.line - 1;
const before = lines[i];
// swap the call expression on that line; .first() etc. preserved by anchoring on the call head
lines[i] = lines[i].replace(/(?:page\.|this\.page\.)?(?:locator|getBy\w+)\([^)]*\)/, `page.${c.selector}`);
c.applied = lines[i] !== before; // record only what actually changed
if (!c.applied) console.warn(`No change at ${file}:${c.line} — locator wrapped or moved; review manually`);
}
fs.writeFileSync(file, lines.join('\n'));
}
fs.writeFileSync('.drift-recovery/candidates.json', JSON.stringify(candidates, null, 2)); // persist applied flags
```
Then validate the **full affected suite** (not just the previously-failing tests):
```bash
npx playwright test --reporter=json > .drift-recovery/post-recovery.json
jq '.stats.unexpected' .drift-recovery/post-recovery.json # 0 = all recovered tests green
```
For any test still failing, revert that one line (`git checkout -p`) and flag it for the PR's manual-review list.
---
## 6. Open the PR
```bash
git checkout -b chore/test-selector-recovery-$(date +%Y%m%d)
git add tests/ .drift-recovery/candidates.json
git commit -m "chore(tests): bulk selector recovery after <refactor>"
git push -u origin HEAD
gh pr create --title "chore(tests): selector recovery after <refactor>" \
--body "$(cat .drift-recovery/pr-body.md)"
```
Body generated from `candidates.json`, filtering on `applied` (set by `apply-recovery.ts`):
```typescript
// scripts/build-pr-body.ts
import fs from 'fs';
const c = JSON.parse(fs.readFileSync('.drift-recovery/candidates.json', 'utf8'));
const recovered = c.filter((x: any) => x.applied); // applied === true
const flagged = c.filter((x: any) => !x.applied && x.score < 3);
const perFile = Object.entries(
recovered.reduce((m: any, r: any) => ((m[r.file] ||= []).push(r), m), {})
).map(([file, rows]: any) =>
`### ${file}\n| Line | Old | New | Score | Shot |\n|---|---|---|---|---|\n` +
rows.map((r: any) => `| ${r.line} | \`${r.oldLocator}\` | \`${r.selector}\` | ${r.score} |  |`).join('\n')
).join('\n\n');
fs.writeFileSync('.drift-recovery/pr-body.md', `## Trigger
<Link to refactor PR>
## Summary
- ${recovered.length} selectors recovered
- ${new Set(recovered.map((r: any) => r.file)).size} test files updated
- ${flagged.length} flagged for manual review
## Per-file changes
${perFile}
## Flagged for review
${flagged.map((f: any) => `- ${f.file}:${f.line} — ${f.oldLocator} (${f.rationale})`).join('\n')}
## How to review
- Check each screenshot: does \`new\` point at the same element as \`old\`?
- For score-3 candidates, verify the region scope is meaningful in the new design.
- For flagged tests, decide: rewrite, delete, or accept a manual update.
`);
```
---
## Cleanup
```bash
rm -rf .drift-recovery/
grep -qxF '.drift-recovery/' .gitignore || echo '.drift-recovery/' >> .gitignore
```
---
## When this workflow is not enough
- **Semantics changed, not just structure.** A "Submit" button that became a multi-step confirmation flow cannot be rescued by a selector swap — the scenario itself must change. Send those to a human.
- **Deeply encapsulated Page Objects.** If the locator lives three levels into a `CheckoutPage` POM, the line in the reporter is the helper, not the test. Walk the POM source (or read the trace action) to find and apply the real locator.
- **SSR with hydration mismatch.** Snapshot AFTER hydration (`await page.waitForLoadState('networkidle')`) or the new-DOM snapshot is the pre-hydration tree and candidates miss client-rendered elements.
## Cypress adaptation
The *shape* of the workflow ports to Cypress, but the mechanics do not: Cypress has no trace-zip artifact, a different reporter JSON (`mochawesome`/`mocha`), and no `getByRole`/`ariaSnapshot` equivalent (use `@testing-library/cypress` `findByRole`, and `cy.document().then(d => d.body.outerHTML)` for raw DOM). Reuse Phases 1, 2, 5, 6 conceptually; rewrite the parsing and candidate-generation scripts for the Cypress reporter and `findBy*` queries.
SKILL.md
---
name: selector-drift-recovery
description: >-
Bulk-regenerate broken test selectors after a UI refactor or redesign. Detects
drift between old and new DOM with an aria-snapshot diff, maps old locators to
new equivalents using role-first + region scoping, validates against the new
build, and produces a single PR with grouped per-file selector updates and
per-change evidence. Assumes Playwright >= 1.50 (trace viewer DOM-snapshot
panel, getByRole filtering, ariaSnapshot). Use when: "UI refactor broke tests,"
"redesign broke tests," "bulk update selectors," "regenerate selectors after
refactor," "selector drift," "fix N broken tests after redesign." Not for:
healing one flaky test at runtime — use test-reliability. Not for: writing a new
test suite from scratch — use playwright-automation. Not for: re-recording tests
after a framework switch (Selenium to Playwright) — use test-migration.
Related: test-reliability, playwright-automation, test-migration, ci-cd-integration, visual-testing.
license: MIT
metadata:
author: kindlmann
version: "2.0"
category: automation
---
<objective>
A redesign shipped and 23 tests now fail with `TimeoutError: locator.* exceeded` — the DOM moved, not the product. This skill closes that maintenance loop: it diffs the old DOM against the new one, regenerates the broken selectors role-first against the new build, validates the whole suite, and ships the diff as one reviewable PR with a confidence score and screenshot per change. The trigger is an event (a refactor merged), not a flake. The output is a PR a human signs off on — not a silent runtime auto-heal.
This is the bulk, offline, batch counterpart to `test-reliability`. They share the multi-attribute locator + confidence scoring primitives but run in opposite directions: `test-reliability` heals one selector at runtime behind a guarded threshold; `selector-drift-recovery` regenerates N selectors offline against the new DOM and bundles them into a PR. If you are doing the second workflow inside `test-reliability`, switch here.
</objective>
## Quick Route
| Situation | Go to |
|-----------|-------|
| One broken test, not a refactor | Stop — use `test-reliability` instead |
| 200+ broken selectors across many files | Split by area first (one PR per page/dir), then Phase 1 |
| Refactor changed flows/semantics, not just structure | Stop — rewrite from specs with `playwright-automation` |
| Framework switch (Selenium → Playwright) | Stop — use `test-migration`, not drift recovery |
| No old-DOM reference exists anywhere | Capture one (Phase 1) or scope down — without it this is "rewrite tests" |
| Have old + new DOM, ready to map | Phase 1 → 6 below |
---
## Discovery Questions
Check `.agents/qa-project-context.md` first — if it exists, use it and skip anything answered there. It identifies your E2E framework, selector strategy, and known fragile areas. Then:
1. **What triggered the drift?** A planned refactor (Storybook can show the new DOM before merge), a shipped redesign (new DOM is in main), a dependency upgrade, or a Tailwind/CSS migration? The trigger decides whether you run pre-emptively or react to CI failures.
2. **What is the blast radius?** A single component, a page, or the whole app? Single component: scope recovery to the test files that touch it. Global: budget half a day to a day, and decide which tests should be rewritten rather than re-selected.
3. **What is your current selector strategy?** If selectors are mostly `data-testid` and the refactor preserved testids, recovery is trivial. If they are CSS-class or XPath based, expect 30–60% to need a new *strategy*, not just a new locator.
4. **Is there a passing baseline?** You need the old DOM *somewhere*: a previous CI trace artifact, a deployed staging build, a Storybook story, or git history of the components. No old-DOM reference means this degrades to "rewrite tests."
5. **Are the broken locators inline or wrapped in a Page Object?** The JSON reporter's `error.location` points at the *failing line*. For inline locators that is the locator itself; for POM-wrapped locators it points at the POM helper, not the test. Know this before you trust the auto-extracted line numbers (see Failure Modes).
6. **Who reviews the resulting PR?** Confidence-scored updates need a human signoff. Decide upfront whether the PR goes to the test author, the engineer who did the refactor, or the QA lead.
---
## Core Principles
1. **Recovery is event-driven, not failure-driven.** Run this when a refactor is planned or just shipped, not when one test goes flaky. One broken test → `test-reliability`. Ten or more from the same event → this skill.
2. **Old DOM, new DOM, mapped pair.** The whole skill rests on a snapshot before the refactor and one after. Everything else is bookkeeping around that pair. Capture both as **aria snapshots** (`await page.locator('body').ariaSnapshot()`), not raw HTML — a role-tree diff is exactly the signal role-first recovery needs and ignores the cosmetic churn (class renames, wrapper divs) that a raw-HTML diff drowns in. If you cannot produce both snapshots, fix that first.
3. **Role-first replacement, every time.** Even when the old test used CSS, the regenerated selector should prefer `getByRole` + accessible name, then `getByLabel` for form fields, then `getByTestId` when the refactor added one. The recovery PR is your chance to ratchet the average selector stability score up (0–5 rubric below, shared with `test-reliability`).
4. **Disambiguate by region scoping, not layout selectors.** When role+name is ambiguous (two "Submit" buttons), narrow with `getByRole('region', { name }).getByRole('button', …)` or `getByRole(...).filter({ hasText })`. Do **not** reach for `:near()` / `:right-of()` — see the Avoid note. A score-3 candidate is one where region scoping has been *applied* and the locator now matches exactly one element.
5. **One PR, grouped by file, with per-change evidence.** Reviewers cannot eyeball 47 selector changes spread across 30 commits. Bundle into one PR, group hunks by test file, attach a confidence score + DOM screenshot per change.
6. **The suite must pass before merge, and dead tests get deleted.** A regenerated selector that doesn't run is a worse version of the original problem; the skill ends with green CI, not a generated diff. And if the refactor removed a feature, prune its tests — do not regenerate selectors for elements that no longer exist.
> **Avoid:** Playwright layout selectors `:near()`, `:right-of()`, `:left-of()`, `:above()`, `:below()` as disambiguators — officially deprecated and "may be removed," because a 1px layout shift changes the match (Playwright docs, 2026). They also contradict the role-first thesis. Use region scoping / `getByRole().filter()` instead.
---
## Workflow
Six phases, each gated by a check before the next.
### Phase 1 — Snapshot the old DOM
You need a snapshot of every page or component the affected tests touch, in its pre-refactor state. **Sources, in preference order:**
1. **Last green CI trace artifact.** Most teams save Playwright traces on failure (`trace: 'on-first-retry'`). Download the green-run artifact, open a trace with `npx playwright show-trace traces/checkout.zip`, select an action, and read the per-action **DOM snapshot panel** for each surface. (The viewer no longer has a "Copy HTML at this step" menu item; you read the snapshot panel or, for a programmatic dump, replay with `page.content()` / `ariaSnapshot()`.)
2. **Storybook at a pre-refactor commit.** `git checkout <PRE_REFACTOR_SHA>`, start Storybook, and dump each story with a tiny `page.content()` / `ariaSnapshot()` script.
3. **A staging build still on the old version.** Navigate the same flows and snapshot.
4. **Git history of the components.** Reconstructable but the most expensive — render in isolation.
Output: `.drift-recovery/old/<page-or-component>.aria.yml` (and `.html` if you also need raw markup) per affected unit.
**Gate:** You can answer "what did this page look like when the tests last passed?" from a snapshot file, not from memory.
### Phase 2 — Snapshot the new DOM
Run the same surfaces in the post-refactor build — a Vercel/Netlify preview deploy is ideal, or a local dev server / the PR branch in CI. Wait for hydration (`await page.waitForLoadState('networkidle')`) before snapshotting, or SSR pages give you the pre-hydration tree and you miss client-rendered elements.
Output: `.drift-recovery/new/<page-or-component>.aria.yml` matching the old set.
**Gate:** Every old snapshot has a matching new one. If a route 404s in the new build, that flow was deleted — mark its tests for the deletion pile in Phase 6.
### Phase 3 — Identify broken selectors and infer intent
For each test file, run against the new build with the JSON reporter, then parse it. Capture, per failure: **file, line, old locator string, error type (timeout vs assertion), and inferred intent**. Group the results by test file.
- **Error classification:** a drift failure is `TimeoutError: locator.* exceeded`. Distinguish it from an assertion failure (`expect(...).toBe`) so you don't try to re-select a locator that resolved fine but failed a value check.
- **Inferred intent is mandatory and not in the reporter.** Read the surrounding test code — what action is taken on the locator, what assertion follows — and store a short intent string ("submit the order", "read the order total"). The candidate generator keys off intent, so this step is load-bearing, not commentary.
- **Page route is also not in the reporter.** Map each locator to the snapshot it should resolve against (which `.drift-recovery/new/*.aria.yml`) so Phase 4 can load the right new DOM.
The result is a per-file table:
| Test file | Line | Old locator | Error type | Page route | Inferred intent |
|---|---|---|---|---|---|
| `tests/checkout.spec.ts` | 42 | `getByTestId('submit-btn')` | timeout | `/checkout` | Submit the order |
| `tests/checkout.spec.ts` | 87 | `locator('.summary > h2')` | timeout | `/checkout` | Read the order total |
See `references/recovery-scripts.md` for `identify-drift.ts`, which produces exactly these rows (with the intent/route fields populated, not stubbed).
**Gate:** Every broken locator has an inferred intent and a page route. If you cannot infer intent, ask the test author or read the original PR — do not guess.
### Phase 4 — Generate replacement candidates
For each row, generate candidates against the **new** DOM snapshot and score each on the 0–5 rubric (shared with `test-reliability`). Strategy ladder, best first:
1. **New `data-testid`** added by the refactor team — the most stable choice they made. Score 5.
2. **`getByRole` + accessible name, unique on the page.** Score 4.
3. **`getByLabel` for a form field**, when the intent is an input and a label exists. Score 4 (use over a bare role when the field has no name otherwise).
4. **Role + name, region-scoped to a single match.** If role+name alone returns >1 element, wrap it — `getByRole('region', { name }).getByRole(role, { name })` or `.filter({ hasText })` — and confirm the scoped locator now matches exactly one. Only score 3 **after** scoping makes it unambiguous.
5. **Visible text only** (`getByText`). Score 2 — fragile to copy changes.
6. **CSS class on the changed structure.** Score 1 — usually still broken.
7. **No safe replacement.** Score 0 — flag for human.
| Score | Replacement strategy | Auto-apply? |
|---|---|---|
| 5 | New `data-testid` exists | yes |
| 4 | `getByRole` + accessible name (or `getByLabel`), unique on page | yes |
| 3 | `getByRole` + name, region-scoped to exactly one match | yes |
| 2 | Visible-text-only | no |
| 1 | CSS class on changed structure | no |
| 0 | No safe replacement found | no — flag for human |
A candidate is **score 3 only if scoping already resolved it to a single element**. A still-ambiguous multi-match (`count > 1`, "needs scoping") is not a 3 — it is unfinished, and must not be auto-applied.
Output: `.drift-recovery/candidates.json` with `{ file, line, oldLocator, selector, score, rationale, screenshotPath }` per change. See `references/recovery-scripts.md` for `generate-candidates.ts`.
**Gate:** Every row has a candidate scored ≥ 3 (and confirmed unique), or is flagged for human review. Score-0/1/2 are never auto-applied.
### Phase 5 — Apply, validate, iterate
1. Apply the score-≥3 replacements to a feature branch. Replace by `(file, line)`, not a content-wide string replace — the reporter's locator string is a *rendered* form (`locator('.summary > h2')`) that rarely matches the source expression verbatim, and a naive `String.replace` hits only the first occurrence and collides on identical locators. Edit the specific line; set `applied: true` on each candidate you actually wrote.
2. Run the **full affected suite**, not just the previously-failing tests — a new selector can match an unintended element and break a previously-passing test.
3. Per test: **passed** → keep the replacement. **failed** → revert that one line, mark the test for human review.
4. Emit a summary: N recovered automatically, M flagged.
**Gate:** Recovered tests pass the suite. Flagged tests are clearly marked, not silently included.
### Phase 6 — Ship the PR
The PR is the deliverable. **Title:** `chore(tests): selector recovery after <refactor description>`. **Body** (generated from `candidates.json`, filtering on `applied`):
```markdown
## Trigger
<Link to the refactor PR / describe the redesign>
## Summary
- N test files updated - M selectors changed
- K tests deleted (feature removed) - L tests flagged for manual review
## Per-file changes
<For each file: a table of line, old, new, score, screenshot URL>
## Flagged for review
<Tests where no candidate scored >= 3, with the inferred intent>
## How to review
- Check each screenshot: does `new` point at the element `old` pointed at?
- For score-3 candidates, verify the region scope is meaningful in the new design.
- For flagged tests, decide: rewrite, delete, or accept a manual selector update.
```
Attach screenshots inline via your team's CI artifact URL pattern. See `references/recovery-scripts.md` for `apply-recovery.ts` (line-anchored) and `build-pr-body.ts`.
**Gate:** PR is reviewable in one sitting. Too large → split by area (one PR per page / component / test directory).
---
## Anti-Patterns
1. **Auto-applying score-0, -1, or -2 candidates.** A score-2 is "we found *some* element." That is gambling, not recovery — and it is the usual cause of a suite-wide stability score *dropping* after a recovery. Auto-apply only score ≥ 3.
2. **Calling an ambiguous multi-match "score 3."** If `getByRole(...)` returns >1 element it is not a 3 until region scoping narrows it to exactly one. Scoring it 3 and auto-applying ships a locator that resolves to the wrong element.
3. **Skipping the screenshots.** A score-4 candidate can still point at the wrong element when the page has two regions with the same role + name. The per-change screenshot is the only check that catches semantic drift; confidence scores alone do not.
4. **Content-wide string replace instead of line-anchored edits.** `content.replace(oldLocator, …)` hits the first occurrence only, collides on duplicate locators, and silently no-ops when the reporter's rendered string differs from the source expression. Edit the specific `(file, line)`.
5. **Trusting auto-extracted line numbers for POM-wrapped locators.** The JSON reporter's `error.location` points at the failing line, which for a Page Object is the helper, not the test. Re-read the locator from the trace action or grep the POM source before applying.
6. **Recovering tests for deleted features.** The refactor may have removed flows. Map deleted routes in Phase 2 and prune those tests — do not regenerate selectors for elements that no longer exist.
7. **Auto-merging the recovery PR in CI.** The PR *is* the artifact; the whole point is a human eyeballing the per-change evidence. Auto-merge once tests pass defeats the purpose — a green suite with a selector pointing at the wrong-but-present element passes and erodes trust. Require a reviewer.
8. **Treating the PR as urgent.** A failed suite feels urgent; a *correctly* recovered one is what matters. Time pressure produces score-2 replacements that quietly degrade the suite.
---
## Failure Modes
| Symptom | Likely cause | Fix or check |
|---|---|---|
| `identify-drift.ts` finds 0 failures despite red CI | Suite errored before producing the JSON report, or you parsed the wrong file | `jq '.stats' .drift-recovery/results.json`; confirm `--reporter=json` redirected to the file |
| Extracted line points at a POM file, not the test | Locator is wrapped in a Page Object | Read the locator from the trace action, or grep the POM source for the rendered string |
| `generate-candidates.ts` reads `undefined` for intent/route | Phase 3 output missing the inferred-intent / page-route fields | Populate them in Phase 3 — they are not in the reporter; the generator cannot infer them |
| New-DOM snapshot is missing client-rendered elements | Snapshotted before hydration | Add `await page.waitForLoadState('networkidle')` before `ariaSnapshot()` |
| Average stability score dropped after recovery | Score-2/CSS candidates auto-applied | Revert candidates with score < 3 in `candidates.json`; only role/label/testid should land (see scorer below) |
---
## Verification
Run these on the recovery branch before opening the PR, smallest first:
```bash
# 1. No applied candidate is below the stability floor (machine-checkable proxy for "ratcheted up")
node references/score-candidates.mjs .drift-recovery/candidates.json
# prints average score + count of applied rows with score < 3 — that count MUST be 0
# 2. The recovered suite is green against the new build
PLAYWRIGHT_TEST_BASE_URL=$PREVIEW_URL npx playwright test --reporter=json \
| jq '.stats.unexpected' # must be 0 (flagged tests excluded via grep/skip)
# 3. The PR exists with the evidence body
gh pr view --json title,body -q '.title' # contains "selector recovery"
```
`references/score-candidates.mjs` reads `candidates.json`, prints the average applied score and the count of `applied && score < 3` rows; a non-zero count means a low-confidence selector leaked in. Step 1 passing + step 2 returning `0` is the proof the recovery worked.
---
## Done When
- Every in-scope test passes on the new build, is flagged for human review with a clear reason, or is deleted because its feature is gone.
- `references/score-candidates.mjs candidates.json` reports **0** applied candidates with score < 3.
- The PR is open (`gh pr view` succeeds) with per-change evidence: confidence score and screenshot per change, grouped by file.
- `npx playwright test --reporter=json | jq '.stats.unexpected'` returns `0` on the recovery branch.
- A short note is added to `.agents/qa-project-context.md` describing the refactor and any new test patterns introduced.
## Reference Files (in `references/`)
- **recovery-scripts.md** — the full playbook with corrected, runnable scripts: aria-snapshot capture, `identify-drift.ts` (populates intent + route), `generate-candidates.ts` (region-scoping ladder, true score-3), line-anchored `apply-recovery.ts`, and `build-pr-body.ts`. Includes the Cypress-adaptation note.
- **score-candidates.mjs** — tiny stability scorer; prints the average applied score and the count of applied rows below score 3. Used by Verification and Done When.
## Related Skills
- **test-reliability** — runtime per-test healing. Use for one flaky test, not a refactor-driven mass update. Shares the 0–5 stability rubric.
- **playwright-automation** — writing new tests from scratch. Use when the refactor removed enough features that tests should be rewritten, not patched.
- **test-migration** — switching frameworks (Selenium → Playwright). A migration re-records tests; it is not selector drift, even though both touch many tests at once.
- **visual-testing** — had this run on every PR, the refactor's visual diff would have flagged before merge. Recovery is the fallback when that coverage is missing.
- **ci-cd-integration** — wires the recovery PR's validation step into CI.