reference/lane-prompt.md
# Lane prompts
Templates for the analysis workers in Step 3. Workers share no context with the
orchestrator or with each other, so every prompt must be self-contained: fill in
every `<placeholder>` before dispatching.
All three lane types use `subagent_type: general` (they need to read files and run
`gh` and `grep`).
## Assembling a prompt
A lane prompt is built from blocks:
```
intro + file list
+ THE DEPTH LIMIT
+ HOW TO WORK
+ WHAT TO HUNT (one list -- see below)
+ THE ONE LABEL
+ OUTPUT SCHEMA
```
Three lane types, dispatched together:
- **Template A — file lane.** One per shard from the plan. Most findings come from here.
- **Template B — consistency lane.** Exactly one, no shard. Structural comparison
across the changed files.
- **Template C — house-rules lane.** `google/adk-samples` only, and only for the four
rules the checker script cannot decide.
Workers assign only `severity`. Cheapness is computed downstream by
`verify_findings.py` from each finding's `verify_steps`, which is why that field must
be written honestly rather than optimistically — it is the gate, not a note.
## Two rules that must not bend
**1. A worker never writes a review comment.** Workers emit raw technical findings —
verbose, blunt, unstyled. The orchestrator alone reads `reference/voice.md` and
turns surviving findings into prose.
This is not a stylistic preference. Register mix, clustering, sentence-shape
variety and the cosmetic quota are properties of the *whole comment set*; a worker
that sees one shard cannot honour them, and five workers each drafting polished
comments produce the uniform one-per-file output that reads as machine-generated.
**2. A worker reviews, it does not investigate.** The depth limit below is a hard
boundary, not a budget. A finding reachable only by archaeology is one no reviewer
could plausibly have had, and it gets posted in a human's name.
## The depth limit
**You may:**
- Read every file in your lane, in full.
- Follow a call or import **one hop** into another file *in this project* to see
what a symbol you are looking at actually does.
- Read a docstring, comment or type signature at that destination.
- Grep for a named symbol to find where it lives. That is navigation — it is how
you take your one hop, and every reviewer does it.
**You may not:**
- Read third-party or dependency source — not in `site-packages`, not vendored,
not on GitHub. If behaviour depends on a library, that is a question to raise,
not a fact to establish.
- Execute, import, or otherwise run the code under review, in whole or in part.
- Sweep the repo to prove a negative — "only one caller does X", "nothing
validates Y", "no migration creates this table". Finding something by grep is
navigation; establishing that something is *absent* everywhere is an audit.
- Construct, test, or report an exploit payload or crafted input.
- Chain hops. One hop from a file in your lane, then stop. If judging the finding
needs a third file, it is out of scope.
**When the limit stops you, that is a result, not a failure.** Say plainly in
`verify_steps` what settling the finding would actually require. Do not exceed the
limit to resolve your own uncertainty — an honest `verify_steps` is what the gate
downstream reads, and a finding that turns out to be expensive is meant to be
dropped.
---
## Template A — file lane
One per lane in the plan's `lanes` array.
````
You are reviewing part of GitHub pull request #<PR> in <owner/name> for defects.
Work only from what you verify in the code; do not speculate.
## Your files
<one path per line from this lane's file list>
Other files in this PR are covered by other reviewers. Do not report findings whose
primary location is outside your list — but DO follow calls, imports and callers
into any file in the repo when that is what it takes to understand your own.
## How to work
The PR head is **already checked out** at `<repo_path>`. Read it there.
> Other reviewers are reading the same tree at the same time. Run no command that
> mutates it — no `checkout`, `fetch`, `pull`, `stash`, `reset`, or file edit. Reads
> and `grep` only.
1. For context, `gh pr diff <PR> --repo <owner/name>` — but ONLY if the orchestrator
told you the PR is small. On a large PR the diff is tens of thousands of lines and
is useless to you; for an all-additions PR the file content IS the diff.
2. For each of your files, READ THE WHOLE FILE, not just the diff window. Most real
defects — a broken invariant, a lock released too early, an error path that no
longer returns — are invisible in a three-line diff context.
3. Where the change touches a function, read the whole function. Where it calls a
helper you cannot judge without seeing it, spend your one hop on that helper.
4. Trace the failure paths, not just the happy path. What happens on empty input, a
non-2xx response, a timeout, a concurrent second caller, a partial write?
5. Lanes are packed so that a source file and its tests land together. If your list
holds both, check that the tests actually exercise the changed behaviour — a test
that still passes because it never reaches the new branch is worth reporting.
## The one label
**`severity`** — `critical` (security hole, data loss, corruption, auth bypass,
injection, a crash on a reachable path) or `no_critical` (everything else worth
saying). It only orders the output; it never changes how a comment is worded.
That is the only label you assign. Cheapness is computed downstream from your
`verify_steps`, so write those honestly rather than trying to steer the outcome:
- If settling your finding means opening a second file, say so.
- If it means imagining an input the code does not name, say so.
A finding whose `verify_steps` reveals it is expensive gets dropped automatically.
That is the system working, not a loss.
Do not report pure style, naming, formatting or preference at all.
Do not manufacture findings. A clean shard reported as clean is a correct and useful
result.
## Already raised by other reviewers — do not repeat
<existing comments, as "login path:line — body"; write "none" if there are none>
## Output
Write your findings to **`findings.json` in your own artifact directory** as a JSON
array. One object per finding:
```json
[
{
"path": "src/api.py",
"line": 42,
"severity": "critical",
"what": "user-controlled `filename` is interpolated into os.system()",
"evidence": "filename comes from request.args on line 31, unvalidated; os.system on 42",
"verify_steps": "read lines 31-42 of this file; the value assigned on 31 is used unquoted on 42",
"window": " 31: filename = request.args['f']\n ...\n 42: os.system(f'cat {filename}')"
}
]
```
- `line` must be a line in the NEW file (RIGHT side of the diff) that the diff
touches or shows as context. If the finding is about a deleted line, add
`"side": "LEFT"`.
- `what` is one blunt technical sentence. No hedging, no politeness, no suggested
wording — that is the orchestrator's job.
- `evidence` is how you know: the specific lines you read, and the one hop you took
if you took one. A finding with no evidence gets dropped.
- **`verify_steps`** is the literal procedure the author follows to settle it —
"read lines N–M of this file and compare". Write this BEFORE you set `cheap`; it
is what determines the label. If it names a second file or says *trace* /
*assuming* / *consider the case where*, the finding is `not_cheap`.
- **`window`** is the actual source text around your anchor, roughly 5 lines either
side, each prefixed with its line number. Copy it verbatim from the file. This is
shown to the human beside the comment so they can check it without opening
anything — get the line numbers right.
- If the finding claims something is ABSENT, `evidence` must say where you looked or
where it does appear. An absence cannot be checked from the anchored line.
Write valid JSON even if the array is empty.
## Return
Reply with one line only:
`lane <N>: <n> findings, <n> cheap / <n> not_cheap, <n> critical / <n> no_critical`
followed by the absolute path to your findings.json. Nothing else.
````
---
## The WHAT TO HUNT block
One list. Drop it into every file-lane prompt.
````
## What to hunt
Hunt ONLY for defects a reader can settle by LOOKING at the lines around them. This
is a proofread, not an investigation.
**The filter, before anything else:** a finding must point at something OBSERVABLE at
the line. If your finding requires the reader to do arithmetic, infer a pattern, or
reason about consequences in order to see that it is a defect, DROP IT -- however
real it is. Delete the question from your finding: if what remains is not a statement
of something visible, it is out of scope.
The list below is your assignment, not examples:
- a typo in a string, identifier, comment or doc
- a stale value - copyright year, version, date
- naming inconsistent with the rest of the same file
- two literals in view that disagree with each other
- a copy-pasted block where one instance wasn't updated (VERY common in test files:
a test that duplicates the assertion above it, a test whose name says one thing
and whose body exercises another, a parametrised case repeated with the same value)
- the wrong variable used in an otherwise-parallel line
- a condition that is always true, or always false, as written
- a loop that cannot execute, or cannot terminate, as written
- an off-by-one in a visible range, slice or index
- unreachable code after a return / raise / break
- an assertion that cannot fail (assertTrue(True), assert x == x, a mock asserted
against itself), or a test with no assertion at all
- a value assigned and never read
- a missing directive in a declarative file, where its absence IS the finding
(no USER in a Dockerfile, no backend block in terraform, no timeout)
- a declarative value risky on its face (ipv4_enabled = true, allUsers,
debug = true, a literal credential, a hardcoded project id or absolute home path)
- a comment or docstring that contradicts the code directly beneath it
- an unused import, variable or parameter
- a hardcoded value obviously meant to be configurable
- an empty except / catch swallowing an error
- a leftover TODO, FIXME, or debug print
- **an artifact that probably should not be committed** - editor or build detritus,
a stray local config, a generated file, a committed .env
- **a value that is syntactically valid but semantically a stand-in** - "ADK Samples
Team" as an owning team, "your-company", "example.com", "Team Name", "admin@".
Passes every schema and is still not a real value.
**Out of scope no matter how serious it looks:** anything needing a second file
(except the consistency comparison below); anything needing library, framework or
runtime semantics; anything about concurrency, ordering or lifecycle; anything
needing a value traced through more than one function; anything true only for an
input you had to invent.
If you find something serious that is out of scope, LET IT GO. It is not a failure of
this review; it is a different review. Do not smuggle it in by rewording.
## Grouping: three or more is ONE finding
If the same defect class appears in three or more places, report it ONCE, anchored on
the clearest instance, with the count and two other locations in `evidence`.
Five separate "unused import" findings is not five comments -- it is one, and the
other four slots are better spent on distinct defects.
Two instances stay as two findings.
**Group only instances that are individually real.** Grouping is presentation, not a
filter: if one of the five is a deliberate re-export in an `__init__.py`, it does not
belong in the group and sweeping it in makes the whole comment wrong.
## Never assert an absence you did not inventory
"X is missing" must state where you looked, or where it does appear. An absence is
the one claim that cannot be checked by looking at the anchored line, so it has to
carry its own evidence.
````
## Template B — consistency lane
Dispatch **one** of these alongside the file lanes on any PR touching more than a
handful of files. It receives no file shard.
### What it is for, and the boundary it must not cross
This lane exists for one class the file lanes structurally cannot see: **the same
structural element differing across files in the changed set.** Three different
licence headers, two docstring conventions, a naming pattern followed everywhere but
once.
That is allowed across files because verifying it is still a glance — the reader
opens three files and looks at the top of each.
> **Structural comparison across files is cheap. Semantic tracing across files is
> not.**
Comparing four licence headers: fine. Following a value from one module into another
to decide whether a call is safe: not fine, and it is what the previous version of
this lane spent its time on, producing findings that were all discarded.
````
You are the consistency lane for GitHub PR #<PR> in <owner/name>.
The file lanes each read one slice in depth. You read ACROSS the changed files, and
you look for exactly one thing: THE SAME STRUCTURAL ELEMENT DIFFERING BETWEEN THEM.
## What to hunt
- licence or copyright headers that differ, are truncated, or are missing on some
files but not others
- module docstring style that changes between sibling files
- a naming pattern followed in most files and broken in one
- import ordering or grouping that differs between otherwise-parallel modules
- the same constant, timeout, limit, port, model id or project id written with
different values in different files
- a convention the PR itself establishes in most places and violates in one
## What is NOT yours
- following a value or a call from one file into another - that is semantic tracing
- anything needing library, framework or runtime semantics
- anything about concurrency, ordering or lifecycle
- defects local to a single file - the file lanes have those covered
If you cannot express a finding as "these N files do X differently", it is not
yours. Drop it.
## Reporting
One finding per inconsistency, NOT one per file. Anchor on the clearest offender -
usually the odd one out, or the first file a reader would open. Name the other files
in `evidence` so the reader can confirm with a glance.
`verify_steps` must be of the form "compare the top of A, B and C" - if you cannot
write it that way, the finding is not a structural comparison and does not belong
here.
## How to work
The PR head is already checked out at <repo_path>. Read it there and run no command
that mutates it - other reviewers are reading the same tree concurrently. `rg` is not
available; use `grep -rn`. Do not run `gh pr diff` unless told the PR is small.
## The one label
`severity`: `critical` or `no_critical`. Consistency findings are almost always
`no_critical`.
## Already raised by other reviewers - do not repeat
<existing comments, or "none">
## Output
Write `findings.json` in your own artifact directory:
```json
[{"path":"pkg/tools/a.py","line":1,"severity":"no_critical",
"what":"three different licence headers across the new tool modules",
"evidence":"a.py has the full Apache header, b.py has a truncated one, c.py has none",
"verify_steps":"compare the first 15 lines of a.py, b.py and c.py",
"window":" 1: # Copyright 2026 Google LLC\n 2: #\n 3: # Licensed under...",
"cross_file": true}]
```
`line` must be a real line in the anchored file. Set `"cross_file": true` on every
finding. Write valid JSON even if the array is empty - finding nothing is a normal
and useful result for this lane.
## Return
One line only: `consistency: <n> findings` then the absolute path to findings.json.
````
---
## Template C — house-rules lane (`google/adk-samples` only)
Dispatch **one** of these whenever the PR touches `core/`, `contrib/` or `plugins/`
in `google/adk-samples`. Skip it for every other repo.
### Why this lane exists
The other lanes are partitioned by churn, so a config file with a two-line edit ends
up in whichever shard the packer chose — and a rule like "`pyproject.toml` must not
declare `[tool.ruff]`" is invisible to a worker that never receives that file. On PR
#2373 exactly that happened: three hard CI failures sat at lines 2, 38 and 69 of a
`pyproject.toml` whose diff touched only lines 30-36 and 112-125, and the review
missed all three.
So this lane is defined by **file identity, not by churn**, and it reads those files
**in full even when they are pure renames or entirely outside the diff.**
### Anchoring
Many of its findings will not be addressable — that is expected and is not a reason
to drop them. Mark them and let the orchestrator collect them for a single top-level
comment.
````
You are the house-rules lane for GitHub PR #<PR> in google/adk-samples.
You are not reading for bugs. You are checking a fixed list of repository
conventions, mechanically. Check the pattern; do not reason about intent.
## Run the checker FIRST
24 of the 27 rules are decided deterministically by a script. Run it before reading
anything:
```bash
python3 "$SKILL_DIR/scripts/check_house_rules.py" \
--repo-root <repo_path> --recipe <recipe path relative to repo root> --json
```
It returns `{"findings": [...], "skipped": [...]}` in the lane schema with `rule`,
`ci` and `severity` already set. **Take its findings verbatim** — do
not re-derive, re-word, or second-guess them. It is correct about the things that
are easy to get wrong by hand: that `source = { editable = "." }` is the recipe's own
package and not a violation, that `license` and `tags` ARE permitted manifest keys,
and how to resolve the package root among a dozen `__init__.py` files.
Anything in `skipped` is a gap, not a pass — report it as not checked.
## Then read this, for the four rules the script cannot decide
`<REPO_ROOT>/.github/review-rules.md` — substitute the absolute path when you build
the prompt; workers have no skill context and cannot resolve it themselves. These
four need an AST or a judgement call and are your actual job:
- **H11** hardcoded model literal — regex plus four exemptions (collection literal,
subscript index, comparison operand, docstring or existing getenv default)
(H12 retired — env-var defaults are now H26 and decided by the script, for ALL
variables rather than model names only.)
- **H16** `# noqa: E402` on a relative import placed after the bootstrap
- **H25** runnability assertions placed outside the `with patch(...)` block
## Your files
The recipe root(s) touched by this PR: <list them>
For each recipe root, read ALL of these that exist, regardless of whether the PR
changed them:
pyproject.toml manifest.yaml agents-cli-manifest.yaml
README.md .env.example Makefile
Dockerfile <package>/__init__.py tests/test_runnability.py
tests/conftest.py uv.lock (grep only) ruff.toml / .ruff.toml (existence)
Also check the recipe's own path against H22/H23/H24, and its file list against H21.
The tree is checked out at <repo_path>. Read it there, run no mutating command.
`rg` is not available; use `grep -rn`.
## How to report
One finding per rule violated, not one per occurrence. If a deprecated model id
appears fifteen times, that is ONE finding on the clearest occurrence, with the
count noted in `what`.
Same schema as the other lanes, plus one extra field:
```json
[{"path":"<repo-relative>","line":69,"severity":"no_critical",
"rule":"H1","ci":"fail",
"what":"pyproject.toml declares [tool.ruff]; recipes must not (CI hard fail)",
"evidence":"lines 69,77,81,85; AGENTS.md:50-52",
"verify_steps":"grep '^\\[tool\\.ruff' in this file",
"window":" 67: ...\n 68: ...\n 69: [tool.ruff]",
"anchorable":false}]
```
- `rule` — the H-number from `.github/review-rules.md`.
- `ci` — `"fail"` or `"advisory"`, copied from the rule. Getting this wrong is worse
than not reporting: never let a comment claim CI will fail when it will not.
- `anchorable` — `true` only if the line is inside a diff hunk of the PR. If you are
unsure, set `false`; the orchestrator re-checks and will promote it.
- `severity` is `no_critical` for essentially all of these; they are conventions, not
security holes.
Report a rule ONLY when you can point at the violating line. Do not report a rule as
satisfied, and do not report rules that do not apply to this recipe's root.
## Return
One line only: `house-rules: <n> findings (<n> ci-fail, <n> advisory, <n> unanchorable)`
followed by the absolute path to your findings.json.
````
reference/rationale.md
# Why these rules exist
Kept out of `SKILL.md` because that file loads on every invocation and this is
background. Read it when a rule looks arbitrary and you are tempted to skip it.
## The house-rules lane reads files outside the diff
Churn-based partitioning cannot see a rule violation on an unchanged line. On PR
#2373 three hard CI failures sat at lines 2, 38 and 69 of a `pyproject.toml` whose
diff touched only lines 30-36 and 112-125. A twelve-lane review missed every one.
## Rules are a script wherever they can be
When the 25 house rules were run by hand, two nearly became false positives —
`source = { editable = "." }` is the recipe's own package, and `license`/`tags` turn
out to be permitted manifest keys — and one **was posted wrong**: an absence
asserted from a truncated `grep | head -6`, which required a public correction.
## The window doubles as a hallucination detector
Findings quote their source verbatim, so it can be diffed against the file. On #2302
that cleared 132 of 134 findings and caught the two that were fabricated. A lane that
invents a finding invents the source line with it.
## Addressability is checked before drafting, not at the gate
GitHub only accepts a comment on a line inside a diff hunk. On #2373 a third of all
findings sat outside every hunk. Discovering that at the posting gate means having
already drafted, and become attached to, comments that cannot go up.
## Comments are posted one at a time, slowly
`POST /pulls/{n}/comments` is what GitHub creates when a person types a comment on a
line. The 10-20s gaps exist because a burst of nine comments in ten seconds is the
single most obvious tell available. A pending review blocks all of it: GitHub allows
one per user, and the endpoint implicitly opens one.
## Where the voice rules come from
`reference/voice.md` is calibrated on two things: a 26-example rating session, and —
more importantly — the real accept/reject outcome of 20 drafted comments on #2373.
Where they disagree, the live outcome wins. That is how the ban on suggesting a fix
was overturned: all four comments the user edited before posting had a remedy added.
reference/voice.md
# Voice calibration — the general review style
The house style for review comments produced by this skill. It is calibrated on two
things: a rated corpus of real review comments on `google/adk-samples`, and the
recorded accept/reject outcome of 20 drafted comments on PR #2373 — live decisions,
not a rating exercise.
Ratings below mean **strong** (on-style), **weak** (recognisably off), or
**rejected** (do not produce this shape).
This file is the single source of truth for how a comment must read. When in doubt,
match an example below rather than inventing a new shape. It is also the file to
replace if a team wants to shift the style — everything else in the skill is
independent of it.
---
## The two registers
Every comment is either Register A or Register B. There is no third option.
### Register A — full sentence, ends in a question mark
Proper capitalisation and punctuation. **Must end in `?`.** Often opens with a hedge.
May carry one short supporting fact before or after the question.
Hedges that tested well: `It seems`, `Perhaps`, `I think`, `I see`, `Is there a reason`,
`Can we`, `Can you please`.
```
Can you please add error handling around this call?
It seems the lock is released before the write finishes. Is that intentional?
Is there a reason we're not reusing `get_client()` here? It seems like it would do the same thing.
```
### Register B — lowercase fragment
2–6 words. No leading capital. No terminal period. Used for obvious defects and for
cross-referencing another comment.
```
missing await here
same issue as above
this'll break if the list is empty
```
Target mix: roughly **60% Register A, 40% Register B**.
---
## The first filter: anchor on a fact
> **A comment must point at something observable at the line. If it asks a question,
> the question is about what to DO with that fact — never about establishing the
> fact by reasoning.**
This is the strongest predictor in the whole file, and unlike everything else here
it comes from live accept/reject decisions rather than a rating session. On PR #2373
it separates all 16 accepted comments from all 4 rejected ones — 16 for 16.
**Rejected — each one asks the reader to derive something:**
| | Comment | What it demands |
|---|---|---|
| R1 | `Is 200 characters enough to stay clear of the keys? The fixed prefix looks like about 64.` | do the arithmetic |
| R2 | `` `DENN` and `CHAL` break the pattern the rest of the map follows `` | infer a pattern across a map |
| R3 | `` Would a failed call pass this? `error` is in the accepted list. `` | reason about test semantics |
| R4 | `Should this `chdir` be restored afterwards?` | reason about consequences |
**Accepted — each one points at something on screen:**
| | Comment | The fact |
|---|---|---|
| A1 | `hardcoded project name here` | the literal is right there |
| A2 | `` `city_clean` isn't used below `` | assigned, never read |
| A3 | `` this `f` has nothing to interpolate `` | an f-string with no braces |
| A4 | `no licence header on this one` | the top of the file |
| A5 | `looks like three files got concatenated` | mid-file copyright comments |
| A6 | `` Is falling back to `TX` right here? The label still says the requested state. `` | the fallback literal, plus the label two lines down |
| A7 | `The padding appends the same query over and over, but the banner below calls them unique. Which is right?` | two visible statements that disagree |
| A8 | `Does `$DEPLOY_CMD` still have the key values in it at this point?` | a question the author answers from what they built |
Note A6–A8: a question is fine, and so is a supporting clause. What matters is that
the **fact is visible** and the question is about its consequence or intent — not a
request for the reader to work the fact out.
**The test:** delete the question from your comment. Is what remains a statement of
something visible at the line? If nothing remains, or what remains is an inference,
the comment fails.
## The hard rule
> **A full-sentence comment must end in a question mark.
> A declarative multi-clause statement is never acceptable.**
This single rule accounts for almost every rejection in the calibration set. Compare:
| | Comment | Verdict |
|---|---|---|
| 18 | `The lock is released before the write finishes.` | weak — declarative, no question |
| 19 | `It seems the lock is released before the write finishes. Is that intentional?` | **strong** |
| 20 | `It seems the lock is released before the write finishes, which means two concurrent callers could interleave and corrupt the file. Perhaps move the release after the flush?` | **rejected** — too dense |
18 → 19 → 20 is the density ruler. **19 is the target.** 18 is under-committed
(states a fact but asks nothing). 20 narrates the full causal chain and prescribes a
fix — that is the single most common AI-review failure mode.
**One thought per sentence.** Do not explain the mechanism, the consequence, and the
remedy in one breath. That is what makes 20 fail — a single sentence doing three
jobs, not the presence of a suggestion.
### A remedy is welcome — as its own sentence
This is a correction to the rating session, made from live evidence. Of the 16
comments accepted on PR #2373, **four were edited before posting, and all four edits
added a suggested fix**:
| Drafted | Posted |
|---|---|
| `the licence header stops mid-sentence` | `…mid-sentence. I suggest cleaning up all the headers in a more consistent form. You may borrow from other existing recipes under core/ if you wish.` |
| `same default as the advisor module` | `…advisor module. You may add these default values to .env.example` |
| `Should this endpoint come from an env var rather than being pinned here?` | `…pinned here? It also has hardcoded project and other things. Please clean up.` |
| `real project id committed here` | `Real project ID is committed here.` |
So: **observation first, then a short plain sentence saying what to do.** Not
"Consider…", not "It would be better to…" — the shapes above: *"I suggest…"*, *"You
may…"*, *"Please…"*.
The distinction from 20 is structural, not topical:
- **20 (rejected):** `It seems the lock is released before the write finishes, which means two concurrent callers could interleave and corrupt the file. Perhaps move the release after the flush?` — one sentence carrying mechanism *and* consequence, then a fix.
- **Accepted shape:** `the licence header stops mid-sentence. I suggest cleaning up all the headers in a more consistent form.` — a fact, then a remedy. Two sentences, one job each.
Omit the remedy when the fix is a judgement call about how to restructure something.
Include it when it is obvious, or when it points at an existing example to copy.
The fourth edit is a smaller signal: a comment that reads as a full clause gets
capitalised and given a full stop. `no backend block here` stayed lowercase;
`Real project ID is committed here.` did not. Fragments stay fragments; clauses
become sentences.
### Never assert an absence you did not inventory
A claim that something is **missing** must say where you looked, or where it *does*
appear. This rule exists because of a comment this skill posted and had to publicly
correct:
> **Wrong:** *"`load_dotenv()` is only in `tests/` and `eval/`."* — asserted from a
> truncated `grep | head -6`. It was in fact called from four package modules, and
> the PR author would rightly have replied "it's right there in `agent.py`".
>
> **Right:** *"`load_dotenv()` is called from `agent.py`, `real_estate_advisor.py`,
> `universal_whitepaper_orchestrator.py` and `dynamic_search_harvester.py`, but not
> from `economic_research/__init__.py`."*
An absence is the one claim you cannot verify by looking at the anchored line, so it
must carry its own evidence.
---
## Say what you are pointing at
Brevity is not the same as being elliptical, and that difference separates
`missing await here` (**strong**) from `prefix ends up as just bash` (unusable). Both
are short. The first describes something on the anchored line and is therefore
complete. The second is a short comment about a *distant mechanism* — the worst
combination available: too terse to explain itself, too remote to check.
Three rules, and they bind harder than the register mix.
### Name the referent
Every comment must name the identifier, setting or directive it is about. A bare
demonstrative — "this", "the prefix", "the setting above" — is allowed only when the
thing is unambiguous **on the anchored line itself**.
| | Comment | Verdict |
|---|---|---|
| N1 | `different user ids can collide here` | unusable — collide into what? |
| N2 | `Could two different user ids end up with the same sanitised value here?` | **strong** |
| N3 | `old version still holds the value` | unusable — which version? |
| N4 | `Does the previous version get destroyed anywhere, or does the old value stay readable?` | **strong** |
### Carry the verification path
If the comment depends on anything not on the anchored line, **name it and say
where**. The rated corpus already does this, and it is what makes a comment checkable
rather than merely assertive:
- `I see timeout is set to 5 here but 30 in config.py. Which one is correct?`
- `Perhaps this should be user_id instead of userId? The rest of the file uses snake_case.`
Both tell the reader exactly where to look. Compare `this contradicts the setting
above` — the same finding with the path removed, made expensive to verify purely
through wording.
### Assertions must be certain; questions need not be
State a defect flatly **only** when it is visible on the anchored line. Anything
requiring inference goes in question form.
This is a cost rule, not a politeness one. `different user ids can collide here` is a
claim that is embarrassing if wrong. `Could two user ids collide here?` costs nothing
if the answer is no — the author replies "no, because X" and everyone moves on. Bare
Register B fragments are assertions, which is precisely why they are reserved for
things plainly on screen.
### The cold-read test
Before presenting any comment, look at **only** the anchored line ±10 and the comment
text. Can you say what it refers to, and how you would check it? If not, rewrite or
drop it. Every comment, not a sample.
---
## Plausibility — could the reviewer have known this?
Everything above governs how a comment *sounds*. This section governs whether a
human could plausibly have *found* it. A comment can satisfy every rule in this
file and still be obviously machine-generated, because the giveaway is not the
wording — it is how much the commenter would have had to know.
Before writing any comment, ask: **reading these files, would a person have
arrived here?** If the honest answer is no, the comment is unpostable at any level
of polish.
### The depth tell
The finding is unreachable from the code the reviewer read. It needed a
dependency's internals, a repo-wide grep, or a cross-file comparison nobody makes
while reading a diff.
```
IdentityMiddleware resolves the caller but doesn't compare it to the {user_id}
in ADK's session routes. Should it?
```
Correct, serious, and impossible. Knowing that ADK's `ApiServer` route table
carries `{user_id}` requires reading `api_server.py` inside the installed package.
Ask what you could actually wonder instead:
```
Does this cover ADK's own session routes too, or only ours?
```
Same line, same defect, same outcome once the author looks. No claim to knowledge
the reviewer never had.
### The proof tell — the worse one
The comment quotes a crafted input. **You only cite a working bypass if you built
and ran one.** A reviewer has a suspicion, not a proof of concept.
| | Comment | Verdict |
|---|---|---|
| P1 | `The \`sudo\` branch above skips past the flags but the \`env\` branch doesn't. Does \`env -i rm -rf /\` still classify?` | rejected — payload proves it was executed |
| P2 | `The \`sudo\` branch above skips past the flags but this one doesn't. Is that deliberate?` | **strong** |
| P3 | `Perhaps resolve the path before the \`.agents/skills/\` check? \`x/../../../evil.sh\` gets through as it stands.` | rejected — "gets through as it stands" asserts a verified traversal |
| P4 | `Does \`_is_skills_path\` resolve \`..\` before it checks the prefix?` | **strong** |
Banned outright: exploit strings, payloads, "I tried X and got Y", any phrasing
that reports the result of running the code.
### Cheap to verify
Separate from whether a comment is *true* is what it costs the author to find out.
That cost is the dominant variable, because of an asymmetry:
- cheap and wrong — the author glances, says no, loses five seconds. Harmless.
- cheap and right — the author sees it immediately and fixes it. Ideal.
- **expensive and wrong** — the author burns twenty minutes, finds nothing, and
discounts everything else you wrote. The worst outcome available.
- expensive and right — a real bug, bought at twenty minutes and some goodwill.
A reviewer who is 80% accurate but always cheap is a pleasure. A reviewer who is
80% accurate and always expensive is a liability. **Optimise the cost, not the
hit rate.**
> **Cheap-to-verify:** looking only at the ~10 lines around the anchor, plus at
> most one in-file lookup the comment explicitly names, the author can settle the
> comment
>
> - without inventing an input, value or scenario the code does not already name,
> - without relying on any library, framework or service behaviour not visible there,
> - in one pass, without taking notes.
>
> A minute is the sanity check, not the criterion. If it takes longer, one of the
> three conditions above was already broken — find which.
**Score the final comment text, not the underlying finding.** The same defect can
be cheap or expensive depending on how it is written:
| | Comment | Cost |
|---|---|---|
| C1 | `` I think `ty` is on 3.10 here while `requires-python` says 3.11? `` | **cheap** — names both sides, so the one lookup is free |
| C2 | `this contradicts the setting above` | expensive — same defect, but the reader has to hunt |
| C3 | `` The `sudo` branch above skips past the flags but this one doesn't. Is that deliberate? `` | **cheap** — both branches on screen, claim is the asymmetry |
| C4 | `prefix ends up as just `bash`` | expensive — the proof is in another file |
The trap: **everything can be on screen and the comment still expensive.** A
four-line function is fully visible, but if settling the comment means inventing
two inputs and running them through a regex in your head, it is not cheap. The
test is not "can I see the code" — it is "must I supply anything the code does
not already name".
Conditions written in the code are free to reason about (`if allowlist:` invites
you to consider an empty allowlist). Conditions you bring yourself are not.
### The question mark is epistemic, not decorative
The hard rule above says a full sentence ends in `?`. That rule exists because the
reviewer is genuinely asking. **A question mark applied to something you have
already verified is a lie about your own certainty**, and the confidence leaks
through the surrounding phrasing every time — in the specificity, in the
supporting clause, in the crafted example.
If you know the answer, you are not asking a question, and the comment needs to be
rewritten around what you could actually have suspected. This is the single most
useful test in this file: **write the comment you would write before doing the
work, not after.**
---
## Severity does not change tone
The most counterintuitive finding, and the easiest one to get wrong.
The user handles command injection with exactly the politeness used for a missing
try/catch. Severity changes **what you point at**, never **how you sound**.
| | Comment | Verdict |
|---|---|---|
| 13 | `This is user input going straight into a shell command. Can we use `subprocess.run()` with a list instead?` | **strong** |
| 14 | `this is a command injection` | weak |
| 15 | `I don't think we can ship this one — `filename` comes from the request and goes straight into `os.system()`.` | weak |
| 16 | `Command injection here. `filename` is user-controlled.` | rejected |
| 17 | `Is `filename` validated anywhere upstream? If not this is a command injection.` | rejected |
**13 is the model for critical findings**: plain observation of what the code does,
then the concrete fix offered as a `Can we …?` question.
Do not use blocking language (`I don't think we can ship this`), and do not lead with
a severity noun-phrase (`Command injection here.`). 17 shows that even a question
fails if it is a conditional accusation.
---
## Banned outright
Every one of these appeared in a rejected example or contradicts the rules above.
- Bold labels and severity prefixes — `**Critical:**`, `Security issue:`, `nit:` is borderline (8 rated only weak)
- Emoji of any kind. The user writes `:-)`, never 🔴/✅/⚠️
- Markdown headers, bullet lists, numbered lists inside a comment
- ` ```suggestion ` blocks (24 rejected)
- `Consider …`, `It would be better to …`, `I recommend …` — note these are banned as
*phrasings*, not because suggesting a fix is banned. `I suggest …`, `You may …` and
`Please …` are all attested in real posted comments.
- `… to improve readability and maintainability` and similar abstract-quality prose
- Referencing line numbers in prose (`before line 42`) — the comment is already anchored to a line
- Multi-clause causal explanation in a single sentence (see 20). A remedy in its own
sentence is fine — see "A remedy is welcome".
- Any comment whose fact must be derived by the reader rather than seen (R1–R4 above)
- Asserting something is missing without saying where you looked
- Polished cross-references (26 rejected: `Related to my comment above about the timeout.`)
Use Register B instead (22: `same issue as above`)
- Crafted payloads and exploit strings — see the proof tell above
- Any claim that rests on code the reviewer did not read — see the depth tell above
---
## Formatting habits
- Backticks around every identifier, path, and filename: `` `user_id` ``, `` `config.py` ``
- `:-)` occasionally, only on positive or social comments, never on a defect
- No sign-offs, no greetings, no "Thanks for the PR!"
---
## Full rated corpus
Round 1 — general shapes.
| # | Comment | Verdict |
|---|---|---|
| 1 | `Can you please add error handling around this call?` | **strong** |
| 2 | `**Critical:** This introduces a race condition. Consider using a mutex to guard access to the shared counter.` | rejected |
| 3 | `Perhaps this should be `user_id` instead of `userId`? The rest of the file uses snake_case.` | **strong** |
| 4 | `this'll break if the list is empty` | **strong** |
| 5 | `I think this will throw if `results` comes back empty. Should we guard against that?` | weak |
| 6 | `Consider refactoring this function to improve readability and maintainability. It currently handles multiple responsibilities.` | rejected |
| 7 | `Is there a reason we're not reusing `get_client()` here? It seems like it would do the same thing.` | **strong** |
| 8 | `nit: typo in "recieve"` | weak |
| 9 | `I see `timeout` is set to 5 here but 30 in `config.py`. Which one is correct?` | **strong** |
| 10 | `🔴 Security issue: this SQL query is vulnerable to injection.` | rejected |
| 11 | `This looks like it would leak the connection if the exception fires before line 42. Perhaps wrap it in a `try/finally`?` | rejected |
| 12 | `The reason we avoid `eval()` here is that the input comes straight from the request body. Can we use `json.loads()` instead?` | weak |
Round 2 — severity register, density, fragments, suggestions, cross-refs.
| # | Comment | Verdict |
|---|---|---|
| 13 | `This is user input going straight into a shell command. Can we use `subprocess.run()` with a list instead?` | **strong** |
| 14 | `this is a command injection` | weak |
| 15 | `I don't think we can ship this one — `filename` comes from the request and goes straight into `os.system()`.` | weak |
| 16 | `Command injection here. `filename` is user-controlled.` | rejected |
| 17 | `Is `filename` validated anywhere upstream? If not this is a command injection.` | rejected |
| 18 | `The lock is released before the write finishes.` | weak |
| 19 | `It seems the lock is released before the write finishes. Is that intentional?` | **strong** |
| 20 | `It seems the lock is released before the write finishes, which means two concurrent callers could interleave and corrupt the file. Perhaps move the release after the flush?` | rejected |
| 21 | `missing await here` | **strong** |
| 22 | `same issue as above` | **strong** |
| 23 | `is this still needed?` | **strong** |
| 24 | ` ```suggestion ` block | rejected |
| 25 | `Same as the one in `utils.py` — probably worth fixing both.` | weak |
| 26 | `Related to my comment above about the timeout.` | rejected |
---
## Authentic comments from the real corpus
Verbatim, previously posted by the user. Highest-fidelity reference available.
```
Can you please add a real team here?
```
```
Perhaps you want to change these 2025 years to 2026?
```
```
I see. The reason we introduced `team` was to make sure no sample/recipe will become
orphaned (since we had to deal with a number of them before). We wanted to make sure a
team owns the recipe in case the poc leaves. If you think you will be able to remain
the owner and continue to maintain it, we can accept you as the team.
```
```
Oh okay. Sounds good. I will approve your PR.
Congratulations on submitting the very first recipe to our newly restructured contrib/python by the way. :-)
```
Note the third example: longer explanatory prose **is** in range when the user is
justifying a policy in a back-and-forth reply. That is a reply register, not a
first-pass review register. A first-pass review comment stays short.
Note also `Perhaps you want to change these 2025 years to 2026?` — a stale copyright
year. Confirms that small cosmetic findings are genuinely in scope, in strict
moderation (see the quota in SKILL.md Step 4).
scripts/build_report.py
#!/usr/bin/env python3
"""Render reviewed candidates as a markdown report.
The point of the report is that the reader can settle a comment WITHOUT
opening a file, so the code window travels next to the comment everywhere.
build_report.py --candidates cands.json --repo owner/name --pr 123 \
--out-md <dir>/pr-123-review.md
`--out-csv` is optional and off by default: it carried the same rows as the
markdown tables, so writing both left two copies of one report on disk.
Input JSON: an array of candidate objects.
path repo-relative path (required)
line int (required)
comment the exact text that will be posted (required; `body` also accepted)
severity critical | no_critical
window source text at the anchor, one "NNN: code" per line
verify_steps what the author does to settle it
what the blunt technical finding behind the comment
evidence how the lane knew
Markdown layout is summary-table-first, detail-sections-below: the table is for
scanning and settling the easy ones, the sections are for when it isn't enough.
"""
import argparse
import csv
import json
import os
import re
import sys
TABLE_CODE_MAX = 68 # chars of source shown in a table cell
SEV_SHORT = {"critical": "crit", "no_critical": "no_crit"}
def common_prefix(paths):
"""Longest shared directory prefix, so the table isn't 60 chars of boilerplate."""
if not paths:
return ""
parts = [p.split("/")[:-1] for p in paths]
shared = []
# strict=False is the point: paths of different depth stop at the
# shortest shared prefix, which is exactly the common directory.
for chunk in zip(*parts, strict=False):
if len(set(chunk)) != 1:
break
shared.append(chunk[0])
return "/".join(shared) + "/" if shared else ""
def cell(text):
"""Make a string safe inside a markdown table cell."""
return (
str(text)
.replace("\\", "\\\\")
.replace("|", "\\|") # an unescaped pipe silently eats the row
.replace("\n", " ")
.strip()
)
def anchor_line(window, line):
"""The single source line at the anchor, pulled out of the window block."""
if not window:
return ""
for raw in str(window).split("\n"):
m = re.match(r"\s*(\d+)\s*[:|]\s?(.*)$", raw)
if m and int(m.group(1)) == line:
return m.group(2).strip()
# No numbered match — fall back to the longest non-empty line.
lines = [ln.strip() for ln in str(window).split("\n") if ln.strip()]
return max(lines, key=len) if lines else ""
def truncate(code):
code = code.strip()
return (
code
if len(code) <= TABLE_CODE_MAX
else code[: TABLE_CODE_MAX - 1] + "…"
)
def code_block_cell(text):
"""A multi-line code window inside one table cell.
Markdown fences can't live in a cell, so use <code> with <br>. Everything
must be HTML-escaped first or a stray < eats the rest of the row.
"""
if not text:
return ""
esc = (
str(text)
.rstrip()
.replace("&", "&")
.replace("<", "<")
.replace(">", ">")
.replace("|", "|") # pipe would split the cell
.replace("`", "`") # source containing ``` would open a fence
)
lines = [ln.replace(" ", " ") for ln in esc.split("\n")]
return "<code>" + "<br>".join(lines) + "</code>"
def csv_safe(value):
"""Defuse a spreadsheet formula before it reaches the triage CSV.
Every string in these rows is derived from the PR's own diff, and the CSV
exists to be opened in a spreadsheet. Excel and Sheets both execute a cell
beginning with = + - or @, so a contributor could land a line of source
that runs a formula on the reviewer's machine. Quoting does not stop it --
the leading tab does, and it survives a round-trip as the same text.
"""
s = "" if value is None else str(value)
return "\t" + s if s[:1] in ("=", "+", "-", "@") else s
def group_key(c):
return c.get("severity", "no_critical")
def render_table(rows, prefix, start_index):
out = [
"| # | file | line | sev | code at that line | comment |",
"|---|---|---|---|---|---|",
]
for i, c in enumerate(rows, start_index):
short = (
c["path"][len(prefix) :]
if c["path"].startswith(prefix)
else c["path"]
)
code = truncate(anchor_line(c.get("window", ""), c["line"]))
out.append(
f"| {i} | `{cell(short)}` | {c['line']} "
f"| {SEV_SHORT.get(c.get('severity'), '?')} "
# <code>, not backticks: source lines contain backticks of their own
# (RST double-backticks especially) and would close the span early.
f"| {code_block_cell(code)} | {cell(c['comment'])} |"
)
return out
def render_detail_table(rows, prefix, start_index):
"""Second table: the full window and how to check it. Still a table."""
out = [
"| # | location | code window | to check | underlying finding |",
"|---|---|---|---|---|",
]
for i, c in enumerate(rows, start_index):
short = (
c["path"][len(prefix) :]
if c["path"].startswith(prefix)
else c["path"]
)
finding = cell(c.get("what", ""))
if c.get("unchecked"):
finding += f" **Not checked:** {cell(c['unchecked'])}"
out.append(
f"| {i} | `{cell(short)}:{c['line']}` "
f"| {code_block_cell(c.get('window', ''))} "
f"| {cell(c.get('verify_steps', ''))} "
f"| {finding} |"
)
return out
def counts(rows):
from collections import Counter
return Counter(group_key(c) for c in rows)
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--candidates", required=True)
ap.add_argument("--repo", required=True)
ap.add_argument("--pr", required=True)
ap.add_argument("--out-md", required=True)
ap.add_argument(
"--out-csv",
default=None,
help="optional; same rows as the markdown, for spreadsheet triage",
)
ap.add_argument(
"--dropped",
type=int,
default=0,
help="how many findings the gate discarded",
)
ap.add_argument("--title", default="")
args = ap.parse_args()
with open(args.candidates, encoding="utf-8") as fh:
cands = json.load(fh)
for c in cands:
if "comment" not in c and "body" in c:
c["comment"] = c["body"]
# Candidates come from model-authored JSON, and a lane that omits
# `line` used to take the whole report down with a KeyError six
# frames from the cause. Normalise once, here, so every `c["line"]`
# below is safe and the un-anchorable row still names the file.
try:
c["line"] = int(c.get("line") or 0)
except (TypeError, ValueError):
c["line"] = 0
missing = [c for c in cands if not c.get("comment") or not c.get("path")]
if missing:
sys.exit(f"{len(missing)} candidate(s) missing path or comment")
cands.sort(key=lambda c: (group_key(c) != "critical", c["path"], c["line"]))
# Un-anchorable findings are real but sit outside every diff hunk, so they can
# never be inline comments; they are offered as one top-level comment instead.
unanchorable = [c for c in cands if c.get("anchorable") is False]
inline = [c for c in cands if c.get("anchorable") is not False]
prefix = common_prefix([c["path"] for c in cands])
# ---------------- markdown ----------------
m = []
m.append(f"# PR #{args.pr} review")
m.append("")
m.append(f"**Repo** {args.repo} ")
if args.title:
m.append(f"**PR** {args.title} ")
n_crit = sum(1 for c in inline if c.get("severity") == "critical")
m.append(
f"**Comments** {len(inline)} ({n_crit} critical)"
+ (f", plus {len(unanchorable)} un-anchorable" if unanchorable else "")
+ " "
)
if args.dropped:
m.append(
f"**Gated out** {args.dropped} findings, not cheaply verifiable "
)
m.append("**Status** nothing posted")
m.append("")
if prefix:
m.append(f"Paths below are relative to `{prefix}`")
m.append("")
m.append(
"Every row carries the source line it is anchored to. If you cannot "
"settle a row from the table alone, that is a defect in the comment, "
"not in you — say so and it gets cut."
)
m.append("")
if unanchorable:
nfail = sum(1 for c in unanchorable if c.get("ci") == "fail")
m.append(
f"The {len(unanchorable)} un-anchorable "
f"({nfail} CI-failing) can only go up as one top-level comment — "
"see the last table."
)
m.append("")
m.append("---")
m.append("")
if inline:
m.append("## Comments")
m.append("")
m += render_table(inline, prefix, 1)
m.append("")
if unanchorable:
m.append("## Un-anchorable — one top-level comment")
m.append("")
m.append(
"Real findings whose lines sit outside every diff hunk, so GitHub "
"cannot take them as inline comments. CI-failing first. Do **not** "
"pin these to a nearby line to force them through."
)
m.append("")
ua = sorted(
unanchorable,
key=lambda c: (c.get("ci") != "fail", c["path"], c["line"]),
)
m.append("| # | file | line | CI | rule | finding |")
m.append("|---|---|---|---|---|---|")
for i, c in enumerate(ua, 1):
short = (
c["path"][len(prefix) :]
if c["path"].startswith(prefix)
else c["path"]
)
ci = "**FAIL**" if c.get("ci") == "fail" else "advisory"
m.append(
f"| {i} | `{cell(short)}` | {c['line']} | {ci} "
f"| {cell(c.get('rule', ''))} | {cell(c.get('what') or c.get('comment', ''))} |"
)
m.append("")
m.append("---")
m.append("")
m.append("## Detail")
m.append("")
m.append("Same rows, with the full window and the check procedure.")
m.append("")
if inline:
m += render_detail_table(inline, prefix, 1)
m.append("")
os.makedirs(
os.path.dirname(os.path.expanduser(args.out_md)) or ".", exist_ok=True
)
with open(os.path.expanduser(args.out_md), "w", encoding="utf-8") as fh:
fh.write("\n".join(m) + "\n")
# ---------------- csv (opt-in) ----------------
if args.out_csv:
os.makedirs(
os.path.dirname(os.path.expanduser(args.out_csv)) or ".",
exist_ok=True,
)
with open(
os.path.expanduser(args.out_csv),
"w",
newline="",
encoding="utf-8",
) as fh:
wr = csv.writer(fh, quoting=csv.QUOTE_ALL)
wr.writerow(
[
"path",
"line",
"severity",
"anchorable",
"ci",
"rule",
"comment",
"code_at_line",
"verify_steps",
"code_window",
]
)
for c in inline + unanchorable:
wr.writerow(
[
csv_safe(c["path"]),
c["line"],
csv_safe(c.get("severity", "")),
"no" if c.get("anchorable") is False else "yes",
csv_safe(c.get("ci", "")),
csv_safe(c.get("rule", "")),
csv_safe(c["comment"]),
csv_safe(anchor_line(c.get("window", ""), c["line"])),
csv_safe(c.get("verify_steps", "")),
csv_safe(c.get("window", "")),
]
)
print(f"markdown -> {args.out_md}")
if args.out_csv:
print(f"csv -> {args.out_csv}")
print(
f"{len(inline)} comments"
+ (f", {len(unanchorable)} un-anchorable" if unanchorable else "")
)
if __name__ == "__main__":
main()
scripts/check_house_rules.py
#!/usr/bin/env python3
"""Check a google/adk-samples recipe against the house rules in .github/review-rules.md.
check_house_rules.py --repo-root /path/to/checkout --recipe contrib/python/foo
check_house_rules.py --repo-root . --recipe contrib/python/foo --json
Emits findings in the same schema the analysis lanes use, with `rule`, `ci` and
`anchorable` pre-populated. Deterministic: no model, no judgement, no tokens.
Every reportable rule in `.github/review-rules.md` is either implemented here or
declared in MODEL_JUDGED below, and `tests/test_house_rules_drift.py` fails if a
rule is in neither. That test is why this list is short: seven reportable rules
(H39-H47) were in the doc, unimplemented, and NOT on any declared exception list,
so nothing was checking them and nothing said so.
Three rules here have traps that a naive grep gets wrong, and each is called out at
its implementation:
H9 `source = { editable = "." }` is the recipe's OWN package, not a violation
H15 needs package-root resolution among many __init__.py, and is a NEGATIVE
H19 needs the real JSON schema; `license`/`tags` ARE permitted keys
"""
import argparse
import ast
import json
import os
import re
import subprocess
import sys
try:
import tomllib
except ModuleNotFoundError: # py<3.11
import tomli as tomllib
CI_FAIL = "fail"
CI_ADV = "advisory"
# Reportable rules this script deliberately leaves to a model, and why. Anything
# in `.github/review-rules.md` under "Report these" that is neither implemented
# here nor listed here is an unchecked rule, and the drift test says so by name.
# A wrong "this fails CI" costs more than a missed nit, so the bar for adding a
# rule here rather than implementing it is "a regex gets it wrong in practice".
MODEL_JUDGED = {
"H11": "hardcoded model literal — four AST exemptions (collection literal, "
"subscript index, comparison operand, docstring)",
"H16": "noqa E402 on a trailing import — needs 'after the first "
"non-import statement', which the import graph alone does not give",
"H25": "runnability assert placement — needs the `with patch(...)` block "
"structure, not just the presence of an assert",
}
# Rules that could not be evaluated this run. Always reported -- a silent skip
# makes every "no violations" result a lie.
SKIPPED = []
def _scalar_value(raw):
"""The value part of a YAML scalar line, with any trailing comment removed.
The manifest TEMPLATE ships every enum as `type: "standalone" # Options:
[standalone | module]`, so a fallback parser that keeps the whole remainder
reads the value as `"standalone" # Options: [standalone | module]` and H19
then reports three confident CI-FAILs saying the enum is invalid. That is
the most expensive comment this skill can produce, and it fired on
contrib/python/clause-agent.
"""
raw = raw.strip()
q = re.match(r"""^(['"])(.*?)\1""", raw)
if q:
return q.group(2)
# Unquoted: ` #` starts a comment, a bare `#` mid-token does not.
return re.split(r"(?:^|\s)#", raw, maxsplit=1)[0].strip()
def _parse_manifest(text):
"""(data, degraded). Falls back to a top-level-key scan without pyyaml."""
if not text:
return {}, False
try:
import yaml
return (yaml.safe_load(text) or {}), False
except ModuleNotFoundError:
pass
except Exception:
return None, False
data = {}
for line in text.split("\n"):
if line.lstrip().startswith("#"):
continue
m = re.match(r"^([A-Za-z_][A-Za-z0-9_]*):\s*(.*)$", line)
if not m:
continue
k, v = m.group(1), _scalar_value(m.group(2))
data[k] = v if v else {}
return data, True
# Rules that describe the recipe as a whole rather than a specific file. They are
# only the PR's responsibility when the PR is adding the recipe.
WHOLE_RECIPE_RULES = {"H21", "H22", "H23"}
# H24 is the exception: modifying ANY file under a frozen path is itself the
# violation, so a small edit to a legacy recipe does trigger it.
# H48 is the other: a junk ownership.team is the one thing nobody else catches
# (the schema only asks for a non-empty string), and letting it through once is
# what produced a fleet of recipes owned by "Google". It is always reported;
# when manifest.yaml is outside the diff the finding lands in the un-anchorable
# bucket rather than being dropped.
ALWAYS_RULES = {"H24", "H48"}
CHANGED = None # set from --changed-files; None means "audit everything"
NEW_RECIPE = True # set False when the PR only edits an existing recipe
FILTERED = [] # rules suppressed as pre-existing, for reporting
# Rules whose `path` names a DIRECTORY rather than a file: the recipe root, a
# pruned build directory, a misplaced vertical. `CHANGED` holds files, so a
# plain `path in CHANGED` is false for every one of them and the rule can
# never fire in CI. H41, H44 and H47 were implemented, tested, declared
# covered by the drift test -- and dead on arrival in their only production
# caller, which is a worse state than being unimplemented, because the drift
# test reports them as done.
DIRECTORY_RULES = {"H22", "H23", "H41", "H44", "H47"}
def _is_ours(rule, path):
"""Is this violation something the PR introduced, or pre-existing noise?"""
if CHANGED is None:
return True
if rule in ALWAYS_RULES:
return True
# H22 and H23 sit in BOTH sets, and the two questions are independent:
# "is this PR adding a recipe at all" and "is THIS the recipe it adds".
# Returning on the first set made the second unaskable, so a PR that added
# recipe A and touched one file in recipe B reported B's folder name and
# skill depth as if the PR had authored them.
if rule in WHOLE_RECIPE_RULES and not NEW_RECIPE:
return False
if rule in DIRECTORY_RULES:
prefix = path.rstrip("/") + "/"
return any(changed.startswith(prefix) for changed in CHANGED)
if rule in WHOLE_RECIPE_RULES:
return True
return path in CHANGED
def fetch_changed_files(repo, pr):
"""Every changed path, paged.
`gh pr view --json files` silently caps at 100. On PR #2302 (787 files) that
truncation left pyproject.toml out of the list, so every finding in it was
filtered away as "pre-existing" on a brand-new recipe.
"""
out, page = set(), 1
while True:
proc = subprocess.run(
[
"gh",
"api",
f"/repos/{repo}/pulls/{pr}/files?per_page=100&page={page}",
],
capture_output=True,
text=True,
check=False,
)
if proc.returncode != 0:
raise SystemExit(f"gh api failed: {proc.stderr.strip()[:200]}")
batch = json.loads(proc.stdout or "[]")
if not batch:
break
out.update(f["filename"] for f in batch)
if len(batch) < 100:
break
page += 1
return out
def find(out, rule, ci, path, line, what, evidence, verify):
if not _is_ours(rule, path):
FILTERED.append((rule, path))
return
out.append(
{
"rule": rule,
"ci": ci,
"path": path,
"line": line,
"severity": "no_critical",
"confidence": "high",
"cheap": "cheap",
"what": what,
"evidence": evidence,
"verify_steps": verify,
"window": "",
"unchecked": None,
}
)
# GitHub accepts a 100 MB file, every reader here splits what it reads into
# lines (roughly doubling it), and the recipe subtree is walked six times. No
# rule needs more than this to decide anything.
MAX_READ_BYTES = 2_000_000
def read(p):
"""A file's text, or None if it cannot or should not be read.
Guards OSError rather than just missing files: a dangling symlink, a
permission error and a directory in place of a file all reach here, and
any one of them escaping discards every finding for the whole recipe.
"""
try:
if os.path.getsize(p) > MAX_READ_BYTES:
return None
with open(p, encoding="utf-8", errors="replace") as handle:
return handle.read()
except OSError:
return None
def lineno_of(text, pattern):
for i, line in enumerate(text.split("\n"), 1):
if re.search(pattern, line):
return i
return 1
def _table(data, key):
"""`data[key]` when it is a mapping, else an empty one."""
value = data.get(key) if isinstance(data, dict) else None
return value if isinstance(value, dict) else {}
def load_toml(p):
try:
with open(p, "rb") as fh:
return tomllib.load(fh)
except Exception:
return None
# --------------------------------------------------------------------------- #
# --------------------------------------------------------------- H26 / H27
APACHE_HEADER = [
"Copyright {year} Google LLC",
"",
'Licensed under the Apache License, Version 2.0 (the "License");',
"you may not use this file except in compliance with the License.",
"You may obtain a copy of the License at",
"",
" https://www.apache.org/licenses/LICENSE-2.0",
"",
"Unless required by applicable law or agreed to in writing, software",
'distributed under the License is distributed on an "AS IS" BASIS,',
"WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.",
"See the License for the specific language governing permissions and",
"limitations under the License.",
]
HEADER_EXT = {
".py": "#",
".sh": "#",
".tf": "#",
".yaml": "#",
".yml": "#",
".ts": "//",
".tsx": "//",
".js": "//",
".jsx": "//",
}
# Source files are EXPECTED to carry the header, so a missing one is a violation
# however many others also lack it. Config formats only have to be self-consistent
# -- if no .tf in the recipe has a header, that is the convention there.
HEADER_REQUIRED = {".py", ".ts", ".tsx", ".js", ".jsx", ".sh"}
# Blank separator lines carry no text, so only the prose lines are matchable.
# Deriving this rather than hardcoding it: a literal 10 was unreachable against
# the 9 real lines, so every file scored "partial" and the check silently died.
_HEADER_LINES = [w for w in APACHE_HEADER[1:] if w]
_FULL_AT = len(_HEADER_LINES) - 1 # tolerate one reflowed line
_ENV_READERS = {"getenv", "environ"}
def _env_read_defaults(src, path):
"""[(line, call_text, why)] for env reads that carry a hardcoded default.
AST, not regex: a regex flags the same call inside a docstring or a comment,
and cannot tell `os.environ["X"]` (fine -- cannot carry a default) from
`os.environ.get("X", "d")` (not fine).
"""
try:
tree = ast.parse(src)
except (SyntaxError, ValueError, RecursionError, MemoryError):
# RecursionError, not SyntaxError, is what a generated constant table
# of 200k terms raises -- a plausible accident, not just an attack.
# Anything escaping here discards the whole recipe's findings.
return []
def env_call(node):
"""('getenv'|'get'|'setdefault') if this Call reads the environment."""
fn = node.func
if not isinstance(fn, ast.Attribute):
return None
if (
fn.attr == "getenv"
and isinstance(fn.value, ast.Name)
and fn.value.id == "os"
):
return "getenv"
if fn.attr in ("get", "setdefault"):
v = fn.value
if isinstance(v, ast.Attribute) and v.attr == "environ":
return fn.attr
if isinstance(v, ast.Name) and v.id == "environ":
return fn.attr
return None
out = []
for node in ast.walk(tree):
if isinstance(node, ast.Call):
kind = env_call(node)
if not kind:
continue
if len(node.args) >= 2 or any(
k.arg == "default" for k in node.keywords
):
out.append(
(
node.lineno,
kind,
"second argument is a hardcoded default",
)
)
# `os.getenv("X") or "fallback"` -- semantically a default.
elif isinstance(node, ast.BoolOp) and isinstance(node.op, ast.Or):
first = node.values[0]
if isinstance(first, ast.Call) and env_call(first):
rest = node.values[1:]
if any(
isinstance(v, ast.Constant) and v.value not in (None, "")
for v in rest
):
out.append(
(node.lineno, "or", "`or` fallback after an env read")
)
return out
def check_env_defaults(out, root, rel):
"""H26 -- an env read must not carry a hardcoded default."""
recipe_abs = os.path.join(root, rel)
hits = []
for dirpath, dirnames, filenames in os.walk(recipe_abs):
dirnames[:] = [
d
for d in dirnames
if d not in ("__pycache__", ".venv", "node_modules", ".git")
]
for fn in filenames:
if not fn.endswith(".py"):
continue
fp = os.path.join(dirpath, fn)
relp = os.path.relpath(fp, root)
if CHANGED is not None and relp not in CHANGED:
continue
src = read(fp)
if not src:
continue
for line, kind, why in _env_read_defaults(src, relp):
hits.append((relp, line, kind, why))
if not hits:
return
# One finding, not one per hit -- the grouping rule.
relp, line, kind, why = hits[0]
others = sorted({h[0] for h in hits})
find(
out,
"H26",
CI_ADV,
relp,
line,
f"env read carries a hardcoded default ({why}); "
f"{len(hits)} occurrence(s) across {len(others)} file(s). Defaults belong "
"in .env.example, not in the code",
"; ".join(f"{h[0]}:{h[1]}" for h in hits[:6]),
f"read line {line}; the call has a second argument",
)
def _header_state(text, marker):
"""('full'|'partial'|'none', matched_lines)."""
lines = [ln.rstrip() for ln in text.split("\n")[:25]]
body = []
for ln in lines:
s = ln.strip()
if s.startswith(marker):
body.append(s[len(marker) :].strip())
elif not s and body:
body.append("")
elif body:
break
if not body:
return "none", 0
joined = " ".join(body)
matched = sum(1 for want in _HEADER_LINES if want.strip()[:40] in joined)
has_copyright = "Copyright" in joined and "Google LLC" in joined
if matched >= _FULL_AT:
return "full", matched
if has_copyright or matched:
return "partial", matched
return "none", 0
def _anchor_in_diff(paths):
"""The first of these paths the PR actually changed, else the first.
H27 counts the whole recipe -- it is a claim about the recipe's
convention -- but `_is_ours` filters it on `path in CHANGED`. Anchoring
on the first offender in WALK order therefore threw the finding away
unless that exact file happened to be in the diff, which on a sixteen-file
offender list is about a one-in-eight chance.
"""
if CHANGED is None:
return paths[0]
return next((p for p in paths if p in CHANGED), paths[0])
def check_license_headers(out, root, rel):
"""H27 -- the licence header must be consistent WITHIN each file type.
Compared per extension, not pooled. Every .tf file in this recipe lacks a
header; that is the convention for terraform here, not 12 violations. Pooling
them against 662 headed .py files produced exactly that false positive.
"""
recipe_abs = os.path.join(root, rel)
by_ext = {}
for dirpath, dirnames, filenames in os.walk(recipe_abs):
dirnames[:] = [
d
for d in dirnames
if d
not in (
"__pycache__",
".venv",
"node_modules",
".git",
"dist",
"build",
".next",
)
]
for fn in filenames:
ext = os.path.splitext(fn)[1]
marker = HEADER_EXT.get(ext)
if not marker:
continue
fp = os.path.join(dirpath, fn)
relp = os.path.relpath(fp, root)
# NOT filtered by CHANGED here. H27 is a statement about the
# recipe's convention -- "N differ from the M carrying the full
# block" -- so the tally has to be over the whole recipe. Counting
# only changed files made a PR that edits two of the sixteen
# unheaded files in a 465-file recipe report "no .py file in this
# recipe carries the standard Apache header", with 449 of them
# carrying it. The filter chose the branch as well as the numbers.
# _is_ours still decides whether the finding is reported.
if fn.endswith((".gen.ts", ".gen.tsx")) or "generated" in relp:
continue
text = read(fp)
if text is None or not text.strip():
continue
state, _ = _header_state(text, marker)
by_ext.setdefault(ext, {"full": [], "partial": [], "none": []})
by_ext[ext][state].append(relp)
for ext, g in sorted(by_ext.items()):
total = sum(len(v) for v in g.values())
full = len(g["full"])
if ext in HEADER_REQUIRED:
if total < 2:
continue # a lone file establishes nothing
# Config formats: consistency only. No convention, no finding.
elif total < 5 or full == 0 or full / total < 0.6:
continue
dominant_full = full / total >= 0.6
if g["partial"] or g["none"]:
if dominant_full:
# A convention exists and a minority breaks it.
if g["partial"]:
find(
out,
"H27",
CI_ADV,
_anchor_in_diff(g["partial"]),
1,
f"licence header is truncated here; {len(g['partial'])} "
f"{ext} file(s) differ from the {full} carrying the full "
"Apache block",
"; ".join(g["partial"][:6]),
f"compare the first 15 lines of {g['partial'][0]} "
"with a sibling",
)
if g["none"]:
find(
out,
"H27",
CI_ADV,
_anchor_in_diff(g["none"]),
1,
f"no licence header on this file; {len(g['none'])} {ext} "
f"file(s) have none while {full} carry the full block",
"; ".join(g["none"][:6]),
f"read the first 15 lines of {g['none'][0]}",
)
elif full:
# Two competing shapes. Say that, rather than implying the
# majority is a broken version of the minority.
find(
out,
"H27",
CI_ADV,
_anchor_in_diff(g["partial"] or g["none"]),
1,
f"the {ext} files carry two different headers: "
f"{len(g['partial'])} a shorter notice, {len(g['none'])} none, "
f"and {full} the full Apache block",
"; ".join((g["partial"] + g["none"])[:6]),
f"compare the first 15 lines of {(g['partial'] or g['none'])[0]} "
f"against {g['full'][0]}",
)
else:
find(
out,
"H27",
CI_ADV,
_anchor_in_diff(g["partial"] or g["none"]),
1,
f"no {ext} file in this recipe carries the standard Apache "
f"header ({len(g['partial'])} have a shorter notice, "
f"{len(g['none'])} have none)",
"; ".join((g["partial"] + g["none"])[:6]),
f"read the first 15 lines of {(g['partial'] or g['none'])[0]}",
)
# Roots whose recipe name is namespaced by the segment above it. Mirrors
# NAMESPACED_ROOTS in .github/scripts/check_recipe_pyproject.py, which is what
# actually fails the build.
_NAMESPACED_ROOTS = {"plugins"}
def _expected_project_name(rel, recipe_name):
"""The [project].name CI will demand for this recipe."""
parts = rel.strip("/").split("/")
if len(parts) == 3 and parts[0] in _NAMESPACED_ROOTS:
return f"{parts[1]}-{parts[2]}"
return recipe_name
def check_pyproject(out, root, rel, recipe_name):
p = os.path.join(root, rel, "pyproject.toml")
text = read(p)
if text is None:
return
r = os.path.join(rel, "pyproject.toml")
data = load_toml(p) or {}
# `.get(k, {})` is not enough: a key present with the WRONG type returns
# that value, and every reader below then calls .get() on a str or a list.
# A mistyped table is exactly what these rules exist to catch, so crashing
# on one loses the recipe's whole review to the defect it was looking for.
proj = _table(data, "project")
# H1 -- ruff config belongs to the repo root only
hits = [
i
for i, ln in enumerate(text.split("\n"), 1)
if re.match(r"^\s*\[tool\.ruff(\.|\])", ln)
]
if hits:
find(
out,
"H1",
CI_FAIL,
r,
hits[0],
f"declares {len(hits)} [tool.ruff*] table(s); recipes must not "
f"(lines {', '.join(map(str, hits))})",
"AGENTS.md:50-52; python-validate-recipe.yml:261-266",
"grep '^\\[tool\\.ruff' in this file",
)
# H3 -- [project].name. Under plugins/ the expected value is
# <vertical>-<solution>, not the bare basename: check_recipe_pyproject's
# NAMESPACED_ROOTS namespaces that root. Comparing against the basename
# fired on both shipped vertical plugins and told each author to set the
# one value CI would reject.
name = proj.get("name")
expected = _expected_project_name(rel, recipe_name)
if name and isinstance(name, str) and name != expected:
find(
out,
"H3",
CI_FAIL,
r,
lineno_of(text, r"^\s*name\s*="),
f'[project].name is "{name}"; this recipe\'s name is "{expected}"',
"check_recipe_pyproject.py:182-215",
f"compare name= against {expected}",
)
# H4 -- must accept 3.11 exactly. Specifier logic, not a string match.
rp = proj.get("requires-python")
ln = lineno_of(text, r"^\s*requires-python\s*=")
if not rp:
find(
out,
"H4",
CI_FAIL,
r,
1,
"no requires-python declared",
"check_recipe_pyproject.py:117-197",
"read [project]",
)
elif not isinstance(rp, str):
find(
out,
"H4",
CI_FAIL,
r,
ln,
f"requires-python is {type(rp).__name__}, not a string; "
"`requires-python = 3.11` without quotes is a float",
"check_recipe_pyproject.py:117-197",
"read the requires-python line",
)
else:
bad = None
for part in [s.strip() for s in rp.split(",")]:
# Bounded: int() on a string of more than 4300 digits raises
# ValueError, and a contributor could disable the deterministic
# lane for their own recipe with one long line.
m = re.match(r"(>=|>|~=|==)\s*3\.(\d{1,6})", part)
if not m:
continue
op, minor = m.group(1), int(m.group(2))
if op in (">=", "~=", "==") and minor < 11:
bad = f"{rp} permits Python below 3.11"
if op in (">=", "~=", "==") and minor > 11:
bad = f"{rp} excludes Python 3.11 (CI pins 3.11 for uv lock --check)"
if bad:
find(
out,
"H4",
CI_FAIL,
r,
ln,
bad,
"check_recipe_pyproject.py:117-197",
"read the requires-python specifier",
)
# H5 -- [[tool.uv.index]] array-of-tables, default=true, public PyPI
idx = _table(_table(data, "tool"), "uv").get("index")
ok_urls = {"https://pypi.org/simple", "https://pypi.org/simple/"}
if idx is None:
find(
out,
"H5",
CI_FAIL,
r,
1,
"no [[tool.uv.index]] declared",
"check_recipe_pyproject.py:262-340",
"read [tool.uv]",
)
elif not isinstance(idx, list):
# single-bracket [tool.uv.index] instead of array-of-tables
find(
out,
"H5",
CI_FAIL,
r,
lineno_of(text, r"tool\.uv\.index"),
"[tool.uv.index] must be an array-of-tables [[tool.uv.index]]",
"check_recipe_pyproject.py:298-308",
"check the bracket count",
)
else:
defaults = [
e for e in idx if isinstance(e, dict) and e.get("default") is True
]
if not defaults:
find(
out,
"H5",
CI_FAIL,
r,
lineno_of(text, r"tool\.uv\.index"),
"no [[tool.uv.index]] entry has default = true",
"check_recipe_pyproject.py:262-340",
"read the index entries",
)
elif defaults[0].get("url") not in ok_urls:
find(
out,
"H5",
CI_FAIL,
r,
lineno_of(text, r"tool\.uv\.index"),
f'default index is "{defaults[0].get("url")}", must be public PyPI',
"check_recipe_pyproject.py:262-340",
"read the default index url",
)
# H6 -- python-dotenv in [project].dependencies (dev group does NOT count)
deps = proj.get("dependencies") or []
if isinstance(deps, str) or not isinstance(deps, (list, tuple)):
deps = []
names = {
re.match(r"^\s*([A-Za-z0-9_.\-]+)", d).group(1).lower()
for d in deps
# A dependency written as a table rather than a PEP 508 string is a
# realistic mistake, and re.match on a dict raises TypeError.
if isinstance(d, str) and re.match(r"^\s*([A-Za-z0-9_.\-]+)", d)
}
if "python-dotenv" not in names:
find(
out,
"H6",
CI_ADV,
r,
lineno_of(text, r"^\s*dependencies\s*="),
"python-dotenv is not in [project].dependencies "
"(a dev dependency-group does not count)",
"extract_env_vars.py:1797-1839",
"read [project].dependencies",
)
# H7 -- build-system with both keys
bs = _table(data, "build-system")
if not bs or not bs.get("requires") or not bs.get("build-backend"):
find(
out,
"H7",
CI_ADV,
r,
lineno_of(text, r"\[build-system\]"),
"[build-system] missing or lacks requires / build-backend",
"align_pyproject.py:667",
"read [build-system]",
)
# H8 -- testpaths, if present, must collect the runnability test
tp = _table(_table(_table(data, "tool"), "pytest"), "ini_options").get(
"testpaths"
)
if tp:
if isinstance(tp, str):
entries = [tp]
elif isinstance(tp, (list, tuple)):
entries = [e for e in tp if isinstance(e, str)]
else:
entries = []
ok = {"", ".", "tests", "tests/test_runnability.py"}
if not any(e.strip().rstrip("/") in ok for e in entries):
find(
out,
"H8",
CI_ADV,
r,
lineno_of(text, r"testpaths"),
f"testpaths {entries} never collects tests/test_runnability.py",
"align_pyproject.py:1063-1121",
"read testpaths",
)
def check_uv_lock(out, root, rel, recipe_name, pyproject_name):
"""H9. TRAP: `source = { editable = "." }` is the recipe's OWN package."""
p = os.path.join(root, rel, "uv.lock")
text = read(p)
if text is None:
return
r = os.path.join(rel, "uv.lock")
own = {pyproject_name, recipe_name, recipe_name.replace("-", "_")}
pkg = None
for i, line in enumerate(text.split("\n"), 1):
m = re.match(r'^\s*name\s*=\s*"([^"]+)"', line)
if m:
pkg = m.group(1)
continue
m = re.match(r"^\s*source\s*=\s*\{\s*(\w+)\s*=", line)
if m and m.group(1) in ("git", "editable", "directory"):
if m.group(1) in ("editable", "directory") and pkg in own:
continue # the recipe self-reference -- correct, not a defect
find(
out,
"H9",
CI_FAIL,
r,
i,
f'package "{pkg}" uses a {m.group(1)} source; '
"recipes must depend only on published PyPI releases",
"python-dependency-policy.yml:224-303",
f"read line {i}",
)
def check_dotenv_bootstrap(out, root, rel):
"""H15. TRAP: needs package-root resolution, and asserts a NEGATIVE.
The package root is the directory holding __init__.py that is a *direct child*
of the recipe root -- not any of the nested ones, and not tests/ or eval/.
"""
recipe_abs = os.path.join(root, rel)
pkg_dirs = []
try:
for entry in sorted(os.listdir(recipe_abs)):
d = os.path.join(recipe_abs, entry)
if (
os.path.isdir(d)
and entry not in ("tests", "eval", "scripts", "docs")
and not entry.startswith(".")
and os.path.exists(os.path.join(d, "__init__.py"))
):
pkg_dirs.append(entry)
except OSError:
return
if not pkg_dirs:
return # no package -- H15 does not apply
pkg = pkg_dirs[0]
init_rel = os.path.join(rel, pkg, "__init__.py")
init_text = read(os.path.join(root, init_rel)) or ""
if "load_dotenv" in init_text:
return # compliant
# Where IS it called? Reporting the absence alone is what got this wrong before.
callers = []
for dirpath, dirnames, filenames in os.walk(os.path.join(recipe_abs, pkg)):
dirnames[:] = [d for d in dirnames if d not in ("__pycache__", ".venv")]
for fn in filenames:
if not fn.endswith(".py"):
continue
fp = os.path.join(dirpath, fn)
t = read(fp) or ""
if re.search(r"^\s*load_dotenv\s*\(", t, re.M):
callers.append(os.path.relpath(fp, recipe_abs))
if not callers:
return # reads no dotenv at all -- not this rule's business
find(
out,
"H15",
CI_ADV,
init_rel,
1,
f"load_dotenv() is called from {len(callers)} module(s) "
f"({', '.join(sorted(callers)[:4])}) but not from the package __init__.py",
"docs/recipe-handbook/languages/python.md:107-110",
f"grep load_dotenv in {pkg}/ and check {pkg}/__init__.py",
)
# ------------------------------------------------------------------------ H48
# Values that name no owner. Normalised (lowercased, punctuation and a trailing
# "team"/"group" stripped) before lookup, so "Google, LLC" and "ADK Samples Team"
# both land here. Deliberately a closed list: an unfamiliar name like "OpenEAGO"
# or "attenu-io" is somebody's real org, and guessing "that looks like a handle"
# from its shape flags those far more often than it flags a cheat.
GENERIC_TEAMS = {
"google",
"google llc",
"google inc",
"google cloud",
"google cloud platform",
"googler",
"googlers",
"gcp",
"alphabet",
"adk",
"adk samples",
"adk-samples",
"adk sample",
"contrib",
"open source",
"opensource",
"team",
"the",
"my",
"our",
"n/a",
"na",
"none",
"null",
"nil",
"unknown",
"tbd",
"todo",
"xxx",
"placeholder",
"test",
"testing",
"demo",
"example",
"self",
"me",
"myself",
"personal",
"individual",
"independent",
"solo",
"misc",
"other",
"others",
}
def _norm_team(value):
"""Lowercase, de-punctuate, and drop a trailing team/group/org noun.
The stripper is why the generic list must stay short: "Cloud Org" strips
to "cloud", so putting a plausible team word on the list flags every real
team whose name ends in it. Words removed for exactly that reason:
DevRel, Community, Engineering, Cloud, Samples and their variants -- each
of them names a real team somewhere, and one of them names a real team
that owns five recipes in this repository.
"""
v = re.sub(r"[^a-z0-9&/ -]", " ", str(value).lower())
v = re.sub(r"\s+", " ", v).strip(" -&/")
stripped = re.sub(
r"\b(team|group|org|organisation|organization|llc|inc)\b\s*$", "", v
).strip(" -&/")
# "Team" on its own strips to nothing; keep the original so it still matches.
return re.sub(r"\s+", " ", stripped or v)
def _ownership(data, text):
"""(team, poc, [contributors]) with a regex fallback for degraded parses."""
own = data.get("ownership")
if isinstance(own, dict) and own:
contrib = own.get("contributors")
return (
own.get("team"),
own.get("poc"),
contrib if isinstance(contrib, list) else [],
)
# Degraded parse (no pyyaml): the top-level scan yields `ownership: {}`, so
# read the two scalars off the indented lines. A trailing `# comment` is part
# of the line and not part of the value.
def scalar(key):
m = re.search(rf"^\s+{key}:[ \t]*(.+?)[ \t]*$", text, re.M)
return _scalar_value(m.group(1)) or None if m else None
return scalar("team"), scalar("poc"), []
def check_ownership_team(out, r, text, data):
"""H48 -- ownership.team must name a team, not an org or a person.
The schema asks only for minLength 1, so "Google" and the author's own GitHub
handle both sail through every deterministic check in the repo. Neither tells
a future maintainer who to page.
"""
team, poc, contributors = _ownership(data, text)
if not team or not isinstance(team, str):
return # absent/typed wrong -- H19 and the schema
raw = team.strip()
if not raw:
return
# A live placeholder is H17's finding; two comments on one line is noise.
if raw.upper().startswith("TODO") or "TODO:" in raw:
return
line = lineno_of(text, r"^\s+team:")
norm = _norm_team(raw)
handles = {
str(h).strip().lstrip("@").lower() for h in [poc, *contributors] if h
}
verify = f"read the ownership.team value at line {line}"
if norm in GENERIC_TEAMS:
find(
out,
"H48",
CI_ADV,
r,
line,
f'ownership.team is "{raw}" -- that names an organisation, not a '
"team. It must be the team that will maintain the recipe, specific "
"enough that someone can find them",
".github/schemas/manifest-schema.json (ownership.team)",
verify,
)
return
if raw.strip().lstrip("@").lower() in handles:
who = (
"poc"
if str(poc or "").strip().lstrip("@").lower()
== raw.strip().lstrip("@").lower()
else "a contributor"
)
find(
out,
"H48",
CI_ADV,
r,
line,
f'ownership.team and {who} are both "{raw}". One of the two is '
"wrong: team is the owning team and poc is a person, so a single "
"value cannot be both, and whichever one is the personal handle "
"leaves the recipe unowned the moment that person moves",
".github/schemas/manifest-schema.json (ownership.team)",
verify,
)
return
# A group alias is not a person and does not leave when one does, which
# is the exact failure this rule exists to prevent. Only a personal
# address is a problem, and telling the two apart is not something to
# guess at, so no email is reported.
if re.search(r"https?://|github\.com/", raw):
find(
out,
"H48",
CI_ADV,
r,
line,
f'ownership.team is "{raw}" -- a URL, not a team name',
".github/schemas/manifest-schema.json (ownership.team)",
verify,
)
return
# `isalnum()`, not an ASCII character class: [^A-Za-z0-9]+ matched any
# name written wholly in a non-Latin script, so チーム, Команда and
# 大数据平台组 were each reported as naming nobody.
if len(raw) < 2 or not any(c.isalnum() for c in raw):
find(
out,
"H48",
CI_ADV,
r,
line,
f'ownership.team is "{raw}", which names nobody',
".github/schemas/manifest-schema.json (ownership.team)",
verify,
)
def check_manifest(out, root, rel, schema_path):
p = os.path.join(root, rel, "manifest.yaml")
text = read(p)
if text is None:
return
r = os.path.join(rel, "manifest.yaml")
# H17 -- the two canonical placeholders. Never suggest a value for these.
for ph in (
"TODO: Replace with your team name",
"TODO: Replace with your GitHub user ID",
):
if ph in text:
find(
out,
"H17",
CI_FAIL,
r,
lineno_of(text, re.escape(ph)),
f'ownership placeholder still present: "{ph}"',
"validate_manifest.py:140-151",
"grep the literal string",
)
# A manifest is flat enough that a missing pyyaml must not silently disable
# H18/H19 -- a checker that quietly skips rules is worse than no checker.
data, degraded = _parse_manifest(text)
if data is not None and not isinstance(data, dict):
# Anything that is not a mapping. Written as a blocklist of (list,
# str, int, float, bool) it missed the rest of what yaml.safe_load
# returns -- a bare date, a timestamp, !!set, !!binary -- and each of
# those still reached .get() and discarded the recipe's whole review.
# The question is "is it a mapping", so ask that.
SKIPPED.append(
(
"H17/H18/H19/H48",
f"manifest.yaml is a {type(data).__name__}, not a mapping",
)
)
return
if data is None:
SKIPPED.append(("H18/H19", "could not parse manifest.yaml"))
return
if degraded:
SKIPPED.append(
("H19-nested", "pyyaml missing; only top-level keys checked")
)
# H48 -- ownership.team. Runs before H18 so the ownership comment is first
# in the manifest's findings.
check_ownership_team(out, r, text, data)
# H18 -- description
raw_desc = data.get("description")
# A YAML block scalar with nested keys parses to a dict, and an unquoted
# year parses to an int. Both reached .strip().
desc = raw_desc.strip() if isinstance(raw_desc, str) else ""
if raw_desc is not None and not isinstance(raw_desc, str):
find(
out,
"H18",
CI_FAIL,
r,
lineno_of(text, r"^description:"),
f"description is {type(raw_desc).__name__}, not a string",
"validate_manifest.py:159-166",
"read the description value",
)
if desc.upper().startswith("TODO") or len(desc) < 10:
find(
out,
"H18",
CI_FAIL,
r,
lineno_of(text, r"^description:"),
"description is a TODO placeholder or shorter than 10 characters",
"validate_manifest.py:159-166",
"read the description value",
)
# H19 -- schema keys/enums. TRAP: license/tags/deployable/large ARE permitted.
schema = None
if schema_path and os.path.exists(schema_path):
try:
schema = json.load(open(schema_path))
except Exception:
schema = None
if schema:
allowed = set(schema.get("properties", {}))
for raw_key in data:
# str(): YAML turns `on:` into True, a bare year into an int and
# `2026-01-01:` into a date, and re.escape on any of them raises
# TypeError -- which the lane catches per recipe, discarding every
# finding for it and reporting the PR clean. The key IS invalid
# under additionalProperties: false, so the rule should say so
# rather than die on it.
k = str(raw_key)
if k not in allowed:
find(
out,
"H19",
CI_FAIL,
r,
lineno_of(text, rf"^{re.escape(k)}:"),
f'"{k}" is not a key in manifest-schema.json',
"manifest-schema.json (additionalProperties: false)",
"compare keys against the schema",
)
for k, spec in schema.get("properties", {}).items():
if k in data and "enum" in spec and data[k] not in spec["enum"]:
find(
out,
"H19",
CI_FAIL,
r,
lineno_of(text, rf"^{re.escape(k)}:"),
f'{k} = "{data[k]}" is not one of {spec["enum"]}',
"manifest-schema.json",
"compare against the enum",
)
def check_readme(out, root, rel):
p = os.path.join(root, rel, "README.md")
text = read(p)
if text is None:
return
r = os.path.join(rel, "README.md")
words = len(text.split())
if words < 100:
find(
out,
"H20",
CI_FAIL,
r,
1,
f"README is {words} words, minimum is 100",
"validate_readme.py:42",
"word-count the file",
)
if "TODO:" in text:
find(
out,
"H20",
CI_FAIL,
r,
lineno_of(text, r"TODO:"),
"README still contains TODO: text",
"validate_readme.py:108",
"grep TODO:",
)
if not re.search(r"^```", text, re.M):
find(
out,
"H20",
CI_FAIL,
r,
1,
"README has no fenced code block",
"validate_readme.py:132",
"grep for a ``` fence",
)
heads = re.findall(r"^#+ .*$", text, re.M)
setup = r"setup|prerequisit|installation|install|requirement|configuration|getting started|before you begin|environment"
run = (
r"\brun\b|running|usage|quickstart|quick start|\bstart\b|deploy|launch"
)
if not any(re.search(setup, h, re.I) for h in heads):
find(
out,
"H20",
CI_FAIL,
r,
1,
"README has no setup/prerequisites heading",
"validate_readme.py:116",
"scan the headings",
)
if not any(re.search(run, h, re.I) for h in heads):
find(
out,
"H20",
CI_FAIL,
r,
1,
"README has no run/usage heading",
"validate_readme.py:124",
"scan the headings",
)
LANGUAGE_DIRS = {"python", "java", "go", "kotlin", "typescript"}
# Basenames validate_structure prunes. A recipe directory with one of these
# names is not "badly named" -- it is unvalidated.
PRUNED_DIR_NAMES = {
"bin",
"build",
"dist",
"vendor",
"target",
"out",
"coverage",
".cache",
}
# Committed files nothing else reports, because the size checks exempt them.
# Deliberately narrow: `key.json` and `token.json` are common test fixtures, and
# a false "you committed a credential" is an alarming comment to get wrong.
_JUNK_EXACT = {
".env": "a committed .env; secrets belong in the environment, and "
".env.example is the file that ships",
".DS_Store": "macOS directory metadata, committed by accident",
"credentials.json": "a committed credentials file",
"client_secret.json": "a committed OAuth client secret",
}
# `.pem` alone is NOT here. A public CA bundle (certs/server-ca.pem is the
# documented Cloud SQL proxy fixture) and a test certificate are both ordinary
# committed files, and "you committed a private key" is the most alarming
# thing this checker can say. Only names that say PRIVATE KEY on their face.
_JUNK_SUFFIX = {
"-key.pem": "a committed private key",
"_key.pem": "a committed private key",
"privkey.pem": "a committed private key",
"private.pem": "a committed private key",
".pfx": "a committed key store",
".p12": "a committed key store",
".pyc": "a compiled artefact, not source",
}
def _junk_file(fn):
"""Why this basename should not be in the repo, or None."""
if fn in _JUNK_EXACT:
return _JUNK_EXACT[fn]
# `.env.local.example` and `.env.test.sample` are examples too -- match the
# suffix, not an allowlist of the three spellings someone thought of.
if fn.startswith(".env.") and not re.search(
r"\.(example|sample|template|dist)$", fn
):
return f"{fn} is a real environment file; only .env.example ships"
for suffix, why in _JUNK_SUFFIX.items():
if fn.endswith(suffix):
return why
# A template or an example is the file a recipe SHOULD ship. The .env
# branch above already knew that; this one did not, so
# service-account-template.json was reported as a committed key.
if re.fullmatch(r"service[-_]account.*\.json", fn) and not re.search(
r"(example|sample|template|fake|dummy|test)", fn
):
return "a committed service-account key"
return None
# H43 is a claim about what is IN THE REPO, and the filesystem cannot answer
# that: a developer who ran the recipe has a .env and a .DS_Store sitting in
# the tree, both gitignored. On this repo, checking the filesystem produced 17
# "you committed a .env" findings and the true count of tracked ones is zero.
# There is no more alarming comment to get wrong.
#
# Keyed by (root, rel), not one bare set: the CI lane checks every recipe a PR
# touches in ONE process, and a single cached set would answer the second
# recipe with the first recipe's files.
_TRACKED = {}
def _tracked_files(root, rel):
"""The set of git-tracked paths under the recipe, or None if git cannot say."""
key = (root, rel)
if key not in _TRACKED:
proc = subprocess.run(
["git", "-C", root, "ls-files", "-z", "--", rel],
capture_output=True,
text=True,
check=False,
)
_TRACKED[key] = (
None
if proc.returncode != 0
else {p for p in proc.stdout.split("\0") if p}
)
return _TRACKED[key]
def _manifest_language(root, rel):
"""manifest.language, or None. Works with or without pyyaml."""
text = read(os.path.join(root, rel, "manifest.yaml"))
if not text:
return None
data, _ = _parse_manifest(text)
if isinstance(data, dict) and isinstance(data.get("language"), str):
return data["language"].strip()
m = re.search(r"^language:\s*(.+?)\s*$", text, re.M)
return _scalar_value(m.group(1)) if m else None
def check_pr_shape(out, root, rel):
"""H42 -- a PR must not mix repo-skill changes with recipe changes.
Needs the changed-file list; with no --changed-files/--pr there is no PR to
have a shape, so this is silently not applicable rather than skipped.
A property of the PR, not of a recipe, so it is emitted ONCE however many
recipes the run covers. Called per recipe, it produced N identical
comments on one line -- and at exactly three recipes the grouping pass
collapsed them into "the same thing in 2 other places in this PR", which
is false: it is the same place, three times.
"""
if CHANGED is None:
return
if any(f["rule"] == "H42" for f in out):
return
skill_files = sorted(c for c in CHANGED if c.startswith(".agents/skills/"))
recipe_files = sorted(
c for c in CHANGED if c.startswith(("core/", "contrib/", "plugins/"))
)
if not skill_files or not recipe_files:
return
# Anchor inside the recipe, not the skill: the recipe half is what the
# author is here for, and it is the half a maintainer will be reading.
# Prefer this recipe's own manifest: the finding is anchored somewhere a
# maintainer will be reading, and `recipe_files[0]` could be a file that
# merely sorts first, such as contrib/README.md.
anchor = next(
(
c
for c in recipe_files
if c.startswith(rel + "/") and c.endswith("manifest.yaml")
),
next(
(c for c in recipe_files if c.endswith("manifest.yaml")),
recipe_files[0],
),
)
find(
out,
"H42",
CI_ADV,
anchor,
1,
f"this PR changes {len(skill_files)} file(s) under .agents/skills/ "
f"alongside {len(recipe_files)} recipe file(s). Repo skills and "
"recipes are separate concerns and belong in separate PRs",
"AGENTS.md:14",
f"compare {skill_files[0]} against {anchor}",
)
def _rule_sources(root):
"""Which rule definitions actually exist in the tree under review.
The checker must not apply a rule whose source is absent. PR #1994's head
predates AGENTS.md and .github/policy.yml entirely, so reporting "deprecated
model" or "frozen path" against it applies rules that did not exist when the
branch was written. Absent source -> skip and say so.
"""
return {
"agents_md": os.path.exists(os.path.join(root, "AGENTS.md")),
"policy": os.path.exists(os.path.join(root, ".github/policy.yml")),
}
# Mirrors .github/policy.yml `required_files`. Read from policy.yml when it is
# present in the tree under review, so the two cannot drift; these are the
# fallback for a checkout that predates a key.
_REQUIRED_ALWAYS = ["README.md", ".env.example"]
_REQUIRED_BY_ROOT = {
"core": ["AGENTS.md"],
"contrib": [],
"plugins": ["SKILL.md", "EVAL.yaml"],
}
_REQUIRED_BY_LANGUAGE = {
"python": [
"pyproject.toml",
"uv.lock",
"tests/test_runnability.py",
],
"go": ["go.mod"],
"java": [("pom.xml", "build.gradle", "build.gradle.kts")],
"kotlin": ["build.gradle.kts"],
"typescript": [
"package.json",
(
"package-lock.json",
"pnpm-lock.yaml",
"yarn.lock",
"bun.lockb",
"bun.lock",
),
],
}
def _case_insensitive_files(root):
"""Names policy.yml says CI accepts in any case (today: EVAL.yaml)."""
for base in (_OWN_REPO, root):
path = os.path.join(base, ".github/policy.yml")
if not os.path.exists(path):
continue
try:
import yaml
with open(path, "rb") as handle:
names = (yaml.safe_load(handle) or {}).get(
"case_insensitive_files"
)
except Exception:
continue
if isinstance(names, list):
return {str(n).lower() for n in names}
return {"eval.yaml"}
def _missing(recipe_abs, rel_name, lenient):
"""Is this required file absent?
`eval.yaml` satisfies `EVAL.yaml`: validate_structure.py reads
policy.case_insensitive_files and accepts either spelling, and the lane
runs on a case-sensitive filesystem where a bare os.path.exists does not.
A false "required file missing" on a file that is right there is the most
confusing comment this rule can produce.
"""
if os.path.exists(os.path.join(recipe_abs, rel_name)):
return False
if rel_name.lower() not in lenient:
return True
directory, base = os.path.split(os.path.join(recipe_abs, rel_name))
try:
return base.lower() not in {e.lower() for e in os.listdir(directory)}
except OSError:
return True
def _required_files(root, rel, recipe_abs):
"""The files THIS recipe must have: always + by root + by its language.
A recipe's language comes from its manifest, not its path: under plugins/
the middle folder is a vertical, so the path cannot say.
"""
policy = _load_policy_required_files(root)
always = policy.get("always", _REQUIRED_ALWAYS)
by_root = policy.get("by_root", _REQUIRED_BY_ROOT)
by_language = policy.get("by_language", _REQUIRED_BY_LANGUAGE)
area = rel.strip("/").split("/")[0]
required = list(always) + list(by_root.get(area) or [])
language = (_manifest_language(root, rel) or "").strip().lower()
if not language:
parts = rel.strip("/").split("/")
if area in ("core", "contrib") and len(parts) >= 2:
language = parts[1].lower()
required += list(by_language.get(language) or [])
# A YAML list entry means "any one of these"; tuples keep it hashable for
# the dedupe below. Converted for every source, as validate_structure.py
# does, so an alternative added under `always` or `by_root` cannot crash.
required = [tuple(i) if isinstance(i, list) else i for i in required]
seen = set()
deduped = []
for item in required:
if item not in seen:
seen.add(item)
deduped.append(item)
return deduped
# This file lives at <repo>/.agents/skills/github-pr-review/scripts/, so the
# repository holding it is four levels up. That repository is the BASE
# checkout when the CI lane runs, which is the whole point: policy.yml decides
# what a recipe must contain, so reading it out of the tree under review would
# let a PR edit the policy that judges it. One line added to its own
# policy.yml would turn H21 from five CI-FAILs into none.
_OWN_REPO = os.path.dirname(os.path.abspath(__file__))
for _ in range(4):
_OWN_REPO = os.path.dirname(_OWN_REPO)
def _load_policy_required_files(root):
"""`required_files` from the checker's OWN repository, or {}.
`root` is consulted only when this script runs standalone against a tree
that is not its own — a developer pointing it somewhere else — and never
in preference to the base copy.
"""
for base in (_OWN_REPO, root):
path = os.path.join(base, ".github/policy.yml")
if not os.path.exists(path):
continue
try:
import yaml
with open(path, "rb") as handle:
section = (yaml.safe_load(handle) or {}).get("required_files")
except Exception:
continue
if not isinstance(section, dict):
continue
# Each key by its OWN expected type. `isinstance(v, (list, dict))`
# let `by_root:` written as a list through, and _required_files then
# called .get() on it -- crashing the rule that exists to catch
# exactly that kind of mistake, and taking the recipe's whole review
# with it.
clean = {}
if isinstance(section.get("always"), list):
clean["always"] = section["always"]
for key in ("by_root", "by_language"):
value = section.get(key)
if isinstance(value, dict):
clean[key] = {
k: v for k, v in value.items() if isinstance(v, list)
}
if base is not _OWN_REPO:
# Reached only when the checker lives outside a repository -- the
# skill documents installing it in ~/.agents/skills. The tree
# under review is then the only policy available, and a PR can
# edit it, so the reader deserves to know.
SKIPPED.append(
(
"H21",
"required_files came from the tree under review, not from "
"a base checkout; a PR can edit that file",
)
)
return clean
return {}
def check_layout(out, root, rel, recipe_name):
recipe_abs = os.path.join(root, rel)
src = _rule_sources(root)
# H2 -- standalone ruff config anywhere in the subtree
for dirpath, dirnames, filenames in os.walk(recipe_abs):
dirnames[:] = [
d
for d in dirnames
if d not in ("__pycache__", ".venv", "node_modules")
]
for fn in filenames:
if fn in ("ruff.toml", ".ruff.toml"):
rp = os.path.relpath(os.path.join(dirpath, fn), root)
find(
out,
"H2",
CI_FAIL,
rp,
1,
"standalone ruff config in a recipe; config lives in the repo root",
"python-validate-recipe.yml:268-278",
"check the file exists",
)
# H21 -- required files, SCOPED BY LANGUAGE AND ROOT.
#
# This used to hardcode the Python list for every recipe in every
# language, so a new TypeScript or Kotlin recipe collected four confident
# CI-FAIL comments demanding a pyproject.toml, a uv.lock, an .env.example
# and a pytest file it should never have -- the failure mode this whole
# lane exists to avoid, produced deterministically on every non-Python
# recipe. policy.yml has scoped these under `by_language` all along.
lenient = _case_insensitive_files(root)
for f in _required_files(root, rel, recipe_abs):
if isinstance(f, tuple):
if all(_missing(recipe_abs, alt, lenient) for alt in f):
find(
out,
"H21",
CI_FAIL,
os.path.join(rel, f[0]),
1,
f"required file missing: any of {', '.join(f)}",
".github/policy.yml required_files",
"check one of the files exists",
)
elif _missing(recipe_abs, f, lenient):
find(
out,
"H21",
CI_FAIL,
os.path.join(rel, f),
1,
f"required file missing: {f}",
".github/policy.yml required_files",
"check the file exists",
)
# H22 -- folder name
if not re.fullmatch(r"[a-z][a-z-]*", recipe_name):
find(
out,
"H22",
CI_FAIL,
rel,
1,
f'folder name "{recipe_name}" must match ^[a-z][a-z-]*$',
"validate_structure.py:84",
"read the directory name",
)
elif recipe_name.endswith("-"):
find(
out,
"H22",
CI_ADV,
rel,
1,
f'folder name "{recipe_name}" ends with a hyphen '
"(CI permits it, the scaffolder rejects it)",
"scaffold.py:25",
"read the directory name",
)
if len(recipe_name) > 30:
find(
out,
"H22",
CI_FAIL,
rel,
1,
f"folder name is {len(recipe_name)} chars, max is 30",
".github/policy.yml max_folder_name_length",
"count the characters",
)
# H23 / H41 / H47 -- plugins/<vertical>/<solution>. One condition, three
# different consequences; emit exactly one, or a misplaced plugin collects
# three comments saying the same thing in different words.
parts = rel.strip("/").split("/")
if parts[0] == "plugins":
if len(parts) == 2:
# H41: the depth CI checks and the depth the Python validation
# matrix reads are different things. This passes and is unvalidated.
find(
out,
"H41",
CI_ADV,
rel,
1,
f"a solution directly under plugins/ ({rel}) gets no per-recipe "
"Python validation at all -- it must be "
"plugins/<vertical>/<solution>/",
".github/scripts/recipe_manifests.py",
"count the path segments",
)
elif len(parts) != 3:
find(
out,
"H23",
CI_FAIL,
rel,
1,
f"plugins recipes must be at plugins/<vertical>/<solution>/, got {rel}",
"validate_placement.py:51-92",
"count the path segments",
)
elif parts[1] in LANGUAGE_DIRS | {"js", "javascript", "ts"}:
# H47: right depth, so CI passes it. Wrong meaning.
find(
out,
"H47",
CI_ADV,
rel,
1,
f'"{parts[1]}" is a language, but that folder is the VERTICAL '
"(retail, finance, healthcare). The language comes from "
"manifest.language",
"tools/validate_placement.py:67",
"read the middle path segment",
)
# H24 -- frozen legacy roots. Only meaningful if policy.yml declares them.
if not src["policy"]:
SKIPPED.append(
(
"H24",
"no .github/policy.yml in the reviewed tree; "
"frozen paths undefined here",
)
)
elif re.match(r"^(python|java|go|kotlin|typescript)/agents/", rel):
find(
out,
"H24",
CI_FAIL,
rel,
1,
f"{rel} is under a frozen legacy path; use contrib/ or core/",
".github/policy.yml frozen_paths",
"read the path",
)
# H44 -- a directory whose basename validation silently prunes. Everything
# inside it is invisible to every check in the repo, which is a far worse
# outcome than a badly named folder.
# From git, not the filesystem — the same mistake H43 was rewritten to
# fix, twenty lines away. dist/, build/, out/ and coverage/ are exactly
# what a local build leaves behind and .gitignore hides, so on a
# developer's own tree (SKILL.md invokes this checker there) every one of
# them was a finding.
tracked_dirs = set()
for rp in _tracked_files(root, rel) or ():
parts = os.path.dirname(rp).split("/")
for i, part in enumerate(parts):
if part in PRUNED_DIR_NAMES:
tracked_dirs.add("/".join(parts[: i + 1]))
for rp in sorted(tracked_dirs):
d = rp.rsplit("/", 1)[-1]
find(
out,
"H44",
CI_ADV,
rp,
1,
f'"{d}/" is silently pruned from validation, so nothing inside '
"it is checked by anything. Rename it",
".github/policy.yml:103-121",
"read the directory name",
)
# H43 -- a file that should never be committed. Size-exempt, so no other
# check reports them. TRACKED files only; see _tracked_files.
junk: list[tuple[str, str]] = []
tracked = _tracked_files(root, rel)
if tracked is None:
SKIPPED.append(
(
"H43",
"git could not list tracked files here; refusing "
"to call an untracked .env 'committed'",
)
)
else:
for rp in sorted(tracked):
head, fn = os.path.split(rp)
why = _junk_file(fn)
if why:
junk.append((rp, why))
elif ".idea" in head.split("/"):
# .vscode is deliberately excluded: `launch.json` and
# `extensions.json` are routinely committed as recommended
# project configuration, and calling that "editor state
# committed by accident" is wrong more often than it is right.
junk.append((rp, "JetBrains project state, not project config"))
# ONE finding with a count, like H10 and H26. A recipe with twelve stray
# files should not collect twelve comments.
if junk:
rp, why = junk[0]
extra = (
f"; {len(junk)} such file(s) are tracked here"
if len(junk) > 1
else ""
)
find(
out,
"H43",
CI_ADV,
rp,
1,
f"{why}{extra}",
".gitignore",
"check the file is tracked: git ls-files -- " + rp,
)
# H40 -- manifest.language must agree with the path. The two consumers
# resolve it differently and Python validation is skipped entirely when
# they disagree, so the recipe looks green while nothing has run.
# Only when the path segment IS a language directory: a recipe at
# core/<name> with no language segment is a placement problem, and calling
# it a language mismatch describes the wrong defect.
if (
parts[0] in ("core", "contrib")
and len(parts) >= 2
and parts[1] in LANGUAGE_DIRS
):
declared = _manifest_language(root, rel)
if declared and declared.lower() != parts[1].lower():
find(
out,
"H40",
CI_ADV,
os.path.join(rel, "manifest.yaml"),
1,
f'manifest.language is "{declared}" but the recipe sits under '
f"{parts[0]}/{parts[1]}/. The path and the manifest disagree, "
"and per-language validation follows the path",
"tools/validate_structure.py",
"compare the two",
)
# H39. A value that reads as real and is not. Exact matches only, lowercased:
# a substring rule flags `PROJECT_ID_HELP_URL=https://example.com/docs`, and a
# shape rule flags every legitimately short value.
_STUB_VALUES = {
"your-project-id",
"my-project-id",
"your-project",
"my-project",
"project-id",
"your_project_id",
"my_project_id",
"sample-project",
"changeme",
"change-me",
"change_me",
"replace-me",
"replaceme",
"example.com",
"www.example.com",
"https://example.com",
"http://example.com",
"user@example.com",
"foo@example.com",
"foo",
"bar",
"baz",
"foobar",
"qux",
"asdf",
"test123",
"adk samples team",
"your-team",
"my-team",
"your-name",
"my-name",
"your-api-key",
"my-api-key",
"api-key-here",
"insert-key-here",
"your-bucket",
"my-bucket",
"bucket-name",
"your-region",
"us-central1-placeholder",
"0000000000",
"1234567890",
}
def _looks_like_a_stub(val):
v = val.strip().strip("\"'").lower()
return bool(v) and v in _STUB_VALUES
# A line that names a deprecated model in order to BAN it is documentation,
# not a use. Two recipes in this repo carry exactly that in their own
# AGENTS.md -- "don't use deprecated ones (`gemini-2.0-flash`,
# `gemini-2.5-flash`)" -- and reporting it is a confidently wrong comment
# from the one lane whose reason for existing is that it cannot produce one.
_PROHIBITION = re.compile(
r"\b(deprecated|do not|don't|dont|never|avoid|instead of|no longer|"
r"rather than|forbidden|banned|not use)\b",
re.IGNORECASE,
)
def _forbids_rather_than_uses(line):
return bool(_PROHIBITION.search(line))
def check_text_wide(out, root, rel):
"""H10, H13, H14 and H39 -- literal scans across the recipe."""
recipe_abs = os.path.join(root, rel)
if not os.path.exists(os.path.join(root, "AGENTS.md")):
SKIPPED.append(
(
"H10",
"no AGENTS.md in the reviewed tree; the deprecated "
"model list is undefined here",
)
)
banned = None
else:
# The negative lookahead is the whole point: `gemini-2.5-flash-image`
# is a CURRENT model and `gemini-2.5-flash-lite` another, and a
# prefix match told the author of plugins/retail/virtual-tryon to
# replace a correct image model with a text one, 21 times. A trailing
# `-` followed by a letter starts a different model name; a digit
# (`-001`) is a version pin of the same deprecated one, so that still
# matches.
banned = re.compile(r"gemini-2\.[05]-flash(?!-[a-zA-Z])")
hits = []
for dirpath, dirnames, filenames in os.walk(recipe_abs) if banned else []:
dirnames[:] = [
d
for d in dirnames
if d not in ("__pycache__", ".venv", "node_modules", ".git")
]
for fn in filenames:
if fn in ("uv.lock", "poetry.lock"):
continue
fp = os.path.join(dirpath, fn)
try:
if os.path.getsize(fp) > MAX_READ_BYTES:
continue
except OSError:
# A dangling symlink is trivially committable -- git stores
# mode 120000 and never checks the target. Unguarded, the
# FileNotFoundError escapes to the lane, which drops EVERY
# finding for this recipe and reports it as clean. One file
# would evade the entire deterministic review.
continue
t = read(fp)
if not t:
continue
for i, line in enumerate(t.split("\n"), 1):
if banned.search(line) and not _forbids_rather_than_uses(line):
hits.append((os.path.relpath(fp, root), i))
# Sorted, so the same pull request always produces the same comment.
# os.walk yields filesystem order, not alphabetical, so which file the
# finding anchored on varied with the order the files happened to be
# created in.
hits.sort()
if hits:
# ONE finding, not one per hit -- anchored on a file the PR touched,
# like every other whole-recipe rule. Anchored on the first hit in
# walk order, _is_ours dropped it whenever an earlier file happened
# to mention a banned id, which a recipe's own AGENTS.md often does.
anchored = _anchor_in_diff([h[0] for h in hits])
p, first_line = next((h for h in hits if h[0] == anchored), hits[0])
find(
out,
"H10",
CI_ADV,
p,
first_line,
f"deprecated model id (use gemini-3.5-flash); {len(hits)} occurrence(s) "
f"across {len({h[0] for h in hits})} file(s)",
"AGENTS.md:40",
"grep gemini-2.0-flash / gemini-2.5-flash",
)
envx = os.path.join(recipe_abs, ".env.example")
t = read(envx)
if t:
for i, line in enumerate(t.split("\n"), 1):
m = re.match(
r"^\s*(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(.*)$", line
)
if not m:
continue
# _scalar_value, not split("#"): `NAME="foo # bar"` is a value
# containing a hash, not a value plus a comment. Splitting it
# produced '"foo" is a stub committed as if it were a real value',
# stray quote included.
var, val = m.group(1), _scalar_value(m.group(2))
if var != var.upper():
find(
out,
"H13",
CI_FAIL,
os.path.join(rel, ".env.example"),
i,
f'"{var}" is not UPPER_SNAKE_CASE',
"extract_env_vars.py:444",
f"read line {i}",
)
if val.lower() in (
"<changeme>",
"changeme",
"todo",
"<todo>",
"xxx",
):
find(
out,
"H14",
CI_ADV,
os.path.join(rel, ".env.example"),
i,
'placeholder should be the exact string "<TODO: update-this-value>"',
"extract_env_vars.py:91",
f"read line {i}",
)
elif _looks_like_a_stub(val):
# H39. Distinct from H14: this one does NOT look like a
# placeholder, so a reader copying .env.example gets a value
# that is syntactically fine and simply wrong.
find(
out,
"H39",
CI_ADV,
os.path.join(rel, ".env.example"),
i,
f'"{val}" is a stub committed as if it were a real value. '
"Someone copying this file has no way to tell it needs "
"replacing; use <TODO: update-this-value>",
".agents/skills/extract-python-environment-variables/",
f"read line {i}",
)
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--repo-root", required=True)
ap.add_argument(
"--recipe", required=True, help="path relative to repo root"
)
ap.add_argument(
"--schema",
default=None,
help="path to manifest-schema.json (defaults to <root>/.github/schemas/)",
)
ap.add_argument(
"--repo",
help="owner/name; with --pr, fetches the changed "
"files itself (paged) so nothing is truncated",
)
ap.add_argument("--pr", help="PR number; use with --repo")
ap.add_argument(
"--changed-files",
help="override: file with one PR-changed path per line. Without "
"this or --repo/--pr the whole recipe is audited, which is "
"right for a NEW recipe and wrong for a small edit.",
)
ap.add_argument("--json", action="store_true")
args = ap.parse_args()
root = os.path.abspath(os.path.expanduser(args.repo_root))
rel = args.recipe.strip("/")
if not os.path.isdir(os.path.join(root, rel)):
sys.exit(f"not a directory: {os.path.join(root, rel)}")
recipe_name = os.path.basename(rel)
schema = args.schema or os.path.join(
root, ".github/schemas/manifest-schema.json"
)
global CHANGED, NEW_RECIPE
if args.changed_files:
CHANGED = {ln.strip() for ln in open(args.changed_files) if ln.strip()}
elif args.repo and args.pr:
CHANGED = fetch_changed_files(args.repo, args.pr)
if CHANGED is not None:
# A PR that adds the recipe's manifest or pyproject is creating it.
NEW_RECIPE = any(
c.startswith(rel + "/")
and c.endswith(("manifest.yaml", "pyproject.toml"))
for c in CHANGED
)
pj = load_toml(os.path.join(root, rel, "pyproject.toml")) or {}
pj_name = _table(pj, "project").get("name", "")
out = []
check_pyproject(out, root, rel, recipe_name)
check_uv_lock(out, root, rel, recipe_name, pj_name)
check_dotenv_bootstrap(out, root, rel)
check_manifest(out, root, rel, schema)
check_readme(out, root, rel)
check_layout(out, root, rel, recipe_name)
check_text_wide(out, root, rel)
check_env_defaults(out, root, rel)
check_license_headers(out, root, rel)
check_pr_shape(out, root, rel)
out.sort(key=lambda f: (f["ci"] != "fail", f["rule"], f["path"]))
if args.json:
print(json.dumps({"findings": out, "skipped": SKIPPED}, indent=1))
else:
nf = sum(1 for f in out if f["ci"] == "fail")
print(
f"{rel}: {len(out)} finding(s), {nf} CI-failing"
if out
else f"{rel}: no house-rule violations"
)
if out:
print()
for f in out:
tag = "FAIL" if f["ci"] == "fail" else "adv "
print(f" [{tag}] {f['rule']:<4} {f['path']}:{f['line']}")
print(f" {f['what']}")
if FILTERED:
from collections import Counter
c = Counter(r for r, _ in FILTERED)
print(
f"\n {len(FILTERED)} pre-existing violation(s) not attributed to "
f"this PR: {', '.join(f'{k}x{v}' for k, v in sorted(c.items()))}"
)
if SKIPPED:
print("\n NOT CHECKED this run:")
for rule, why in SKIPPED:
print(f" {rule}: {why}")
print("\n Never checked by this script (need an AST or judgement):")
for rule, why in sorted(MODEL_JUDGED.items()):
print(f" {rule}: {why.splitlines()[0]}")
if __name__ == "__main__":
main()
scripts/existing_comments.py
#!/usr/bin/env python3
"""Fetch everything already said on a PR, so the review doesn't repeat it.
existing_comments.py --repo owner/name --pr 123 --out existing.json
This exists so a PR can be re-reviewed -- by you again, or after a bot or a
colleague has been through it -- and still produce fresh comments.
Three sources, because no single endpoint has all of it:
GET /pulls/{n}/comments inline comments. Carries `line` AND `original_line`;
GitHub nulls `line` once a comment goes outdated, and
only `original_line` survives.
GET /issues/{n}/comments top-level comments. No line anchor at all, so they can
only ever match on text.
GraphQL reviewThreads the ONLY source of isResolved / isOutdated. The REST
comment payload does not carry either flag.
Emitted records are consumed by verify_findings.py --existing.
"""
import argparse
import json
import subprocess
import sys
GRAPHQL = """
query($owner:String!, $name:String!, $pr:Int!, $cursor:String) {
repository(owner:$owner, name:$name) {
pullRequest(number:$pr) {
reviewThreads(first:100, after:$cursor) {
pageInfo { hasNextPage endCursor }
nodes {
isResolved
isOutdated
comments(first:1) { nodes { path line originalLine body } }
}
}
}
}
}
"""
def gh(args):
proc = subprocess.run(
["gh", *args], capture_output=True, text=True, check=False
)
if proc.returncode != 0:
raise SystemExit(
f"gh {' '.join(args[:2])} failed: {proc.stderr.strip()[:200]}"
)
return json.loads(proc.stdout or "null")
def paged(path):
out, page = [], 1
while True:
batch = gh(
[
"api",
f"{path}{'&' if '?' in path else '?'}per_page=100&page={page}",
]
)
if not batch:
break
out.extend(batch)
if len(batch) < 100:
break
page += 1
return out
def thread_state(repo, pr):
"""(path, line) -> {resolved, outdated}, keyed on the thread's first comment."""
owner, name = repo.split("/")
state, cursor = {}, None
while True:
data = gh(
[
"api",
"graphql",
"-f",
f"query={GRAPHQL}",
"-F",
f"owner={owner}",
"-F",
f"name={name}",
"-F",
f"pr={pr}",
]
+ (["-F", f"cursor={cursor}"] if cursor else [])
)
rt = data["data"]["repository"]["pullRequest"]["reviewThreads"]
for node in rt["nodes"]:
c = (node.get("comments") or {}).get("nodes") or [{}]
c = c[0]
key = (c.get("path"), c.get("line") or c.get("originalLine"))
state[key] = {
"resolved": bool(node.get("isResolved")),
"outdated": bool(node.get("isOutdated")),
}
if not rt["pageInfo"]["hasNextPage"]:
break
cursor = rt["pageInfo"]["endCursor"]
return state
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--repo", required=True)
ap.add_argument("--pr", required=True)
ap.add_argument("--out")
args = ap.parse_args()
inline = paged(f"/repos/{args.repo}/pulls/{args.pr}/comments")
issues = paged(f"/repos/{args.repo}/issues/{args.pr}/comments")
try:
states = thread_state(args.repo, args.pr)
except SystemExit as e:
print(
f"warning: thread state unavailable ({e}); "
"treating all threads as open",
file=sys.stderr,
)
states = {}
out = []
for c in inline:
line = c.get("line")
orig = c.get("original_line")
st = states.get((c.get("path"), line or orig), {})
user = c.get("user") or {}
out.append(
{
"kind": "inline",
"path": c.get("path"),
"line": line,
"original_line": orig,
"side": c.get("side") or "RIGHT",
"body": c.get("body") or "",
"author": user.get("login"),
# A bot may be typed as Bot, or be a User account with a [bot] suffix.
"is_bot": user.get("type") == "Bot"
or str(user.get("login", "")).endswith("[bot]"),
"resolved": st.get("resolved", False),
"outdated": st.get("outdated", False),
}
)
for c in issues:
user = c.get("user") or {}
out.append(
{
"kind": "issue",
"path": None,
"line": None,
"original_line": None,
"side": None,
"body": c.get("body") or "",
"author": user.get("login"),
"is_bot": user.get("type") == "Bot"
or str(user.get("login", "")).endswith("[bot]"),
"resolved": False,
"outdated": False,
}
)
if args.out:
json.dump(out, open(args.out, "w"), indent=1)
n_inline = sum(1 for c in out if c["kind"] == "inline")
n_bot = sum(1 for c in out if c["is_bot"])
n_res = sum(1 for c in out if c["resolved"])
n_out = sum(1 for c in out if c["outdated"])
print(
f"{len(out)} existing comment(s): {n_inline} inline, "
f"{len(out) - n_inline} top-level"
)
print(f" {n_bot} from bots, {n_res} resolved, {n_out} outdated")
if n_out:
print(
" outdated threads do NOT block new findings (the code moved), "
"but their text still counts"
)
if args.out:
print(f"-> {args.out}")
if __name__ == "__main__":
main()
scripts/plan_review.py
#!/usr/bin/env python3
"""Turn a PR into a review plan: what to skip, how many comments, how to shard.
Produces the same plan every run, so the checkpoint the user sees is consistent
instead of hand-rolled from raw JSON each time.
plan_review.py --repo owner/name --pr 123
plan_review.py --repo owner/name --pr 123 --json
Human output is a size table plus the lane assignments. --json emits:
{
"repo": ..., "pr": ..., "head_sha": ...,
"churn": {"total": 1430, "reviewable": 1180, "skipped": 250},
"budget": {"low": 8, "high": 12, "mid": 10},
"fan_out": true,
"lanes": [{"id": 1, "churn": 320, "files": ["src/a.py", ...]}, ...],
"skipped": [{"path": "pnpm-lock.yaml", "churn": 250, "reason": "lockfile"}],
"post_eta_seconds": 250
}
The budget is computed on REVIEWABLE churn only -- a PR that is 1,400 lines of
regenerated lockfile and 80 lines of hand-written code earns a small-PR budget.
"""
import argparse
import json
import math
import re
import subprocess
import sys
# --- what never earns a review comment -------------------------------------
# (pattern, reason) -- matched against the full path, case-insensitively.
SKIP_PATTERNS = [
(
r"(^|/)(package-lock\.json|pnpm-lock\.yaml|yarn\.lock|poetry\.lock|Cargo\.lock|Gemfile\.lock|composer\.lock|go\.sum|uv\.lock)$",
"lockfile",
),
(r"(^|/)(vendor|node_modules|third_party|external)/", "vendored"),
(r"(^|/)(dist|build|out|target)/", "build output"),
(r"(^|/)__snapshots__/", "snapshot"),
(r"\.snap$", "snapshot"),
(r"\.min\.(js|css)$", "minified"),
(r"\.(pb|pb2)\.(go|py|js|ts|cc|h)$", "generated protobuf"),
(r"_pb2(_grpc)?\.pyi?$", "generated protobuf"),
(r"\.generated\.[a-z]+$", "generated"),
(r"(^|/)generated/", "generated"),
(
r"\.(png|jpe?g|gif|svg|ico|webp|pdf|woff2?|ttf|eot|zip|tar|gz|jar|so|dylib|dll)$",
"binary asset",
),
(r"(^|/)testdata/", "test fixture"),
(r"(^|/)fixtures?/", "test fixture"),
]
# A data file this large is a fixture dump whatever it is named.
BULK_DATA_EXT = {
".json",
".csv",
".tsv",
".yaml",
".yml",
".xml",
".sql",
".txt",
".ndjson",
".jsonl",
}
BULK_DATA_CHURN = 500
# Comment budget by reviewable churn: (max_churn, low, high). Mirrors SKILL.md.
BUDGET_TABLE = [
(50, 2, 3),
(200, 3, 5),
(600, 5, 8),
(1500, 8, 12),
(math.inf, 12, 20),
]
# Fan out only when the work is big enough to pay for the spawn overhead.
FAN_OUT_MIN_CHURN = 400
FAN_OUT_MIN_FILES = 8
CHURN_PER_LANE = 500
# 12, not 5: a 100k-line PR at 5 lanes gives each worker 20k lines, which is not a
# review, it is a skim. Lanes are cheap; unread code is not.
MAX_LANES = 12
# post_comments.py sleeps a random 10-20s between posts.
AVG_POST_GAP_SECONDS = 15
def gh_api(path):
proc = subprocess.run(
["gh", "api", path], capture_output=True, text=True, check=False
)
if proc.returncode != 0:
raise SystemExit(f"gh api {path} failed: {proc.stderr.strip()}")
return json.loads(proc.stdout) if proc.stdout.strip() else {}
def fetch_files(repo, pr):
files, page = [], 1
while True:
batch = gh_api(
f"/repos/{repo}/pulls/{pr}/files?per_page=100&page={page}"
)
if not batch:
break
files.extend(batch)
if len(batch) < 100:
break
page += 1
return files
# Optional scope carve-outs, driven by the Step 2 question. Tests are the richest
# source of cheaply-verifiable defects, so they are IN by default; --no-tests is
# for a security-focused complete review where the production surface matters more.
TEST_DIR_RE = re.compile(r"(^|/)(tests?|__tests__|spec)/", re.IGNORECASE)
WEB_DIR_RE = re.compile(r"(^|/)(web|frontend|client|ui)/", re.IGNORECASE)
def skip_reason(path, churn, *, include_tests=True, include_web=True):
for pattern, reason in SKIP_PATTERNS:
if re.search(pattern, path, re.IGNORECASE):
return reason
if not include_tests and TEST_DIR_RE.search(path):
return "tests excluded"
if not include_web and WEB_DIR_RE.search(path):
return "web excluded"
ext = (
path[path.rfind(".") :].lower()
if "." in path.rsplit("/", 1)[-1]
else ""
)
if ext in BULK_DATA_EXT and churn >= BULK_DATA_CHURN:
return "bulk data"
return None
def budget_for(churn):
for ceiling, low, high in BUDGET_TABLE:
if churn <= ceiling:
return low, high
return 12, 20
def lane_count(churn, file_count):
"""1 means review inline; >1 means fan out."""
if churn < FAN_OUT_MIN_CHURN and file_count < FAN_OUT_MIN_FILES:
return 1
n = max(2, math.ceil(churn / CHURN_PER_LANE))
return max(1, min(n, MAX_LANES, file_count))
# Directory components that describe layout rather than subject matter, so a test
# and the thing it tests resolve to the same affinity key.
STRUCTURAL_DIRS = {
"tests",
"test",
"__tests__",
"spec",
"specs",
"src",
"lib",
"scripts",
}
TEST_AFFIXES = [
(r"^test_", ""),
(r"_test$", ""),
(r"^Test", ""),
(r"\.test$", ""),
(r"\.spec$", ""),
(r"_spec$", ""),
]
def affinity_key(path):
"""Group a source file with its tests, so one lane reviews both together.
`tools/validate.py` and `tools/tests/test_validate.py` both key to
`tools/validate`. A reviewer holding only one of the pair cannot tell whether
the test still covers the code.
"""
parts = path.split("/")
name = parts[-1]
stem = name[: name.rfind(".")] if "." in name else name
for pattern, repl in TEST_AFFIXES:
stem = re.sub(pattern, repl, stem)
dirs = [d for d in parts[:-1] if d.lower() not in STRUCTURAL_DIRS]
return "/".join([*dirs, stem.lower()])
def pack(files, n):
"""Longest-processing-time-first bin packing over affinity groups.
Groups keep related files in one lane; LPT keeps the lanes finishing together.
"""
groups = {}
for f in files:
groups.setdefault(affinity_key(f["path"]), []).append(f)
# A group heavier than twice an even share would wreck the balance on its own;
# cohesion is worth less than a lane that never finishes.
ideal = sum(f["churn"] for f in files) / n if n else 0
units = []
for members in groups.values():
churn = sum(f["churn"] for f in members)
if churn > 2 * ideal and len(members) > 1:
units.extend([{"churn": f["churn"], "files": [f]} for f in members])
else:
units.append({"churn": churn, "files": members})
lanes = [{"id": i + 1, "churn": 0, "files": []} for i in range(n)]
for unit in sorted(units, key=lambda u: -u["churn"]):
lane = min(lanes, key=lambda x: x["churn"])
# Heaviest file first, so the lane's prompt leads with the substantive work.
lane["files"].extend(
f["path"] for f in sorted(unit["files"], key=lambda f: -f["churn"])
)
lane["churn"] += unit["churn"]
return [x for x in lanes if x["files"]]
def build_plan(repo, pr, *, include_tests=True, include_web=True):
meta = gh_api(f"/repos/{repo}/pulls/{pr}")
raw = fetch_files(repo, pr)
reviewable, skipped = [], []
for f in raw:
churn = f.get("additions", 0) + f.get("deletions", 0)
entry = {
"path": f["filename"],
"churn": churn,
"status": f.get("status"),
}
reason = skip_reason(
f["filename"],
churn,
include_tests=include_tests,
include_web=include_web,
)
if reason:
skipped.append({**entry, "reason": reason})
elif f.get("status") == "removed":
skipped.append({**entry, "reason": "file deleted"})
elif f.get("status") == "renamed" and churn == 0:
# Byte-identical, just moved. Nobody re-reviews code that didn't
# change; on a migration PR this is most of the diff.
skipped.append({**entry, "reason": "pure rename"})
else:
reviewable.append(entry)
r_churn = sum(f["churn"] for f in reviewable)
s_churn = sum(f["churn"] for f in skipped)
low, high = budget_for(r_churn)
n = lane_count(r_churn, len(reviewable))
return {
"repo": repo,
"pr": pr,
"title": meta.get("title"),
"head_sha": meta.get("head", {}).get("sha"),
"state": meta.get("state"),
"churn": {
"total": r_churn + s_churn,
"reviewable": r_churn,
"skipped": s_churn,
},
"budget": {"low": low, "high": high, "mid": (low + high) // 2},
"fan_out": n > 1,
"lanes": pack(reviewable, n),
"skipped": sorted(skipped, key=lambda f: -f["churn"]),
"post_eta_seconds": ((low + high) // 2) * AVG_POST_GAP_SECONDS,
}
def render(plan):
out = []
churn = plan["churn"]
n_files = sum(len(lane["files"]) for lane in plan["lanes"])
out.append(f"PR #{plan['pr']} - {plan['title']}")
out.append(
f" {churn['total']} changed lines; {churn['reviewable']} reviewable "
f"across {n_files} file(s)"
)
if plan["skipped"]:
by_reason = {}
for f in plan["skipped"]:
by_reason.setdefault(f["reason"], 0)
by_reason[f["reason"]] += 1
summary = ", ".join(f"{v} {k}" for k, v in sorted(by_reason.items()))
out.append(
f" skipped {len(plan['skipped'])} file(s) / {churn['skipped']} lines: {summary}"
)
b = plan["budget"]
out.append(
f" budget: {b['low']}-{b['high']} comments "
f"(post run ~{plan['post_eta_seconds'] // 60} min)"
)
out.append("")
if plan["fan_out"]:
out.append(
f"Fan out: {len(plan['lanes'])} file lanes + 1 cross-cutting lane"
)
for lane in plan["lanes"]:
out.append(
f" lane {lane['id']} {lane['churn']:>5} lines "
f"{len(lane['files'])} file(s)"
)
for p in lane["files"]:
out.append(f" {p}")
else:
out.append("Below fan-out threshold: review inline, no sub-agents.")
for lane in plan["lanes"]:
for p in lane["files"]:
out.append(f" {p}")
return "\n".join(out)
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--repo", required=True, help="owner/name")
ap.add_argument("--pr", required=True, type=int)
ap.add_argument("--json", action="store_true", help="emit the plan as JSON")
ap.add_argument(
"--no-tests",
action="store_true",
help="exclude test directories from review scope",
)
ap.add_argument(
"--no-web",
action="store_true",
help="exclude web/frontend directories from review scope",
)
args = ap.parse_args()
plan = build_plan(
args.repo,
args.pr,
include_tests=not args.no_tests,
include_web=not args.no_web,
)
if plan["state"] != "open":
print(f"warning: PR #{args.pr} is {plan['state']}", file=sys.stderr)
if not plan["lanes"]:
sys.exit(
"Nothing reviewable in this PR -- every changed file was skipped."
)
print(json.dumps(plan, indent=2) if args.json else render(plan))
if __name__ == "__main__":
main()
scripts/post_comments.py
#!/usr/bin/env python3
"""Post PR review comments one at a time, paced, like a human typing them.
Each comment is posted via POST /repos/{owner}/{repo}/pulls/{n}/comments, which
creates a standalone review comment with its own implicit review id -- the exact
shape GitHub produces when a person types a comment on a line in the web UI.
(Verified against real manual comments: each carries a distinct
pull_request_review_id.)
Every comment is validated against the PR's diff hunks BEFORE anything is posted,
so an unaddressable line fails the whole run up front rather than halfway through
leaving a partial review behind.
Usage:
post_comments.py --repo owner/name --pr 123 --file comments.json [--dry-run]
post_comments.py --repo owner/name --pr 123 --file comments.json --min-gap 10 --max-gap 20
comments.json:
[
{"path": "src/api.py", "line": 42, "body": "missing await here"},
{"path": "src/api.py", "line": 88, "body": "Can we use `subprocess.run()` here?"}
]
Optional per-comment keys:
"side": "RIGHT" (default) or "LEFT" for a deleted line
"start_line": for a multi-line comment (must also be in the diff)
State: writes <file>.posted alongside the input recording which comments landed,
so re-running after an interruption skips them instead of double-posting.
"""
import argparse
import json
import os
import random
import re
import subprocess
import sys
import time
HUNK_RE = re.compile(r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@")
def gh_api(path, method="GET", payload=None):
"""Call the GitHub API through gh, returning parsed JSON."""
cmd = ["gh", "api", path]
if method != "GET":
cmd += ["--method", method, "--input", "-"]
proc = subprocess.run( # noqa: PLW1510 -- returncode is inspected below
cmd,
input=json.dumps(payload) if payload is not None else None,
capture_output=True,
text=True,
)
if proc.returncode != 0:
raise RuntimeError(proc.stderr.strip() or f"gh api {path} failed")
return json.loads(proc.stdout) if proc.stdout.strip() else {}
def fetch_pr(repo, pr):
return gh_api(f"/repos/{repo}/pulls/{pr}")
def fetch_files(repo, pr):
"""All changed files, paginated."""
files, page = [], 1
while True:
batch = gh_api(
f"/repos/{repo}/pulls/{pr}/files?per_page=100&page={page}"
)
if not batch:
break
files.extend(batch)
if len(batch) < 100:
break
page += 1
return files
def commentable_lines(patch):
"""Line numbers addressable by a review comment, from a unified diff patch.
GitHub accepts a comment on any line that appears in the diff -- added lines
and untouched context lines both count (RIGHT side). Deleted lines are only
addressable on the LEFT side, tracked separately.
"""
right, left = set(), set()
if not patch:
return right, left
old_ln = new_ln = 0
for raw in patch.split("\n"):
m = HUNK_RE.match(raw)
if m:
old_ln, new_ln = int(m.group(1)), int(m.group(3))
continue
if not raw:
continue
tag = raw[0]
if tag == "+":
right.add(new_ln)
new_ln += 1
elif tag == "-":
left.add(old_ln)
old_ln += 1
elif tag == " ":
right.add(new_ln)
left.add(old_ln)
old_ln += 1
new_ln += 1
# '\' (no newline at EOF) and any other marker advance nothing
return right, left
def load_state(state_path):
if os.path.exists(state_path):
with open(state_path) as fh:
return json.load(fh)
return {"posted": []}
def save_state(state_path, state):
with open(state_path, "w") as fh:
json.dump(state, fh, indent=2)
def key_of(c):
return f"{c['path']}:{c.get('line')}:{hash(c['body']) & 0xFFFFFF}"
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--repo", required=True, help="owner/name")
ap.add_argument("--pr", required=True, type=int)
ap.add_argument("--file", required=True, help="JSON array of comments")
ap.add_argument(
"--dry-run", action="store_true", help="validate only, post nothing"
)
ap.add_argument(
"--min-gap", type=int, default=10, help="min seconds between posts"
)
ap.add_argument(
"--max-gap", type=int, default=20, help="max seconds between posts"
)
args = ap.parse_args()
with open(args.file) as fh:
comments = json.load(fh)
if not isinstance(comments, list) or not comments:
sys.exit("comments file must be a non-empty JSON array")
pr = fetch_pr(args.repo, args.pr)
head_sha = pr["head"]["sha"]
if pr.get("state") != "open":
sys.exit(f"PR #{args.pr} is {pr.get('state')}, refusing to comment")
files = fetch_files(args.repo, args.pr)
index = {}
for f in files:
index[f["filename"]] = commentable_lines(f.get("patch"))
# ---- validate everything before posting anything ----
errors = []
for i, c in enumerate(comments):
where = f"comment {i + 1}"
if not c.get("body", "").strip():
errors.append(f"{where}: empty body")
continue
path = c.get("path")
if path not in index:
errors.append(
f"{where}: '{path}' is not in this PR's changed files"
)
continue
line, side = c.get("line"), c.get("side", "RIGHT")
if not isinstance(line, int):
errors.append(f"{where}: missing integer 'line'")
continue
right, left = index[path]
valid = right if side == "RIGHT" else left
if line not in valid:
nearby = sorted(valid)[:1] + sorted(valid)[-1:] if valid else []
hint = (
f" (addressable {side} lines range {nearby})" if nearby else ""
)
errors.append(f"{where}: {path}:{line} is not in the diff{hint}")
if errors:
print("Validation failed -- nothing was posted:\n", file=sys.stderr)
for e in errors:
print(f" - {e}", file=sys.stderr)
sys.exit(1)
print(
f"Validated {len(comments)} comment(s) against PR #{args.pr} @ {head_sha[:8]}"
)
# A pending (unsubmitted) review blocks every individual comment: POST
# .../comments implicitly opens a pending review and GitHub permits only one
# per user per PR. Catch it here rather than as a bare 422 on comment 1.
try:
me = gh_api("/user").get("login")
reviews = gh_api(f"/repos/{args.repo}/pulls/{args.pr}/reviews")
stuck = [
r
for r in reviews
if r.get("state") == "PENDING"
and (r.get("user") or {}).get("login") == me
]
except Exception:
stuck = [] # never block posting on a probe failure
if stuck:
rid = stuck[0]["id"]
print(
f"\nBlocked: you have a PENDING review on this PR (id {rid}).\n"
"GitHub allows one pending review per user, and it prevents posting\n"
"individual comments. Submit or discard it first:\n"
f" submit: gh api -X POST /repos/{args.repo}/pulls/{args.pr}/reviews/{rid}/events "
"-f event=COMMENT\n"
f" discard: gh api -X DELETE /repos/{args.repo}/pulls/{args.pr}/reviews/{rid}\n"
"Then re-run this command.",
file=sys.stderr,
)
sys.exit(1)
if args.dry_run:
for c in comments:
print(f"\n--- {c['path']}:{c['line']}\n{c['body']}")
print("\nDry run -- nothing posted.")
return
state_path = args.file + ".posted"
state = load_state(state_path)
already = set(state["posted"])
pending = [c for c in comments if key_of(c) not in already]
if len(pending) < len(comments):
print(
f"Resuming: {len(comments) - len(pending)} already posted, {len(pending)} to go"
)
for i, c in enumerate(pending):
payload = {
"body": c["body"],
"commit_id": head_sha,
"path": c["path"],
"line": c["line"],
"side": c.get("side", "RIGHT"),
}
if "start_line" in c:
payload["start_line"] = c["start_line"]
payload["start_side"] = c.get("start_side", payload["side"])
try:
res = gh_api(
f"/repos/{args.repo}/pulls/{args.pr}/comments", "POST", payload
)
except RuntimeError as e:
print(f"\nFailed on {c['path']}:{c['line']}: {e}", file=sys.stderr)
print(
f"Progress saved to {state_path}; re-run to resume.",
file=sys.stderr,
)
save_state(state_path, state)
sys.exit(1)
state["posted"].append(key_of(c))
save_state(state_path, state)
print(
f"[{i + 1}/{len(pending)}] {c['path']}:{c['line']} -> {res.get('html_url', 'posted')}"
)
if i < len(pending) - 1:
gap = random.randint(args.min_gap, args.max_gap)
print(f" waiting {gap}s...")
time.sleep(gap)
print(f"\nDone. {len(pending)} comment(s) posted.")
if os.path.exists(state_path):
os.remove(state_path)
if __name__ == "__main__":
main()
scripts/rejections.py
#!/usr/bin/env python3
"""Remember which drafted comments the user cut, so they don't come back.
# after the user approves a subset at Step 6
rejections.py --record --repo owner/name --pr 123 \
--candidates all-drafted.json --approved what-they-said-yes-to.json
rejections.py --show --repo owner/name --pr 123
Suppression of *posted* comments (existing_comments.py) only covers what made it
onto the PR. A comment the user looked at and cut was never posted, so GitHub has
no record of it -- and on the next review of that PR the same finding comes back
and has to be rejected again.
Observed on the PR #2373 re-review: two findings the user had already cut
reappeared, because nothing remembered the decision.
The ledger lives outside the skill directory on purpose. The skill may be kept
untracked inside a repo checkout, where `git clean -xfd` would delete it; losing
the skill is recoverable, losing a record of the user's decisions is not.
"""
import argparse
import json
import os
import time
from pathlib import Path
def ledger_dir():
return (
Path(
os.environ.get(
"GH_PR_REVIEW_STATE",
Path.home() / ".local" / "state" / "github-pr-review",
)
)
/ "rejected"
)
def ledger_path(repo, pr):
return ledger_dir() / f"{repo.replace('/', '__')}__{pr}.json"
def load(repo, pr):
p = ledger_path(repo, pr)
if not p.exists():
return []
try:
return json.load(open(p))
except (json.JSONDecodeError, OSError):
return []
def record(repo, pr, rejected):
"""Append, de-duplicating on (path, line, comment)."""
p = ledger_path(repo, pr)
p.parent.mkdir(parents=True, exist_ok=True)
existing = load(repo, pr)
seen = {(e.get("path"), e.get("line"), e.get("comment")) for e in existing}
added = 0
for r in rejected:
key = (r.get("path"), r.get("line"), r.get("comment"))
if key in seen:
continue
existing.append(
{
"path": r.get("path"),
"line": r.get("line"),
"comment": r.get("comment") or r.get("body") or "",
"what": r.get("what", ""),
"rejected_at": time.strftime(
"%Y-%m-%dT%H:%M:%SZ", time.gmtime()
),
}
)
seen.add(key)
added += 1
json.dump(existing, open(p, "w"), indent=1)
return added, p
def _key(c):
return (
c.get("path"),
c.get("line"),
(c.get("comment") or c.get("body") or ""),
)
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--repo", required=True)
ap.add_argument("--pr", required=True)
ap.add_argument("--record", action="store_true")
ap.add_argument("--show", action="store_true")
ap.add_argument("--candidates", help="everything that was drafted")
ap.add_argument(
"--approved", help="what the user approved; the rest is rejected"
)
args = ap.parse_args()
if args.show or not args.record:
entries = load(args.repo, args.pr)
if not entries:
print(f"no recorded rejections for {args.repo}#{args.pr}")
return
print(f"{len(entries)} previously rejected on {args.repo}#{args.pr}:")
for e in entries:
print(f" {e['path']}:{e['line']} {e['comment'][:70]}")
print(f"\n{ledger_path(args.repo, args.pr)}")
return
if not args.candidates:
raise SystemExit("--record needs --candidates")
cands = json.load(open(args.candidates))
approved = json.load(open(args.approved)) if args.approved else []
ok = {_key(c) for c in approved}
rejected = [c for c in cands if _key(c) not in ok]
if not rejected:
print("nothing rejected; ledger unchanged")
return
added, path = record(args.repo, args.pr, rejected)
print(f"recorded {added} rejection(s) for {args.repo}#{args.pr}")
for r in rejected:
print(
f" {r.get('path')}:{r.get('line')} "
f"{(r.get('comment') or r.get('body') or '')[:70]}"
)
print(f"-> {path}")
if __name__ == "__main__":
main()
scripts/verify_findings.py
#!/usr/bin/env python3
"""Machine-verify lane findings before any of them becomes a comment.
verify_findings.py --findings raw.json --repo-root /tmp/pr-123 \
--repo owner/name --pr 123 --out verified.json
Every check here is one I have done by hand during a review and would otherwise
forget. Prose in SKILL.md saying "verify the windows" is not a control; this is.
Four checks:
1. **Window vs file.** A lane reports the source lines at its anchor. If those lines
do not match the real file, the finding is fabricated -- a lane that invents a
finding invents the source too. REJECTED, not downgraded.
2. **Addressability.** GitHub only accepts an inline comment on a line inside a diff
hunk. Findings outside are marked `anchorable: false` and kept for a single
top-level comment. On PR #2373 that was a third of the pool.
3. **`verify_steps` gate.** If the stated check procedure names a second file or says
"trace" / "assuming" / "consider the case where", the finding is not cheaply
verifiable whatever the lane labelled it. Forces `cheap: not_cheap`.
4. **Schema.** Missing path/line/what, or a line outside the file, is a broken
finding, not a judgement call.
Exit code is 0 even when findings are rejected -- rejection is a normal result. It
is non-zero only when the input itself cannot be processed.
"""
import argparse
import json
import os
import re
import subprocess
import sys
import unicodedata
# Phrases in verify_steps that mean the reader must leave the anchored lines.
NOT_CHEAP_MARKERS = re.compile(
r"\btrace\b|\bassum\w*\b|consider the case|if an attacker|"
r"\bsimulat\w*\b|\bimagine\b|run the code|execute\b|another file|"
r"\bgrep the repo\b|across (the )?(repo|codebase)",
re.IGNORECASE,
)
HUNK_RE = re.compile(r"^@@ -\d+(?:,\d+)? \+(\d+)(?:,\d+)? @@")
WINDOW_LINE_RE = re.compile(r"^\s*(\d+)\s*[:|]\s?(.*)$")
ESCAPE_SEQ = re.compile(
r"\\u[0-9a-fA-F]{4}|\\U[0-9a-fA-F]{8}|\\x[0-9a-fA-F]{2}"
)
def norm(s):
"""Reduce a source line to a comparable ASCII skeleton.
A lane emits an emoji as the literal escape text `\\u26a0`; the file holds the
real character. Round-tripping through unicode_escape does NOT reconcile these
-- it mojibakes the file side (`⚠` becomes `â\\x9a\\xa0`) and every emoji line
then looks fabricated. Three false rejections on PR #2373 came from exactly that.
So compare the ASCII skeleton instead: drop escape sequences, drop non-ASCII,
drop whitespace. Real fabrication still differs in the ASCII text, which is
where the substance of a source line lives.
"""
s = ESCAPE_SEQ.sub("", s)
s = unicodedata.normalize("NFKC", s)
return "".join(ch for ch in s if ch.isascii() and not ch.isspace())
def addressable_lines(repo, pr):
"""Map path -> set of RIGHT-side line numbers GitHub will accept."""
out, page = {}, 1
while True:
proc = subprocess.run(
[
"gh",
"api",
f"/repos/{repo}/pulls/{pr}/files?per_page=100&page={page}",
],
capture_output=True,
text=True,
check=False,
)
if proc.returncode != 0:
raise SystemExit(f"gh api failed: {proc.stderr.strip()[:200]}")
batch = json.loads(proc.stdout or "[]")
if not batch:
break
for f in batch:
right, new = set(), 0
for raw in (f.get("patch") or "").split("\n"):
m = HUNK_RE.match(raw)
if m:
new = int(m.group(1))
continue
if not raw:
continue
if raw[0] in "+ ":
right.add(new)
new += 1
out[f["filename"]] = right
if len(batch) < 100:
break
page += 1
return out
def _window_pairs(window):
"""[(claimed_line_no, text)] from a window block."""
out = []
for raw in str(window or "").split("\n"):
m = WINDOW_LINE_RE.match(raw)
if not m:
continue
# Lanes abbreviate long lines. A Unicode ellipsis vanishes in norm() as
# non-ASCII, but an ASCII "..." survives and turns a valid prefix into a
# mismatch -- which rejected 6 of 146 real findings on PR #2302.
out.append(
(int(m.group(1)), re.sub(r"(\.{3}|\u2026)\s*$", "", m.group(2)))
)
return out
def _matches_at(pairs, lines, offset):
"""Do all window lines match the file when shifted by `offset`?"""
checked = 0
for n, claimed in pairs:
i = n - 1 + offset
if not (0 <= i < len(lines)):
return False, 0
a, b = norm(claimed), norm(lines[i])
if not a:
continue
# Prefix, not substring: a one-character file line is a substring of
# almost anything and matched spuriously. Below MIN_PREFIX require
# equality, so short lines cannot carry a false match.
if a == b:
pass
elif min(len(a), len(b)) >= MIN_PREFIX and (
a.startswith(b) or b.startswith(a)
):
pass
else:
return False, 0
checked += 1
return checked > 0, checked
def check_window(finding, repo_root, max_drift=3):
"""(ok, reason). False means the window contradicts the file on disk.
A window that matches at a CONSISTENT offset is line drift, not fabrication --
the finding is real and only its anchor is wrong, so repair it rather than
discard it. On PR #2302 four findings were off by exactly one line.
"""
path = os.path.join(repo_root, finding["path"])
if not os.path.exists(path):
return False, "file does not exist in the checkout"
try:
lines = (
open(path, encoding="utf-8", errors="replace").read().split("\n")
)
except OSError as e:
return False, f"unreadable: {e}"
line = finding.get("line")
if not isinstance(line, int) or not (1 <= line <= len(lines)):
return False, f"line {line} outside file (1..{len(lines)})"
pairs = _window_pairs(finding.get("window"))
if not pairs:
return True, "no window supplied" # tolerated, not verified
ok, _ = _matches_at(pairs, lines, 0)
if ok:
return True, "window matches file"
for offset in [d for k in range(1, max_drift + 1) for d in (k, -k)]:
ok, n = _matches_at(pairs, lines, offset)
if ok and n >= 2: # one line could match by chance
new_line = line + offset
if 1 <= new_line <= len(lines):
finding["line"] = new_line
finding["_line_corrected"] = f"{line} -> {new_line}"
return (
True,
f"window matched at offset {offset:+d}; anchor corrected",
)
n, claimed = pairs[0]
actual = lines[n - 1] if 1 <= n <= len(lines) else ""
return False, (
f"window line {n} says {claimed.strip()[:40]!r}, "
f"file has {actual.strip()[:40]!r}"
)
# ---------------------------------------------------------------- existing work
PROXIMITY = 2 # same line, or within 2 -- tight, to avoid eating fresh findings
MIN_PREFIX = 8 # below this, a window line must equal the file line exactly
_STOPWORDS = set(
"""a an the is are was were be been being this that these those it
its of to in on for with from by at as and or not no any some all each every into
out here there which what when where line lines file files code value values never
only still also just even than then them they their should does did has have""".split()
)
def _tokens(text):
return {
w
for w in re.findall(r"[a-z_][a-z_0-9]{3,}", str(text).lower())
if w not in _STOPWORDS
}
def rejected_as_exclusions(entries):
"""Previously-cut comments, shaped like existing comments for reuse.
Marked `kind: inline` so they create line-zones, and never `outdated` -- a
decision the user already made does not expire because the code moved.
"""
return [
{
"kind": "inline",
"path": e.get("path"),
"line": e.get("line"),
"original_line": e.get("line"),
"body": e.get("comment", ""),
"author": "you",
"is_bot": False,
"resolved": False,
"outdated": False,
"_was_rejected": True,
}
for e in entries or []
]
def build_exclusions(existing, proximity=PROXIMITY):
"""(line_zones, texts) from comments already on the PR.
An OUTDATED thread contributes no line-zone: the code moved out from under it,
so that spot deserves a fresh look. Its text still counts, so the same
observation is not repeated verbatim somewhere else.
A RESOLVED thread does block -- it was already discussed, and re-raising a
settled point is worse than missing it.
Bots block exactly like humans: if a linter already flagged line 42, saying it
again adds nothing regardless of who said it first.
"""
zones, texts = {}, []
for c in existing or []:
body = c.get("body") or ""
if body.strip():
texts.append((_tokens(body), c))
if c.get("kind") != "inline" or c.get("outdated"):
continue
path = c.get("path")
for ln in (c.get("line"), c.get("original_line")):
if not path or not ln:
continue
for d in range(-proximity, proximity + 1):
zones.setdefault(path, {}).setdefault(ln + d, c)
return zones, texts
def already_raised(finding, zones, texts, sim=0.55):
"""(True, reason) if this finding repeats something already on the PR."""
hit = zones.get(finding.get("path"), {}).get(finding.get("line"))
if hit:
if hit.get("_was_rejected"):
return True, "you cut this comment on a previous review"
who = "a bot" if hit.get("is_bot") else (hit.get("author") or "someone")
at = hit.get("line") or hit.get("original_line")
state = " (resolved)" if hit.get("resolved") else ""
where = "this line" if at == finding.get("line") else f"line {at}"
return True, f"{who} already commented on {where}{state}"
mine = _tokens(f"{finding.get('what', '')} {finding.get('comment', '')}")
if len(mine) >= 4:
for toks, c in texts:
if len(toks) < 4:
continue
if len(mine & toks) / min(len(mine), len(toks)) >= sim:
if c.get("_was_rejected"):
return True, "very similar to a comment you cut previously"
who = (
"a bot"
if c.get("is_bot")
else (c.get("author") or "someone")
)
return True, f"{who} already said something very similar"
return False, ""
# ------------------------------------------------------------ fact anchoring
# Vocabulary that appears when a comment asks the reader to derive, not to see.
# Bare "pattern" was here and fired on legitimate GROUPED findings ("the same
# silent-swallow pattern appears twice more"), which are encouraged. Only the
# "breaks the pattern" shape demands that the reader infer the pattern first.
INFERENCE_WORDS = re.compile(
r"\bbreaks? the pattern\b|"
r"\benough\b|\bwould\b|\brestored?\b|\bimpl(y|ies)\b|"
r"\bpresumably\b|\beffectively\b|\bin practice\b|\bends up\b",
re.IGNORECASE,
)
def fact_anchor_lint(finding):
"""Advisory. Flags comments that likely ask the reader to derive the defect.
It cannot decide the rule -- that is a judgement -- but it catches the two
shapes behind every comment rejected on PR #2373: a claim about identifiers not
present in the window, and inference vocabulary.
"""
text = finding.get("comment") or finding.get("what") or ""
window = str(finding.get("window") or "")
notes = []
# Only single-token identifiers. A multi-word backticked phrase is prose
# quoting code (`make streamlit`, `except Exception: continue`) and will not
# appear verbatim in the window -- flagging those was pure noise on #2373.
ticked = [
tok
for tok in re.findall(r"`([^`]+)`", text)
if not re.search(r"[\s:(){}\[\]]", tok)
]
if window and ticked:
flat = re.sub(r"\W", "", window)
missing = [
tok
for tok in ticked
if re.sub(r"\W", "", tok) and re.sub(r"\W", "", tok) not in flat
]
if missing:
notes.append(
"names "
+ ", ".join(f"`{m}`" for m in missing[:3])
+ " which is not in the window"
)
m = INFERENCE_WORDS.search(text)
if m:
notes.append(f"inference word {m.group(0)!r}")
return notes
def red_workflows(repo, sha):
"""Basenames of the workflow files that FAILED on this head commit.
The rules doc keeps a hand-maintained "Already enforced" table so a reviewer
does not repeat a red check. A table drifts; the PR's own results do not.
Only `failure` counts -- a run still queued tells us nothing, and suppressing
on it would silently drop a real finding.
"""
proc = subprocess.run(
[
"gh",
"api",
f"/repos/{repo}/actions/runs?head_sha={sha}&per_page=100",
"--jq",
".workflow_runs[] | [.path, .conclusion] | @tsv",
],
capture_output=True,
text=True,
check=False,
)
if proc.returncode != 0:
return None # cannot tell; caller must not suppress
red = set()
for line in proc.stdout.splitlines():
path, _, conclusion = line.partition("\t")
if conclusion.strip() == "failure" and path.strip():
red.add(os.path.basename(path.strip()))
return red
# A finding already cites the check that enforces it in `evidence`, e.g.
# "python-validate-recipe.yml:261-266". Reusing that citation means no second
# rule-to-check mapping table has to be kept in step with anything.
_EVIDENCE_WORKFLOW = re.compile(r"([A-Za-z0-9_.\-]+\.ya?ml)")
def already_red(finding, red):
"""Is a check already failing this PR for exactly this? (reason | None)
Advisory findings are never suppressed: nothing is failing for them, so the
comment is the only way the author hears it at all.
"""
if not red or finding.get("ci") != "fail":
return None
for wf in _EVIDENCE_WORKFLOW.findall(str(finding.get("evidence") or "")):
if wf in red:
return (
f"{wf} is already failing on this PR with a precise message; "
"a comment repeating it lands on an author who is already "
"looking at a red check"
)
return None
def verify(findings, repo_root, addr, existing=None, red=None):
zones, texts = build_exclusions(existing or [])
verified, rejected, suppressed = [], [], []
for f in findings:
if not f.get("path") or not f.get("what"):
rejected.append({**f, "_reason": "missing path or what"})
continue
ok, reason = check_window(f, repo_root)
if not ok:
rejected.append({**f, "_reason": reason})
continue
f["_window_check"] = reason
# Addressability: unknown file means no patch, so not addressable.
f["anchorable"] = f["line"] in addr.get(f["path"], set())
# Compute cheapness from verify_steps. Single source of truth: workers
# no longer assign this, prose no longer re-scores it.
steps = str(f.get("verify_steps") or "")
if not steps.strip():
f["cheap"] = "not_cheap"
f["_cheap_reason"] = "no verify_steps supplied"
elif NOT_CHEAP_MARKERS.search(steps):
f["cheap"] = "not_cheap"
f["_cheap_reason"] = NOT_CHEAP_MARKERS.search(steps).group(0)
else:
f["cheap"] = "cheap"
dup, why = already_raised(f, zones, texts)
if dup:
suppressed.append({**f, "_reason": why})
continue
why_red = already_red(f, red)
if why_red:
suppressed.append({**f, "_reason": why_red})
continue
notes = fact_anchor_lint(f)
if notes:
f["_anchor_warnings"] = notes
verified.append(f)
return verified, rejected, suppressed
# Words too generic to identify a defect class.
_STOP = set(
"""a an the is are was were be been being this that these those it its
of to in on for with from by at as and or not no any some all each every into out
here there which what when where line lines file files code value values never
only still also just even than then them they their there's is-a""".split()
)
def cluster(findings, min_size=3):
"""Group findings whose `what` shares a distinctive vocabulary.
Crude on purpose -- it flags candidates for the human to group, it does not
group anything itself. Five separate "unused import" findings is one comment,
and on PR #2373 twenty findings collapsed to five classes that way, freeing
fifteen of twenty budget slots.
"""
def sig(f):
words = re.findall(r"[a-z_]{4,}", str(f.get("what", "")).lower())
return frozenset(w for w in words if w not in _STOP)
sigs = [(f, sig(f)) for f in findings]
groups, used = [], set()
for i, (_fa, sa) in enumerate(sigs):
if i in used or not sa:
continue
members = [i]
for j, (_fb, sb) in enumerate(sigs[i + 1 :], start=i + 1):
if j in used or not sb:
continue
overlap = len(sa & sb) / max(1, min(len(sa), len(sb)))
if overlap >= 0.5:
members.append(j)
if len(members) >= min_size:
used.update(members)
groups.append([sigs[k][0] for k in members])
return groups
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--findings", required=True)
ap.add_argument("--repo-root", required=True)
ap.add_argument("--repo", help="owner/name; omit to skip addressability")
ap.add_argument("--pr")
ap.add_argument("--out", help="write verified findings here")
ap.add_argument(
"--existing", help="existing_comments.py output; suppress repeats"
)
ap.add_argument(
"--no-ledger",
action="store_true",
help="ignore previously-rejected comments for this PR",
)
ap.add_argument(
"--head-sha",
help="PR head commit; with --repo, suppresses CI-FAIL "
"findings whose enforcing workflow is already red",
)
ap.add_argument(
"--no-ci-status",
action="store_true",
help="do not read the PR's check results",
)
ap.add_argument(
"--json", action="store_true", help="machine-readable summary"
)
args = ap.parse_args()
raw = json.load(open(args.findings))
if isinstance(raw, dict):
raw = raw.get("findings", [])
addr = {}
if args.repo and args.pr:
addr = addressable_lines(args.repo, args.pr)
existing = []
if args.existing:
existing = json.load(open(args.existing))
# Previously-cut comments load automatically -- the user should not have to
# remember to pass them, and forgetting means re-proposing rejected findings.
n_rejected = 0
if args.repo and args.pr and not args.no_ledger:
try:
sys.path.insert(0, os.path.dirname(os.path.abspath(__file__)))
import rejections
prior = rejections.load(args.repo, args.pr)
existing = existing + rejected_as_exclusions(prior)
n_rejected = len(prior)
except Exception as e: # never block a review on this
print(
f"warning: could not load rejection ledger: {e}",
file=sys.stderr,
)
red = None
if args.repo and args.head_sha and not args.no_ci_status:
red = red_workflows(args.repo, args.head_sha)
if red is None:
print(
"warning: could not read check results; not suppressing "
"anything as already-red",
file=sys.stderr,
)
verified, rejected, suppressed = verify(
raw, os.path.expanduser(args.repo_root), addr, existing, red
)
n_unanchor = sum(1 for f in verified if not f.get("anchorable"))
n_down = sum(1 for f in verified if f.get("cheap") == "not_cheap")
n_warn = sum(1 for f in verified if f.get("_anchor_warnings"))
n_fixed = sum(1 for f in verified if f.get("_line_corrected"))
summary = {
"input": len(raw),
"verified": len(verified),
"rejected": len(rejected),
"suppressed_as_duplicate": len(suppressed),
"line_corrected": n_fixed,
"unanchorable": n_unanchor,
"not_cheap": n_down,
"anchor_warnings": n_warn,
}
if args.out:
json.dump(verified, open(args.out, "w"), indent=1)
if args.json:
print(
json.dumps(
{
"summary": summary,
"rejected": rejected,
"suppressed": suppressed,
},
indent=1,
)
)
return
print(
f"{summary['input']} findings in, {summary['verified']} verified, "
f"{summary['rejected']} REJECTED"
)
if n_rejected:
print(
f" ({n_rejected} previously-cut comment(s) loaded from the ledger)"
)
if rejected:
print(
"\nrejected (window contradicts the file -- treat as fabricated):"
)
for f in rejected:
print(f" {f.get('path')}:{f.get('line')} {f['_reason']}")
if n_unanchor:
print(
f"\n{n_unanchor} not addressable (outside every diff hunk) -- "
"these can only go up as one top-level comment:"
)
for f in verified:
if not f.get("anchorable"):
print(f" {f['path']}:{f['line']}")
if n_fixed:
print(f"\n{n_fixed} anchor(s) corrected for line drift:")
for f in verified:
if f.get("_line_corrected"):
print(f" {f['path']} {f['_line_corrected']}")
if suppressed:
print(f"\n{len(suppressed)} suppressed — already raised on this PR:")
for f in suppressed:
print(f" {f['path']}:{f['line']} {f['_reason']}")
if n_down:
print(
f"\n{n_down} not cheaply verifiable (from their own verify_steps):"
)
for f in verified:
if f.get("cheap") == "not_cheap":
print(
f" {f['path']}:{f['line']} ({f.get('_cheap_reason', '')})"
)
if n_warn:
print(
f"\n{n_warn} fact-anchoring warning(s) — check these read as "
"pointing at something visible:"
)
for f in verified:
for note in f.get("_anchor_warnings", []):
print(f" {f['path']}:{f['line']} {note}")
groups = cluster(verified)
if groups:
print(
f"\n{len(groups)} repeated class(es) — group each into ONE comment:"
)
for g in groups:
where = ", ".join(
f"{x['path'].rsplit('/', 1)[-1]}:{x['line']}" for x in g[:3]
)
print(f" {len(g)}x {g[0]['what'][:70]}")
print(f" {where}{' …' if len(g) > 3 else ''}")
if args.out:
print(f"\nverified findings -> {args.out}")
if __name__ == "__main__":
main()
SKILL.md
---
name: github-pr-review
description: Review a GitHub pull request and leave inline comments in a natural human reviewing voice, indistinguishable from comments typed by hand on the GitHub web UI. Deliberately bounded to defects a reader can settle by looking at the anchored line, rather than a deep audit, so every comment is cheap for the author to check. Scales to 2-20 comments by PR size, parallelises analysis across sub-agents, drafts for approval, then posts individually with human pacing. Use when the user says "review this PR", "review PR 123", pastes a github.com/.../pull/N link, or asks for comments on a pull request. Don't use for reviewing local uncommitted changes or a diff against a branch (use the `review` skill for that).
---
# GitHub PR review
Reviews a pull request on GitHub and posts inline comments that read as though a
person typed them by hand. The style is defined in `reference/voice.md`; replace that
file to shift it.
## Install
Drop this folder anywhere your agent discovers skills — `~/.agents/skills/`,
`~/.config/cloudcode/skills/`, or any absolute path registered under `skills.paths`
in `~/.config/cloudcode/cloudcode.json`.
**Set `SKILL_DIR` before running any command below**, to wherever you put it. Every
script invocation in this file is relative to it:
```bash
SKILL_DIR=~/.agents/skills/github-pr-review # adjust to your install
```
Requires `gh` (authenticated) and `python3`. No other dependencies; `pyyaml` is used
if present and degrades gracefully with a reported warning if not.
<details><summary>Living inside the google/adk-samples checkout</summary>
This copy is tracked in `google/adk-samples` at `.agents/skills/github-pr-review/`,
which has two consequences worth knowing before you edit it.
**Its tests run in that repo's CI.** The root `pyproject.toml` puts `.agents/skills`
in `testpaths`, and `tools-tests.yml` fires on any `.agents/**` change, so
`tests/` here is collected by `uv run pytest` alongside the repo's own tooling
suite. A broken test in this skill turns a PR red. Run it before pushing.
`python-format.yml` scopes ruff to `core/`, `contrib/` and `plugins/`, so nothing
here is linted — the skill's own style is its own business.
**It is deliberately absent from `docs/recipe-handbook/skills-catalog.md`.** That
catalog is for skills a recipe contributor should reach for; this one is a
maintainer's reviewing tool, and advertising it invites PR authors to run a
reviewer against their own PR. Keep it out when you edit the catalog.
</details>
Five things make this work, and all five are easy to get wrong:
1. **A comment must point at a fact, not ask the author to derive one.** This is
the strongest predictor of what gets posted: on PR #2373 it separated all 16
accepted comments from all 4 rejected ones. What a comment costs the author to
check follows from it — cheap and wrong loses five seconds, expensive and wrong
burns twenty minutes and spends credibility that took months to earn. Findings
that fail this are dropped, not downgraded. See `reference/voice.md`.
2. **This is a review, not an audit.** The analysis is deliberately bounded to what
a person reading the PR could have seen — see [The depth limit](#the-depth-limit).
A finding reachable only by archaeology is unusable here no matter how real it is.
3. **Analysis and voice are separate phases.** Finding real defects and sounding
human are different jobs; doing them at once yields thin findings in a nice
accent. Analyse first with no style constraints, then voice the survivors.
4. **The voice is calibrated, not improvised.** `reference/voice.md` holds the rules
and a rated corpus. Read it before writing any comment. Do not invent comment
shapes that aren't in it.
5. **Analysis parallelises; voice does not.** On a large PR, fan the analysis out
across sub-agents (Step 3). Never fan out labelling or voicing — those depend on
holding the whole comment set at once (Steps 4 and 5).
This is a long-running skill. A big PR is minutes of analysis followed by up to
twenty minutes of paced posting. **Silence is a failure mode**: announce the plan
before starting, keep a todo list current, and checkpoint between phases. See
[Progress reporting](#progress-reporting).
## When NOT to use
- Reviewing local/uncommitted work or `git diff` against a branch -> the `review` skill.
- The user wants a thorough audit report *for themselves* -> the depth limit below
is exactly wrong for that. Review without this skill.
## What this skill is for, and what it isn't
Everything here optimises for comments that read as hand-typed: the pacing, the
register mix, the sentence-shape variety, the cosmetic quota. That machinery exists
for one reason — **a reviewer who genuinely reviewed should not have their own
comments look machine-written.** It is not a licence to appear to have reviewed
something you did not.
Three things follow, and they are load-bearing:
- **The user reads every comment before anything is posted.** Step 6 is a hard stop,
not a formality. The approval step is where they take ownership of the review as
their own, which is the thing that makes posting under their name honest.
- **Never post on the user's behalf without that approval** — not to save a round
trip, not because the findings look obviously correct.
- **A comment the user would not defend if challenged should not go up.** That is
the whole reason cheap-to-verify beats comprehensive: they can actually check the
set they are signing.
If you are ever asked to post a review the user has not read, or to make an
AI-generated review look human specifically so that its origin is concealed from the
author, that is outside what this skill is for. Say so.
## The depth limit
Binding on every phase, and on every sub-agent.
**You may:** read every changed file in full; follow a call or import **one hop**
into another file in the same project; grep for a named symbol to find where it
lives (that is navigation, and it is how you take the hop).
**You may not:** read third-party or dependency source, installed or on GitHub;
execute the code under review; sweep the repo to prove a negative ("nothing
validates X", "only one caller does Y"); construct or test an exploit payload;
chain hops beyond the first.
The reason, because a rule without one gets rationalised around: **these comments
go out under a human's name.** A finding that required reading a dependency's
internals, or running the code, misrepresents how the reviewer found it — and that
shows in the writing no matter how the comment is phrased. `reference/voice.md`
covers the two ways it leaks (the depth tell and the proof tell).
When the limit stops you, that is a result. Say plainly in `verify_steps` what
settling the finding would actually require. **Do not exceed the limit to resolve
your own uncertainty** — an honest `verify_steps` is what the gate reads in Step 4,
and a finding that turns out to be expensive is meant to be dropped.
## Step 1 - Gather
Resolve the PR from a number, a URL, or the current branch.
```bash
gh pr view <PR> --repo <owner/name> --json number,title,body,headRefOid,additions,deletions,changedFiles,state
python3 "$SKILL_DIR/scripts/existing_comments.py" \
--repo <owner/name> --pr <PR> --out /tmp/pr-<PR>-existing.json
```
Don't fetch the file list here — `plan_review.py` does it in Step 2, with paging.
**Do not run `gh pr diff` on a large PR.** Above roughly 5,000 changed lines it is
useless as context and expensive to carry — #2302's was 122,000 lines. Work from the
checked-out tree instead; for an all-additions PR the file content *is* the diff.
Below that threshold it's fine and often the quickest way to see the shape.
**Read the existing comments, and keep that file** — Step 4 feeds it to the verifier,
which drops anything already raised. This is what makes a PR re-reviewable: run it
again after your own pass, or after a bot or a colleague, and you get only new
material.
Suppression rules, applied automatically:
- same line, or **within 2 lines** → dropped
- similar wording to any existing comment, including top-level ones with no line → dropped
- **resolved** threads still block — already discussed
- **outdated** threads do not block the line (the code moved, so it deserves a fresh
look) but their text still counts
- **bots block exactly like humans**
- **comments the user cut on a previous review** are blocked too, from the ledger
written at Step 6. A rejection never expires as outdated — a decision the user made
does not lapse because the code moved.
Also paste the existing comments into each lane prompt. The verifier is the backstop;
a worker that never generates the duplicate is cheaper than one that gets filtered.
Then get the PR head into a local tree, and note its absolute path:
```bash
gh repo clone <owner/name> /tmp/pr-<PR> -- --depth 1 --no-tags && \
cd /tmp/pr-<PR> && git fetch --depth 1 origin pull/<PR>/head && git checkout FETCH_HEAD
# or, against a clone you already have:
git -C <local-clone> fetch origin pull/<PR>/head && git -C <local-clone> checkout FETCH_HEAD
```
Do this **once, here**. Analysis workers read this tree concurrently and must never
mutate it — five sub-agents each running `gh pr checkout` in the same directory is a
race that leaves the tree on an arbitrary ref mid-review.
## Step 2 - Plan, ask, announce
```bash
python3 "$SKILL_DIR/scripts/plan_review.py" \
--repo <owner/name> --pr <PR> --json
```
The plan gives you, deterministically: which files to skip and why, the comment
budget, whether to fan out, the lane assignments, and the post-run ETA.
It skips lockfiles, generated and vendored code, `dist/`, snapshots, binaries, large
data fixtures, deleted files, and **pure renames** — a file moved with zero content
change has nothing to review, which on a migration PR is most of the diff (65 of 123
files on #2373).
Lanes are packed so a **source file and its tests land together** (`tools/x.py` with
`tools/tests/test_x.py`), because a reviewer holding only one of the pair cannot tell
whether the test still covers the code. Lane sizes stay balanced regardless.
### Ask before analysing — scope
Ask here rather than at the start, because the answer depends on the PR's size and
shape, which you only know now:
```
PR #2302 "add the Horizon long-horizon agent recipe"
787 files, 122,547 lines, all additions. Skipping 16 (lockfiles, binaries).
Budget: 12-20 comments.
Include tests/ (316 files) and web/ (238)? [yes / no]
```
Argue for including `tests/`: test files are the richest source of instantly-checkable
defects — a duplicated assertion, a test named for one thing that exercises another —
and they are usually the least-reviewed part of a PR.
**Act on the answer — re-run the planner with the flags.** The default includes
everything; the question is worthless if you don't pass it through:
```bash
python3 "$SKILL_DIR/scripts/plan_review.py" \
--repo <owner/name> --pr <PR> --json [--no-tests] [--no-web]
```
Use the re-planned lane assignments, not the first run's.
**House rules — `google/adk-samples` only.** If the PR is in that repo and touches
`core/`, `contrib/` or `plugins/`, `.github/review-rules.md` is in force. Say so in
the checkpoint. For every other repo, ignore it — the rules are repo-specific and
applying them elsewhere produces confident nonsense.
Then:
1. **Seed the todo list** — one item per phase, one per lane (see
[Progress reporting](#progress-reporting)).
2. **Print the checkpoint**, so the user knows the shape of the work and that
nothing will be posted without them:
```
Tests included. 771 files across 10 lanes + consistency lane.
Analysis takes a few minutes. Nothing gets posted without your approval.
```
The budget comes from **reviewable** churn, not total — a PR that is 1,400 lines of
regenerated lockfile plus 80 lines of hand-written code earns a small-PR budget.
| Reviewable lines | Comments |
|---|---|
| < 50 | 2–3 |
| 50–200 | 3–5 |
| 200–600 | 5–8 |
| 600–1500 | 8–12 |
| > 1500 | 12–20 |
Hard cap **20**, whatever the size.
## Step 3 - Analysis phase (no voice constraints)
Hunt for defects and write them down verbosely and technically. This output is
internal — never shown to the user, never posted.
**Work inside [the depth limit](#the-depth-limit)**, and within it read the **full
changed files, not just the diff** — a broken invariant or an error path that no
longer returns is invisible in a diff window. Where the diff touches a function, read
the whole function; where it calls a helper you cannot judge blind, spend the hop.
**Use the `What to hunt` block from `reference/lane-prompt.md` verbatim.** It is one
list, and the filter at the top of it is the load-bearing part: a finding must point
at something observable at the line. If seeing that it is a defect requires the
reader to do arithmetic, infer a pattern, or reason about consequences, it is out of
scope *however serious it looks*.
**Group at 3+.** The same defect class in three or more places is ONE finding, on the
clearest instance, with the count in `evidence`. Five "unused import" findings is one
comment; the other four slots buy distinct defects. Two instances stay two.
**The grouped comment's claim is about the anchored instance**; the count is context,
never something the reader must open files to confirm. `a few unused imports in here`
is checkable at the anchor; `unused imports here, also in X and Y` is not. Group only
instances that are individually real — sweeping a deliberate `__init__.py` re-export
into an "unused imports" group makes the whole comment wrong.
**In `google/adk-samples`, run the house-rules checker:**
```bash
python3 "$SKILL_DIR/scripts/check_house_rules.py" \
--repo-root <repo_path> --recipe <recipe> --json
```
24 of the 27 rules are decided deterministically there — take its findings
verbatim. Two of them, H26 (env-read defaults, AST-based) and H27 (licence header
consistency), report one grouped finding with a count rather than one per hit.
Dispatch the house-rules lane (Template C) for the remaining three (H11, H16,
H25), which need a judgement call. One — **H48**, a junk `ownership.team` — is
reported whether or not the PR touched `manifest.yaml`, and is never trimmed;
see [Must-post findings](#must-post-findings).
Prefer the script over a lane wherever a rule is decidable — a script cannot
hallucinate a violation, and a false *"this will fail CI"* is the most expensive
comment this skill can produce.
The lane is defined by *file identity*, not churn: it receives the recipe's config
surface (`pyproject.toml`, `manifest.yaml`, `README.md`, `.env.example`, the package
`__init__.py`, `Makefile`, `Dockerfile`) and reads them **in full even when they are
pure renames or wholly outside the diff** — see `reference/rationale.md`.
Workers assign **one** label:
- `severity`: **critical** (security hole, data loss, corruption, auth bypass,
injection, a crash on a reachable path) or **no_critical** (everything else worth
saying). It orders the output; it never changes how a comment is worded.
Every finding also carries `verify_steps` (the literal procedure the author follows
to settle it) and `window` (the real source lines at the anchor). **`verify_steps` is
the gate** — `verify_findings.py` computes cheapness from it in Step 4, so a worker
that writes it honestly is doing the right thing even when the finding then gets
dropped. `window` is what gets shown to the user beside the comment.
Pure style, naming and formatting are not findings at any severity — discard them.
**A finding claiming something is absent must say where you looked**, or where it
does appear. That is the one claim the anchored line cannot support, and it is the
defect that produced the single wrong comment this skill has posted.
### 3a - Small PR (`fan_out: false`)
Do it inline, yourself. Below ~400 reviewable lines and ~8 files, spawning workers
costs more than it saves.
### 3b - Large PR (`fan_out: true`)
One `batch_task` call. Lanes are the file shards from the plan:
```
batch_task({
description: "PR 2302 analysis",
concurrency: 10,
verify: false,
subtasks: [ ...one per file lane..., consistency lane ]
})
```
Build each prompt from `reference/lane-prompt.md` — Template A for file lanes,
Template B for the consistency lane. `subagent_type: general` for all of them.
Fill in **every** placeholder: workers share no context with you or each other, so
`<PR>`, `<owner/name>`, `<repo_path>` (the tree from Step 1), the lane's file list,
and the existing-comment list all have to be spelled out in each prompt.
**The consistency lane is not optional.** It gets no file shard. It exists for the
one class a per-file split structurally cannot see: the same element differing
between files — three licence headers, two docstring styles, a constant written with
different values in two places. Shard workers cannot see these; one worker comparing
across files can.
It is bounded to **structural comparison**, never semantic tracing. Comparing four
licence headers is a glance; following a value between modules is not, and that is
what the previous version of this lane spent its time on, producing findings that
were all discarded.
`verify: false` on purpose — the batch verifier would re-audit findings you are about
to re-rank yourself in Step 4, adding a full agent pass of latency to the one phase
this whole mechanism exists to make faster.
**Merging the findings.** Read each lane's `findings.json`. Then:
- Dedupe. Two lanes reaching the same defect from different directions is common;
it is one comment.
- On any conflict about a cross-file inconsistency, the consistency lane wins — it
saw every side.
- **Spot-check `severity`, don't re-derive it.** Sanity-check that each `critical`
really is one. Re-reading the cited code is inside the limit; going further to
rescue a finding the gate will drop is not.
- A lane that failed is not a blocker: note it, and either re-run that one lane with
`task` or cover its files yourself.
Then checkpoint (see [Progress reporting](#progress-reporting)).
## Step 4 - Gate or tag, then group
**No budget cut happens here.**
### First, run the verifier — this is not optional
```bash
python3 "$SKILL_DIR/scripts/verify_findings.py" \
--findings /tmp/pr-<N>-raw.json --repo-root <repo_path> \
--repo <owner/name> --pr <N> --head-sha <headRefOid from Step 1> \
--existing /tmp/pr-<N>-existing.json \
--out /tmp/pr-<N>-verified.json
```
It does five things that were previously prose, done by hand, and sometimes skipped:
1. **Window vs file.** Each finding's quoted source is diffed against the real file.
A mismatch means the lane fabricated it — **rejected**, not downgraded.
2. **Addressability.** Marks `anchorable: false` for any line outside every diff hunk.
3. **Cheap gate.** Computes cheapness from `verify_steps`; a procedure that names a
second file or says *trace* / *assuming* is not cheaply verifiable.
4. **Duplicate suppression.** Drops anything already raised on the PR (see Step 1)
**and anything the user cut on a previous review** — the rejection ledger loads
automatically from `--repo`/`--pr`. `--no-ledger` disables it.
5. **Already-red suppression.** With `--head-sha`, drops a **CI-FAIL** finding whose
enforcing workflow is already failing on that commit, matched from the workflow
file the finding cites in `evidence`. The author is looking at that red check
already, and its message is more precise than ours. Advisory findings are never
suppressed this way — nothing is failing for them, so the comment is the only way
the author hears it. An unreadable status suppresses nothing. `--no-ci-status`
disables it.
It also emits a fact-anchoring lint and a clustering hint for grouping. Both are
advisory; read them, don't obey them blindly.
Take its output as the input to everything below. Findings it rejected do not come
back — a fabricated window is not a labelling problem.
**Un-anchorable findings are set aside, not discarded.** A finding outside every hunk
cannot be an inline comment, but it can still be true and important — the three hard
CI failures on PR #2373 were all un-anchorable. They stay in the `unanchorable`
bucket, carried to Step 6 and offered as **one** top-level issue comment. Never
silently drop them, and never anchor them to an unrelated nearby line to force them
through: a comment about `requires-python` pinned to a dependency line twenty rows
away is worse than no comment.
**The gate drops every `not_cheap` finding.** Report a bare count at the checkpoint
(`23 findings dropped as not cheaply verifiable`) — not a list. Handing over the list
would pull the user into evaluating exactly the material the gate exists to spare
them.
**Then apply the fact-anchoring filter yourself**, on the survivors. `verify_steps`
catches findings that are expensive to check; it does not catch findings that are
cheap to check but ask the reader to *derive* the defect. Delete the question from
the comment: if what remains is not a statement of something visible at the line,
drop it. This is the filter that separated all 16 accepted comments on #2373 from all
4 rejected ones — see the rated outcomes in `reference/voice.md`.
**Cheapness is a property of the final comment text**, not the raw finding.
`` `ty` is on 3.10 while `requires-python` says 3.11 `` is cheap because it names
both sides; *"this contradicts the setting above"* is the same defect and expensive.
Re-check at the end of Step 5 once the wording exists.
Survivors are ordered `critical` first. There is no other label — `confidence` was
removed because after the verifier runs, everything surviving is high-confidence by
construction, so the column never discriminated.
Severity orders the output and nothing else. It never changes how a comment is
worded (`reference/voice.md`).
Rules that still apply at this stage:
- **If there are not enough real issues, produce fewer.** Padding produces exactly
the vacuous filler that reads as machine-generated. If a PR is genuinely clean,
say so and post nothing.
- **≤ 2** cosmetic comments (stale years, a constant that disagrees, a naming
inconsistency) — all `no_critical` by definition.
- **1–2** positive remarks when genuinely warranted, **suppressed entirely if
anything `critical` was found**. Nobody compliments the styling on a PR with a
security hole.
## Step 5 - Voice phase
> **Never delegate this step.** Not to a sub-agent, not split across two passes.
> Everything below is a property of the comment set *as a whole* — the register mix,
> the clustering, the sentence-shape variety, the cosmetic quota. An agent holding
> one shard cannot honour any of them, and the failure is invisible per-comment and
> glaring in aggregate.
Read `reference/voice.md` now. Summary of the binding rules:
- Every full-sentence comment **ends in a question mark**. Declarative multi-clause
statements are banned.
- Or use a lowercase fragment of 2–6 words with no terminal period.
- **Tone never escalates with severity.** Command injection is raised as politely as
a missing try/catch.
- One thought per comment. No causal chains, no prescriptions stacked on diagnoses,
no line numbers in prose.
- Banned: bold labels, severity prefixes, emoji, headers, bullets, `suggestion`
blocks, "Consider …", "It would be better to …".
- Backticks on identifiers and paths.
- **No crafted payloads, and no claim resting on code you didn't read** — the proof
tell and the depth tell. Write the comment you would have written *before* doing
the work, not after.
- **Name the referent, and carry the verification path.** A comment must say what it
points at, and if it depends on anything off the anchored line it must name that
and say where. `old version still holds the value` is unusable; `Does the previous
version get destroyed anywhere?` is not.
Aim for ~60% Register A / 40% Register B.
Findings arrive from workers as blunt technical prose (`user-controlled filename is
interpolated into os.system()`). That is raw material, not a draft. Rewrite every
one from scratch against the corpus.
**Severity must not leak into the wording.** A `critical` comment is not sharpened
and a `no_critical` one is not softened. The user reads the ordering; the PR author
reads only the comment, and it has to look the same either way.
### Distribution matters as much as wording
- **Cluster on the substantive files.** Uniform coverage — one comment per file,
evenly spread — is a machine signature. Real reviewers dwell where the risk is and
skim the rest. Fanning out by file makes this *worse* by default, since every lane
returns findings: when you select, deliberately let some lanes contribute nothing.
- Do not let two comments share a sentence shape. Three comments opening with
`It seems` is a tell.
- Vary length. A review where every comment is one line, or every comment is two
sentences, looks generated.
- Cross-reference in Register B when the same issue recurs (`same issue as above`)
rather than restating it.
**Before leaving this step**, re-check the `cheap` label against the wording you
actually wrote (Step 4), and run the cold-read test from `reference/voice.md` on
every comment: anchored line ±10 plus the comment text, nothing else — can you tell
what it refers to and how you'd check it? If not, rewrite or drop it.
## Step 6 - Present, take a decision
Generate the report, print the summary, then **stop and wait.**
The report goes to `~/Documents/agents/pr-reviews/`. Don't ask — that folder is the
destination unless the user names another path in the same breath.
```bash
python3 "$SKILL_DIR/scripts/build_report.py" \
--candidates /tmp/pr-<N>-candidates.json --repo <owner/name> --pr <N> \
--out-md ~/Documents/agents/pr-reviews/pr-<N>-review.md
```
That writes one markdown report — **summary table first, detail sections below** —
carrying the code window beside every comment, which is the whole point: the user
must be able to run the check without opening a file.
`--out-csv` still exists for spreadsheet triage but is off by default; the CSV held
the same rows as the markdown tables, so writing both left two copies of one report
on disk.
### Present the set
One table, ordered `critical` first, then by path. Every row carries the source line
it is anchored to, so the user can settle it without opening a file.
```
16 comments ready (2 critical), ~14 min to post.
3 more findings are un-anchorable — see below.
47 findings dropped as not cheaply verifiable.
Report: ~/Documents/agents/pr-reviews/pr-2373-review.md
```
**Recommend, don't decide.**
- Don't propose "criticals only" by reflex. A review that is entirely critical
security findings is itself implausible; real ones mix a sandbox escape with a
stale Python version.
- Point out anything you are unsure of rather than burying it. The user cutting a
comment costs nothing; a bad comment posted costs their credibility.
**Then apply the size budget from Step 2.** If the set exceeds the cap, trim and say
exactly what was dropped. Never trim silently.
#### Must-post findings
**H48** — an `ownership.team` naming a company, a filler word, or the author's own
GitHub handle — is exempt from the budget and from the "is it worth a slot?" cut.
It survives trimming even on a PR already at its cap.
It is the only field in the repo where a wrong value passes every deterministic
check: the schema asks for a non-empty string and stops there. Nothing downstream
notices, and the recipe ships with no reachable owner. So the comment goes out
every time the checker fires, and when `manifest.yaml` is outside the diff hunks
it goes in the un-anchorable top-level comment rather than being dropped.
Ask who maintains the recipe. Never propose a team name — the same reason as H17.
Never post without explicit approval, and let the user cut or reword anything. A
wrong comment posted in their name costs their credibility with the PR author, so
this step is where they take ownership of the review as their own.
**Record what they cut**, right after they choose and before posting:
```bash
python3 "$SKILL_DIR/scripts/rejections.py" --record \
--repo <owner/name> --pr <PR> \
--candidates /tmp/pr-<PR>-candidates.json \
--approved /tmp/pr-<PR>-comments.json
```
`--candidates` is everything drafted, `--approved` is what they said yes to; the
difference is the rejection. **Pass the matching pair from this run** — mismatched
files record the wrong set and poison the ledger.
Step 1's suppression only sees comments that were *posted*. One the user read and cut
left no trace on GitHub, so without this it returns on the next review and has to be
rejected again — which happened three times on the #2373 re-review. The verifier
loads the ledger automatically; nobody has to remember to pass it.
### Un-anchorable findings
If the `unanchorable` bucket is non-empty, present it as its own short table and
offer **one** top-level issue comment covering all of them:
```
Un-anchorable (3) — real, but the lines aren't in any diff hunk:
pyproject.toml:69 [tool.ruff] tables CI-FAIL
pyproject.toml:38 requires-python >=3.10 CI-FAIL
pyproject.toml:2 name != folder basename CI-FAIL
Offer as one top-level comment?
```
This is the one permitted exception to "no summary comment" (Step 7). Keep it to a
bulleted list of the findings, with no preamble about the review as a whole and no
restatement of the inline comments.
Order them **CI-failures first** — an author will act on "this blocks the build"
and may not act on a convention nit. Mark advisory items as such, and never claim
CI will fail when `.github/review-rules.md` says the check is a `::notice`.
## Step 7 - Post
Write the approved set to JSON and dry-run it first — blocking, it's fast, and it is
the gate:
```bash
python3 "$SKILL_DIR/scripts/post_comments.py" \
--repo <owner/name> --pr <PR> --file /tmp/pr-comments.json --dry-run
```
It validates every line against the diff hunks and fails loudly if a line isn't
addressable. Then run the real thing **as a background job** and hand control back:
```bash
python3 "$SKILL_DIR/scripts/post_comments.py" \
--repo <owner/name> --pr <PR> --file /tmp/pr-comments.json
```
```
Posting 9 comments, ~8 min. Running in the background - you'll be notified when
it's done, so carry on with whatever you like.
```
Do not sleep-and-poll it. You get a completion notification; `tail` the job only if
the user asks how far along it is.
The script posts each comment individually, in file order, sleeping a random 10–20s
between them. **Do not shorten the gaps** — the pacing is the point
(`reference/rationale.md`). If it fails partway, the `.posted` state file records
what landed; re-run the identical command to resume.
**A pending review blocks everything** — GitHub allows one per user and this endpoint
implicitly opens one, so every comment 422s. The script preflights and stops with
instructions. Do **not** submit or discard their pending review for them: submitting
posts publicly under their name and may carry a verdict; discarding destroys drafts
they wrote. Surface it and let them choose.
**No verdict and no summary comment.** Do not approve, do not request changes, do not
leave a top-level recap. The user handles the verdict themselves.
The single exception is the un-anchorable list from Step 6, when the user approves
it — a bulleted list of findings whose lines are in no diff hunk. Not a recap: it
says nothing about the review as a whole and repeats no inline comment.
comments.json format:
```json
[
{"path": "src/api.py", "line": 42, "body": "missing await here"},
{"path": "src/api.py", "line": 88, "body": "Can we use `subprocess.run()` with a list here?"}
]
```
`line` must be a line present in the diff. Add `"side": "LEFT"` to comment on a
deleted line.
## Progress reporting
The user cannot see analysis happening. On a large PR the gap between "review this"
and the draft is minutes, and the post run is minutes more. Fill it.
**Todo list**, seeded in Step 2 and kept current — this is the live progress bar:
```
- Gather PR + existing comments
- Plan lanes and budget
- Lane 1 analysis (4 files, 320 lines)
- Lane 2 analysis (6 files, 295 lines)
- ...
- Consistency lane
- Merge, dedupe and label findings
- Draft comments in voice
- Present table and await approval
- Post (background)
```
Mark each done as it lands, not in batches. On a small PR collapse the lane items
into a single "Analyse changed files".
**Checkpoints.** Four short messages, no more. Prose, not tables.
1. *After the plan* (Step 2) — the mode/scope question, then size, skips, budget,
lane count, and the promise that nothing posts without approval.
2. *After analysis* (Step 3/4) — the funnel, so the shape of the cut is visible:
```
Analysis done. 74 findings across 12 lanes. 3 rejected by the verifier,
25 un-anchorable, 26 dropped as not cheaply verifiable or style.
20 candidates, drafting now.
```
3. *At the decision* (Step 6) — the table, the un-anchorable list, the report path,
the ETA.
4. *At posting* (Step 7) — backgrounded, ETA, notification promise.
`batch_task` does not stream partial results, so nothing arrives between checkpoints
1 and 2. That makes checkpoint 1 load-bearing: it must state that analysis takes a
few minutes, or the silence reads as a hang.
If anything degrades — a lane fails, the diff is unfetchable, the PR is closed, every
file is skipped — say so immediately rather than quietly working around it.
tests/conftest.py
"""Make this skill's scripts/ importable from its tests.
Mirrors the shim the sibling repo skills use, so the skill stays a
self-contained bundle rather than depending on repo-root pytest config.
NOTE: the root pytest config lists `.agents/skills` in testpaths and
tools-tests.yml fires on any `.agents/**` change, so CI runs these. A failure
here turns a PR red -- run them before pushing:
uv run pytest .agents/skills/github-pr-review/tests
"""
import sys
from pathlib import Path
SCRIPTS_DIR = Path(__file__).parent.parent / "scripts"
sys.path.insert(0, str(SCRIPTS_DIR))
tests/fixtures/pr2373_outcomes.json
{
"source": "google/adk-samples#2373",
"note": "Real accept/reject decisions by the user, not a rating session. 16 posted (12 verbatim, 4 with a remedy appended), 4 cut. Home paths in captured windows are redacted to /Users/<dev>/; the finding is that a machine-specific path is hardcoded, not whose.",
"comments": [
{
"n": 1,
"path": "contrib/python/economic-research-agent/economic_research/agent.py",
"line": 443,
"comment": "This looks like a local path from someone's machine \u2014 should it be configurable?",
"window": " 438: import json\n 439: cleaned_eval = eval_text.replace(\"```json\", \"\").replace(\"```\", \"\").strip()\n 440: primitives = json.loads(cleaned_eval)\n 441: \n 442: # Write to a session metadata log file\n 443: log_dir = \"/Users/<dev>/.gemini/jetski/scratch/observability\"\n 444: os.makedirs(log_dir, exist_ok=True)\n 445: import uuid\n 446: session_id = str(uuid.uuid4())\n 447: log_path = os.path.join(log_dir, f\"{session_id}.json\")\n 448: with open(log_path, \"w\") as f:",
"verdict": "keep",
"rationale": "anchors on something visible at the line"
},
{
"n": 2,
"path": "contrib/python/economic-research-agent/deploy.sh",
"line": 103,
"comment": "Does `$DEPLOY_CMD` still have the key values in it at this point?",
"window": " 97: if [ ! -z \"$UPDATE_VARS\" ]; then\n 98: DEPLOY_CMD=\"${DEPLOY_CMD} --update-env-vars=\\\"${UPDATE_VARS}\\\"\"\n 99: fi\n100: \n101: \n102: echo \"\ud83d\ude80 Executing Agent Runtime deployment...\"\n103: echo \"Command: $DEPLOY_CMD\"\n104: eval \"$DEPLOY_CMD\"\n105: \n106: # Step 4: Execute registration to Gemini Enterprise (Unless skipped or dry-run)\n107: if [ \"$NO_REGISTER\" = false ] && [ \"$DRY_RUN\" = false ]; then",
"verdict": "keep",
"rationale": "anchors on something visible at the line"
},
{
"n": 3,
"path": "contrib/python/economic-research-agent/tests/deploy_with_keys.py",
"line": 37,
"comment": "Is 200 characters enough to stay clear of the keys? The fixed prefix looks like about 64.",
"window": " 32: \"--no-confirm-project\",\n 33: \"--update-env-vars\", env_str\n 34: ]\n 35: \n 36: print(\"\\ud83d\\ude80 Starting deployment to Agent Runtime with local API keys...\")\n 37: print(f\"Command: {' '.join(cmd)[:200]}... [truncated keys]\")\n 38: \n 39: result = subprocess.run(cmd, capture_output=False)\n 40: if result.returncode == 0:\n 41: print(\"\\u2705 Deployed and configured with environment variables successfully!\")\n 42: else:",
"verdict": "cut",
"rationale": "asks the reader to do arithmetic (200 vs a ~64-char prefix)"
},
{
"n": 4,
"path": "contrib/python/economic-research-agent/deployment_metadata.json",
"line": 2,
"comment": "real project id committed here",
"window": " 1: {\n 2: \"remote_agent_runtime_id\": \"projects/697625214430/locations/us-east1/reasoningEngines/3370909136714727424\",\n 3: \"deployment_target\": \"agent_runtime\",\n 4: \"is_a2a\": false,\n 5: \"deployment_timestamp\": \"2026-07-10T17:08:22.632797\"\n 6: }",
"verdict": "keep",
"rationale": "anchors on something visible at the line"
},
{
"n": 5,
"path": "contrib/python/economic-research-agent/economic_research/advisors/real_estate_advisor.py",
"line": 20,
"comment": "hardcoded project name here",
"window": " 15: # Load ERA Environment\n 16: env_path = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), \".env\")\n 17: load_dotenv(env_path)\n 18: \n 19: if not os.getenv(\"GOOGLE_CLOUD_PROJECT\"):\n 20: os.environ[\"GOOGLE_CLOUD_PROJECT\"] = os.getenv(\"PROJECT_ID\", \"project-maui\")\n 21: \n 22: logger = logging.getLogger(__name__)\n 23: \n 24: from economic_research.tools.dynamic_entity_resolver import resolve_fips\n 25: from economic_research.tools.hud_skill import fetch_hud_fmr_data, fetch_hud_income_limits",
"verdict": "keep",
"rationale": "anchors on something visible at the line"
},
{
"n": 6,
"path": "contrib/python/economic-research-agent/economic_research/orchestrators/universal_whitepaper_orchestrator.py",
"line": 19,
"comment": "same default as the advisor module",
"window": " 14: # Load ERA Environment\n 15: env_path = os.path.join(os.path.dirname(os.path.dirname(os.path.dirname(__file__))), \".env\")\n 16: load_dotenv(env_path)\n 17:\n 18: if not os.getenv(\"GOOGLE_CLOUD_PROJECT\"):\n 19: os.environ[\"GOOGLE_CLOUD_PROJECT\"] = os.getenv(\"PROJECT_ID\", \"project-maui\")\n 20: if not os.getenv(\"GOOGLE_GENAI_USE_VERTEXAI\"):\n 21: os.environ[\"GOOGLE_GENAI_USE_VERTEXAI\"] = \"1\"\n 22:\n 23: logger = logging.getLogger(__name__)",
"verdict": "keep",
"rationale": "anchors on something visible at the line"
},
{
"n": 7,
"path": "contrib/python/economic-research-agent/tests/run_remote_simulations.py",
"line": 10,
"comment": "Should this endpoint come from an env var rather than being pinned here?",
"window": " 5: import json\n 6: import subprocess\n 7: import google.auth\n 8: \n 9: # Deployed Agent Engine ID\n 10: REMOTE_ENGINE_ID = \"https://us-east1-aiplatform.googleapis.com/v1beta1/projects/697625214430/locations/us-east1/reasoningEngines/8517890192101605376\"\n 11: \n 12: # A selection of 100 unique representative queries (from FRED, BLS, CENSUS, HUD, etc. + README WOW queries)\n 13: WOW_QUERIES = [",
"verdict": "keep",
"rationale": "anchors on something visible at the line"
},
{
"n": 8,
"path": "contrib/python/economic-research-agent/tests/run_remote_simulations.py",
"line": 44,
"comment": "The padding appends the same query over and over, but the banner below calls them unique. Which is right?",
"window": " 39: txt = cases[i][\"conversation\"][0][\"user_content\"][\"parts\"][0][\"text\"]\n 40: if txt not in sim_queries:\n 41: sim_queries.append(txt)\n 42: \n 43: # Fallback to make sure we have 100\n 44: while len(sim_queries) < 100:\n 45: sim_queries.append(\"What is the unemployment rate in Austin for the last year?\")\n 46: \n 47: return sim_queries[:100]",
"verdict": "keep",
"rationale": "anchors on something visible at the line"
},
{
"n": 9,
"path": "contrib/python/economic-research-agent/economic_research/tools/labor_shift_skill.py",
"line": 190,
"comment": "this `f` has nothing to interpolate",
"window": "185: affected_roles = [\"Customer Service Representatives\", \"Logistics Clerks\", \"Billing Specialists\"]\n186: driver = f\"Strong concentration of transaction-oriented and logistics roles ({prof_share + info_share:.1f}% knowledge-sector share). High risk of automation in support operations.\"\n187: elif vulnerability_index > 50:\n188: displacement = \"Medium (10-12%)\"\n189: affected_roles = [\"Financial Clerks\", \"Insurance Underwriters\", \"Operations Assistants\"]\n190: driver = f\"Balanced economy with corporate operations presence. Moderate displacement risk in back-office processing.\"\n191: else:\n192: displacement = \"Low (<4%)\"\n193: affected_roles = [\"Software Developers\", \"Data Analysts\", \"Digital Marketing\"]\n194: driver = f\"High concentration of advanced knowledge sectors ({prof_share + info_share:.1f}% knowledge-sector share) acting as validators and creators of AI workflows.\"\n195: ",
"verdict": "keep",
"rationale": "anchors on something visible at the line"
},
{
"n": 10,
"path": "contrib/python/economic-research-agent/economic_research/tools/labor_shift_skill.py",
"line": 14,
"comment": "`DENN` and `CHAL` break the pattern the rest of the map follows",
"window": " 9: MSA_CODES = {\n 10: \"Austin\": \"AUST448\",\n 11: \"Raleigh\": \"RALE937\",\n 12: \"San Francisco\": \"SANF806\",\n 13: \"Dallas\": \"DALL148\",\n 14: \"Denver\": \"DENN508\",\n 15: \"Seattle\": \"SEAT653\",\n 16: \"Atlanta\": \"ATLA013\",\n 17: \"Charlotte\": \"CHAL837\",\n 18: \"Columbus\": \"COLU139\"\n 19: }",
"verdict": "cut",
"rationale": "asks the reader to infer a pattern across a map"
},
{
"n": 11,
"path": "contrib/python/economic-research-agent/economic_research/tools/eia_skill.py",
"line": 34,
"comment": "Should this default be the EIA sector id rather than the name?",
"window": " 29: sector_map = {\n 30: \"industrial\": \"IND\",\n 31: \"commercial\": \"COM\",\n 32: \"residential\": \"RES\",\n 33: }\n 34: s_id = sector_map.get(sector.lower(), \"industrial\")\n 35: \n 36: for state in state_codes:\n 37: state = state.upper().strip()\n 38: url = (\n 39: f\"https://api.eia.gov/v2/electricity/retail-sales/data/?api_key={eia_key}\"\n 40: f\"&frequency=monthly&data[0]=price\"\n 41: f\"&facets[stateid][]={state}\"\n 42: f\"&facets[sectorid][]={s_id}\"",
"verdict": "keep",
"rationale": "anchors on something visible at the line"
},
{
"n": 12,
"path": "contrib/python/economic-research-agent/economic_research/tools/utility_logistics_skill.py",
"line": 28,
"comment": "Is falling back to `TX` right here? The label still says the requested state.",
"window": " 23: for state in state_names:\n 24: import us\n 25: from economic_research.tools.eia_skill import fetch_state_electricity_rates\n 26: \n 27: state_obj = us.states.lookup(state)\n 28: state_code = state_obj.abbr if state_obj else \"TX\"\n 29: \n 30: raw_eia = fetch_state_electricity_rates([state_code], sector=\"industrial\")\n 31: try:\n 32: parsed_eia = json.loads(raw_eia)\n 33: if isinstance(parsed_eia, list) and len(parsed_eia) > 0 and \"Avg Price (cents/kWh)\" in parsed_eia[0]:",
"verdict": "keep",
"rationale": "anchors on something visible at the line"
},
{
"n": 13,
"path": "contrib/python/economic-research-agent/economic_research/tools/lifestyle_logistics_incentives_skills.py",
"line": 34,
"comment": "looks like three files got concatenated",
"window": " 29: \n 30: return json.dumps(results, indent=2)\n 31: \n 32: \n 33: # Copyright 2025 Google LLC.\n 34: \"\"\"ADK Skill: Lifestyle Density & Amenity Scoring (Google Places/WalkScore). Talent Retention.\"\"\"\n 35: \n 36: \n 37: class LifestyleRequest(BaseModel):\n 38: city_names: list[str] = Field(\n 39: ..., description=\"List of city names to fetch lifestyle benchmarks for.\"",
"verdict": "keep",
"rationale": "anchors on something visible at the line"
},
{
"n": 14,
"path": "contrib/python/economic-research-agent/economic_research/agent_runtime_app.py",
"line": 2,
"comment": "the licence header stops mid-sentence",
"window": " 1: # Copyright 2026 Google LLC\n 2: # Licensed under the Apache License, Version 2.0 (the \"License\");\n 3: \n 4: from vertexai.preview.reasoning_engines import AdkApp\n 5: from economic_research.agent import ERAAgent\n 6: \n 7: # Instantiate the agent",
"verdict": "keep",
"rationale": "anchors on something visible at the line"
},
{
"n": 15,
"path": "contrib/python/economic-research-agent/economic_research/tools/dynamic_entity_resolver.py",
"line": 1,
"comment": "no licence header on this one",
"window": " 1: \"\"\"ADK Tool: Dynamic Entity and Geography Resolver (FIPS, MSA, and HS Codes).\n 2: Evolved autonomously by AlphaEvolve using live Serper.dev integration to provide infinite geographic coverage.\n 3: \"\"\"\n 4: \n 5: import json\n 6: import logging\n 7: import os\n 8: import re\n 9: import urllib.request\n 10: from typing import Any",
"verdict": "keep",
"rationale": "anchors on something visible at the line"
},
{
"n": 16,
"path": "contrib/python/economic-research-agent/economic_research/orchestrators/universal_whitepaper_orchestrator.py",
"line": 217,
"comment": "Should the traceback go back to the caller as the whitepaper text?",
"window": " 212:\n 213: except Exception as e:\n 214: import traceback\n 215: tb_str = traceback.format_exc()\n 216: logger.error(f\"Failed Universal Whitepaper Orchestration: {tb_str}\")\n 217: return f\"Error executing universal whitepaper pipeline: {tb_str}\"",
"verdict": "keep",
"rationale": "anchors on something visible at the line"
},
{
"n": 17,
"path": "contrib/python/economic-research-agent/tests/integration/test_full_golden_suite.py",
"line": 144,
"comment": "Would a failed call pass this? `error` is in the accepted list.",
"window": " 139: assert \"N/A\" not in full_response or \"Source\" in full_response, (\n 140: f\"Possible Grounding failure in {scenario['source']}.\"\n 141: )\n 142:\n 143: # Verify sources are cited or agent correctly declared limitations/declined\n 144: declined_keywords = [\"cannot directly\", \"do not have access\", \"limitations\", \"unverifiable\", \"error\", \"unable to retrieve\", \"does not utilize\", \"did not find\"]\n 145: assert any(\n 146: keyword in full_response\n 147: for keyword in [\n 148: \"Source\",\n 149: \"Data\",",
"verdict": "cut",
"rationale": "asks the reader to reason about test semantics"
},
{
"n": 18,
"path": "contrib/python/economic-research-agent/economic_research/tools/real_estate_skill.py",
"line": 33,
"comment": "`city_clean` isn't used below",
"window": " 28: # Note: These are usually retrieved from a 'Real Estate' BigQuery table or a direct CoStar API.\n 29: # Current implementation provides grounded benchmarks for site-selection comparison.\n 30: results = []\n 31:\n 32: for city in city_names:\n 33: city_clean = city.split(\",\")[0].strip()\n 34:\n 35: from economic_research.tools.dynamic_search_harvester import harvest_real_estate_roi\n 36: harvested = harvest_real_estate_roi(city, property_type)\n 37: results.append(harvested)",
"verdict": "keep",
"rationale": "anchors on something visible at the line"
},
{
"n": 19,
"path": "contrib/python/economic-research-agent/economic_research/tools/macro_foundation_skill.py",
"line": 23,
"comment": "Is there a reason this requires `BEA_API_KEY` when nothing below calls BEA?",
"window": " 18: def get_state_macro_health(state_names: list[str]) -> str:\n 19: \"\"\"\n 20: Fetches GDP and Personal Income (BEA) along with Demographic shifts (Census) for states.\n 21: This provides the 'Top-Line' economic context for site selection.\n 22: \"\"\"\n 23: bea_key = os.getenv(\"BEA_API_KEY\")\n 24: if not bea_key:\n 25: return \"ERROR: BEA_API_KEY is missing.\"\n 26: \n 27: results = []\n 28: ",
"verdict": "keep",
"rationale": "anchors on something visible at the line"
},
{
"n": 20,
"path": "contrib/python/economic-research-agent/tests/integration/test_golden_questions.py",
"line": 65,
"comment": "Should this `chdir` be restored afterwards?",
"window": " 60:\n 61:\n 62: @pytest.fixture(scope=\"function\")\n 63: def engine():\n 64: # Loophole to run project tests from inside scratch workspace\n 65: os.chdir(PROJECT_ROOT)\n 66: from dotenv import load_dotenv\n 67:\n 68: load_dotenv()\n 69:\n 70: from economic_research.agent import export_agent",
"verdict": "cut",
"rationale": "pure consequence reasoning; nothing visible remains if you delete the question"
}
]
}tests/test_build_report.py
"""Escaping regressions. All three of these silently corrupted a real report."""
import json
import subprocess
import sys
from pathlib import Path
import build_report as br
SCRIPT = Path(__file__).parent.parent / "scripts" / "build_report.py"
def run(tmp_path, candidates, **flags):
cj = tmp_path / "c.json"
cj.write_text(json.dumps(candidates))
md = tmp_path / "r.md"
args = [
sys.executable,
str(SCRIPT),
"--candidates",
str(cj),
"--repo",
"o/r",
"--pr",
"1",
"--out-md",
str(md),
]
for k, v in flags.items():
args += [f"--{k.replace('_', '-')}", str(v)]
proc = subprocess.run(args, capture_output=True, text=True, check=False)
assert proc.returncode == 0, proc.stderr
return md.read_text()
def cand(**kw):
base = {
"path": "a/b.py",
"line": 3,
"severity": "no_critical",
"comment": "c",
"window": " 3: x = 1",
"verify_steps": "read line 3",
}
base.update(kw)
return base
# ------------------------------------------------------------- real bugs
def test_pipe_in_source_does_not_split_the_row(tmp_path):
"""A raw | in a code window silently ate the rest of the table row."""
md = run(tmp_path, [cand(window=" 3: x = a | b")])
rows = [ln for ln in md.split("\n") if ln.startswith("| 1 ")]
assert rows and "|" in rows[0]
assert rows[0].count("|") - rows[0].count("\\|") == 7
def test_backticks_in_source_do_not_open_a_fence(tmp_path):
"""RST double-backticks in test_web_search.py produced a stray ``` fence
that swallowed the remainder of the document."""
md = run(tmp_path, [cand(window=" 3: * see ``root_agent.tools``")])
assert md.count("```") == 0 # no fences at all now
def test_triple_backtick_in_source_is_neutralised(tmp_path):
md = run(tmp_path, [cand(window=' 3: t.replace("```json", "")')])
assert md.count("```") == 0
def test_angle_brackets_are_escaped(tmp_path):
"""A raw < opens an HTML tag and eats everything after it."""
md = run(tmp_path, [cand(window=" 3: if a < b and c > d:")])
assert "<" in md and ">" in md
# --------------------------------------------------------------- behaviour
def test_unanchorable_leaves_the_cheap_split(tmp_path):
md = run(
tmp_path,
[
cand(),
cand(
line=4,
anchorable=False,
ci="fail",
rule="H1",
window=" 4: y = 2",
),
],
)
assert "## Un-anchorable" in md
assert "**FAIL**" in md
def test_ci_failures_sort_before_advisory(tmp_path):
md = run(
tmp_path,
[
cand(
line=3,
anchorable=False,
ci="advisory",
rule="H9",
what="adv one",
),
cand(
line=4,
anchorable=False,
ci="fail",
rule="H1",
what="fail one",
window=" 4: y = 2",
),
],
)
ua = md[md.index("## Un-anchorable") :]
assert ua.index("fail one") < ua.index("adv one")
def test_no_csv_is_written_by_default(tmp_path):
"""The CSV duplicated the markdown tables; only the report lands now."""
run(tmp_path, [cand()])
assert list(tmp_path.glob("*.csv")) == []
def test_criticals_sort_first(tmp_path):
md = run(
tmp_path,
[
cand(
path="a/z.py", line=9, comment="minor one", window=" 9: z = 1"
),
cand(path="a/a.py", severity="critical", comment="serious one"),
],
)
body = md[md.index("## Comments") :]
assert body.index("serious one") < body.index("minor one")
def test_report_is_all_tables_no_prose_sections(tmp_path):
"""The user asked for tables only; detail sections used to be H3 + fences."""
md = run(tmp_path, [cand()])
assert "### " not in md
assert md.count("```") == 0
# ------------------------------------------------------------------- units
def test_anchor_line_extracts_the_right_line():
assert br.anchor_line(" 1: aaa\n 2: bbb\n 3: ccc", 2) == "bbb"
def test_anchor_line_falls_back_when_unnumbered():
assert br.anchor_line("just some text", 5) == "just some text"
def test_common_prefix_strips_shared_directories():
assert br.common_prefix(["x/y/a.py", "x/y/b.py"]) == "x/y/"
def test_common_prefix_empty_when_nothing_shared():
assert br.common_prefix(["a/x.py", "b/y.py"]) == ""
def test_a_candidate_without_a_line_does_not_kill_the_report(tmp_path):
"""Candidates are model-authored JSON. A lane that omitted `line` raised
KeyError inside the sort comparator, six frames from the cause, and the
whole review -- every other lane's findings included -- was lost."""
c = cand()
del c["line"]
md = run(tmp_path, [c, cand(path="a/c.py", line=9)])
# Both findings survive, and the one with no line reports 0 rather than
# taking the other one down with it.
assert "`b.py`" in md and "`c.py`" in md
assert "**Comments** 2" in md
def test_a_non_numeric_line_is_coerced_rather_than_crashing(tmp_path):
md = run(tmp_path, [cand(line="not a number"), cand(line=None)])
assert "**Comments** 2" in md
def test_csv_defuses_a_leading_formula(tmp_path):
"""The CSV is opened in a spreadsheet, and every string in it comes from
the PR's own diff. A source line starting with = ran as a formula."""
csv_path = tmp_path / "out.csv"
run(
tmp_path,
[cand(comment="=cmd|'/c calc'!A1", window=" 3: =HYPERLINK(x)")],
out_csv=csv_path,
)
text = csv_path.read_text(encoding="utf-8")
assert '"\t=cmd' in text, "formula comment was not defused"
assert "=cmd" in text, "the original text must survive, just inert"
def test_csv_leaves_ordinary_values_alone(tmp_path):
csv_path = tmp_path / "out.csv"
run(tmp_path, [cand(comment="plain text")], out_csv=csv_path)
text = csv_path.read_text(encoding="utf-8")
assert '"plain text"' in text
assert "\tplain" not in text
def test_csv_safe_covers_every_formula_lead():
for lead in ("=", "+", "-", "@"):
assert br.csv_safe(f"{lead}x") == f"\t{lead}x"
assert br.csv_safe("x=1") == "x=1"
assert br.csv_safe(None) == ""
def test_report_is_written_as_utf8_regardless_of_locale(tmp_path):
"""truncate() emits U+2026. Under a non-UTF-8 locale a bare open(..., "w")
picks ASCII, raises UnicodeEncodeError on that character, and the run ends
with no report at all -- every finding lost to the reviewer's LANG."""
import os
cj = tmp_path / "c.json"
cj.write_text(json.dumps([cand(window=" 3: " + "x" * 400)]))
md = tmp_path / "r.md"
csv_out = tmp_path / "r.csv"
env = {
**os.environ,
"LC_ALL": "C",
"LANG": "C",
"PYTHONUTF8": "0",
"PYTHONCOERCECLOCALE": "0",
}
proc = subprocess.run(
[
sys.executable,
str(SCRIPT),
"--candidates",
str(cj),
"--repo",
"o/r",
"--pr",
"1",
"--out-md",
str(md),
"--out-csv",
str(csv_out),
],
capture_output=True,
text=True,
check=False,
env=env,
)
assert proc.returncode == 0, proc.stderr
assert "…" in md.read_text(encoding="utf-8")
assert csv_out.exists()
tests/test_check_house_rules.py
"""House-rule checks, plus the three traps I fell into running them by hand.
The traps matter more than the happy paths: a false "this will fail CI" is the
most expensive comment this skill can produce.
"""
import json
import os
import textwrap
from pathlib import Path
import check_house_rules as chr
import pytest
def recipe(tmp_path, name="my-recipe", **files):
root = tmp_path / "contrib" / "python" / name
root.mkdir(parents=True)
for rel, text in files.items():
p = root / rel
p.parent.mkdir(parents=True, exist_ok=True)
p.write_text(textwrap.dedent(text), encoding="utf-8")
return str(tmp_path), f"contrib/python/{name}"
def rules(out):
return {f["rule"] for f in out}
# ------------------------------------------------------- the three traps
def test_editable_self_reference_is_not_a_violation(tmp_path):
"""TRAP: `source = { editable = "." }` is the recipe's OWN package.
A naive grep for `editable` flags it. I nearly filed this on PR #2373.
"""
root, rel = recipe(
tmp_path,
**{
"pyproject.toml": '[project]\nname = "my-recipe"\n',
"uv.lock": '[[package]]\nname = "my-recipe"\nsource = { editable = "." }\n',
},
)
out = []
chr.check_uv_lock(out, root, rel, "my-recipe", "my-recipe")
assert out == []
def test_third_party_editable_source_is_a_violation(tmp_path):
root, rel = recipe(
tmp_path,
**{
"uv.lock": '[[package]]\nname = "other-lib"\nsource = { editable = "../x" }\n'
},
)
out = []
chr.check_uv_lock(out, root, rel, "my-recipe", "my-recipe")
assert rules(out) == {"H9"}
def test_licence_and_tags_are_permitted_manifest_keys(tmp_path):
"""TRAP: I expected these to fail the schema. They do not."""
schema = tmp_path / "schema.json"
schema.write_text(
json.dumps(
{
"properties": {
k: {}
for k in (
"type",
"status",
"language",
"description",
"ownership",
"license",
"tags",
"deployable",
"large",
"architecture",
)
},
"required": ["type"],
}
)
)
root, rel = recipe(
tmp_path,
**{
"manifest.yaml": 'type: standalone\ndescription: "A real description here"\n'
"license: Apache-2.0\ntags:\n - finance\n"
},
)
out = []
chr.check_manifest(out, root, rel, str(schema))
assert "H19" not in rules(out)
def test_unknown_manifest_key_is_a_violation(tmp_path):
schema = tmp_path / "schema.json"
schema.write_text(json.dumps({"properties": {"type": {}}, "required": []}))
root, rel = recipe(
tmp_path, **{"manifest.yaml": "type: standalone\nowner: someone\n"}
)
out = []
chr.check_manifest(out, root, rel, str(schema))
assert "H19" in rules(out)
def test_load_dotenv_reports_where_it_is_called_not_just_absence(tmp_path):
"""TRAP: I posted "only in tests/ and eval/" from a truncated grep. Wrong.
It was called from four package modules. The finding must name them, or the
author replies "it's right there in agent.py" and discounts the comment.
"""
root, rel = recipe(
tmp_path,
**{
"mypkg/__init__.py": "# no bootstrap here\n",
"mypkg/agent.py": "from dotenv import load_dotenv\nload_dotenv()\n",
"mypkg/tools.py": "from dotenv import load_dotenv\nload_dotenv()\n",
},
)
out = []
chr.check_dotenv_bootstrap(out, root, rel)
assert rules(out) == {"H15"}
what = out[0]["what"]
assert "2 module(s)" in what
assert "agent.py" in what and "tools.py" in what
def test_load_dotenv_in_package_init_is_compliant(tmp_path):
root, rel = recipe(
tmp_path,
**{
"mypkg/__init__.py": "from dotenv import load_dotenv\nload_dotenv()\n",
"mypkg/agent.py": "import os\n",
},
)
out = []
chr.check_dotenv_bootstrap(out, root, rel)
assert out == []
def test_recipe_that_never_reads_dotenv_is_not_flagged(tmp_path):
root, rel = recipe(
tmp_path, **{"mypkg/__init__.py": "", "mypkg/a.py": "x = 1\n"}
)
out = []
chr.check_dotenv_bootstrap(out, root, rel)
assert out == []
# -------------------------------------------------------------- pyproject
def test_ruff_table_in_recipe_is_ci_fail(tmp_path):
root, rel = recipe(
tmp_path,
**{
"pyproject.toml": '[project]\nname = "my-recipe"\nrequires-python = ">=3.11"\n[tool.ruff]\nline-length = 100\n'
},
)
out = []
chr.check_pyproject(out, root, rel, "my-recipe")
assert "H1" in rules(out)
assert next(f for f in out if f["rule"] == "H1")["ci"] == "fail"
def test_name_must_equal_folder_basename(tmp_path):
root, rel = recipe(
tmp_path, **{"pyproject.toml": '[project]\nname = "other"\n'}
)
out = []
chr.check_pyproject(out, root, rel, "my-recipe")
assert "H3" in rules(out)
def test_requires_python_310_permits_older(tmp_path):
root, rel = recipe(
tmp_path,
**{
"pyproject.toml": '[project]\nname = "my-recipe"\nrequires-python = ">=3.10"\n'
},
)
out = []
chr.check_pyproject(out, root, rel, "my-recipe")
assert "H4" in rules(out)
def test_requires_python_312_excludes_311(tmp_path):
root, rel = recipe(
tmp_path,
**{
"pyproject.toml": '[project]\nname = "my-recipe"\nrequires-python = ">=3.12"\n'
},
)
out = []
chr.check_pyproject(out, root, rel, "my-recipe")
assert "H4" in rules(out)
def test_requires_python_311_range_is_accepted(tmp_path):
root, rel = recipe(
tmp_path,
**{
"pyproject.toml": '[project]\nname = "my-recipe"\nrequires-python = ">=3.11,<3.14"\n'
},
)
out = []
chr.check_pyproject(out, root, rel, "my-recipe")
assert "H4" not in rules(out)
def test_dotenv_in_dev_group_does_not_count(tmp_path):
root, rel = recipe(
tmp_path,
**{
"pyproject.toml": '[project]\nname = "my-recipe"\nrequires-python = ">=3.11"\n'
'dependencies = ["google-adk>=2.0"]\n'
'[dependency-groups]\ndev = ["python-dotenv>=1.0.0"]\n'
},
)
out = []
chr.check_pyproject(out, root, rel, "my-recipe")
assert "H6" in rules(out)
def test_single_bracket_uv_index_is_flagged(tmp_path):
root, rel = recipe(
tmp_path,
**{
"pyproject.toml": '[project]\nname = "my-recipe"\nrequires-python = ">=3.11"\n'
'[tool.uv.index]\nurl = "https://pypi.org/simple"\ndefault = true\n'
},
)
out = []
chr.check_pyproject(out, root, rel, "my-recipe")
assert "H5" in rules(out)
# ------------------------------------------------------------ model ids
def test_deprecated_model_id_is_one_finding_not_one_per_hit(tmp_path):
root, rel = recipe(
tmp_path,
**{
"a.py": 'm = "gemini-2.5-flash"\n',
"b.py": 'n = "gemini-2.5-flash"\n',
"c.md": "use gemini-2.0-flash\n",
},
)
# H10 only applies where AGENTS.md declares the deprecated list.
(Path(root) / "AGENTS.md").write_text("Do NOT use gemini-2.5-flash.\n")
out = []
chr.check_text_wide(out, root, rel)
h10 = [f for f in out if f["rule"] == "H10"]
assert len(h10) == 1
assert "3 occurrence(s)" in h10[0]["what"]
def test_lockfile_is_not_scanned_for_model_ids(tmp_path):
root, rel = recipe(tmp_path, **{"uv.lock": "gemini-2.5-flash\n"})
(Path(root) / "AGENTS.md").write_text("Do NOT use gemini-2.5-flash.\n")
out = []
chr.check_text_wide(out, root, rel)
assert "H10" not in rules(out)
# ------------------------------------------------------- skip reporting
def test_skipped_rules_are_reported_never_silent(tmp_path):
"""A checker that hides what it did not check makes every clean run a lie."""
chr.SKIPPED.clear()
root, rel = recipe(tmp_path, **{"manifest.yaml": "type: standalone\n"})
chr.check_manifest([], root, rel, "/nonexistent/schema.json")
# pyyaml absent -> degraded parse must be recorded, not swallowed
import importlib.util
if importlib.util.find_spec("yaml") is None:
assert any("H19" in r for r, _ in chr.SKIPPED)
# --------------------------------- change-awareness and rule-source presence
def test_preexisting_violation_is_not_attributed_to_a_small_edit(tmp_path):
"""PR #1994 exposed this: a 16-line model-name change was being blamed for
[tool.ruff] tables and missing test files it never touched."""
root, rel = recipe(
tmp_path,
**{
"pyproject.toml": '[project]\nname = "my-recipe"\nrequires-python = ">=3.10"\n[tool.ruff]\nx = 1\n'
},
)
changed = tmp_path / "changed.txt"
changed.write_text(f"{rel}/some_other_file.py\n")
chr.CHANGED = {f"{rel}/some_other_file.py"}
chr.NEW_RECIPE = False
chr.FILTERED.clear()
try:
out = []
chr.check_pyproject(out, root, rel, "my-recipe")
assert out == [], (
"pre-existing pyproject violations must not be reported"
)
assert chr.FILTERED, "and they must be counted, not silently dropped"
finally:
chr.CHANGED = None
chr.NEW_RECIPE = True
def test_new_recipe_is_audited_in_full(tmp_path):
root, rel = recipe(
tmp_path,
**{
"pyproject.toml": '[project]\nname = "wrong"\nrequires-python = ">=3.10"\n'
},
)
chr.CHANGED = {f"{rel}/pyproject.toml"}
chr.NEW_RECIPE = True
try:
out = []
chr.check_pyproject(out, root, rel, "my-recipe")
assert {f["rule"] for f in out} >= {"H3", "H4"}
finally:
chr.CHANGED = None
def test_h10_is_skipped_when_agents_md_is_absent(tmp_path):
"""Applying today's deprecated-model list to a branch that predates it is
anachronistic. PR #1994's head has no AGENTS.md at all."""
root, rel = recipe(tmp_path, **{"a.py": 'm = "gemini-2.5-flash"\n'})
chr.SKIPPED.clear()
out = []
chr.check_text_wide(out, root, rel)
assert "H10" not in rules(out)
assert any(r == "H10" for r, _ in chr.SKIPPED)
def test_h10_fires_when_agents_md_declares_the_list(tmp_path):
root, rel = recipe(tmp_path, **{"a.py": 'm = "gemini-2.5-flash"\n'})
(Path(root) / "AGENTS.md").write_text("Do NOT use gemini-2.5-flash.\n")
chr.SKIPPED.clear()
out = []
chr.check_text_wide(out, root, rel)
assert "H10" in rules(out)
def test_h24_is_skipped_without_policy_yml(tmp_path):
root, _rel = recipe(tmp_path, name="legacy")
frozen_rel = "python/agents/legacy"
(Path(root) / "python" / "agents" / "legacy").mkdir(parents=True)
chr.SKIPPED.clear()
out = []
chr.check_layout(out, root, frozen_rel, "legacy")
assert "H24" not in rules(out)
assert any(r == "H24" for r, _ in chr.SKIPPED)
def test_new_recipe_detection_needs_the_complete_file_list(tmp_path):
"""PR #2302 regression: `gh pr view --json files` caps at 100.
With a truncated list, pyproject.toml was absent, NEW_RECIPE came out False,
and every finding on a brand-new 787-file recipe was filtered as pre-existing.
"""
root, rel = recipe(
tmp_path,
**{
"pyproject.toml": '[project]\nname = "my-recipe"\nrequires-python = ">=3.10"\n'
},
)
truncated = {f"{rel}/some/other/file{i}.py" for i in range(100)}
chr.CHANGED, chr.NEW_RECIPE = (
truncated,
any(
c.startswith(rel + "/")
and c.endswith(("manifest.yaml", "pyproject.toml"))
for c in truncated
),
)
chr.FILTERED.clear()
try:
out = []
chr.check_pyproject(out, root, rel, "my-recipe")
assert out == [], (
"truncated list should look like an edit, not a new recipe"
)
finally:
chr.CHANGED, chr.NEW_RECIPE = None, True
complete = truncated | {f"{rel}/pyproject.toml"}
chr.CHANGED, chr.NEW_RECIPE = complete, True
try:
out = []
chr.check_pyproject(out, root, rel, "my-recipe")
assert "H4" in rules(out), (
"complete list must attribute findings to the PR"
)
finally:
chr.CHANGED, chr.NEW_RECIPE = None, True
# ------------------------------------------------ H26: env-read defaults
def env_hits(src):
return chr._env_read_defaults(src, "a.py")
def test_getenv_with_a_default_is_flagged():
assert env_hits('import os\nx = os.getenv("X", "d")\n')
def test_environ_get_with_a_default_is_flagged():
assert env_hits('import os\nx = os.environ.get("X", "d")\n')
def test_setdefault_is_flagged():
assert env_hits('import os\nos.environ.setdefault("X", "d")\n')
def test_keyword_default_is_flagged():
assert env_hits('import os\nx = os.getenv("X", default="d")\n')
def test_or_fallback_is_flagged():
assert env_hits('import os\nx = os.getenv("X") or "fallback"\n')
def test_bare_getenv_is_clean():
assert not env_hits('import os\nx = os.getenv("X")\n')
def test_dict_style_access_is_clean():
"""os.environ["X"] cannot carry a default, so it is the encouraged shape."""
assert not env_hits('import os\nx = os.environ["X"]\n')
def test_or_none_is_not_a_default():
assert not env_hits('import os\nx = os.environ.get("X") or None\n')
def test_a_default_in_a_docstring_is_not_a_finding():
"""AST, not regex -- prose describing the pattern must not fire."""
assert not env_hits(
'"""Call os.getenv("X", "d") to read it."""\nimport os\n'
)
def test_an_unrelated_dot_get_is_not_an_env_read():
assert not env_hits('d = {}\nx = d.get("X", "default")\n')
def test_syntax_error_does_not_crash_the_check():
assert env_hits("def broken(:\n") == []
# ------------------------------------------------ H27: licence headers
FULL = "\n".join(
"# " + ln if ln else "#"
for ln in [
"Copyright 2026 Google LLC",
"",
'Licensed under the Apache License, Version 2.0 (the "License");',
"you may not use this file except in compliance with the License.",
"You may obtain a copy of the License at",
"",
" https://www.apache.org/licenses/LICENSE-2.0",
"",
"Unless required by applicable law or agreed to in writing, software",
'distributed under the License is distributed on an "AS IS" BASIS,',
"WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.",
"See the License for the specific language governing permissions and",
"limitations under the License.",
]
)
def test_full_header_is_recognised():
assert chr._header_state(FULL + "\nimport os\n", "#")[0] == "full"
def test_one_line_copyright_is_partial():
assert (
chr._header_state("# Copyright 2026 Google LLC\nimport os\n", "#")[0]
== "partial"
)
def test_no_header_is_none():
assert chr._header_state("import os\n", "#")[0] == "none"
def test_header_after_a_ruff_pragma_still_counts():
assert chr._header_state("# ruff: noqa\n" + FULL, "#")[0] == "full"
def test_slash_comment_style_is_supported():
js = FULL.replace("#", "//")
assert chr._header_state(js, "//")[0] == "full"
def test_a_type_with_no_headers_anywhere_is_not_drift(tmp_path):
"""Every .tf file lacking a header is the convention, not 12 violations.
Pooling extensions produced exactly this false positive on PR #2302.
"""
root, rel = recipe(
tmp_path,
**{f"terraform/f{i}.tf": 'resource "x" {}\n' for i in range(6)},
)
out = []
chr.check_license_headers(out, root, rel)
assert out == []
def test_a_minority_breaking_the_convention_is_flagged(tmp_path):
files = {f"m{i}.py": FULL + "\nimport os\n" for i in range(8)}
files["odd.py"] = "# Copyright 2026 Google LLC\nimport os\n"
root, rel = recipe(tmp_path, **files)
out = []
chr.check_license_headers(out, root, rel)
assert "H27" in rules(out)
assert "1 .py file(s) differ" in out[0]["what"]
def test_minority_breaking_a_clear_convention_says_so(tmp_path):
files = {f"m{i}.py": FULL + "\nimport os\n" for i in range(8)}
files["odd.py"] = "# Copyright 2026 Google LLC\nimport os\n"
root, rel = recipe(tmp_path, **files)
out = []
chr.check_license_headers(out, root, rel)
assert "differ from the 8" in out[0]["what"]
def test_two_competing_conventions_are_described_as_such(tmp_path):
"""PR #2373: 78 files use a one-line notice, 9 use Apache. Calling the 78
'truncated Apache headers' misdescribes what is actually there."""
files = {
f"short{i}.py": "# Copyright 2026 Google LLC\nimport os\n"
for i in range(8)
}
files.update({f"full{i}.py": FULL + "\nimport os\n" for i in range(2)})
root, rel = recipe(tmp_path, **files)
out = []
chr.check_license_headers(out, root, rel)
assert "two different headers" in out[0]["what"]
def test_no_apache_header_anywhere_is_reported_once(tmp_path):
files = {f"m{i}.py": "import os\n" for i in range(6)}
root, rel = recipe(tmp_path, **files)
out = []
chr.check_license_headers(out, root, rel)
assert len(out) == 1
assert "no .py file in this recipe" in out[0]["what"]
def test_config_formats_stay_consistency_only(tmp_path):
"""Required for source, consistency-only for config: 6 header-less .tf files
are the terraform convention, not six violations."""
root, rel = recipe(
tmp_path,
**{f"terraform/f{i}.tf": 'resource "x" {}\n' for i in range(6)},
)
out = []
chr.check_license_headers(out, root, rel)
assert out == []
def test_a_single_source_file_establishes_nothing(tmp_path):
root, rel = recipe(tmp_path, **{"only.py": "import os\n"})
out = []
chr.check_license_headers(out, root, rel)
assert out == []
# ------------------------------------------------------------------ H48
def manifest_with(tmp_path, ownership, name="my-recipe"):
return recipe(
tmp_path,
name=name,
**{
"manifest.yaml": "type: standalone\n"
'description: "A real description here"\n'
f"ownership:\n{ownership}"
},
)
def h48(out):
return [f for f in out if f["rule"] == "H48"]
def test_company_name_as_team_is_reported(tmp_path):
for value in (
'"google"',
'"Google LLC"',
'"Google Cloud"',
'"ADK Samples Team"',
'"n/a"',
'"me"',
'"TEAM"',
):
out = []
root, rel = manifest_with(
tmp_path / value.strip('"'), f' team: {value}\n poc: "someone"\n'
)
chr.check_manifest(out, root, rel, None)
assert h48(out), f"{value} should be flagged as a non-team"
def test_team_equal_to_the_poc_handle_is_reported(tmp_path):
out = []
root, rel = manifest_with(
tmp_path, ' team: "lspataroG"\n poc: "lspatarog"\n'
)
chr.check_manifest(out, root, rel, None)
assert "poc" in h48(out)[0]["what"]
assert "cannot be both" in h48(out)[0]["what"]
def test_team_equal_to_a_contributor_handle_is_reported(tmp_path):
out = []
root, rel = manifest_with(
tmp_path,
' team: "someone"\n poc: "other"\n contributors:\n - "someone"\n',
)
chr.check_manifest(out, root, rel, None)
assert h48(out), "a team matching a contributor handle should be flagged"
def test_real_team_names_already_in_the_repo_are_not_flagged(tmp_path):
"""Calibration. A 'that looks like a username' heuristic flags every one of
these, and each is a real owning team in google/adk-samples today."""
for value in (
'"DEE"',
'"octo"',
'"adk-kotlin"',
'"attenu-io"',
'"OpenEAGO"',
'"RobustAI"',
'"Verizon"',
"FDE/Blackbelt",
'"GS&I AI Apps and Platforms"',
'"Developer Evangelism & Engineering (DevRel) - Cloud AI"',
):
out = []
root, rel = manifest_with(
tmp_path / value.strip('"').replace("/", "-"),
f' team: {value}\n poc: "someone"\n',
)
chr.check_manifest(out, root, rel, None)
assert not h48(out), f"{value} is a real team and must not be flagged"
def test_placeholder_team_stays_h17_only(tmp_path):
"""Two comments on one line is noise; the placeholder is H17's finding."""
out = []
root, rel = manifest_with(
tmp_path,
' team: "TODO: Replace with your team name"\n poc: "someone"\n',
)
chr.check_manifest(out, root, rel, None)
assert "H17" in rules(out) and not h48(out)
def test_h48_survives_the_changed_file_filter(tmp_path, monkeypatch):
"""It is reported even when the PR did not touch manifest.yaml -- the whole
point of the rule is that nothing else ever catches the value."""
monkeypatch.setattr(chr, "CHANGED", {"contrib/python/my-recipe/agent.py"})
monkeypatch.setattr(chr, "NEW_RECIPE", False)
out = []
root, rel = manifest_with(tmp_path, ' team: "google"\n poc: "someone"\n')
chr.check_manifest(out, root, rel, None)
assert h48(out)
# ---------------------------------------------- the degraded (no pyyaml) parse
TEMPLATE_MANIFEST = """\
type: "standalone" # Options: [standalone | module]
status: "active" # Options: [active | inactive]
language: "python" # Options: [python | java | go | kotlin | typescript]
description: "A real description of a real recipe, long enough to pass."
# deployable: true # (optional) omit if false
ownership:
team: "google"
poc: "someone"
"""
FULL_SCHEMA = {
"properties": {
"type": {"enum": ["standalone", "module"]},
"status": {"enum": ["active", "inactive"]},
"language": {"enum": ["python", "java", "go", "kotlin", "typescript"]},
"description": {},
"ownership": {},
"deployable": {},
"license": {},
"tags": {},
"architecture": {},
"large": {},
},
"required": ["type", "status", "language", "description", "ownership"],
}
def test_inline_comments_do_not_fake_an_enum_violation(tmp_path, monkeypatch):
"""The manifest TEMPLATE ships every enum with a trailing `# Options: [...]`
comment. A fallback parser that keeps it reads the value as
'"standalone" # Options: [standalone | module]' and H19 reports three
CI-FAILs that are pure fiction. It did, on contrib/python/clause-agent."""
monkeypatch.setitem(
__import__("sys").modules, "yaml", None
) # force degraded
schema = tmp_path / "schema.json"
schema.write_text(json.dumps(FULL_SCHEMA))
root, rel = recipe(tmp_path, **{"manifest.yaml": TEMPLATE_MANIFEST})
out = []
chr.check_manifest(out, root, rel, str(schema))
assert "H19" not in rules(out), [f["what"] for f in out]
def test_degraded_parse_still_reads_the_ownership_block(tmp_path, monkeypatch):
"""Degrading must not silently disable H48 -- a checker that quietly skips
a rule is worse than no checker."""
monkeypatch.setitem(__import__("sys").modules, "yaml", None)
root, rel = recipe(tmp_path, **{"manifest.yaml": TEMPLATE_MANIFEST})
out = []
chr.check_manifest(out, root, rel, None)
assert h48(out) and 'is "google"' in h48(out)[0]["what"]
def test_a_commented_out_key_is_not_a_key(tmp_path):
"""`# deployable: true` in the template is documentation, not a declaration."""
assert (
chr._parse_manifest("# deployable: true\ntype: standalone\n")[0].get(
"deployable"
)
is None
)
def test_scalar_values_survive_a_hash_inside_them():
"""A `#` that is not preceded by whitespace is part of the value."""
assert chr._scalar_value("issue-#42 # a real comment") == "issue-#42"
assert chr._scalar_value('"quoted # hash" # comment') == "quoted # hash"
assert chr._scalar_value("plain value") == "plain value"
# --------------------------------- H39 H40 H41 H42 H43 H44 H47 (the silent gap)
def test_h39_a_stub_committed_as_a_real_value(tmp_path):
root, rel = recipe(
tmp_path,
**{
".env.example": "GOOGLE_CLOUD_PROJECT=your-project-id\nREAL_ONE=us-central1\n"
},
)
out = []
chr.check_text_wide(out, root, rel)
hits = [f for f in out if f["rule"] == "H39"]
assert len(hits) == 1 and "your-project-id" in hits[0]["what"]
def test_h39_does_not_fire_on_a_stub_shaped_substring(tmp_path):
"""A URL that CONTAINS example.com is a real value; the stub is the whole
value. A substring rule flags every docs link in every recipe."""
root, rel = recipe(
tmp_path,
**{
".env.example": "DOCS_URL=https://example.com/docs/getting-started\n"
"REGION=us-central1\nTIMEOUT=30\n"
},
)
out = []
chr.check_text_wide(out, root, rel)
assert not [f for f in out if f["rule"] == "H39"]
def test_h39_and_h14_never_both_fire_on_one_line(tmp_path):
root, rel = recipe(tmp_path, **{".env.example": "KEY=changeme\n"})
out = []
chr.check_text_wide(out, root, rel)
assert [f["rule"] for f in out] == ["H14"]
def test_h40_language_disagrees_with_the_path(tmp_path):
root = tmp_path
(root / "core" / "python" / "thing").mkdir(parents=True)
(root / "core" / "python" / "thing" / "manifest.yaml").write_text(
'type: standalone\nlanguage: "go"\n'
)
out = []
chr.check_layout(out, str(root), "core/python/thing", "thing")
hits = [f for f in out if f["rule"] == "H40"]
assert len(hits) == 1 and '"go"' in hits[0]["what"]
def test_h40_stays_quiet_when_the_path_has_no_language_segment(tmp_path):
"""core/<name>/ is a placement problem, not a language mismatch. Calling it
one describes the wrong defect to the author."""
root = tmp_path
(root / "core" / "thing").mkdir(parents=True)
(root / "core" / "thing" / "manifest.yaml").write_text(
'type: standalone\nlanguage: "python"\n'
)
out = []
chr.check_layout(out, str(root), "core/thing", "thing")
assert not [f for f in out if f["rule"] == "H40"]
def test_h41_and_h23_and_h47_never_double_report(tmp_path):
"""One misplaced skill, one comment -- not three saying the same thing."""
cases = {
"plugins/store-ops": "H41", # too shallow
"plugins/a/b/c": "H23", # too deep
"plugins/python/store-ops": "H47", # right depth, language folder
"plugins/retail/store-ops": None, # correct
}
for rel, expected in cases.items():
root = tmp_path / rel.replace("/", "_")
(root / rel).mkdir(parents=True)
out = []
chr.check_layout(out, str(root), rel, rel.rsplit("/", 1)[-1])
got = [f["rule"] for f in out if f["rule"] in ("H23", "H41", "H47")]
assert got == ([expected] if expected else []), f"{rel} -> {got}"
def test_h42_mixed_pr_is_reported_once_anchored_in_the_recipe(
tmp_path, monkeypatch
):
monkeypatch.setattr(
chr,
"CHANGED",
{
".agents/skills/some-skill/SKILL.md",
"contrib/python/my-recipe/manifest.yaml",
"contrib/python/my-recipe/agent.py",
},
)
out = []
root, rel = recipe(tmp_path)
chr.check_pr_shape(out, root, rel)
assert len(out) == 1
assert out[0]["rule"] == "H42"
assert out[0]["path"] == "contrib/python/my-recipe/manifest.yaml"
def test_h42_silent_on_a_skill_only_or_recipe_only_pr(tmp_path, monkeypatch):
root, rel = recipe(tmp_path)
for changed in (
{".agents/skills/x/SKILL.md"},
{"contrib/python/my-recipe/agent.py"},
None,
):
monkeypatch.setattr(chr, "CHANGED", changed)
out = []
chr.check_pr_shape(out, root, rel)
assert out == [], changed
def test_h43_only_reports_files_git_actually_tracks(tmp_path, monkeypatch):
"""An untracked .env is a developer who ran the recipe. Reporting it as
committed is the most alarming false positive this script could produce --
on the real repo the filesystem shows 17 and git tracks none."""
root, rel = recipe(
tmp_path, **{".env": "SECRET=x\n", ".env.example": "A=b\n"}
)
monkeypatch.setattr(chr, "_TRACKED", {})
monkeypatch.setattr(chr, "_tracked_files", lambda r, s: set())
out = []
chr.check_layout(out, root, rel, "my-recipe")
assert not [f for f in out if f["rule"] == "H43"]
monkeypatch.setattr(
chr,
"_tracked_files",
lambda r, s: {f"{rel}/.env", f"{rel}/.env.example"},
)
out = []
chr.check_layout(out, root, rel, "my-recipe")
hits = [f for f in out if f["rule"] == "H43"]
assert len(hits) == 1 and hits[0]["path"].endswith("/.env")
def test_h43_example_variants_are_not_real_env_files(tmp_path, monkeypatch):
root, rel = recipe(tmp_path)
monkeypatch.setattr(chr, "_TRACKED", {})
monkeypatch.setattr(
chr,
"_tracked_files",
lambda r, s: {
f"{rel}/.env.local.example",
f"{rel}/.env.test.sample",
f"{rel}/.env.template",
},
)
out = []
chr.check_layout(out, root, rel, "my-recipe")
assert not [f for f in out if f["rule"] == "H43"]
def test_h43_says_so_when_git_cannot_answer(tmp_path, monkeypatch):
monkeypatch.setattr(chr, "SKIPPED", [])
monkeypatch.setattr(chr, "_tracked_files", lambda r, s: None)
root, rel = recipe(tmp_path)
chr.check_layout([], root, rel, "my-recipe")
assert any(r == "H43" for r, _ in chr.SKIPPED)
def test_h44_pruned_directory_name(tmp_path, monkeypatch):
"""From git, not the filesystem: dist/ and build/ are exactly what a local
build leaves behind and .gitignore hides, and SKILL.md invokes this
checker against a developer's own tree."""
root, rel = recipe(tmp_path, **{"dist/thing.py": "x = 1\n"})
monkeypatch.setattr(chr, "_TRACKED", {})
monkeypatch.setattr(
chr, "_tracked_files", lambda r, s: {f"{rel}/dist/thing.py"}
)
out = []
chr.check_layout(out, root, rel, "my-recipe")
hits = [f for f in out if f["rule"] == "H44"]
assert len(hits) == 1 and hits[0]["path"].endswith("/dist")
def test_h44_ignores_an_untracked_build_directory(tmp_path, monkeypatch):
root, rel = recipe(tmp_path, **{"dist/thing.py": "x = 1\n"})
monkeypatch.setattr(chr, "_TRACKED", {})
monkeypatch.setattr(chr, "_tracked_files", lambda r, s: set())
out = []
chr.check_layout(out, root, rel, "my-recipe")
assert not [f for f in out if f["rule"] == "H44"]
def test_h43_reports_one_grouped_finding_not_one_per_file(
tmp_path, monkeypatch
):
"""H10 and H26 both group; a recipe with twelve stray files should not
collect twelve comments."""
root, rel = recipe(tmp_path)
monkeypatch.setattr(chr, "_TRACKED", {})
monkeypatch.setattr(
chr,
"_tracked_files",
lambda r, s: {f"{rel}/.env", f"{rel}/.DS_Store", f"{rel}/a.pyc"},
)
out = []
chr.check_layout(out, root, rel, "my-recipe")
hits = [f for f in out if f["rule"] == "H43"]
assert len(hits) == 1
assert "3 such file(s)" in hits[0]["what"]
def test_h43_does_not_call_every_pem_a_private_key(tmp_path, monkeypatch):
"""A public CA bundle is an ordinary committed file, and "you committed a
private key" is the most alarming thing this checker can say."""
root, rel = recipe(tmp_path)
monkeypatch.setattr(chr, "_TRACKED", {})
monkeypatch.setattr(
chr,
"_tracked_files",
lambda r, s: {
f"{rel}/certs/server-ca.pem",
f"{rel}/service-account-template.json",
f"{rel}/.vscode/launch.json",
},
)
out = []
chr.check_layout(out, root, rel, "my-recipe")
assert not [f for f in out if f["rule"] == "H43"]
def test_h43_still_catches_a_real_private_key(tmp_path, monkeypatch):
root, rel = recipe(tmp_path)
monkeypatch.setattr(chr, "_TRACKED", {})
monkeypatch.setattr(
chr, "_tracked_files", lambda r, s: {f"{rel}/server-key.pem"}
)
out = []
chr.check_layout(out, root, rel, "my-recipe")
assert [f for f in out if f["rule"] == "H43"]
def test_tracked_file_cache_is_per_recipe(tmp_path):
"""The CI lane checks several recipes in one process. A single cached set
answers the second recipe with the first recipe's files."""
chr._TRACKED.clear()
chr._TRACKED[("/root", "a")] = {"a/.env"}
chr._TRACKED[("/root", "b")] = set()
assert chr._tracked_files("/root", "b") == set()
assert chr._tracked_files("/root", "a") == {"a/.env"}
def test_words_that_name_real_teams_are_not_generic(tmp_path):
"""The trailing-noun stripper is why this list must stay short: "Cloud
Org" strips to "cloud", so a plausible team word on the list flags every
real team whose name ends in it. `DevRel` is the everyday short form of a
team that owns five recipes in this repository."""
for value in (
"DevRel",
"Community",
"Engineering",
"Cloud Org",
"Samples",
"Developer Relations",
"Public Sector",
):
out = []
root, rel = manifest_with(
tmp_path / value.replace(" ", "-"),
f' team: "{value}"\n poc: "someone"\n',
)
chr.check_manifest(out, root, rel, None)
assert not h48(out), f"{value} is a plausible real team"
def test_a_group_alias_is_an_owner(tmp_path):
"""A mailing list is the one value immune to the failure this rule exists
to prevent: it does not leave when a person does."""
out = []
root, rel = manifest_with(
tmp_path, ' team: "adk-samples-team@google.com"\n poc: "someone"\n'
)
chr.check_manifest(out, root, rel, None)
assert not h48(out)
def test_a_url_is_still_not_a_team(tmp_path):
out = []
root, rel = manifest_with(
tmp_path, ' team: "https://github.com/orgs/x/teams/y"\n poc: "a"\n'
)
chr.check_manifest(out, root, rel, None)
assert h48(out)
# ------------------------------------ crash inputs that erased a whole recipe
def _all_checks(root, rel, name="my-recipe"):
"""Every check the CI lane runs, in the order the lane runs them.
AGENTS.md is written first because check_text_wide skips its whole
directory walk without one -- and that walk is where the file-size and
symlink handling lives. A crash test that never reaches the walk passes
for no reason at all.
"""
Path(root, "AGENTS.md").write_text("Use gemini-3.5-flash instead.\n")
out = []
chr.check_pyproject(out, root, rel, name)
chr.check_uv_lock(out, root, rel, name, "")
chr.check_dotenv_bootstrap(out, root, rel)
chr.check_manifest(out, root, rel, None)
chr.check_readme(out, root, rel)
chr.check_layout(out, root, rel, name)
chr.check_text_wide(out, root, rel)
chr.check_env_defaults(out, root, rel)
chr.check_license_headers(out, root, rel)
return out
def test_a_dangling_symlink_does_not_erase_the_recipes_review(tmp_path):
"""git stores mode 120000 and never checks the target, so a dangling
symlink is trivially committable. os.path.getsize on it raised
FileNotFoundError, which the lane catches per recipe — so the whole
recipe went unreviewed and the PR was reported clean. One file evaded the
entire deterministic lane."""
root, rel = recipe(tmp_path, **{"manifest.yaml": "type: standalone\n"})
os.symlink("/nonexistent/target", Path(root) / rel / "dangling.txt")
findings = _all_checks(root, rel)
assert findings, "the recipe produced nothing at all"
def test_a_deep_expression_does_not_erase_the_recipes_review(tmp_path):
"""ast.parse raises RecursionError, not SyntaxError, on a generated
constant table — a plausible accident, not only an attack."""
root, rel = recipe(
tmp_path,
**{
"manifest.yaml": "type: standalone\n",
"generated.py": "x = " + "1+" * 60000 + "1\n",
},
)
assert _all_checks(root, rel), "the recipe produced nothing at all"
@pytest.mark.parametrize(
("name", "content"),
[
("manifest.yaml", "- a list\n- where a mapping belongs\n"),
("manifest.yaml", "just a string\n"),
],
)
def test_a_manifest_that_is_not_a_mapping_does_not_crash(
tmp_path, name, content
):
root, rel = recipe(tmp_path / content[:6].strip(), **{name: content})
assert isinstance(_all_checks(root, rel), list)
@pytest.mark.parametrize(
"pyproject",
[
'[tool.uv]\nindex = "https://pypi.org/simple"\n', # not array-of-tables
'project = "oops"\n', # not a table
'[project]\nname = "x"\n[build-system]\nrequires = "x"\n',
],
)
def test_a_mistyped_pyproject_does_not_crash_the_rule_that_catches_it(
tmp_path, pyproject
):
"""H5 exists to catch a malformed index. A malformed index killing the run
instead means the recipe is reported clean."""
root, rel = recipe(
tmp_path / str(abs(hash(pyproject)))[:6],
**{"manifest.yaml": "type: standalone\n", "pyproject.toml": pyproject},
)
assert isinstance(_all_checks(root, rel), list)
# ------------------------------------------- H21 is scoped by language, not path
@pytest.mark.parametrize(
("rel", "language", "expected", "forbidden"),
[
(
"contrib/typescript/ts-thing",
"typescript",
{"README.md", ".env.example", "package.json"},
{"pyproject.toml", "uv.lock"},
),
(
"contrib/go/go-thing",
"go",
{"README.md", ".env.example", "go.mod"},
{"pyproject.toml", "uv.lock"},
),
(
"contrib/python/py-thing",
"python",
{"README.md", ".env.example", "pyproject.toml", "uv.lock"},
set(),
),
(
"plugins/retail/store-ops",
"typescript",
{"README.md", ".env.example", "SKILL.md", "EVAL.yaml"},
{"pyproject.toml", "uv.lock"},
),
],
)
def test_h21_asks_each_language_for_its_own_files(
tmp_path, rel, language, expected, forbidden
):
"""It hardcoded the Python list for every recipe in every language, so a
new TypeScript, Go, Java or Kotlin recipe collected four confident
CI-FAIL comments demanding files it should never have. policy.yml has
scoped these under by_language all along."""
root = tmp_path
(root / rel).mkdir(parents=True)
(root / rel / "manifest.yaml").write_text(
f'type: standalone\nlanguage: "{language}"\n'
)
out = []
chr.check_layout(out, str(root), rel, rel.rsplit("/", 1)[-1])
asked = {f["path"].rsplit("/", 1)[-1] for f in out if f["rule"] == "H21"}
assert expected <= asked, f"{language}: missing {expected - asked}"
assert not (forbidden & asked), (
f"{language}: wrongly asked for {forbidden & asked}"
)
@pytest.mark.parametrize(
("language", "present", "missing_group"),
[
("java", "build.gradle", None),
("java", None, "pom.xml"),
("typescript", "yarn.lock", None),
("typescript", None, "package-lock.json"),
],
)
def test_h21_any_one_alternative_satisfies_the_group(
tmp_path, language, present, missing_group
):
"""A list entry in policy.yml means any ONE file will do. One file from
the group must silence H21 for it; none must raise a single finding
naming every alternative, not one per file."""
rel = f"contrib/{language}/thing"
(tmp_path / rel).mkdir(parents=True)
(tmp_path / rel / "manifest.yaml").write_text(
f'type: standalone\nlanguage: "{language}"\n'
)
if present:
(tmp_path / rel / present).write_text("")
out = []
chr.check_layout(out, str(tmp_path), rel, "thing")
group = [f for f in out if f["rule"] == "H21" and "any of" in f["what"]]
if missing_group is None:
assert group == []
else:
assert len(group) == 1
assert group[0]["path"].endswith("/" + missing_group)
def test_h21_alternative_under_always_does_not_crash(tmp_path, monkeypatch):
monkeypatch.setattr(
chr,
"_load_policy_required_files",
lambda root: {"always": ["README.md", ["LICENSE", "COPYING"]]},
)
rel = "contrib/go/thing"
(tmp_path / rel).mkdir(parents=True)
(tmp_path / rel / "manifest.yaml").write_text('language: "go"\n')
(tmp_path / rel / "COPYING").write_text("")
out = []
chr.check_layout(out, str(tmp_path), rel, "thing")
assert not [f for f in out if f["rule"] == "H21" and "any of" in f["what"]]
@pytest.mark.parametrize(
("label", "pyproject", "manifest"),
[
(
"requires-python unquoted",
'[project]\nname = "my-recipe"\nrequires-python = 3.11\n',
None,
),
(
"requires-python a list",
'[project]\nname = "my-recipe"\nrequires-python = [">=3.11"]\n',
None,
),
(
"dependency as a table",
'[project]\nname = "my-recipe"\ndependencies = [{name = "x"}]\n',
None,
),
(
"testpaths a number",
"[tool.pytest.ini_options]\ntestpaths = 3\n",
None,
),
(
"description a mapping",
None,
"type: standalone\ndescription:\n text: hello\n",
),
("description a number", None, "type: standalone\ndescription: 2026\n"),
],
)
def test_a_mistyped_value_never_erases_the_recipes_review(
tmp_path, label, pyproject, manifest
):
"""Every one of these is a realistic typo — forgetting quotes around
3.11 most of all — and every one of them raised out of the checker, which
the lane catches per recipe. The recipe then goes unreviewed and the PR is
reported clean, losing every other finding in it.
H4's own code path crashed on exactly the defect H4 exists to catch."""
files = {"manifest.yaml": manifest or "type: standalone\n"}
if pyproject:
files["pyproject.toml"] = pyproject
root, rel = recipe(tmp_path / label.replace(" ", "-"), **files)
findings = _all_checks(root, rel)
assert isinstance(findings, list), label
def test_an_unquoted_requires_python_is_reported_not_swallowed(tmp_path):
root, rel = recipe(
tmp_path,
**{
"manifest.yaml": "type: standalone\n",
"pyproject.toml": '[project]\nname = "my-recipe"\n'
"requires-python = 3.11\n",
},
)
out = []
chr.check_pyproject(out, root, rel, "my-recipe")
h4 = [f for f in out if f["rule"] == "H4"]
assert h4 and "not a string" in h4[0]["what"]
def test_h3_expects_the_namespaced_name_for_a_vertical_skill(tmp_path):
"""check_recipe_pyproject namespaces plugins/ as <vertical>-<solution>, so
comparing against the bare basename told the author of every vertical
plugin to set the one value CI would then reject. It fired on both shipped
plugins in this repo."""
rel = "plugins/retail/store-ops"
root = tmp_path
(root / rel).mkdir(parents=True)
(root / rel / "pyproject.toml").write_text(
'[project]\nname = "retail-store-ops"\n'
)
out = []
chr.check_pyproject(out, str(root), rel, "store-ops")
assert not [f for f in out if f["rule"] == "H3"], (
"the correct namespaced name was reported as wrong"
)
(root / rel / "pyproject.toml").write_text(
'[project]\nname = "store-ops"\n'
)
out = []
chr.check_pyproject(out, str(root), rel, "store-ops")
h3 = [f for f in out if f["rule"] == "H3"]
assert h3 and "retail-store-ops" in h3[0]["what"]
def test_h3_is_unchanged_for_an_ordinary_recipe(tmp_path):
rel = "contrib/python/thing"
(tmp_path / rel).mkdir(parents=True)
(tmp_path / rel / "pyproject.toml").write_text(
'[project]\nname = "thing"\n'
)
out = []
chr.check_pyproject(out, str(tmp_path), rel, "thing")
assert not [f for f in out if f["rule"] == "H3"]
def _case_sensitive_exists(monkeypatch):
"""Make os.path.exists case-sensitive for the duration of a test.
The CI runner is Linux; this developer machine is macOS, where
os.path.exists("EVAL.yaml") is already True when eval.yaml is on disk. A
test written against the local filesystem therefore passes whether the
fix is present or not, which is how the first version of this one shipped
green and useless.
"""
real = os.path.exists
def exists(path):
if not real(path):
return False
directory, base = os.path.split(path)
try:
return base in os.listdir(directory or ".")
except OSError:
return real(path)
monkeypatch.setattr(chr.os.path, "exists", exists)
def test_h21_accepts_a_lowercase_eval_yaml(tmp_path, monkeypatch):
"""policy.case_insensitive_files lists EVAL.yaml and validate_structure
honours it, so `eval.yaml` passes CI. A bare os.path.exists on a
case-sensitive runner reported it missing — a file that is right there."""
rel = "plugins/retail/store-ops"
(tmp_path / rel).mkdir(parents=True)
for name in ("README.md", "SKILL.md", "eval.yaml"):
(tmp_path / rel / name).write_text("x\n")
(tmp_path / rel / "manifest.yaml").write_text(
'type: standalone\nlanguage: "typescript"\n'
)
_case_sensitive_exists(monkeypatch)
out = []
chr.check_layout(out, str(tmp_path), rel, "store-ops")
missing = {f["path"].rsplit("/", 1)[-1] for f in out if f["rule"] == "H21"}
assert "EVAL.yaml" not in missing, f"still asked for EVAL.yaml: {missing}"
def test_h21_still_reports_a_genuinely_absent_eval_yaml(tmp_path, monkeypatch):
rel = "plugins/retail/no-eval"
(tmp_path / rel).mkdir(parents=True)
for name in ("README.md", "SKILL.md"):
(tmp_path / rel / name).write_text("x\n")
(tmp_path / rel / "manifest.yaml").write_text(
'type: standalone\nlanguage: "typescript"\n'
)
_case_sensitive_exists(monkeypatch)
out = []
chr.check_layout(out, str(tmp_path), rel, "no-eval")
missing = {f["path"].rsplit("/", 1)[-1] for f in out if f["rule"] == "H21"}
assert "EVAL.yaml" in missing
@pytest.mark.parametrize(
"key_line",
[
"on: true",
"no: something",
"2026: notes",
"1.0: notes",
"2026-01-01: notes",
"~: notes",
],
)
def test_a_manifest_key_yaml_reads_as_a_non_string_does_not_crash(
tmp_path, key_line
):
"""`on:` is YAML 1.1's Norway problem — it parses as the boolean True —
and a bare year is an int, a date is a date. re.escape on any of them
raises TypeError, which the lane catches per recipe: every finding for
that recipe is discarded and the PR reported clean. The key IS invalid
under additionalProperties: false, so the rule must say so."""
schema = tmp_path / "schema.json"
schema.write_text(json.dumps(FULL_SCHEMA))
root, rel = recipe(
tmp_path / key_line.split(":")[0].strip("~ "),
**{
"manifest.yaml": "type: standalone\nstatus: active\n"
'language: python\ndescription: "A real description here."\n'
f"{key_line}\n"
},
)
out = []
chr.check_manifest(out, root, rel, str(schema))
assert isinstance(out, list)
assert any(f["rule"] == "H19" for f in out), (
"the invalid key went unreported"
)
def test_h27_anchors_on_a_file_the_pr_changed(tmp_path, monkeypatch):
"""H27 counts the whole recipe, but _is_ours filters it on
`path in CHANGED`. Anchoring on the first offender in WALK order threw
the finding away unless that exact file happened to be in the diff — a
one-in-eight chance on a sixteen-file offender list, so the fix to the
arithmetic quietly traded a false positive for a false negative."""
files = {f"headed{i}.py": FULL + "\nimport os\n" for i in range(8)}
files.update({f"bare{i}.py": "import os\n" for i in range(3)})
root, rel = recipe(tmp_path, **files)
# The PR touches the LAST bare file, not the first in walk order.
changed = f"{rel}/bare2.py"
monkeypatch.setattr(chr, "CHANGED", {changed})
monkeypatch.setattr(chr, "FILTERED", [])
out = []
chr.check_license_headers(out, root, rel)
h27 = [f for f in out if f["rule"] == "H27"]
assert h27, "the finding was filtered away as pre-existing"
assert h27[0]["path"] == changed
assert "8" in h27[0]["what"], "the count is not the whole-recipe tally"
def test_h27_counts_the_whole_recipe_not_just_the_changed_files(
tmp_path, monkeypatch
):
"""The tally is a claim about the RECIPE's convention. Counting only
changed files made a PR touching two of the unheaded files report "no .py
file in this recipe carries the standard Apache header" while most of
them did — the filter chose the branch as well as the numbers."""
files = {f"headed{i}.py": FULL + "\nimport os\n" for i in range(9)}
files.update({f"bare{i}.py": "import os\n" for i in range(2)})
root, rel = recipe(tmp_path, **files)
monkeypatch.setattr(chr, "CHANGED", {f"{rel}/bare0.py", f"{rel}/bare1.py"})
monkeypatch.setattr(chr, "FILTERED", [])
out = []
chr.check_license_headers(out, root, rel)
h27 = [f for f in out if f["rule"] == "H27"]
assert h27
what = h27[0]["what"]
assert "9" in what, f"the 9 headed files were not counted: {what}"
assert "no .py file in this recipe" not in what
@pytest.mark.parametrize(
("label", "body"),
[
("a bare date", "2026-01-01\n"),
("a timestamp", "2026-01-01 10:00:00\n"),
("a set", "!!set\n? a\n? b\n"),
("binary", "!!binary |\n aGVsbG8=\n"),
("a list", "- a\n- b\n"),
("a scalar", "just a string\n"),
],
)
def test_any_non_mapping_manifest_is_skipped_not_fatal(tmp_path, label, body):
"""The guard was a blocklist of the types someone thought of — list, str,
int, float, bool — and yaml.safe_load also returns dates, timestamps, sets
and bytes. Each of those still reached .get() and discarded the recipe's
entire review. The question is "is it a mapping", so it asks that now."""
root, rel = recipe(
tmp_path / label.replace(" ", "-"), **{"manifest.yaml": body}
)
chr.SKIPPED = []
out = []
chr.check_manifest(out, root, rel, None)
assert any(r.startswith("H17") for r, _ in chr.SKIPPED), (
"the unreadable manifest was not reported as unchecked"
)
def test_an_absurdly_long_version_does_not_crash_the_checker(tmp_path):
"""int() on a string of more than 4300 digits raises, and this one is
reachable straight from PR content: a one-line way for a contributor to
make the deterministic lane skip their own recipe."""
root, rel = recipe(
tmp_path,
**{
"manifest.yaml": "type: standalone\n",
"pyproject.toml": '[project]\nname = "my-recipe"\n'
'requires-python = ">=3.' + "9" * 5000 + '"\n',
},
)
out = []
chr.check_pyproject(out, root, rel, "my-recipe")
assert isinstance(out, list)
@pytest.mark.parametrize(
"team", ["チーム", "Команда", "大数据平台组", "Ελλάδα", "Équipe Données"]
)
def test_a_team_name_in_a_non_latin_script_is_a_team(tmp_path, team):
"""`[^A-Za-z0-9]+` is ASCII-only, so any name written wholly in another
script matched "names nobody" — the exact false positive the closed
GENERIC_TEAMS list exists to avoid."""
out = []
root, rel = manifest_with(
tmp_path / str(abs(hash(team)))[:6],
f' team: "{team}"\n poc: "someone"\n',
)
chr.check_manifest(out, root, rel, None)
assert not h48(out), f"{team} was reported as naming nobody"
@pytest.mark.parametrize("team", ["---", "!!", "..."])
def test_a_punctuation_only_team_still_names_nobody(tmp_path, team):
out = []
root, rel = manifest_with(
tmp_path / str(abs(hash(team)))[:6],
f' team: "{team}"\n poc: "someone"\n',
)
chr.check_manifest(out, root, rel, None)
assert h48(out)
def test_h10_does_not_flag_documentation_that_forbids_the_model(tmp_path):
"""Two recipes in this repo carry "don't use deprecated ones
(gemini-2.0-flash, gemini-2.5-flash)" in their own AGENTS.md. Reporting
it is a confidently wrong comment from the one lane whose reason for
existing is that it cannot produce one. The existing tests wrote
AGENTS.md at the repo root, outside the scanned subtree, which is why
they never saw it."""
root, rel = recipe(
tmp_path,
**{
"AGENTS.md": "Do NOT use gemini-2.0-flash or gemini-2.5-flash — "
"both are deprecated.\n",
},
)
Path(root, "AGENTS.md").write_text("Use gemini-3.5-flash instead.\n")
out = []
chr.check_text_wide(out, root, rel)
assert not [f for f in out if f["rule"] == "H10"]
def test_h10_still_flags_a_real_use(tmp_path):
root, rel = recipe(tmp_path, **{"agent.py": 'MODEL = "gemini-2.5-flash"\n'})
Path(root, "AGENTS.md").write_text("Use gemini-3.5-flash instead.\n")
out = []
chr.check_text_wide(out, root, rel)
assert [f for f in out if f["rule"] == "H10"]
def test_h10_anchors_on_a_file_the_pr_changed(tmp_path, monkeypatch):
"""It anchored on the first hit in walk order, and _is_ours then dropped
the finding unless the PR happened to touch that exact file — which a
recipe's own docs mentioning the ids made routine."""
# zzz FIRST: os.walk yields filesystem order, which here is creation
# order, so an unsorted implementation anchors on the changed file by
# luck and the test cannot tell the difference. Sorted, aaa comes first
# and the anchor has to be chosen deliberately.
root, rel = recipe(
tmp_path,
**{
"zzz_changed.py": 'MODEL = "gemini-2.5-flash"\n',
"aaa_first.py": 'FALLBACK = "gemini-2.0-flash"\n',
},
)
Path(root, "AGENTS.md").write_text("Use gemini-3.5-flash instead.\n")
# BOTH changed, so `_anchor_in_diff` cannot decide it: what is left is
# the order of `hits`, and os.walk's order is the filesystem's. Sorted,
# the same pull request always produces the same comment.
monkeypatch.setattr(
chr, "CHANGED", {f"{rel}/zzz_changed.py", f"{rel}/aaa_first.py"}
)
monkeypatch.setattr(chr, "FILTERED", [])
out = []
chr.check_text_wide(out, root, rel)
h10 = [f for f in out if f["rule"] == "H10"]
assert h10, "the finding was filtered away as pre-existing"
assert h10[0]["path"].endswith("aaa_first.py"), (
f"anchored on {h10[0]['path']} — the order os.walk happened to give"
)
def test_h10_anchors_on_a_changed_file_when_only_one_changed(
tmp_path, monkeypatch
):
root, rel = recipe(
tmp_path,
**{
"aaa_first.py": 'FALLBACK = "gemini-2.0-flash"\n',
"zzz_changed.py": 'MODEL = "gemini-2.5-flash"\n',
},
)
Path(root, "AGENTS.md").write_text("Use gemini-3.5-flash instead.\n")
monkeypatch.setattr(chr, "CHANGED", {f"{rel}/zzz_changed.py"})
monkeypatch.setattr(chr, "FILTERED", [])
out = []
chr.check_text_wide(out, root, rel)
h10 = [f for f in out if f["rule"] == "H10"]
assert h10 and h10[0]["path"].endswith("zzz_changed.py")
@pytest.mark.parametrize(
"model",
[
"gemini-2.5-flash-image",
"gemini-2.5-flash-lite",
"gemini-2.5-flash-preview",
],
)
def test_h10_does_not_flag_a_current_model_by_prefix(tmp_path, model):
"""`gemini-2.5-flash-image` is a CURRENT model and the pattern had no
right-hand boundary, so it matched the prefix: plugins/retail/virtual-tryon
carries 21 occurrences and not one deprecated id, and any PR touching it
was told to replace a correct image model with a text one."""
root, rel = recipe(tmp_path / model, **{"agent.py": f'MODEL = "{model}"\n'})
Path(root, "AGENTS.md").write_text("Use gemini-3.5-flash instead.\n")
out = []
chr.check_text_wide(out, root, rel)
assert not [f for f in out if f["rule"] == "H10"], f"{model} was flagged"
def test_h10_still_flags_a_pinned_deprecated_version(tmp_path):
"""A trailing `-001` is a version pin of the deprecated model itself; a
trailing letter starts a different model name."""
root, rel = recipe(
tmp_path, **{"agent.py": 'MODEL = "gemini-2.5-flash-001"\n'}
)
Path(root, "AGENTS.md").write_text("Use gemini-3.5-flash instead.\n")
out = []
chr.check_text_wide(out, root, rel)
assert [f for f in out if f["rule"] == "H10"]
def test_whole_recipe_rules_are_also_scoped_to_their_own_directory():
"""H22/H23 sit in WHOLE_RECIPE_RULES and DIRECTORY_RULES both, and the
two conditions are independent. `_is_ours` returned on the first set, so
the directory branch was unreachable for them: a PR that ADDS recipe A
set NEW_RECIPE=True, and every OTHER recipe the walker reached -- none of
whose files the PR touches at all -- collected CI-FAILs for a folder name
and a skill depth that predate the PR by years."""
added = "contrib/python/new-recipe"
untouched = "contrib/python/old-recipe"
chr.CHANGED = {f"{added}/manifest.yaml", f"{added}/agent.py"}
chr.NEW_RECIPE = True
try:
for rule in ("H22", "H23"):
assert chr._is_ours(rule, added), (
f"{rule} must still fire on the recipe the PR is adding"
)
out = []
chr.find(out, rule, chr.CI_FAIL, untouched, 1, "w", "e", "v")
assert out == [], (
f"{rule} fired on {untouched}, which the PR never touches"
)
finally:
chr.CHANGED = None
chr.NEW_RECIPE = True
def test_whole_recipe_rules_stay_off_an_edit_only_pr():
"""The original guard must survive: NEW_RECIPE=False silences them even
though every file under the recipe is in CHANGED."""
rel = "contrib/python/old-recipe"
chr.CHANGED = {f"{rel}/agent.py"}
chr.NEW_RECIPE = False
try:
for rule in ("H21", "H22", "H23"):
assert not chr._is_ours(rule, rel)
finally:
chr.CHANGED = None
chr.NEW_RECIPE = True
def test_h21_is_not_directory_scoped():
"""H21 is whole-recipe only. It must not acquire a directory condition
from the restructure -- required files can be missing from a recipe whose
own directory has no changed file under the exact prefix."""
chr.CHANGED = {"contrib/python/new-recipe/agent.py"}
chr.NEW_RECIPE = True
try:
assert chr._is_ours("H21", "contrib/python/other-recipe")
finally:
chr.CHANGED = None
chr.NEW_RECIPE = True
def test_parse_manifest_tolerates_empty_and_missing_text():
"""`text.split` on None is an AttributeError that takes the whole run
down. Both callers guard today; the parser must not depend on that."""
assert chr._parse_manifest(None) == ({}, False)
assert chr._parse_manifest("") == ({}, False)
tests/test_house_rules_drift.py
"""Pin .github/review-rules.md and check_house_rules.py to the repo's real validators.
check_house_rules.py reimplements logic that lives in six places under .github/
and tools/. Nothing keeps them in step. When the repo changes a rule, this script
keeps confidently reporting the old one -- and it is the component most likely to
be believed, because it is a script rather than a model.
These tests read the ACTUAL repo files and fail when they diverge. They are
deliberately loose: they assert the anchor facts each rule depends on, not exact
source text, so ordinary refactors don't trip them.
Skipped automatically when run outside an adk-samples checkout.
"""
import json
import os
import re
from pathlib import Path
import pytest
def _find_repo():
"""Locate an adk-samples checkout, wherever the skill happens to be installed.
Works when the skill lives inside the checkout (.agents/skills/<name>/tests) and
when it lives anywhere else, via ADK_SAMPLES_ROOT. Returns None if neither, and
the whole module then skips -- these tests are only meaningful against the real
repo, and a teammate running the suite outside one should not see failures.
"""
env = os.environ.get("ADK_SAMPLES_ROOT")
if env and (Path(env) / "AGENTS.md").exists():
return Path(env)
for parent in Path(__file__).resolve().parents:
if (parent / "AGENTS.md").exists() and (parent / ".github").is_dir():
return parent
return None
REPO = _find_repo()
RULES_MD = (REPO / ".github" / "review-rules.md") if REPO else None
pytestmark = pytest.mark.skipif(
REPO is None,
reason="no adk-samples checkout found; set ADK_SAMPLES_ROOT to run these",
)
def read(rel):
p = REPO / rel
if not p.exists():
pytest.skip(f"{rel} not present in this checkout")
return p.read_text(encoding="utf-8", errors="replace")
# ------------------------------------------------------------- H3 / H4 / H5
def test_h4_python_311_is_still_the_minimum():
"""H4 hardcodes 3.11. If the repo bumps it, the rule and script are wrong."""
agents = read("AGENTS.md")
assert re.search(r"[Mm]inimum python version:\s*3\.11", agents), (
"AGENTS.md no longer says 3.11 -- update H4 and check_house_rules.check_pyproject"
)
def test_h3_h4_h5_validator_still_exists():
"""The three CI-FAIL pyproject rules all cite this script."""
src = read(".github/scripts/check_recipe_pyproject.py")
for marker in ("name", "requires-python", "index"):
assert marker in src, (
f"check_recipe_pyproject.py no longer mentions {marker}"
)
# -------------------------------------------------------------------- H1 / H2
def test_h1_h2_ruff_config_ban_is_still_repo_policy():
agents = read("AGENTS.md")
assert "ruff.toml" in agents and "root" in agents.lower(), (
"AGENTS.md no longer bans standalone ruff config -- recheck H1/H2"
)
# ------------------------------------------------------------------------ H10
def test_h10_deprecated_model_ids_match_agents_md():
"""The banned list and the replacement both come from AGENTS.md."""
agents = read("AGENTS.md")
rules = RULES_MD.read_text(encoding="utf-8")
for model in ("gemini-2.0-flash", "gemini-2.5-flash"):
assert model in agents, (
f"{model} no longer listed as deprecated in AGENTS.md"
)
assert model in rules, (
f"{model} missing from .github/review-rules.md H10"
)
m = re.search(r"[Uu]se\s+`?(gemini-[\w.\-]+)`?\s+instead", agents)
assert m, "AGENTS.md no longer names a replacement model"
assert m.group(1) in rules, (
f"AGENTS.md now recommends {m.group(1)}; .github/review-rules.md H10 says otherwise"
)
# ------------------------------------------------------------------------ H19
def test_h19_manifest_schema_keys_are_current():
"""H19 validates against the live schema, but the doc lists enums inline."""
schema = json.loads(read(".github/schemas/manifest-schema.json"))
rules = RULES_MD.read_text(encoding="utf-8")
assert schema.get("additionalProperties") is False, (
"schema no longer forbids extra keys -- H19's premise is gone"
)
for key in schema.get("required", []):
assert key in rules, f"required manifest key {key!r} missing from H19"
def test_h19_enums_in_the_doc_match_the_schema():
schema = json.loads(read(".github/schemas/manifest-schema.json"))
rules = RULES_MD.read_text(encoding="utf-8")
for field in ("type", "status", "language"):
spec = schema.get("properties", {}).get(field, {})
for value in spec.get("enum", []):
assert value in rules, (
f"{field} enum value {value!r} is in the schema but not in H19"
)
# ------------------------------------------------------------------------ H21
def test_h21_required_files_match_policy_yml():
text = read(".github/policy.yml")
for required in (
"README.md",
"pyproject.toml",
"uv.lock",
".env.example",
"test_runnability.py",
):
assert required in text, (
f"{required} no longer in policy.yml -- H21's list is stale"
)
def test_h22_folder_name_limit_matches_policy():
text = read(".github/policy.yml")
m = re.search(r"max_folder_name_length:\s*(\d+)", text)
if not m:
pytest.skip("policy.yml no longer declares max_folder_name_length")
limit = int(m.group(1))
assert limit == 30, (
f"policy.yml says {limit}; check_house_rules.check_layout hardcodes 30"
)
# ------------------------------------------------------------------------ H24
def test_h24_frozen_paths_are_still_frozen():
text = read(".github/policy.yml")
assert "frozen_paths" in text, (
"policy.yml dropped frozen_paths -- H24 is obsolete"
)
assert "python/agents" in text, (
"python/agents no longer frozen -- recheck H24"
)
# ---------------------------------------------------------------- doc hygiene
def test_every_rule_declares_ci_fail_or_advisory():
"""Getting this wrong produces the most expensive comment the skill can make."""
rules = RULES_MD.read_text(encoding="utf-8")
reportable = rules.split("### Report these")[1].split(
"### Already enforced"
)[0]
found = re.findall(
r"\*\*(H\d+)\*\*\s*·\s*\*\*(CI-FAIL|advisory)\*\*", reportable
)
ids = re.findall(r"^\s*\d+\.\s+\*\*(H\d+)\*\*", reportable, re.M)
assert ids, "no rules found in the 'Report these' section"
tagged = {r for r, _ in found}
missing = [r for r in ids if r not in tagged]
assert not missing, (
f"these reportable rules do not declare CI-FAIL or advisory: {missing}"
)
# ------------------------------------------------------- rule coverage
def _reportable_rule_ids():
"""Every H-id in the 'Report these' section of the rules doc."""
text = RULES_MD.read_text(encoding="utf-8")
section = text.split("### Report these")[1].split("### Already enforced")[0]
return sorted(
set(re.findall(r"\*\*(H\d+)\*\*", section)), key=lambda r: int(r[1:])
)
def _implemented_rule_ids():
src = (
Path(__file__).parent.parent / "scripts" / "check_house_rules.py"
).read_text(encoding="utf-8")
return set(re.findall(r'find\(\s*out,\s*"(H\d+)"', src))
def test_every_reportable_rule_is_implemented_or_declared_model_judged():
"""The gap this test exists to close.
Before it, H39 H40 H41 H42 H43 H44 and H47 were all listed as reportable in
.github/review-rules.md, implemented nowhere, and absent from the script's
own list of rules it deliberately skips. Nothing checked them and nothing
said so -- the worst of the three states, because the script's silence reads
as "no violations".
"""
import check_house_rules
covered = _implemented_rule_ids() | set(check_house_rules.MODEL_JUDGED)
missing = [r for r in _reportable_rule_ids() if r not in covered]
assert not missing, (
f"reportable but unchecked: {missing}. Implement each in "
"check_house_rules.py, or add it to MODEL_JUDGED with the reason a "
"script gets it wrong."
)
def test_model_judged_rules_are_actually_in_the_rules_doc():
"""The reverse drift: a rule retired from the doc but still excused here."""
import check_house_rules
reportable = set(_reportable_rule_ids())
stale = [r for r in check_house_rules.MODEL_JUDGED if r not in reportable]
assert not stale, (
f"MODEL_JUDGED excuses {stale}, which the rules doc no longer asks "
"anyone to report"
)
def test_no_rule_is_both_implemented_and_excused():
import check_house_rules
both = _implemented_rule_ids() & set(check_house_rules.MODEL_JUDGED)
assert not both, (
f"{sorted(both)} are implemented AND listed as model-judged; the lane "
"will report each of them twice"
)
tests/test_plan_review.py
"""Planner behaviours that are load-bearing and were previously undocumented."""
import plan_review as pr
def f(path, churn, status="added"):
return {"path": path, "churn": churn, "status": status}
# ------------------------------------------------- affinity packing
def test_source_and_its_test_land_in_the_same_lane():
"""Split across lanes, neither reviewer can tell if the test covers the code."""
files = [
f("tools/validate.py", 100),
f("tools/tests/test_validate.py", 100),
f("other/a.py", 100),
f("other/b.py", 100),
]
lanes = pr.pack(files, 2)
for lane in lanes:
has_src = "tools/validate.py" in lane["files"]
has_test = "tools/tests/test_validate.py" in lane["files"]
assert has_src == has_test
def test_affinity_key_pairs_common_test_layouts():
pairs = [
("tools/validate.py", "tools/tests/test_validate.py"),
("src/api/client.ts", "src/api/__tests__/client.test.ts"),
("a/scripts/align.py", "a/tests/test_align.py"),
("pkg/thing.go", "pkg/thing_test.go"),
]
for src, test in pairs:
assert pr.affinity_key(src) == pr.affinity_key(test), (src, test)
def test_affinity_key_keeps_unrelated_files_apart():
assert pr.affinity_key("go/README.md") != pr.affinity_key("java/README.md")
def test_lanes_stay_balanced_despite_grouping():
files = [f(f"m{i}.py", 100) for i in range(10)]
lanes = pr.pack(files, 5)
churns = [lane["churn"] for lane in lanes]
assert max(churns) - min(churns) <= 100
def test_oversized_group_is_split_rather_than_wrecking_balance():
files = [f("big.py", 5000), f("tests/test_big.py", 5000)] + [
f(f"s{i}.py", 10) for i in range(4)
]
lanes = pr.pack(files, 3)
assert len([lane for lane in lanes if lane["files"]]) >= 2
# ------------------------------------------------------ skip rules
def test_pure_rename_is_skipped():
"""65 of 123 files on PR #2373. Byte-identical moves have nothing to review."""
plan_files = [f("a.py", 0, "renamed"), f("b.py", 10, "added")]
reviewable = [
x
for x in plan_files
if not (x["status"] == "renamed" and x["churn"] == 0)
]
assert [x["path"] for x in reviewable] == ["b.py"]
def test_renamed_with_edits_is_still_reviewed():
x = f("a.py", 6, "renamed")
assert not (x["status"] == "renamed" and x["churn"] == 0)
def test_lockfiles_and_generated_are_skipped():
for path in (
"pnpm-lock.yaml",
"go.sum",
"vendor/x.go",
"dist/a.js",
"x_pb2.py",
"api.png",
"__snapshots__/a.snap",
):
assert pr.skip_reason(path, 10) is not None, path
def test_real_source_is_not_skipped():
assert pr.skip_reason("src/agent.py", 100) is None
def test_bulk_data_skipped_only_when_large():
assert pr.skip_reason("data/x.json", 900) == "bulk data"
assert pr.skip_reason("data/x.json", 10) is None
def test_scope_flags_exclude_tests_and_web():
assert (
pr.skip_reason("a/tests/x.py", 10, include_tests=False)
== "tests excluded"
)
assert pr.skip_reason("a/tests/x.py", 10, include_tests=True) is None
assert (
pr.skip_reason("web/src/a.ts", 10, include_web=False) == "web excluded"
)
# --------------------------------------------------------- budget
def test_budget_uses_reviewable_churn_not_total():
"""A PR that is 4,800 lines of lockfile plus 2 real lines is a small PR."""
assert pr.budget_for(2) == (2, 3)
assert pr.budget_for(120) == (3, 5)
assert pr.budget_for(3000) == (12, 20)
def test_budget_never_exceeds_twenty():
assert pr.budget_for(10**6)[1] == 20
# ----------------------------------------------------- lane count
def test_small_pr_does_not_fan_out():
assert pr.lane_count(300, 5) == 1
def test_large_pr_fans_out_but_is_capped():
assert pr.lane_count(100_000, 500) == pr.MAX_LANES
def test_lane_count_never_exceeds_file_count():
assert pr.lane_count(50_000, 3) <= 3
tests/test_post_comments.py
"""The posting gate. Every failure here reaches a colleague's PR."""
import post_comments as pc
PATCH = "@@ -1,3 +1,5 @@\n ctx1\n+added2\n+added3\n ctx4\n-removed\n"
def test_commentable_lines_are_added_plus_context():
right, _ = pc.commentable_lines(PATCH)
assert right == {1, 2, 3, 4}
def test_deleted_lines_are_left_side_only():
"""A deleted line is addressable, but only with side=LEFT."""
right, left = pc.commentable_lines(PATCH)
assert 3 in left # the removed line, old numbering
assert 5 not in right
def test_empty_patch_yields_nothing():
assert pc.commentable_lines("") == (set(), set())
def test_multi_hunk_patch_tracks_both_ranges():
patch = "@@ -1,1 +1,1 @@\n ctx\n@@ -50,2 +50,2 @@\n ctx50\n+new51\n"
right, _ = pc.commentable_lines(patch)
assert {1, 50, 51} <= right
assert 25 not in right
def test_no_newline_marker_does_not_advance_numbering():
right, _ = pc.commentable_lines(
"@@ -1,1 +1,1 @@\n+a\n\\ No newline at end of file\n"
)
assert right == {1}
def test_state_roundtrip_enables_resume(tmp_path):
p = tmp_path / "c.json.posted"
pc.save_state(str(p), {"posted": ["a.py:1"]})
assert pc.load_state(str(p))["posted"] == ["a.py:1"]
def test_missing_state_file_starts_empty(tmp_path):
assert pc.load_state(str(tmp_path / "nope"))["posted"] == []
def test_key_of_is_stable_and_distinguishes_lines():
a = {"path": "x.py", "line": 1, "body": "b"}
b = {"path": "x.py", "line": 2, "body": "b"}
assert pc.key_of(a) == pc.key_of(dict(a))
assert pc.key_of(a) != pc.key_of(b)
tests/test_regression_pr2373.py
"""The fact-anchoring rule, held to real accept/reject decisions.
This is the skill's only ground truth about what the user will actually post:
20 drafted comments on google/adk-samples#2373, 16 posted and 4 cut.
The rule itself is a judgement an agent applies. What CAN be tested is the
`fact_anchor_lint` heuristic that approximates it — so if someone loosens the lint,
or the rule drifts, these fail instead of the skill quietly getting worse.
The fixture is small and the rule was fitted to it, so a perfect score here is
expected rather than impressive. Its value is as a *ratchet*: it can only get worse
silently if someone edits the fixture too.
"""
import json
from pathlib import Path
import pytest
import verify_findings as vf
FIXTURE = json.loads(
(Path(__file__).parent / "fixtures" / "pr2373_outcomes.json").read_text()
)
COMMENTS = FIXTURE["comments"]
CUT = [c for c in COMMENTS if c["verdict"] == "cut"]
KEPT = [c for c in COMMENTS if c["verdict"] == "keep"]
def test_fixture_matches_the_recorded_outcome():
assert len(COMMENTS) == 20
assert len(KEPT) == 16 and len(CUT) == 4
@pytest.mark.parametrize("c", CUT, ids=lambda c: f"cut-{c['n']}")
def test_every_cut_comment_has_a_recorded_rationale(c):
"""A cut with no reason teaches nothing."""
assert c["rationale"] and "anchors on" not in c["rationale"]
def test_lint_flags_most_of_what_was_cut():
"""The heuristic should catch the rejected shapes, not just claim to."""
flagged = [c for c in CUT if vf.fact_anchor_lint(c)]
assert len(flagged) == 4, (
f"lint caught {len(flagged)}/4 cut comments, expected 4: "
f"{[c['n'] for c in CUT if not vf.fact_anchor_lint(c)]}"
)
def test_lint_is_quiet_on_most_of_what_was_kept():
"""False positives are the real cost — they suppress comments you wanted."""
noisy = [c for c in KEPT if vf.fact_anchor_lint(c)]
assert len(noisy) <= 1, (
f"lint fires on {len(noisy)}/{len(KEPT)} accepted comments: "
f"{[c['n'] for c in noisy]} — false positives suppress wanted comments"
)
def test_lint_separates_cut_from_kept_better_than_chance():
cut_rate = sum(1 for c in CUT if vf.fact_anchor_lint(c)) / len(CUT)
keep_rate = sum(1 for c in KEPT if vf.fact_anchor_lint(c)) / len(KEPT)
assert cut_rate > keep_rate, (
f"lint fires on {cut_rate:.0%} of cut vs {keep_rate:.0%} of kept — "
"it is not discriminating"
)
def test_every_fixture_comment_has_a_window():
"""Without a window the lint cannot check identifiers, and the user cannot
settle the comment from the table."""
missing = [c["n"] for c in COMMENTS if not c.get("window")]
assert not missing, f"comments without a window: {missing}"
def test_the_four_cuts_are_the_expected_ones():
"""Pinned so a fixture regeneration cannot silently rewrite history."""
assert {c["n"] for c in CUT} == {3, 10, 17, 20}
tests/test_rejections.py
"""The rejection ledger: don't re-propose what the user already cut.
Suppression of posted comments cannot see a cut comment -- it was never posted, so
GitHub has no record of it. Observed on the PR #2373 re-review: three findings the
user had already rejected came back.
"""
import pytest
import rejections
import verify_findings as vf
@pytest.fixture(autouse=True)
def isolated_ledger(tmp_path, monkeypatch):
monkeypatch.setenv("GH_PR_REVIEW_STATE", str(tmp_path))
def c(path="src/a.py", line=2, comment="something"):
return {"path": path, "line": line, "comment": comment}
def test_record_then_load_roundtrip():
rejections.record("o/r", "1", [c()])
got = rejections.load("o/r", "1")
assert len(got) == 1 and got[0]["path"] == "src/a.py"
def test_recording_is_idempotent():
rejections.record("o/r", "1", [c()])
added, _ = rejections.record("o/r", "1", [c()])
assert added == 0
assert len(rejections.load("o/r", "1")) == 1
def test_ledgers_are_per_pr():
rejections.record("o/r", "1", [c()])
assert rejections.load("o/r", "2") == []
def test_ledgers_are_per_repo():
rejections.record("o/r", "1", [c()])
assert rejections.load("other/repo", "1") == []
def test_missing_ledger_is_empty_not_an_error():
assert rejections.load("never/seen", "99") == []
def test_corrupt_ledger_degrades_quietly(tmp_path):
p = rejections.ledger_path("o/r", "1")
p.parent.mkdir(parents=True, exist_ok=True)
p.write_text("{ not json")
assert rejections.load("o/r", "1") == []
def test_entries_carry_a_timestamp():
rejections.record("o/r", "1", [c()])
assert rejections.load("o/r", "1")[0]["rejected_at"].endswith("Z")
# ------------------------------------------------- suppression on re-review
def write(tmp_path, rel, text):
p = tmp_path / rel
p.parent.mkdir(parents=True, exist_ok=True)
p.write_text(text)
return p
def finding(**kw):
base = {
"path": "src/a.py",
"line": 2,
"what": "x",
"comment": "something",
"verify_steps": "read line 2",
"window": " 2: two",
}
base.update(kw)
return base
def test_a_cut_comment_does_not_come_back(tmp_path):
write(tmp_path, "src/a.py", "one\ntwo\nthree\n")
prior = vf.rejected_as_exclusions([c()])
v, _, sup = vf.verify([finding()], str(tmp_path), {"src/a.py": {2}}, prior)
assert not v
assert "you cut this comment on a previous review" in sup[0]["_reason"]
def test_a_cut_comment_blocks_nearby_lines_too(tmp_path):
write(tmp_path, "src/a.py", "\n".join(f"l{i}" for i in range(10)))
prior = vf.rejected_as_exclusions([c(line=2)])
v, _, sup = vf.verify(
[finding(line=4, window=" 4: l3")],
str(tmp_path),
{"src/a.py": {4}},
prior,
)
assert not v and sup
def test_a_rejection_never_expires_as_outdated(tmp_path):
"""A decision the user made does not lapse because the code moved."""
write(tmp_path, "src/a.py", "one\ntwo\n")
entry = vf.rejected_as_exclusions([c()])[0]
assert entry["outdated"] is False
def test_unrelated_finding_survives_the_ledger(tmp_path):
write(tmp_path, "src/a.py", "\n".join(f"l{i}" for i in range(30)))
prior = vf.rejected_as_exclusions(
[c(line=2, comment="licence header truncated")]
)
f = finding(
line=25, what="ipv4_enabled is true on the database", window=" 25: l24"
)
v, _, sup = vf.verify([f], str(tmp_path), {"src/a.py": {25}}, prior)
assert len(v) == 1 and not sup
def test_empty_ledger_suppresses_nothing(tmp_path):
write(tmp_path, "src/a.py", "one\ntwo\n")
v, _, sup = vf.verify(
[finding()],
str(tmp_path),
{"src/a.py": {2}},
vf.rejected_as_exclusions([]),
)
assert len(v) == 1 and not sup
tests/test_verify_findings.py
"""verify_findings is the control that replaces my manual diligence.
Every test here is a mistake actually made during a real review.
"""
import pytest
import verify_findings as vf
def write(tmp_path, rel, text):
p = tmp_path / rel
p.parent.mkdir(parents=True, exist_ok=True)
p.write_text(text, encoding="utf-8")
return p
def finding(**kw):
base = {
"path": "src/a.py",
"line": 2,
"what": "something",
"cheap": "cheap",
"verify_steps": "read line 2",
"window": "",
}
base.update(kw)
return base
# --------------------------------------------------------------- window check
def test_matching_window_verifies(tmp_path):
write(tmp_path, "src/a.py", "one\ntwo\nthree\n")
f = finding(window=" 1: one\n 2: two\n 3: three")
ok, _ = vf.check_window(f, str(tmp_path))
assert ok
def test_fabricated_window_is_rejected(tmp_path):
"""A lane that invents a finding invents the source line with it."""
write(tmp_path, "src/a.py", "one\ntwo\nthree\n")
f = finding(window=" 2: os.system(user_input)")
ok, reason = vf.check_window(f, str(tmp_path))
assert not ok
assert "window line 2" in reason
def test_emoji_escape_is_not_a_fabrication(tmp_path):
"""Regression: three false rejections on PR #2373.
The lane writes the escape text; the file holds the real character. Decoding
via unicode_escape mojibakes the file side and flags every emoji line.
"""
write(
tmp_path, "src/a.py", 'x = 1\nprint("\u26a0\ufe0f [Quota] exhausted")\n'
)
f = finding(line=2, window=r' 2: print("\u26a0\ufe0f [Quota] exhausted")')
ok, reason = vf.check_window(f, str(tmp_path))
assert ok, reason
def test_line_outside_file_is_rejected(tmp_path):
write(tmp_path, "src/a.py", "one\ntwo\n")
ok, reason = vf.check_window(finding(line=99), str(tmp_path))
assert not ok
assert "outside file" in reason
def test_missing_file_is_rejected(tmp_path):
ok, reason = vf.check_window(finding(path="nope.py"), str(tmp_path))
assert not ok
assert "does not exist" in reason
def test_absent_window_is_tolerated_not_rejected(tmp_path):
"""No window is a gap, not evidence of fabrication."""
write(tmp_path, "src/a.py", "one\ntwo\n")
ok, _ = vf.check_window(finding(window=""), str(tmp_path))
assert ok
# ------------------------------------------------------------ addressability
def test_unaddressable_line_is_kept_but_flagged(tmp_path):
"""PR #2373: a third of findings sat outside every hunk. Real, uncommentable."""
write(tmp_path, "src/a.py", "one\ntwo\nthree\n")
addr = {"src/a.py": {1, 3}}
verified, rejected, _ = vf.verify([finding(line=2)], str(tmp_path), addr)
assert not rejected
assert verified[0]["anchorable"] is False
def test_addressable_line_is_marked_true(tmp_path):
write(tmp_path, "src/a.py", "one\ntwo\nthree\n")
verified, _, _ = vf.verify(
[finding(line=2)], str(tmp_path), {"src/a.py": {1, 2, 3}}
)
assert verified[0]["anchorable"] is True
def test_file_absent_from_diff_is_unanchorable(tmp_path):
write(tmp_path, "src/a.py", "one\ntwo\n")
verified, _, _ = vf.verify([finding()], str(tmp_path), {})
assert verified[0]["anchorable"] is False
# ------------------------------------------------------- verify_steps gating
@pytest.mark.parametrize(
"steps",
[
"trace the value through build_client",
"assuming the allowlist is empty",
"consider the case where input is empty",
"if an attacker supplies a crafted path",
"grep the repo for other callers",
"open another file and compare",
"simulate the regex with two ids",
],
)
def test_expensive_verify_steps_force_not_cheap(tmp_path, steps):
write(tmp_path, "src/a.py", "one\ntwo\n")
verified, _, _ = vf.verify(
[finding(verify_steps=steps)], str(tmp_path), {"src/a.py": {2}}
)
assert verified[0]["cheap"] == "not_cheap"
assert verified[0]["_cheap_reason"]
def test_local_verify_steps_stay_cheap(tmp_path):
write(tmp_path, "src/a.py", "one\ntwo\n")
verified, _, _ = vf.verify(
[finding(verify_steps="read lines 1-3 of this file")],
str(tmp_path),
{"src/a.py": {2}},
)
assert verified[0]["cheap"] == "cheap"
assert "_cheap_reason" not in verified[0]
def test_lane_cannot_relabel_its_way_past_the_gate(tmp_path):
"""Workers no longer assign `cheap`; the gate computes it from verify_steps.
Even a worker that smuggles the label in loses to its own procedure.
"""
write(tmp_path, "src/a.py", "one\ntwo\n")
f = finding(cheap="cheap", verify_steps="trace it through two modules")
verified, _, _ = vf.verify([f], str(tmp_path), {"src/a.py": {2}})
assert verified[0]["cheap"] == "not_cheap"
def test_missing_verify_steps_is_not_cheap(tmp_path):
"""No stated procedure means no evidence it is checkable."""
write(tmp_path, "src/a.py", "one\ntwo\n")
verified, _, _ = vf.verify(
[finding(verify_steps="")], str(tmp_path), {"src/a.py": {2}}
)
assert verified[0]["cheap"] == "not_cheap"
# ------------------------------------------------------------------ clustering
def test_repeated_class_is_clustered(tmp_path):
"""Five 'unused import' findings are one comment, not five."""
fs = [
finding(
path=f"src/m{i}.py",
line=1,
what=f"the logging import is created but never used in m{i}",
)
for i in range(4)
]
groups = vf.cluster(fs)
assert len(groups) == 1 and len(groups[0]) == 4
def test_distinct_findings_are_not_clustered():
fs = [
finding(what="the licence header stops mid-sentence"),
finding(what="ipv4_enabled is true on the database"),
finding(what="city_clean is assigned and never read"),
]
assert vf.cluster(fs) == []
def test_two_of_a_kind_stays_below_the_grouping_threshold():
fs = [
finding(what="the logging import is created but never used here"),
finding(what="the logging import is created but never used there"),
]
assert vf.cluster(fs) == []
# --------------------------------------------------------------------- schema
@pytest.mark.parametrize(
"bad", [{"line": 2, "what": "x"}, {"path": "src/a.py", "line": 2}]
)
def test_incomplete_findings_are_rejected(tmp_path, bad):
verified, rejected, _ = vf.verify([bad], str(tmp_path), {})
assert not verified and len(rejected) == 1
def test_norm_ignores_whitespace_and_indentation():
assert vf.norm(" x = 1") == vf.norm("x=1")
def test_norm_keeps_ascii_substance():
assert vf.norm("os.system(x)") != vf.norm("subprocess.run(x)")
# ------------------------------------------------- suppressing existing work
def existing(**kw):
base = {
"kind": "inline",
"path": "src/a.py",
"line": 2,
"original_line": 2,
"body": "already said this",
"author": "someone",
"is_bot": False,
"resolved": False,
"outdated": False,
}
base.update(kw)
return base
def test_exact_line_collision_is_suppressed(tmp_path):
"""The re-review case: my own prior comments must exclude themselves."""
write(tmp_path, "src/a.py", "one\ntwo\nthree\n")
v, _, sup = vf.verify(
[finding()], str(tmp_path), {"src/a.py": {2}}, [existing()]
)
assert not v and len(sup) == 1
assert "already commented on this line" in sup[0]["_reason"]
def test_within_two_lines_is_suppressed(tmp_path):
write(tmp_path, "src/a.py", "\n".join(f"l{i}" for i in range(10)))
v, _, sup = vf.verify(
[finding(line=4)], str(tmp_path), {"src/a.py": {4}}, [existing(line=2)]
)
assert not v and "line 2" in sup[0]["_reason"]
def test_three_lines_away_survives(tmp_path):
"""Proximity is 2, deliberately tight so fresh findings aren't eaten."""
write(tmp_path, "src/a.py", "\n".join(f"l{i}" for i in range(10)))
v, _, sup = vf.verify(
[finding(line=5)], str(tmp_path), {"src/a.py": {5}}, [existing(line=2)]
)
assert len(v) == 1 and not sup
def test_outdated_thread_does_not_block(tmp_path):
"""The code moved out from under it, so the spot gets a fresh look."""
write(tmp_path, "src/a.py", "one\ntwo\nthree\n")
v, _, sup = vf.verify(
[finding()],
str(tmp_path),
{"src/a.py": {2}},
[existing(outdated=True, body="unrelated words entirely")],
)
assert len(v) == 1 and not sup
def test_resolved_thread_still_blocks(tmp_path):
"""Already discussed. Re-raising a settled point is worse than missing it."""
write(tmp_path, "src/a.py", "one\ntwo\nthree\n")
v, _, sup = vf.verify(
[finding()], str(tmp_path), {"src/a.py": {2}}, [existing(resolved=True)]
)
assert not v and "(resolved)" in sup[0]["_reason"]
def test_bot_comments_suppress_like_humans(tmp_path):
write(tmp_path, "src/a.py", "one\ntwo\nthree\n")
v, _, sup = vf.verify(
[finding()],
str(tmp_path),
{"src/a.py": {2}},
[existing(is_bot=True, author="lint[bot]")],
)
assert not v and "a bot" in sup[0]["_reason"]
def test_similar_text_suppresses_even_on_a_different_line(tmp_path):
"""Top-level comments have no line at all, so text is the only signal."""
write(tmp_path, "src/a.py", "\n".join(f"l{i}" for i in range(40)))
f = finding(
line=30, what="hardcoded project identifier committed in the config"
)
prior = existing(
kind="issue",
path=None,
line=None,
original_line=None,
body="hardcoded project identifier committed in the config file",
)
v, _, sup = vf.verify([f], str(tmp_path), {"src/a.py": {30}}, [prior])
assert not v and "similar" in sup[0]["_reason"]
def test_unrelated_text_is_not_suppressed(tmp_path):
write(tmp_path, "src/a.py", "\n".join(f"l{i}" for i in range(40)))
f = finding(line=30, what="the licence header stops mid sentence")
prior = existing(
kind="issue",
path=None,
line=None,
original_line=None,
body="database has a public ipv4 address enabled",
)
v, _, sup = vf.verify([f], str(tmp_path), {"src/a.py": {30}}, [prior])
assert len(v) == 1 and not sup
def test_no_existing_comments_suppresses_nothing(tmp_path):
write(tmp_path, "src/a.py", "one\ntwo\n")
v, _, sup = vf.verify([finding()], str(tmp_path), {"src/a.py": {2}}, [])
assert len(v) == 1 and not sup
# ------------------------------------------------------- fact-anchoring lint
def test_lint_flags_identifier_absent_from_window():
notes = vf.fact_anchor_lint(
{"comment": "does `strip_wrapper` handle this?", "window": " 3: x = 1"}
)
assert notes and "strip_wrapper" in notes[0]
def test_lint_passes_identifier_present_in_window():
assert not vf.fact_anchor_lint(
{
"comment": "`city_clean` isn't used below",
"window": " 3: city_clean = city.split(',')[0]",
}
)
def test_lint_flags_inference_vocabulary():
notes = vf.fact_anchor_lint(
{
"comment": "Is 200 characters enough to stay clear of the keys?",
"window": " 37: print(cmd[:200])",
}
)
assert any("inference word" in n for n in notes)
def test_lint_is_quiet_on_a_plain_observation():
assert not vf.fact_anchor_lint(
{"comment": "this runs as root", "window": ' 35: CMD ["uvicorn"]'}
)
def test_ascii_ellipsis_abbreviation_is_not_a_fabrication(tmp_path):
"""PR #2302 regression: lanes shorten long lines with a trailing ellipsis.
A Unicode ellipsis disappears in norm() as non-ASCII; an ASCII '...' does not,
and turned a valid prefix into a mismatch. Six real findings were rejected.
"""
write(
tmp_path,
"src/a.py",
"x = 1\nthis is a very long line with lots of detail after it\n",
)
f = finding(line=2, window=" 2: this is a very long line with lots...")
ok, reason = vf.check_window(f, str(tmp_path))
assert ok, reason
def test_unicode_ellipsis_abbreviation_also_passes(tmp_path):
write(
tmp_path,
"src/a.py",
"x = 1\nthis is a very long line with lots of detail after it\n",
)
f = finding(line=2, window=" 2: this is a very long line with lots\u2026")
ok, reason = vf.check_window(f, str(tmp_path))
assert ok, reason
def test_truncation_without_a_marker_is_still_rejected(tmp_path):
"""Silent divergence is indistinguishable from fabrication, so it must fail."""
write(tmp_path, "src/a.py", "x = 1\nalpha beta gamma\n")
f = finding(line=2, window=" 2: alpha beta DELTA")
ok, _ = vf.check_window(f, str(tmp_path))
assert not ok
def test_consistent_line_drift_is_repaired_not_rejected(tmp_path):
"""PR #2302: four findings were off by exactly one line. The content was real;
only the numbering was wrong, so discarding them lost real findings."""
write(
tmp_path,
"src/a.py",
"\n".join(["zero", "alpha", "beta", "gamma", "delta"]),
)
f = finding(line=2, window=" 2: alpha\n 3: beta\n 4: gamma")
# file has alpha at 2, beta at 3, gamma at 4 -> offset 0; shift the claim by 1
f = finding(line=3, window=" 3: alpha\n 4: beta\n 5: gamma")
ok, reason = vf.check_window(f, str(tmp_path))
assert ok and "corrected" in reason
assert f["line"] == 2 and f["_line_corrected"] == "3 -> 2"
def test_single_line_match_is_not_enough_to_claim_drift(tmp_path):
"""One line can coincide by chance; two is evidence."""
write(
tmp_path, "src/a.py", "\n".join(["x = 1", "x = 1", "totally different"])
)
f = finding(line=3, window=" 3: x = 1")
ok, _ = vf.check_window(f, str(tmp_path))
assert not ok
def test_drift_beyond_the_search_window_is_still_rejected(tmp_path):
write(tmp_path, "src/a.py", "\n".join(["a"] * 20 + ["alpha", "beta"]))
f = finding(line=2, window=" 2: alpha\n 3: beta")
ok, _ = vf.check_window(f, str(tmp_path))
assert not ok
# ------------------------------------------------- CI-status awareness
def _fail_finding(**kw):
f = {
"path": "contrib/python/x/pyproject.toml",
"line": 4,
"ci": "fail",
"what": "declares a [tool.ruff] table",
"evidence": "python-validate-recipe.yml:261-266",
"verify_steps": "read line 4",
}
f.update(kw)
return f
def test_a_finding_whose_workflow_is_already_red_is_suppressed():
"""The author is already looking at that red check; the comment adds noise
and spends a budget slot."""
why = vf.already_red(_fail_finding(), {"python-validate-recipe.yml"})
assert why and "already failing" in why
def test_a_green_workflow_does_not_suppress():
assert vf.already_red(_fail_finding(), {"go-format.yml"}) is None
def test_an_advisory_finding_is_never_suppressed_by_ci_status():
"""Nothing is failing for an advisory rule, so the comment is the only way
the author hears it -- even if some unrelated workflow is red."""
f = _fail_finding(ci="advisory")
assert vf.already_red(f, {"python-validate-recipe.yml"}) is None
def test_unknown_check_status_suppresses_nothing():
"""gh failed, or the checks have not run. Dropping a real finding because
we could not read a status is the wrong way to be wrong."""
assert vf.already_red(_fail_finding(), None) is None
assert vf.already_red(_fail_finding(), set()) is None
def test_only_a_failure_conclusion_counts(monkeypatch):
rows = "\n".join(
[
".github/workflows/python-validate-recipe.yml\tfailure",
".github/workflows/python-tests.yml\tsuccess",
".github/workflows/go-format.yml\t", # still running
".github/workflows/tools-tests.yml\tcancelled",
]
)
class P:
returncode = 0
stdout = rows
monkeypatch.setattr(vf.subprocess, "run", lambda *a, **k: P())
assert vf.red_workflows("o/r", "abc") == {"python-validate-recipe.yml"}
def test_red_workflows_returns_none_when_gh_fails(monkeypatch):
class P:
returncode = 1
stdout = ""
monkeypatch.setattr(vf.subprocess, "run", lambda *a, **k: P())
assert vf.red_workflows("o/r", "abc") is None
tests/test_voice_corpus.py
"""The voice corpus is the skill's only ground truth. Guard it from rotting.
The 16-for-16 fact-anchoring split and the four remedy edits are real accept/reject
decisions on live comments, not a rating session. If someone trims voice.md these
tests fail loudly rather than the skill quietly losing its calibration.
"""
import re
from pathlib import Path
import pytest
VOICE = Path(__file__).parent.parent / "reference" / "voice.md"
TEXT = VOICE.read_text(encoding="utf-8")
def test_fact_anchoring_is_the_first_filter():
"""It predicts acceptance better than anything else in the file."""
assert "## The first filter: anchor on a fact" in TEXT
i_filter = TEXT.index("anchor on a fact")
i_hard = TEXT.index("## The hard rule")
assert i_filter < i_hard, "fact-anchoring must precede the register rules"
def test_all_four_rejected_examples_are_present():
"""Each was cut by the user on PR #2373 for demanding reasoning."""
for fragment in (
"Is 200 characters enough",
"break the pattern the rest of the map follows",
"is in the accepted list",
"restored afterwards",
):
assert fragment in TEXT, f"rejected example missing: {fragment}"
def test_accepted_examples_are_present():
"""Posted verbatim, so they are the closest thing to a target."""
for fragment in (
"hardcoded project name here",
"isn't used below",
"has nothing to interpolate",
"no licence header on this one",
"looks like three files got concatenated",
):
assert fragment in TEXT, f"accepted example missing: {fragment}"
def test_the_rejected_table_explains_what_each_demanded():
section = TEXT[
TEXT.index("## The first filter") : TEXT.index("## The hard rule")
]
for demand in ("arithmetic", "infer a pattern", "reason about"):
assert demand in section, f"missing rationale: {demand}"
def test_remedy_rule_exists_and_cites_the_real_edits():
assert "A remedy is welcome" in TEXT
for fragment in (
"I suggest cleaning up all the headers",
"You may add these default values",
):
assert fragment in TEXT, f"remedy evidence missing: {fragment}"
def test_remedy_rule_does_not_contradict_the_ban_list():
"""`I suggest` / `You may` are attested; `Consider …` is still banned."""
assert "Consider …" in TEXT
assert "I suggest" in TEXT and "You may" in TEXT
ban = TEXT[TEXT.index("## Banned outright") :]
assert "not because suggesting a fix is banned" in ban
def test_absence_claims_require_an_inventory():
assert "Never assert an absence you did not inventory" in TEXT
assert "load_dotenv" in TEXT, "the real incident should stay as the example"
def test_example_20_is_still_rejected():
"""The remedy rule must not be read as licence for the dense chain."""
assert "20" in TEXT
assert "one sentence carrying mechanism" in TEXT or "too dense" in TEXT
@pytest.mark.parametrize(
"section",
[
"## The two registers",
"## The hard rule",
"## Say what you are pointing at",
"## Plausibility — could the reviewer have known this?",
"## Severity does not change tone",
"## Banned outright",
],
)
def test_core_sections_survive(section):
assert section in TEXT
def test_no_stale_mode_or_confidence_vocabulary():
"""Modes and the confidence label are gone; the corpus must not resurrect them."""
for dead in ("cheap mode", "complete mode", "not_high"):
assert dead not in TEXT, f"stale vocabulary in voice.md: {dead}"
# ------------------------------------------- structural, not string-presence
import json as _json # noqa: E402 -- the corpus below is the module docstring's subject
from pathlib import Path as _Path # noqa: E402
_FIXTURE = _json.loads(
(_Path(__file__).parent / "fixtures" / "pr2373_outcomes.json").read_text()
)
_BY_N = {c["n"]: c for c in _FIXTURE["comments"]}
def _table_rows(heading):
"""Rows of the first markdown table under a heading."""
section = TEXT[TEXT.index(heading) :]
rows = []
for line in section.split("\n"):
if line.startswith("|") and not re.match(r"^\|[\s|:-]+\|$", line):
cells = [c.strip() for c in line.strip("|").split("|")]
rows.append(cells)
elif rows and not line.startswith("|"):
break
return rows[1:] # drop the header row
def test_rejected_table_is_structurally_complete():
"""Four rejected examples, each with an id, the comment, and what it demands."""
rows = _table_rows("## The first filter: anchor on a fact")
ids = [r[0] for r in rows if r and r[0].startswith("R")]
assert ids == ["R1", "R2", "R3", "R4"], ids
for r in rows:
if r and r[0].startswith("R"):
assert len(r) >= 3 and r[2], f"{r[0]} has no stated demand"
def test_every_rejected_example_is_a_real_cut_in_the_fixture():
"""The corpus must not invent rejections that never happened."""
cut_text = " ".join(
c["comment"] for c in _FIXTURE["comments"] if c["verdict"] == "cut"
)
for frag in (
"200 characters",
"break the pattern",
"accepted list",
"restored afterwards",
):
assert frag in cut_text, (
f"{frag!r} is in voice.md but was not actually cut"
)
def test_every_accepted_example_is_a_real_keep_in_the_fixture():
kept_text = " ".join(
c["comment"] for c in _FIXTURE["comments"] if c["verdict"] == "keep"
)
for frag in (
"hardcoded project name here",
"isn't used below",
"has nothing to interpolate",
"no licence header on this one",
"looks like three files got concatenated",
):
assert frag in kept_text, (
f"{frag!r} is in voice.md but was not actually kept"
)
def test_remedy_examples_are_real_edits_not_invented():
"""All four remedy examples must be things the user actually wrote."""
for frag in (
"I suggest cleaning up all the headers",
"You may add these default values",
):
assert frag in TEXT
# and the pre-edit form must match what the skill drafted
drafted = " ".join(c["comment"] for c in _FIXTURE["comments"])
assert "the licence header stops mid-sentence" in drafted
assert "same default as the advisor module" in drafted
def test_corpus_counts_stated_in_prose_match_the_fixture():
"""'16 accepted / 4 rejected' must not drift from the data."""
kept = sum(1 for c in _FIXTURE["comments"] if c["verdict"] == "keep")
cut = sum(1 for c in _FIXTURE["comments"] if c["verdict"] == "cut")
assert f"all {kept} accepted" in TEXT or f"{kept} accepted" in TEXT
assert f"all {cut} rejected" in TEXT or f"{cut} rejected" in TEXT