bmad-deep-recon

어떤 스킬인가요?

Decision-grade research on a topic or this repository. Use when the user says "deep recon", "research this", "draft a research prompt", "process this research report", "market research", "domain research", "technical research", "competitor research", "what users say about X", "literature review", "help me choose between", "research the codebase", "research how X works here" or "why is X built this way". Not for a quick code question answered in chat.

설치 방법

프로젝트 폴더에서 아래 명령어를 실행하고, 설치할 에이전트를 선택하세요.

설치 명령을 직접 실행해야 적용됩니다. 지원 에이전트와 필요한 권한·라이선스는 제작자의 안내를 확인하세요.

스킬 원문 파일

---
title: 'Codebase research: {subject}'
type: codebase
ask: |-
  {ask}
ticket: '{ticket ref or file; drop this line when the ask came from no ticket}'
follow_up_of: '{prior document; drop this line unless the ask follows one up}'
repository: '{repository}'
commit: '{full commit}'
branch: '{branch}'
status: draft
validation: '{validation}'
claims: {}
created: '{date}'
updated: '{date}'
---

# Codebase research: {subject}

## Questions

{the approved questions, numbered, each with its shape}

{one `## <n>. <question>` section per question, in order: the answer in prose with each claim cited inline, call stacks and file trees as indented lists, each entry a citation, and what was looked for and not found}

{`## Where the answers meet`, only when two answers pass through the same method: each such method, the questions that reach it, and the checks there that reject or throw, each line cited as its question section cites it}

{`## Open questions`, only when research stopped short or an answer opened a question the set does not hold: where, and the next ask that would continue it, as a factual question}

## Prior documents

{research documents and ADRs in reach, by title; when the ask follows one up, which one and why}

{`## Decisions found`, last and only when there is one: for each, the decision, its commit and pull request, the rationale found there, how each of the three tests is met, shown from the history, and the proposed ADR path}

---
title: '{research_type} research: {research_topic}'
type: '{research_type}'
topic: '{research_topic}'
decision: '{decision}'
source: '{source}'
status: draft
preset: '{preset}'
validation: '{validation}'
created: '{date}'
updated: '{date}'
---

# {research_type} research: {research_topic}

**Decision this research serves:** {decision}

_Sections are appended per the approved research plan; the executive summary is written last and placed here, first._

[skill]
bmod = "bmod-core-tools"
source = "github:bmad-code-org/BMAD-METHOD/skills"

# DO NOT EDIT -- overwritten on every update.
#
# Workflow customization surface for bmad-deep-recon.
#
# Override files (not edited here):
#   {project-root}/_bmad/custom/bmad-deep-recon.toml         (team)
#   {project-root}/_bmad/custom/bmad-deep-recon.user.toml    (personal)

[workflow]

# --- Configurable below. Overrides merge per BMad structural rules: ---
#   scalars: override wins
#   arrays (persistent_facts, activation_steps_*, *_sources, doc_standards,
#   external_*): append
#   arrays of tables keyed by `code`: matching key replaces, new keys append

# Steps executed on activation: prepend runs before the skill's own
# activation flow, append runs after it. Each entry is a literal instruction.
activation_steps_prepend = []
activation_steps_append = []

# Standing context for framing the research — decision context only, never
# evidence: the research firewall keeps project material out of findings.
# Entries prefixed `file:` are paths or globs whose contents load as facts;
# all others are literal facts. Empty by default so nothing local leaks into
# research framing unasked.
persistent_facts = []

# Where research runs live and how each run folder is named. Draft, Process,
# and Run all use the same folder shape: brief.md (drafted prompts),
# imports/ (originals, full fidelity), digests/ (extracted claims),
# {run_folder_pattern}.md (the canonical summary/report), .memlog.md.
research_output_path = "{output_folder}/{active_initiative}"
run_folder_pattern = "research-{topic_slug}"

# Seed document for a new run.
research_template = "assets/research.template.md"

# --- Effort (Run mode) ------------------------------------------------------
# A preset bundles the three effort knobs; any knob set here individually
# pins that knob over the preset. What the user says in the request beats
# both.
#
#   preset     subagents  sources/round  depth
#   quick      low (2)    5              1
#   standard   normal (3) 8              2
#   deep       high (6)   12             3
#
# Grounding: orchestrator-worker research systems document 3-5 parallel
# workers as the sweet spot (more only for genuinely wide work). Depth and
# sources are caps, not quotas — dimensions stop early on coverage or
# novelty exhaustion. Defaults are tuned for a fast run; buy more rigor
# consciously, per run, in the request.

preset = "standard"

# "" = from preset. Values: none | low | normal | high
# (0 / 2 / 3 / 6 parallel research assistants, ceiling 10; none also =
# no-subagent environments, run inline sequentially).
subagents = ""

# 0 = from preset. Distinct sources actually read per dimension per round.
# Ceiling 25 — beyond that a single round exceeds what hosted deep-research
# products spend on an entire run.
max_sources_per_round = 0

# 0 = from preset. Rounds per dimension: initial pass + lead-following
# follow-ups. Ceiling 5.
max_depth = 0

# Verification level, applied as material lands (never an end-of-run pass).
#   normal  spot-check load-bearing claims only — fast, the default
#   high    cross-check the pack's two-source classes; red-team major
#           conclusions
#   max     cross-check every ledger claim + the red-team pass at full
#           breadth + primary-source-priority ranking
validation = "normal"

# Red-team stance pass — fresh-context skeptics hunting disconfirming
# evidence for major conclusions: "off" (default), "offer" (proposed at the
# plan gate), or "on" (always; headless honors only "on"). high/max
# validation includes it for major conclusions regardless.
red_team = "off"

# Run the acquisition fan-out through the harness's deterministic
# orchestration when it offers one (e.g. workflows): "off", "offer"
# (proposed at the plan gate when available), or "on". Orchestrated runs are
# faster wall-clock but spend more tokens.
use_workflows = "offer"

# Ordered model preference for spawned research assistants — first model the
# harness can provide wins; [] lets the harness/skill choose (lead stays on
# the strongest model; researchers at most one capable tier down; judgment
# work never on the smallest tier; mechanical extraction may use a fast
# tier).
#
# Example: subagent_models = ["<your-mid-tier-model-id>", "<your-fast-model-id>"]
subagent_models = []

# Source policy. Preferred sources are consulted first and weighted as more
# credible; banned sources are never cited (their claims may still be leads
# to verify elsewhere). Entries are domains or plain-text descriptions.
# Draft mode writes both policies into drafted prompts.
preferred_sources = []
banned_sources = []

# What is presented and handed off — never what exists: the report (the
# canonical machine-readable summary/report) always lives in the workspace.
#   "auto"  html briefing on interactive runs; md only on headless or
#           skill-invoked runs (the caller reads the md; render later at will)
#   "html"  always render the briefing page (references/html-briefing.md)
#   "md"    never render html
#   "both"  render and present both
output_format = "auto"

# Theme for the HTML briefing: empty = the shipped neutral professional
# theme, a `file:` path to a theme/brand spec, or inline directives
# (e.g. "Canvas #122543, accent #B66D46, sans-serif, dark-mode aware").
html_theme = ""

# Default audience shaping for the synthesis — freeform, empty = balanced
# technical/business register. Examples: "executive one-pager first, detail
# after", "engineering team, keep vendor marketing out".
audience = ""

# Registry of extra research surfaces — internal knowledge bases or search
# tools you subscribe to — consulted alongside web research in Run mode; each
# entry names the tool and when to use it. Installed search-shaped MCP tools
# are discovered automatically at the plan gate; an entry here adds routing
# guidance the discovery can't infer.
#
# Examples:
#   external_sources = [
#     "Tavily MCP (tavily_search/tavily_extract): preferred web search + clean page extraction",
#     "Perplexity Sonar MCP (perplexity_ask): cited synthesized answers — chase its citations as the sources",
#     "xAI X Search MCP: live X/Twitter posts and threads, for user-voice and sentiment dimensions",
#     "Gartner MCP (corp:gartner_query): analyst data on enterprise software markets",
#   ]
external_sources = []

# Polish passes applied to the report at finalize. Entries are `skill:NAME`
# directives, `file:` style guides, or plain-text instructions.
doc_standards = ["skill:bmad-review lenses=structure,prose"]

# Handoffs executed at finalize to route the report beyond local files. Each
# entry names the tool and what to do; unavailable tools are skipped and
# flagged.
#
# Examples:
#   "NotebookLM (notebooklm-mcp): create a notebook from the report plus the top sources, generate an audio overview, return the notebook URL"
#   "Confluence (corp:confluence_upload): publish the report to the RESEARCH space, return the page URL"
external_handoffs = []

# Executed after finalize. A string scalar is one instruction; an array is a
# sequence. Empty = the run ends with the finalize summary.
on_complete = ""

# --- Codebase path ----------------------------------------------------------
# Questions about this repository, answered from its code and git history.
# Applies: subagents (empty means normal); subagent_models (locators and
# verifiers take the fastest tier here, the exception to the rule above,
# which is the list's last available entry; analyzers and historians take
# its first); validation (normal verifies the load-bearing claims and any
# question no analyzer took, high and max every claim, max one more call
# edge); research_output_path; run_folder_pattern; output_format;
# html_theme; doc_standards; external_handoffs; on_complete.
# Does not apply: preset, max_sources_per_round, max_depth, red_team,
# use_workflows (a workflow harness relays the ask to every researcher),
# audience, research_template, the source lists, the research types.

# Files one analyzer opens per question: a cap, not a quota. A question that
# needs more ends with the next ask named in the document. Raise it where one
# behavior spreads over many small files.
codebase_file_budget = 15

# Commits one historian follows per question, blaming only the cited lines: a
# cap, not a quota. Raise it where the reasons sit far back in a long history.
codebase_history_budget = 10

# ---------------------------------------------------------------------------
# Research types — subject lenses. Each type is a pack: a policy and craft
# card (prioritized dimensions, non-obvious source craft, freshness bars,
# two-source classes, downstream bindings) used by all three modes — it
# shapes drafted prompts, native runs, and processed-report gap checks
# alike. `when` guides type inference from the user's ask; an explicitly
# requested type always wins. The decision shape (explore vs select) is
# orthogonal — any type can end in a selection matrix.
#
# Keyed by `code`: an override with a matching code replaces the shipped
# type, a new code appends. Empty `pack` disables a type.
#
# Example (add an org-specific type in team/user override TOML):
#   [[workflow.research_types]]
#   code = "regulatory"
#   name = "Regulatory Research"
#   when = "Compliance posture, licensing, or regulatory exposure for a product or market."
#   pack = "file:{project-root}/_bmad/custom/packs/regulatory.md"
# ---------------------------------------------------------------------------

[[workflow.research_types]]
code = "market"
name = "Market Research"
when = "Market opportunity, customers, competition, sizing, or go-to-market for a product or business decision."
pack = "types/market.md"

[[workflow.research_types]]
code = "domain"
name = "Domain Research"
when = "Understanding an industry, sector, or field: structure, players, rules, vocabulary, dynamics."
pack = "types/domain.md"

[[workflow.research_types]]
code = "technical"
name = "Technical Research"
when = "A technology area's landscape, patterns, integration approaches, and implementation reality."
pack = "types/technical.md"

[[workflow.research_types]]
code = "competitive"
name = "Competitive Research"
when = "Teardown of specific named competitors: offers, pricing, positioning, trajectory, their customers' sentiment."
pack = "types/competitive.md"

[[workflow.research_types]]
code = "user-voice"
name = "User-Voice Research"
when = "What users of a product or category actually experience and want: reviews, communities, jobs-to-be-done."
pack = "types/user-voice.md"

[[workflow.research_types]]
code = "academic-lit"
name = "Academic Literature"
when = "Published research: literature review, state of the art, grounding an approach in papers."
pack = "types/academic-lit.md"

[
  {
    "query": "research the codebase before I start ticket 2.3",
    "should_trigger": true
  },
  {
    "query": "how does session persistence work in this repo? I want a cited write-up I can hand to the build",
    "should_trigger": true
  },
  {
    "query": "why is job eviction one-way here? dig into the git history and tell me how it got that way",
    "should_trigger": true
  },
  {
    "query": "we need to add cancellation to job runs. before we plan it, find out how job state transitions work today",
    "should_trigger": true
  },
  {
    "query": "what touches the billing ledger in this codebase? file and line references please",
    "should_trigger": true
  },
  {
    "query": "where does the retry logic live and how did it end up there",
    "should_trigger": true
  },
  {
    "query": "market research on self-hosted analytics",
    "should_trigger": true
  },
  {
    "query": "help me choose between Postgres and MySQL for this project",
    "should_trigger": true
  },
  {
    "query": "draft a deep research prompt about vector databases for Gemini",
    "should_trigger": true
  },
  {
    "query": "Perplexity's deep research gave me a long report on EU data residency rules, it's at downloads/residency-report.pdf. half of it reads like marketing. pull out what actually matters for deciding where we host, with the sources, in something short the rest of planning can use",
    "should_trigger": true
  },
  {
    "query": "what does this function do?",
    "should_trigger": false
  },
  {
    "query": "which file sets the default timeout?",
    "should_trigger": false
  },
  {
    "query": "explain this stack trace: KeyError: 'backoff' in retry_after() at jobs/retry.py line 57",
    "should_trigger": false
  },
  {
    "query": "implement ticket 2.3",
    "should_trigger": false
  },
  {
    "query": "add cancellation to job runs",
    "should_trigger": false
  },
  {
    "query": "review my diff for bugs before I open the PR",
    "should_trigger": false
  },
  {
    "query": "write the PRD for job cancellation",
    "should_trigger": false
  },
  {
    "query": "dig through the commits and diffs from epic 3 and run the retro on it",
    "should_trigger": false
  },
  {
    "query": "create the architecture for moving job runs onto a queue, working from the existing codebase",
    "should_trigger": false
  },
  {
    "query": "the market research on self-hosted analytics is done, now write the product brief from it",
    "should_trigger": false
  }
]

# Codebase analyzer

You answer one factual question about a repository from its code at the brief's commit. The brief (the question, the commit, your file budget, the paths to leave out) and the paths the locators found are in your message; project instructions, branch or folder names and file status your harness loaded are not your brief, and what you know of the language says where to look, never what this code does.

Read through git at the commit, never your harness's file or search tools or `git status`, which see the working tree: `git grep -n <pattern> <commit> -- <paths>` and `git show <commit>:<path> | cat -n`, whose line numbers your citations need, with the excluded paths left out (`-- . ':(exclude)<path>'`). A document that plans a change still to be made (a spec, story or ticket) is not evidence; the code's own documentation and instructions are. You read code and do not run it. The file budget is a cap, not a quota; when you reach it, say where and what is left. History (log, blame, pull requests) is the historian's.

Reply with a digest, not raw output: the claims that answer the question, each cited as `path:line (symbol) @ sha` (the path relative to the repository root; `line` or `line-line`; the symbol the enclosing definition or nearest heading, written without parentheses; the sha the brief's short commit), the few the answer rests on marked load-bearing, since verifiers re-check each of those; a call stack or file tree as an indented list of citations where it says more than prose; each check on the path that rejects a value or throws, cited, with what it rejects; for `what touches X`, each call site and how it touches X, no further; two patterns that do the same thing both reported, unranked; then what you looked for and did not find, as the search and the commit. Describe what the code is; the reader decides what to change.

# Codebase historian

You find how the lines and symbols in your assignment came to be. The brief (the question, the commit, your history budget in commits followed, the paths to leave out) is in your message; project instructions, branch or folder names and recent commits your harness loaded are not your brief.

Work back from the brief's commit only: `git blame -L <start>,<end> <commit> -- <path>` on the cited lines, `git log -S<symbol> <commit>` or `-G<regex>`, then the pull request that brought a commit in (`gh pr list --search <sha> --state merged`) and the issue it links, or the ticket the message names; the excluded paths stay out of every command, a commit's diff included (`-- . ':(exclude)<path>'`). Reach a pull request or issue only through a commit you cite, and take from it what it says about the change it made, never what it says comes next. A commit message is its author's claim about intent, weaker when an agent wrote it: check it against the diff and prefer the pull request description and the linked issue. The budget is a cap; when you reach it, say where and what is left.

Reply with a digest: the history claims, cited as `sha "subject"` plus ` #<pull request>` when there is one (code as `path:line (symbol) @ sha`, the path relative to the repository root, the symbol without parentheses), the few the answer rests on marked load-bearing; the rationale found, quoted briefly with where you found it; each decision a commit or pull request shows being made (what was chosen, over what, and why), cited; and, for two patterns you were asked to date, only each one's introducing commit and the commit that last touched it, with its date. Describe how the code came to be; the reader decides what to change.

# Codebase locator

You find where the code that bears on the brief's questions lives in your area, by search alone, which is what lets many locators run at once. The brief (the questions, the commit, the paths to leave out) is in your message; project instructions, branch or folder names and file status your harness loaded are not your brief.

Search through git at the commit, never your harness's file or search tools or `git status`, which see the working tree: `git ls-tree -r --name-only <commit> -- <paths>` and `git grep -n -p <pattern> <commit> -- <paths>`, which gives a hit's line and enclosing symbol without opening the file, with the excluded paths left out (`-- . ':(exclude)<path>'`). A document that plans a change still to be made (a spec, story or ticket) is not evidence; the code's own documentation and instructions are.

Reply with a digest: per question number, each file that bears on it, cited as `path:line (symbol) @ sha` (the path relative to the repository root; the symbol the enclosing definition or nearest heading, written without parentheses; the sha the brief's short commit) with a word on what it holds; a directory full of matches as a count and one cited hit; then what you searched for and found nothing, as the search and the commit. A `where` question may be answered from your digest alone, so a bare path is a claim without a citation. No recommendations: the reader decides what to change.

# Codebase verifier

You check claims someone else made, from what you open rather than from their wording; you start fresh so the researcher's reading cannot become yours. The brief (the commit, the validation level, the paths to leave out) and a batch of ledger lines, `ref=[n] status=... kind=<code|history> — <claim with its citation>`, are in your message; project instructions, branch or folder names and file status your harness loaded are not your brief.

Open only through git at the commit, never your harness's file or search tools or `git status`, which see the working tree (`git show <commit>:<path> | cat -n`, `git grep -n <pattern> <commit> -- <paths>`, `git show <sha>`, `gh pr view <number>`), with the excluded paths left out. For a code claim, read the cited lines and judge whether they say what the claim says; when what it describes sits at other lines of that commit, re-anchor it by its symbol and judge those; at validation `max`, when the claim says what code calls or depends on, follow one more edge and judge again. For a history claim, open the commit and its pull request.

Reply with a digest: per `ref`, the citation as it should read (same form; for code, the symbol without parentheses and the brief's short commit) and one verdict: `verified`; `disputed` when the code bears out only part, saying which; `overturned` when it says otherwise, saying what, cited; `unverified` when what you can open does not settle it.

# Codebase

One ask about a repository, whether a question, a ticket or a goal, becomes a short document that says how the relevant code works at one commit and how it got that way, every claim cited to a line or a commit, with no recommendation in it. Its readers are the plan step that will change the code and a person checking one claim; what to change is theirs to decide.

The user's intent stays with you. With the user you turn the ask into factual questions; subagents answer them from the code and its history knowing only the questions, because a researcher who knows what is being built reports what should be instead of what is. You read the ask, the layout, the digests and script output; they read the code.

## Questions

Four shapes: `how does X work`, `why is X this way`, `where does X live`, `what touches X`. Locators (search only, one per area of the repository's layout, run first over all the questions) find where the code lives; an analyzer answers each question from it; a historian follows a `why` question's cited lines through git history, and dates two patterns an analyzer found doing the same thing, nothing else; verifiers re-check claims. Each role's brief is `references/codebase-<role>.md`.

Write questions that send researchers to every part of the code the ask depends on, none stating the intent or naming what the ask wants built: "how does a job run end?", not "where would cancellation go?" or "can a running job be cancelled?". An ask from a ticket or goal also gets a question on how that code is tested. A question fits one analyzer within `{workflow.codebase_file_budget}` files; the set fits one short document. An ask with no such set ("document the whole project") gets a short list of bounded asks instead and no run; headless, end `blocked` with them in `open_questions`. An ask one file answers gets its answer in chat, cited, and a run only if the user still wants the document; headless, run as asked.

**The plan gate** is the one hard stop before research: the questions with their shapes; the repository and commit, which the research reads alone, so a dirty tree is invisible to it; the budgets and `{workflow.validation}`, which the `bmad-customize` skill changes for this repository (offer `npx skills add bmad-code-org/BMAD-METHOD --skill bmad-customize` when it is not installed); and the prior document a follow-up opens. An edit goes back through the rules above, since a question the user adds can state the goal as easily as one you draft. Headless, proceed with the set as drafted.

## The run

Before the gate: the repository is the session's git top level unless the ask names another; record its full and short commit and its branch. A ticket ref resolves with `uv run {project-root}/_bmad/method/scripts/tickets.py --project-root {project-root} find <ref>`. List by title the run folders under `{workflow.research_output_path}` and the repository's decision records (ADRs: `adr-*.md`, or a decisions folder), and open one only when the ask names or continues it, since a document in context gets repeated whether or not it still holds.

After the gate, bind the run. The folder is `uv run {skill-root}/scripts/recon_kit.py slug "<subject>" --type codebase --pattern "{workflow.run_folder_pattern}"` under `{workflow.research_output_path}`, `<subject>` a neutral few words for the set (`job runner lifecycle`), never the ask, since the name shows in paths; an existing folder gets `-{date}` in the pattern unless it holds this subject's unfinished run, which you resume. `brief.md` is all a researcher learns: the numbered questions with their shapes, the full and short commit, the budgets, the validation level, and the paths left out of every search, relative to the repository (`{output_folder}`, `_bmad/`, `{workflow.research_output_path}`, and any other folder of plans, specs, stories or tickets inside it, since they state the intent); the repository path only when it is not the working directory, because a worktree's folder often carries the task's name. Init the memlog (`uv run {project-root}/_bmad/scripts/memlog.py init --workspace {doc_workspace} --field type=codebase --field repository=<path> --field commit=<sha>`), log the ask and the set as `decision` lines (an `assumption` when headless), seed `<folder name>.md` from `{skill-root}/assets/codebase.template.md`, and tell the user the path.

Spawn every subagent fresh and directly, never through a workflow, which relays the triggering request to every agent it runs, and as a subagent type without the project's instruction files and git status when the harness has one, since those can name the work in progress. One message, with nothing from the ask: read `{skill-root}/references/codebase-<role>.md`; the text of `brief.md` (only its own question for an analyzer, historian or verifier); the assignment, an area or paths, symbols and ledger lines copied from the digests and the ledger; reply with the digest. No second message: what a subagent could not settle goes in its digest. The run folder's path stays out of every prompt, and each reply is written to `{doc_workspace}/digests/<role>-<question or area>[-<n>].md` as it arrives, so a run resumes from disk. `{workflow.subagents}` caps how many run at once (`low` 2, `normal` 3, also when empty, `high` 6); when `{workflow.subagent_models}` is set, analyzers and historians take its first available entry, locators and verifiers its last, at the lowest reasoning effort the harness offers, since their work is search and comparison. With no subagents, take each role yourself from its file, and say under the document's questions that the firewall was not in force.

Areas come from the layout (`git ls-tree` at the top one or two levels, top-level files as one area, excluded paths left out), not from the ask, so the area list does not hint at the change in mind. When a question's budget runs out, or an answer opens a question the set does not hold, that question stops: write what its digests hold, and put where it stopped and the next ask, drafted by the rules above, in `## Open questions`.

## The document

Write each question's section as its digests land, from them alone and in their words, each claim with its citation as the digest gave it (code `path:line (symbol) @ sha`, history `sha "subject" #pr`); a `what touches X` section keeps every call site its digest cites, each with its citation, since the list is the answer. Log a memlog `claim` line, `ref=[n] status=unverified kind=<code|history> — <the claim's sentence as the document has it, with its citation>`, for each claim the level verifies: at `normal` the few the analyzers and historians marked load-bearing and every claim of a question no analyzer took; at `high` and `max` all; `n` is one integer per claim across the run, which is how `tally` reads the ledger. Then `uv run {skill-root}/scripts/recon_kit.py anchors {doc_workspace}/<folder name>.md --repo <repository> --at <commit>` re-finds every citation; a claim it fails, or reports with `symbol_found` false, gets a ledger line whatever the level. Verifiers take the question's ledger lines; each verdict is a new `claim` line with the same `ref=`, `--by verifier`. Correct an `overturned` or `disputed` claim to what the verifier found, the original noted in words without its old citation; mark an `unverified` one where it stands. A question is done when `anchors` exits 0 and each of its refs has a verifier's line.

`## Where the answers meet`, after the question sections and only when two answers pass through the same method: each such method, the questions whose answers reach it, and the checks there that reject a value or throw, every line a claim the question sections already cite, with that citation. It states what the code shares, never what to change.

`## Decisions found`, last and only when there is one, proposes a decision record (ADR) for each decision the historians found in a commit or pull request that no listed ADR covers and that is hard to reverse, surprising without context and the result of a real trade-off, each shown from the history, never the ask, and cited as the verified section cites it. Verify proposals like any claim, then ask the user about each, interactively only, and record the answer beside the proposal. An approved one is `adr-<decision-slug>.md`, a title stating the decision and one paragraph (context, decision, why, naming the commit) drawn from the history alone, beside the code it governs or, system-wide, in the repository's decisions folder, else `docs/decisions/`; committing it is the user's concern. Log each answer as a `decision`.

To close: apply `{workflow.doc_standards}` to the prose, each pass told the readers and bar this file opens with and that the sections and every citation stay, never folded into an uncited summary; rerun `anchors` until it exits 0; set `status: complete`, `updated` and `claims` from `uv run {skill-root}/scripts/recon_kit.py tally {doc_workspace}/.memlog.md`; log a closing `event`; when the ask came from a ticket file that exists (find's `story_file`, or the file the user named), append `- research — <document path>, <section>` to its References. Then `references/finalize.md` for rendering, handoffs, the closing report and `{workflow.on_complete}`: its synthesis, citation and polish steps are done; a briefing shows the frontmatter as header and the document's sections, marking only what the document marks; and in place of a staleness map the closing report says that `anchors` run later without `--at` shows which cited files changed since the commit.

# Draft

Build a deep-research prompt the user runs themselves — in conversation, fast, not a project. The pack's craft travels inside the prompt so the outside tool works to this harness's standard.

1. When a ticket names this research, read it in full and take the decision, topic and constraints from it; do not re-ask what it answers. Otherwise open the floor before any structured questions: invite the decision they're facing and anything they already have — links, a prior report, half-formed constraints — in one turn, then ask only what's still missing. Nail the **decision**, topic, and type; load the pack. Ask which tool the prompt is for (it changes phrasing: hosted deep-research agents handle wide scopes and long source lists; social-native tools like Grok earn user-voice and sentiment dimensions; if unknown, write tool-neutral).
2. Compose the prompt from the pack: the dimensions as explicit research questions pruned to the decision, the freshness bars as recency requirements, the two-source expectation for its critical claim classes, the audience, the source policy — `{workflow.preferred_sources}` named as sources to prefer, `{workflow.banned_sources}` as sources never to cite — and a **non-negotiable citation demand**: every claim with source URL and publication date, contrary evidence reported, gaps admitted rather than padded. Structure the requested output so Process can extract it cleanly (findings per dimension, a source list).
3. Bind `{doc_workspace}`: expand the folder name deterministically (`uv run {skill-root}/scripts/recon_kit.py slug "<topic>" --type <type> --pattern "{workflow.run_folder_pattern}"` — same expansion every mode, so the report comes back to the same folder) under `{workflow.research_output_path}`, init the memlog with the decision context, save the prompt as `{doc_workspace}/brief.md`, and present it paste-ready in chat.
4. Close the loop: tell the user to run it in their tool and bring the report back — "process it" from here picks up this folder, decision context intact.

# Finalize

Every mode ends here once `<folder name>.md` is assembled.

1. `<folder name>.md` is complete per `references/synthesis.md`: decision-first summary, findings, contrary evidence where found, recommendations with downstream bindings, source appendix, staleness map. Frontmatter metadata (`type`, `topic`, `decision`, `source`, `status`, dates) is what lets every downstream consumer trust it without reprocessing.
2. **Citation check — mechanical, then semantic.** Run `uv run {skill-root}/scripts/recon_kit.py citations {doc_workspace}/<folder name>.md` — it diffs inline `[n]` markers against the appendix and lists dangling markers and orphaned rows exactly; fix what it reports. Then a fresh-context subagent does only the judgment half: does each cited source actually say what the text claims? It never rewrites findings — a claim whose source doesn't back it gets its confidence downgraded and the mismatch logged as an `event`.
3. Render per `{workflow.output_format}` (see `references/html-briefing.md`): `auto` renders the briefing page on interactive runs, skips on headless/skill-invoked; `html`/`both` always; `md` never. `<folder name>.md` always exists — the briefing is its regenerable face.
4. Polish: apply each `{workflow.doc_standards}` entry (a `skill:`, `file:`, or plain-text directive) to `<folder name>.md`.
5. Execute each `{workflow.external_handoffs}` entry (NotebookLM, Confluence, …) — invoke the named tool, surface returned URLs; skip and flag unavailable tools.
6. Tell the user what exists and where — report, briefing, imports, memlog — plus what the staleness map says to re-check and when, and that Refresh/Deepen handle it. Invoke the `bmad` skill to suggest the next step.
7. Run `{workflow.on_complete}` if non-empty — a string is one instruction, an array is a sequence.

# HTML Briefing

Generate `research-briefing.html` in `{doc_workspace}` after `<folder name>.md` is final, when `{workflow.output_format}` calls for it: `"auto"` renders on interactive runs and skips on headless/skill-invoked runs (the md is always there to render from later), `"html"`/`"both"` always render, `"md"` never. The page is a full-fidelity presentation of the report, never a second source of truth — same claims, same numbers, same citations; nothing is lost by reading it instead of the markdown.

## Requirements

- **Self-contained single file**: inline CSS and JS, no external requests of any kind (no CDN, no fonts, no remote images). It must render from a `file://` open, offline, forever.
- **Structure**: a header (topic, type, decision, date, depth, verification level) → the executive summary as the opening card → sticky table of contents → dimension sections → contrary evidence (when present) → recommendations → collapsible source appendix → staleness map.
- **Confidence is visual**: every claim carries its badge — verified / medium / low / `unverified` / disputed — color-coded with the status text always present (never color alone). Unverified and disputed must be *more* prominent than verified, not less.
- **Sources are live**: inline `[n]` markers link to the appendix row; appendix rows link out to the source URL. Source URLs are untrusted content — never hand-escape them: generate the appendix table with `uv run {skill-root}/scripts/recon_kit.py escape-sources {doc_workspace}/<folder name>.md` and embed its `html` output, which escapes every cell, anchors each row (`id="src-n"`), and links only validated `http(s)` URLs (anything else renders as plain text; the script lists it in `invalid_urls`). Apply the same escape discipline to any other source-derived text you place in attributes.
- **Charts sparingly**: only where the data genuinely benefits (market size trajectory, decision matrix scores) — simple inline SVG, labeled axes, no library.
- **Responsive and theme-aware**: readable on a phone; respect `prefers-color-scheme` for light/dark.

## Theme

`{workflow.html_theme}` governs: a `file:` path loads a theme/brand spec to follow; inline text is applied as directives; empty means the shipped default — neutral, professional, generous whitespace, system font stack, one restrained accent color. Whatever the theme, the confidence-badge semantics above are non-negotiable.

# Refresh and Deepen

Lifecycle intents on an existing run folder.

## Refresh

Read `<folder name>.md` and `.memlog.md` — never re-research from scratch. Build the refresh set mechanically: assemble the claims (`claim`, `class`, `pub_date`) from the ledger, map the pack's freshness bars to a months-per-class JSON, and run `uv run {skill-root}/scripts/recon_kit.py staleness <claims.json> --windows '<map>'` — the stale flags are the candidate set. Confirm it in one exchange, re-verify just those claims, and deliver a **delta report** (confirmed / changed / overturned, new sources) appended to `<folder name>.md` with the frontmatter `updated` bumped. Claims outside the set keep their status. An overturned load-bearing claim triggers an explicit warning naming the downstream artifacts that consumed it.

## Deepen

Drill into one dimension or add a new one without touching the rest: mini plan gate, acquire → verify for that slice only (or a drafted follow-up prompt when the user's tool is better placed), merge into `<folder name>.md`, update only the synthesis sections the new material affects — a deepening that changes no conclusion says so.

# Process

For a report the user names or drops ("there's a research report at <path>, process it"):

1. **File it.** Find or create the run folder: if a drafted brief for this topic exists, that folder is the target; otherwise infer type and topic from the report (confirm in one line), bind `{doc_workspace}` (expand the folder name with `uv run {skill-root}/scripts/recon_kit.py slug` as in Draft), and init the memlog. Move or copy the original into `{doc_workspace}/imports/` untouched — full fidelity is preserved there, and nowhere else.
2. **Record provenance** in the memlog: what produced it (which tool or firm), when (ask if not evident — production date drives staleness), and what the user wants decided from it.
3. **Extract.** A subagent (fresh context, firewall rules) reads the import and pulls every claim bearing on the decision into digest files under `{doc_workspace}/digests/` — standard shape `{claim, source, publisher, pub_date, accessed, confidence, class}`, keeping the original's citations (the cited source is the publisher; the import is the via). Multiple imports each get their own digest; contradictions between them are findings, not noise.
4. **Check against the pack**: which of the type's dimensions the material covers, which are open, where its claims fall inside two-source classes but rest on one publisher. Verification per the resolved `validation` level (`references/verification.md`) — at `normal` this is a spot-check of the load-bearing claims only, minutes not hours.
5. **Distill** into `<folder name>.md` per `references/synthesis.md` — the succinct, cited, decision-first summary with full metadata frontmatter (topic, type, decision, `source:` provenance, dates, status). This is the artifact downstream skills read; nobody ever reprocesses the import. Open dimensions are listed honestly with a one-line route: draft a follow-up prompt, or a targeted Run on the gap.
6. Finalize per `references/finalize.md`.

# Run

Native research, when chosen: resolve effort, hold the plan gate, then run the acquisition loop once per dimension of the approved plan, in plan order.

## Effort

Three knobs bundled in a **preset**; any knob pins individually, and **what the user says in the request beats both**.

| Preset (`{workflow.preset}`) | subagents | sources/round | depth |
|---|---|---|---|
| `quick` | low (2) | 5 | 1 |
| `standard` (default) | normal (3) | 8 | 2 |
| `deep` | high (6) | 12 | 3 |

- **subagents** — parallel assistants: `none` (0 — inline, sequential; also the no-subagent-harness fallback), `low` (2), `normal` (3), `high` (6, cap 10 — beyond the 3–5 sweet spot only for genuinely wide work).
- **max_sources_per_round** — distinct sources actually read per dimension per round (cap 25).
- **max_depth** — rounds per dimension: initial pass plus lead-following follow-ups (cap 5). A cap, not a quota — dimensions stop early on coverage or novelty exhaustion.
- **validation** (orthogonal to preset, default `normal`) — rigor rises `normal` < `high` < `max`; level semantics live in `references/verification.md`. Verification happens per dimension as material lands, never as an end-of-run rewrite pass.

`{workflow.subagent_models}` is an ordered model preference for assistants — first available wins; empty means harness default. Keep the lead on the strongest model; researchers at most one tier down; judgment work never on the smallest tier.

## The plan gate

The one hard stop, kept light: decision, type and pack-derived dimensions pruned to it, shape, the **decomposition topology** — *breadth-first* (independent sub-questions: assistants split the dimensions), *depth-first* (one question that needs several perspectives: assistants split by angle or methodology, not by dimension), or *straightforward* (a focused ask: one assistant, a handful of calls, no fan-out — never overinvest in a simple query) — knobs in force and where each came from, which search surfaces exist (harness web search; installed search-shaped MCP tools; `{workflow.external_sources}` — check, don't assume), whether to run the fan-out as a workflow when the harness offers orchestration and `{workflow.use_workflows}` allows, and an honest time estimate (a standard run is minutes; deep runs are tens of minutes and many times the tokens).

Present as a compact checklist, get approval, then: bind `{doc_workspace}` under `{workflow.research_output_path}` — expand the folder name with `uv run {skill-root}/scripts/recon_kit.py slug "<topic>" --type <type> --pattern "{workflow.run_folder_pattern}"` so the same topic always resolves to the same folder — seed `<folder name>.md` from `{workflow.research_template}`, init the memlog (`uv run {project-root}/_bmad/scripts/memlog.py init --workspace {doc_workspace} --field topic="<topic>" --field type="<type>" --field decision="<decision>" --field preset="<preset>"`), log the approved plan as a `decision`, and tell the user the path.

Each dimension then runs in **rounds** — up to the resolved `max_depth` — and the report grows as material lands: the user watches the document build, not a spinner. Every digest is written to `{doc_workspace}/digests/` the moment it exists — one file per assistant per round (`<dimension>-r<round>-<n>.md`), the digest shape below, raw enough to re-derive from.

## Rounds and lead-following

Round 1 pursues the plan's questions **broad-first**: short, wide queries to map what exists, narrowing as the shape emerges — not long specific queries that return nothing. After each round, harvest the leads: new entities worth chasing, unexpected connections, contradictions between sources, and questions the round opened. Contradictions get priority. Promising leads become the next round's brief; note mid-course discoveries in the checkpoint so the user sees the turn happening.

A dimension stops before its round cap when either holds:

- **Coverage** — its plan questions are answered, with the critical claims confirmed per the resolved `validation` level.
- **Novelty exhaustion** — a full round surfaced no new load-bearing claim or lead.

Say which one ended it. Hitting the round cap with open questions is reported as an open question, never silently dropped.

**Stop-and-write valve.** If the run is dragging well past the plan gate's estimate — rounds queuing, budgets mostly spent — stop spawning, synthesize from the digests already on disk, and report the remainder as open questions with a route (a Deepen later, or a drafted prompt for the user's own tool). A shorter honest report beats a longer stale one.

## The fan-out

Fan out researcher assistants for the round — concurrency per the resolved `subagents` level, split by the plan's **topology**: breadth-first gives each assistant independent sub-questions; depth-first gives each a distinct perspective or methodology on the *same* question; straightforward is one assistant with a small budget — never fan out what one focused assistant answers. Each assistant runs behind the **research firewall**: it gets its brief and nothing else — no project files, no ambient context. The brief contains:

- the questions it owns, the decision they serve, and the topic
- its search surfaces (specialized tools first — installed search-shaped MCP tools, `{workflow.external_sources}` entries whose directive matches — then generic search), plus `{workflow.preferred_sources}` first / `{workflow.banned_sources}` never
- the pack's source craft and freshness bars, and the source-quality card below
- its budgets — sources (the round's share of `max_sources_per_round`) and tool calls, scaled to its task: under 5 for a simple lookup, ~5 medium, ~10 hard, 15 for genuinely multi-part, 20 never exceeded. Either budget spent → synthesize what it has
- the query craft: short queries (roughly five words or fewer) beat hyper-specific ones that return nothing; broaden when results are sparse, narrow when abundant; never repeat an identical query on the same tool; after every tool result, pause and evaluate — what did this add, what gap remains, what's the best next query — before firing again
- the epistemics rules verbatim, and the return contract: a digest, not raw results — findings as claims, each with `{claim, source, publisher, pub_date, accessed, confidence, class}`, plus leads worth chasing and what it looked for and could not find

**On each return, write the digest to `{doc_workspace}/digests/` before doing anything else with it.**

Spawn assistants on `{workflow.subagent_models}` when set (first available wins); otherwise the harness default — judgment work never drops to the smallest tier. When subagents are unavailable (or `subagents` is `none`), run the same rounds yourself, sequentially, under the same budgets and the same files-first discipline.

When workflow orchestration was approved at the plan gate, run the fan-out as a workflow: dimensions as parallel pipelines, assistants returning structured digests. The budgets, digest contract, firewall, and stopping rules apply unchanged — and however the acquisition parallelizes, digests land as files and the lead alone writes `<folder name>.md`, committing sections in plan order.

## Source quality

One card, applied by every assistant and the lead alike. Prefer **primary sources** — filings, regulator text, official documentation, original papers, a company's own reported numbers — over aggregators and secondary reporting. Red flags that downgrade confidence on sight: speculative language ("could", "may", projections in future tense presented as findings), marketing register, passive voice with unnamed sources, cherry-picked or unsourced numbers, and aggregators recycling a single upstream report (that's one publisher, however many domains echo it). Answer engines (Perplexity Sonar, Grok, and kin) are aggregators too, however good the synthesis: chase their citations and cite those, never the engine. Conflicts resolve by recency, consistency with adjacent established facts, and publisher quality — never by averaging.

## Synthesize the dimension

When a dimension's rounds are done:

1. Verify at landing per `references/verification.md` — at `normal` validation this is a spot-check of the dimension's load-bearing claims, not a sweep.
2. Write the dimension's section per the pack's skeleton from its digest files — findings woven into prose answering the dimension's questions, every load-bearing claim cited inline `[n]`, confidence flagged where below high, contradictions reported with both sides cited. Append to `<folder name>.md` and add its sources to the running source table.
3. Log one memlog line per source batch (`--type source`) and one per load-bearing claim worth tracking for refresh — `--type claim`, text in the machine-readable shape `ref=[n] status=<verified|unverified|disputed|overturned> class=<class> pub=<YYYY-MM> — <claim>` so `scripts/recon_kit.py tally` and `staleness` can read the ledger; a later status change is a fresh claim line with the same `ref=` (last status wins).
4. Checkpoint: one or two lines in chat — what the dimension found, anything surprising, anything unresolved. Keep moving unless the user speaks up; a mid-run scope change is logged as a `decision` and the plan adjusts. Headless: skip checkpoints entirely.

When all dimensions are done, proceed to `references/synthesis.md` for final assembly.

# The Select Shape

When the decision is **choose between candidates** — technologies, vendors, libraries, platforms, agencies, anything — this method layers over whichever research type fits the subject. The type pack still governs sources, craft, and freshness; this shape governs the flow and the verdict.

1. **Requirements.** What must the winner do, under what constraints — scale, compliance, budget, team skills, existing stack, exit-cost tolerance? Split hard gates from weighted preferences and set the weights. Sources: the ticket for this work, read in full, then the companions it names, `{workflow.persistent_facts}`, and the user; with no ticket, the brief, PRD, architecture document and codebase — web research does not set requirements. **Agree the requirements before any candidate research runs**; interactive runs confirm them even though the plan gate approved the dimension list.
2. **Candidate screen.** Establish the credible field — leaders, strong challengers, one wildcard — and cut anything failing a hard gate. Screen to 3–5 finalists; record the cuts and why. Screening sources ≤ 6 months old — this field moves.
3. **Evidence per criterion.** Score finalists against the requirements using the type pack's dimensions and craft, verified against current versions/offerings. Cite every contested cell; where vendor claims and independent experience diverge, the divergence is a finding.
4. **Cost & lock-in.** Total cost over the product's horizon — license/subscription, hosting, operational load, learning curve — and the cost of leaving. Current pricing pages read directly (≤ 3 mo, always); pricing-change history — a vendor that repriced once will again; migration-away accounts for real exit costs.
5. **Verdict.** The weighted decision matrix — show the scoring, not just totals; a matrix the user can re-weight is worth more than a verdict they must trust. Then: the pick; the named runner-up and the conditions under which it wins instead; the strongest argument against the pick (from the red-team pass when it ran); the cheapest reversibility hedge (abstraction seam, pilot scope, exit test).

**Two-source classes (added to the type's own):** pricing figures; performance/scale numbers; any cell that decides between the top two finalists.

**Staleness:** a selection report older than two quarters should be refreshed before anyone acts on it — say so in the report.

# Synthesis

The report answers the decision — whether the material came from a native Run or a processed import. **Succinct is the contract**: findings and verdicts, not essays; rationale lives in the memlog; a reader gets the decision-relevant truth in minutes. For Process mode this is the whole point — the summary is what downstream consumers read so nobody reprocesses the original, and sections with nothing behind them collapse to a line rather than pad.

Assemble `<folder name>.md` in this order, shaped by `{workflow.audience}`:

1. **Executive summary** — decision-first: what the evidence says to do, the two or three findings that drive that answer, and the biggest caveat. One page maximum, readable standalone. Written last, placed first.
2. **Dimension sections** — already written during the loop; now reconciled: consistent terminology, no duplicated ground, verification statuses and any corrections from the pass applied to the text.
3. **Cross-dimension insights** — what only the *combination* shows (e.g. the market is growing but the regulatory dimension caps the reachable segment; the technically superior option loses on ecosystem health). This section is the harness earning its keep — if there are no cross-dimension insights, say so rather than manufacture them.
4. **Contrary evidence** — when the red-team pass ran and found material; the strongest surviving counter-arguments, cited.
5. **Recommendations** — each bound to the decision and, where the project has them, to what consumes it: the ticket whose Requirements it feeds, or the artifact per the pack's `Feeds` entries (brief section, PRD input, architecture constraint). Each recommendation names its confidence basis; a recommendation resting on low-confidence or disputed claims says so in the same sentence.
6. **Open questions** — what the research could not answer, and what it would take to answer each.
7. **Source appendix** — the numbered source table: `[n] | claim/finding it supports | publisher | pub date | accessed | confidence`, the publisher cell a markdown link to the source URL. Every inline `[n]` resolves here.
8. **Staleness map** — the claims that age fastest, computed not hand-derived: build the claims list (`claim`, `class`, `pub_date`) from the ledger, map the pack's freshness bars to months per class, and run `uv run {skill-root}/scripts/recon_kit.py staleness <claims.json> --windows '<map>'` — render its re-check dates and close by noting the earliest. This is Refresh's work order.

Update the frontmatter (`status: complete`, `updated`, and the verified/unverified counts from `uv run {skill-root}/scripts/recon_kit.py tally {doc_workspace}/.memlog.md` — never hand-counted), log a final `event` in the memlog, and proceed to `references/finalize.md`.

# Verification

The trust layer — the same rules whatever produced the material (a native run's digests or a processed import). Verification happens **as material lands**, per dimension, in fresh-context verifier subagents reading digest files — never as an end-of-run rewrite pass over an hour of accumulated context. Late-pass rewrites degrade reports; landing-time checks improve them.

## The claims ledger

The memlog `claim` entries are the ledger: every claim a decision could rest on, with its class (each pack names its classes — quantitative sizes, pricing, versions/compatibility, regulatory assertions, …), source, publisher, publication date, and status. New claims enter `unverified`; on a Refresh or Deepen run, claims outside the run's scope keep their prior status from the memlog — only new and in-scope claims are (re)checked.

## Levels

Per the resolved `validation` level (request > knob > default `normal`):

- **normal** — spot-check the **load-bearing claims only**: the handful per dimension the recommendation actually rests on. One independent-source check each, at landing. Everything else ships with its single source cited and confidence marked honestly. Fast by design.
- **high** — cross-check every claim in the pack's *two-source classes*, and run the red-team pass on major conclusions regardless of `{workflow.red_team}`.
- **max** — cross-check every ledger claim, run the red-team pass below at full breadth (every major conclusion), and primary-source-priority ranking: where a primary source (filing, regulator text, official docs, original paper) exists, secondary reporting alone does not verify.

Verifier assistants run behind the research firewall on `{workflow.subagent_models}` when set; judgment work never drops to the smallest tier.

**Independent** means a different publisher with different underlying data or reporting — not a syndication, quote, or republication of the first source, and not the same vendor's marketing in two places. An imported report counts as one publisher regardless of how many sources it cites internally; two imports from different tools agreeing is genuine confirmation, and their disagreement is a finding.

Outcomes per claim: **verified** (independent source agrees within tolerance — for quantitative claims, same order of magnitude and direction), **disputed** (independent sources materially disagree — report both figures, both cited; never average), **unverified** (no independent check within budget — the claim stays, flagged, and joins the staleness map), or **overturned** (the weight of evidence contradicts it — corrected in the text, original noted). Every status change lands in the memlog as a fresh `claim` line with the same `ref=` and the new status — last status wins, which is how `scripts/recon_kit.py tally` reads the ledger. A verification outcome adjusts status and flags — it never licenses rewriting a finding's substance beyond what the new evidence says.

Confidence rendered in the report: **high** (verified, fresh, credible publishers), **medium** (single credible source, fresh), **low** (stale, weak publisher, or disputed) — plus the explicit `unverified` flag. Confidence is per-claim, never per-section.

## Red-team pass

The single adversarial mechanism — no other verifier duplicates it. Off by default (`{workflow.red_team}` = `"off"`; `"offer"` proposes it at the plan gate, `"on"` always runs; `high` validation includes it for major conclusions, `max` runs it at full breadth). When it runs: for each major conclusion, a **fresh-context** skeptic subagent — the conclusion and a search budget, no supporting evidence, no run context — hunts for disconfirming evidence: the bear case, failed attempts, contrary data, the strongest good-faith argument the conclusion is wrong.

What comes back is weighed, not appended: a conclusion that survives gets its strongest counter-argument acknowledged in the synthesis; one that doesn't is revised before the report states it. Material findings land in a **Contrary Evidence** section with full citation discipline. Zero findings after a real search is itself reportable — say what was searched for and not found.

#!/usr/bin/env python3
# /// script
# requires-python = ">=3.11"
# ///
"""recon_kit — deterministic helpers for bmad-deep-recon.

The mechanical half of the research workflow: everything here is exact,
repeatable work the LLM should never re-derive by hand. All subcommands
print one JSON object to stdout; diagnostics go to stderr. Exit codes:
0 = pass, 1 = findings that need attention, 2 = usage/parse error.

Subcommands:
  citations RESEARCH_MD
      Cross-check inline [n] markers against the source-appendix table:
      dangling markers (no appendix row) and orphaned rows (never cited).
  tally MEMLOG_MD
      Count memlog entries by type, and claim entries by status.
      Claim lines carry `status=<word>` and optionally `ref=[n]`; for a
      given ref the LAST status wins, so status changes are appends.
  staleness CLAIMS_JSON --windows JSON [--today YYYY-MM-DD]
      Given claims [{claim, class, pub_date}] and a months-per-class map
      (e.g. '{"size/growth": 18, "pricing": 3}'), compute each claim's
      re-check date, flag stale ones, and report the earliest re-check.
  slug TOPIC --type TYPE [--pattern P] [--date YYYY-MM-DD]
      Expand the run-folder pattern deterministically so the same topic
      always lands in the same folder across draft -> process -> refresh.
  escape-sources RESEARCH_MD
      Emit the source-appendix table as HTML with every cell escaped and
      only validated http(s) URLs turned into links, for the briefing.
  anchors RESEARCH_MD --repo PATH [--at REV]
      Re-find each `path:line (symbol) @ sha` and `sha "subject" #pr`
      citation in git. A code claim is anchored, moved, incomplete,
      missing or unknown-commit; a history claim is anchored,
      subject-mismatch, pr-mismatch or unknown-commit.
      moved means the file changed between the cited commit and --at (default HEAD), not that the line shifted; head_line says where the cited text now sits, or null when it is gone.
      incomplete means the symbol is empty; pr-mismatch means the subject ends in (#n) and the citation names no PR or another one; unparsed lists citation-shaped text no claim matched. All three fail the run. symbol_found says whether the symbol's last identifier is in the cited lines or the 80 lines above them; it never fails the run.
"""

from __future__ import annotations

import argparse
import calendar
import html
import json
import os
import posixpath
import re
import subprocess
import sys
import unicodedata
from collections import Counter
from datetime import date, datetime
from pathlib import Path
from urllib.parse import urlparse

MARKER_RE = re.compile(r"\[(\d+)\](?!\()")  # [3] but not a [3](url) link
# URLs may hold one level of balanced parentheses, as Wikipedia's often do.
MD_LINK_RE = re.compile(r"\[([^\]]*)\]\(((?:[^\s()]|\([^\s()]*\))+)\)")
BARE_URL_RE = re.compile(r"https?://(?:[^\s()|\]]|\([^\s()]*\))+")
ROW_ID_RE = re.compile(r"\[(\d+)\]")  # appendix row id: [n], never a bare number
FENCE_RE = re.compile(r"^\s*(`{3,}|~{3,})")  # any indent: fences nest in list items


def out(payload: dict, exit_code: int) -> int:
    print(json.dumps(payload, indent=2, ensure_ascii=False, default=str))
    return exit_code


def read_text(path_arg: str) -> str:
    if path_arg == "-":
        # Decode the bytes ourselves: the locale's encoding may not be UTF-8.
        buffer = getattr(sys.stdin, "buffer", None)
        return buffer.read().decode("utf-8") if buffer is not None else sys.stdin.read()
    return Path(path_arg).read_text(encoding="utf-8")


def strip_fences(text: str) -> str:
    """Blank out fenced code blocks so their contents never count as markers or rows.

    Fences pair CommonMark-style: a block closes only on a line of the opener's
    character, at least as long, with nothing after it.
    """
    lines, open_fence = [], None  # (char, length) while inside a fenced block
    for ln in text.splitlines():
        fence = FENCE_RE.match(ln)
        if open_fence is None:
            if fence:
                open_fence = (fence.group(1)[0], len(fence.group(1)))
            lines.append("" if fence else ln)
            continue
        marker = fence.group(1) if fence else ""
        if marker[:1] == open_fence[0] and len(marker) >= open_fence[1] and ln.strip() == marker:
            open_fence = None
        lines.append("")
    return "\n".join(lines)


def table_cells(line: str) -> list[str]:
    return [c.strip() for c in line.strip().strip("|").split("|")]


def appendix_rows(text: str) -> dict[int, list[str]]:
    """Source-appendix rows: markdown table rows whose first cell is [n]."""
    rows: dict[int, list[str]] = {}
    for ln in text.splitlines():
        stripped = ln.strip()
        if not stripped.startswith("|"):
            continue
        cells = table_cells(stripped)
        if not cells or len(cells) < 2:
            continue
        m = ROW_ID_RE.fullmatch(cells[0])
        if m:
            rows[int(m.group(1))] = cells
    return rows


# --- citations ---------------------------------------------------------------


def cmd_citations(args) -> int:
    text = strip_fences(read_text(args.file))
    rows = appendix_rows(text)
    markers: set[int] = set()
    for ln in text.splitlines():
        stripped = ln.strip()
        if stripped.startswith("|"):
            cells = table_cells(stripped)
            if cells and ROW_ID_RE.fullmatch(cells[0]):
                continue  # an appendix row is not a citation of itself
        markers.update(int(n) for n in MARKER_RE.findall(ln))
    dangling = sorted(markers - set(rows))
    orphaned = sorted(set(rows) - markers)
    ok = not dangling and not orphaned
    return out(
        {
            "markers": sorted(markers),
            "appendix_rows": sorted(rows),
            "dangling_markers": dangling,
            "orphaned_rows": orphaned,
            "ok": ok,
        },
        0 if ok else 1,
    )


# --- tally -------------------------------------------------------------------

ENTRY_RE = re.compile(r"^- (?:\(([\w-]+)(?: by [^)]*)?\)\s*)?(.*)$")


def cmd_tally(args) -> int:
    text = read_text(args.file)
    body = text.split("---", 2)[-1] if text.startswith("---") else text
    by_type: dict[str, int] = {}
    by_ref: dict[int, str] = {}
    unref_status: dict[str, int] = {}
    entries = 0
    for ln in body.splitlines():
        m = ENTRY_RE.match(ln)
        if not m or not ln.startswith("- "):
            continue
        entries += 1
        etype = m.group(1) or "note"
        by_type[etype] = by_type.get(etype, 0) + 1
        if etype == "claim":
            status_m = re.search(r"status=([\w-]+)", m.group(2))
            status = status_m.group(1) if status_m else "unknown"
            ref_m = re.search(r"ref=\[?(\d+)\]?", m.group(2))
            if ref_m:
                by_ref[int(ref_m.group(1))] = status  # last status wins per ref
            else:
                unref_status[status] = unref_status.get(status, 0) + 1
    claims: dict[str, int] = dict(unref_status)
    for status in by_ref.values():
        claims[status] = claims.get(status, 0) + 1
    return out(
        {
            "entries": entries,
            "by_type": dict(sorted(by_type.items())),
            "claims": dict(sorted(claims.items())),
            "claims_total": sum(claims.values()),
        },
        0,
    )


# --- staleness ---------------------------------------------------------------


def parse_date(raw: str) -> date:
    raw = raw.strip()
    for fmt in ("%Y-%m-%d", "%Y-%m", "%Y"):
        try:
            return datetime.strptime(raw, fmt).date()
        except ValueError:
            continue
    raise ValueError(f"unparseable date: {raw!r} (want YYYY[-MM[-DD]])")


def add_months(d: date, months: int) -> date:
    total = d.month - 1 + months
    year, month = d.year + total // 12, total % 12 + 1
    return date(year, month, min(d.day, calendar.monthrange(year, month)[1]))


def cmd_staleness(args) -> int:
    try:
        payload = json.loads(read_text(args.file))
        raw_windows = json.loads(args.windows)
        if not isinstance(raw_windows, dict):
            raise ValueError("--windows must be a JSON object of class -> months")
        windows = {k.lower(): int(v) for k, v in raw_windows.items()}
        today = parse_date(args.today) if args.today else date.today()
    except (TypeError, ValueError) as e:
        print(f"error: {e}", file=sys.stderr)
        return 2
    claims = payload.get("claims") if isinstance(payload, dict) else payload
    if not isinstance(claims, list) or not all(isinstance(c, dict) for c in claims):
        print('error: claims must be a JSON array of objects, or {"claims": [...]}', file=sys.stderr)
        return 2
    results, no_window, stale_count = [], set(), 0
    earliest: date | None = None
    for c in claims:
        cls = str(c.get("class", "")).lower()
        try:
            pub = parse_date(str(c["pub_date"]))
        except (KeyError, ValueError) as e:
            print(f"error in claim {c!r}: {e}", file=sys.stderr)
            return 2
        months = windows.get(cls)
        if months is None:
            no_window.add(cls)
            results.append({**c, "recheck": None, "stale": None})
            continue
        recheck = add_months(pub, months)
        stale = recheck <= today
        stale_count += stale
        earliest = recheck if earliest is None or recheck < earliest else earliest
        results.append({**c, "recheck": recheck.isoformat(), "stale": stale})
    return out(
        {
            "today": today.isoformat(),
            "claims": results,
            "stale_count": stale_count,
            "earliest_recheck": earliest.isoformat() if earliest else None,
            "no_window_classes": sorted(no_window),
        },
        1 if stale_count else 0,
    )


# --- slug --------------------------------------------------------------------


def slugify(text: str, max_len: int = 40) -> str:
    text = unicodedata.normalize("NFKD", text).encode("ascii", "ignore").decode()
    text = re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-")
    return re.sub(r"-{2,}", "-", text)[:max_len].rstrip("-")


def cmd_slug(args) -> int:
    slug = slugify(args.topic)
    if not slug:
        print("error: topic slugified to an empty string", file=sys.stderr)
        return 2
    folder = (
        args.pattern.replace("{research_type}", args.type)
        .replace("{topic_slug}", slug)
        .replace("{date}", args.date or date.today().isoformat())
    )
    return out({"topic_slug": slug, "folder": folder}, 0)


# --- escape-sources ----------------------------------------------------------


def safe_url(raw: str) -> str | None:
    parsed = urlparse(raw)
    return raw if parsed.scheme in ("http", "https") and parsed.netloc else None


def cell_html(cell: str, invalid: list[str]) -> str:
    """Escape a cell; a markdown link or bare URL becomes an <a> only when http(s)."""
    link = MD_LINK_RE.search(cell)
    if link:
        url = safe_url(link.group(2))
        label = html.escape(link.group(1) or link.group(2))
        if url:
            return (
                html.escape(cell[: link.start()])
                + f'<a href="{html.escape(url, quote=True)}" target="_blank" rel="noopener">{label}</a>'
                + html.escape(cell[link.end() :])
            )
        invalid.append(link.group(2))
        return html.escape(cell.replace(link.group(0), link.group(1) or link.group(2)))
    bare = BARE_URL_RE.search(cell)
    if bare:
        url = safe_url(bare.group(0))
        if url:
            escaped = html.escape(url, quote=True)
            return (
                html.escape(cell[: bare.start()])
                + f'<a href="{escaped}" target="_blank" rel="noopener">{escaped}</a>'
                + html.escape(cell[bare.end() :])
            )
        invalid.append(bare.group(0))
    return html.escape(cell)


def cmd_escape_sources(args) -> int:
    text = strip_fences(read_text(args.file))
    rows = appendix_rows(text)
    if not rows:
        print("error: no source-appendix table rows found", file=sys.stderr)
        return 2
    invalid: list[str] = []
    body_rows = []
    for n in sorted(rows):
        cells = rows[n]
        tds = "".join(f"<td>{cell_html(c, invalid)}</td>" for c in cells[1:])
        body_rows.append(f'<tr id="src-{n}"><td>[{n}]</td>{tds}</tr>')
    table = '<table class="sources"><tbody>' + "".join(body_rows) + "</tbody></table>"
    return out({"rows": len(rows), "invalid_urls": invalid, "html": table}, 1 if invalid else 0)


# --- anchors -----------------------------------------------------------------

# A path starts a token, so a URL is never read as one, and may hold a balanced (group) as Next.js routes do.
PATH_CHAR = r"(?:[^\s:`'\"()<>|*]|\([^\s:`'\"()<>|*/]*\))"
CITED_PATH = rf"(?<![\w./:\\)-])({PATH_CHAR}+)"
# A path may hold spaces, so this takes up to five words before it; spaced_path keeps those the commit's tree has.
CODE_CLAIM_RE = re.compile(
    rf"(?<![\w./:\\)-])({PATH_CHAR}+(?: {PATH_CHAR}+){{0,5}})"
    + r":(\d+)(?:-(\d+))?[ \t]*\(([^()\n]*)\)[ \t]*@[ \t]*([0-9a-f]{7,40})\b"
)
# Not after "@ ": that sha ends a code claim, and a quote after it is not a subject.
# A subject may hold quotes (Revert "x"): it ends at a quote not followed by a word or another quote.
HISTORY_CLAIM_RE = re.compile(r"(?<![\w./:\\-])(?<!@ )([0-9a-f]{7,40})[ \t]+\"(.+?)\"(?![\w\"])(?:[ \t]+#(\d+))?")
PR_SUFFIX_RE = re.compile(r"\s*\(#(\d+)\)$")
SUBJECT_END_RE = re.compile(r"\"(?:[ \t]+#(\d+))?")
# Citation-shaped text: what no claim consumed is reported as unparsed.
LOOSE_CODE_RE = re.compile(
    CITED_PATH + r":\d+(?:[-:]\d+)*\b(?:[ \t]*\((?:[^()\n]|\([^()\n]*\))*\))?(?:[ \t]*@[ \t]*[0-9a-fA-F]{7,40}\b)?"
)
FILE_EXT_RE = re.compile(r"\.\w*[A-Za-z]\w*$")
SHA_TAIL_RE = re.compile(r"(?<![\w.@/-])@[ \t]*[0-9a-fA-F]{7,40}\b")  # not user@host or action@sha
CURLY_SUBJECT_RE = re.compile(
    r"(?<![\w./:\\-])[0-9a-fA-F]{7,40}[ \t]+[\u2018\u2019\u201c\u201d][^\n]*?(?:[\u2018\u2019\u201c\u201d]|$)", re.M
)
RANGE_DASH_RE = re.compile(r"(:\d+)[\u2013\u2014](?=\d)")
CURLY_DOUBLE = str.maketrans("\u201c\u201d", '""')
CURLY_ANY = str.maketrans("\u201c\u201d\u2018\u2019", "\"\"''")
# What git prints when the object or path is not there; any other failure is an error.
GIT_MISSING_RE = re.compile(
    r"not a valid object name|does not exist in|exists on disk, but not in|bad file|outside repository", re.I
)


class GitError(Exception):
    """git failed for a reason other than a missing object or path."""

    def __init__(self, proc: subprocess.CompletedProcess[str]):
        super().__init__(f"git {' '.join(proc.args[1:])} exited {proc.returncode}: {proc.stderr.strip()}")


def git(repo: Path, *args: str) -> subprocess.CompletedProcess[str]:
    # Drop GIT_DIR and the rest: a git hook exports them, and they would override --repo.
    env = {k: v for k, v in os.environ.items() if not k.startswith("GIT_")}
    env["LC_ALL"] = "C"  # untranslated messages, which GIT_MISSING_RE reads
    return subprocess.run(
        ["git", *args], cwd=repo, env=env, capture_output=True, text=True, encoding="utf-8", errors="replace"
    )


def raise_unless_missing(proc: subprocess.CompletedProcess[str]) -> None:
    if not GIT_MISSING_RE.search(proc.stderr):
        raise GitError(proc)


def repo_root(path: Path) -> Path | None:
    if not path.exists():
        return None
    proc = git(path if path.is_dir() else path.parent, "rev-parse", "--show-toplevel")
    top = proc.stdout.strip()
    return Path(top) if proc.returncode == 0 and top else None


def blob_lines(repo: Path, rev: str, path: str) -> list[str] | None:
    """The file's lines at rev, or None when rev holds no file at path."""
    proc = git(repo, "cat-file", "blob", f"{rev}:{path}")  # not git show, which lists a directory with exit 0
    if proc.returncode:
        raise_unless_missing(proc)
        return None
    lines = proc.stdout.split("\n")  # as git counts lines; splitlines() also breaks on form feeds
    return lines[:-1] if lines[-1] == "" else lines


def is_commit(repo: Path, sha: str) -> bool:
    proc = git(repo, "cat-file", "-e", f"{sha}^{{commit}}")
    if proc.returncode:
        raise_unless_missing(proc)
    return proc.returncode == 0


def tree_paths(repo: Path, sha: str, cache: dict[str, set[str]]) -> set[str]:
    """Every file path at sha, or none when sha is not a commit."""
    if sha not in cache:
        proc = git(repo, "ls-tree", "-r", "--name-only", "-z", sha) if is_commit(repo, sha) else None
        if proc is not None and proc.returncode:
            raise_unless_missing(proc)
        cache[sha] = set(proc.stdout.split("\0")) if proc is not None and not proc.returncode else set()
    return cache[sha]


def spaced_path(repo: Path, m: re.Match, cache: dict[str, set[str]]) -> tuple[str, int]:
    """The cited path and where it starts: the most trailing words of the match's path the commit's tree has, else the last."""
    words = m[1].split(" ")
    paths = tree_paths(repo, m[5], cache) if len(words) > 1 else set()
    for i in range(len(words) - 1):
        if posixpath.normpath(" ".join(words[i:])) in paths:
            break
    else:
        i = len(words) - 1
    return " ".join(words[i:]), m.start(1) + len(" ".join(words[:i])) + (1 if i else 0)


def history_claim(repo: Path, m: re.Match) -> tuple[str, str, int | None, tuple[int, int]]:
    """sha, subject, pr and span of a history match, its subject run on to the last quote on the line that keeps it a prefix of git's own."""
    sha, subject, pr, span = m[1], m[2], int(m[3]) if m[3] else None, m.span()
    text = m.string
    quotes = [m.end(2) + 1 + i for i, c in enumerate(text[m.end(2) + 1 :].split("\n", 1)[0]) if c == '"']
    if quotes and is_commit(repo, sha):
        log = git(repo, "log", "-1", "--no-show-signature", "--format=%s", sha, "--")
        if log.returncode:
            raise GitError(log)
        actual = norm_subject(log.stdout)
        for q in reversed(quotes):
            if actual.startswith(norm_subject(text[m.start(2) : q])):
                end = SUBJECT_END_RE.match(text, q)
                subject, pr, span = text[m.start(2) : q], int(end[1]) if end[1] else None, (m.start(), end.end())
                break
    return sha, subject, pr, span


def find_block(lines: list[str], block: list[str]) -> int | None:
    """The 1-based line where block starts in lines, or None."""
    return next((i + 1 for i in range(len(lines) - len(block) + 1) if lines[i : i + len(block)] == block), None)


def symbol_found(symbol: str, lines: list[str]) -> bool:
    """Whether the symbol's last identifier stands as a whole word in lines."""
    parts = [part for part in re.split(r"::|[.#\s]", symbol) if part]
    return bool(parts) and re.search(rf"(?<!\w){re.escape(parts[-1])}(?!\w)", "\n".join(lines)) is not None


def check_code(repo: Path, at: str, sha: str, path: str, line: int, end_line: int | None, symbol: str) -> dict:
    unread = {"line_text": None, "changed_since": None, "head_line": None, "symbol_found": None}
    if not is_commit(repo, sha):
        return {"status": "unknown-commit", **unread}
    path = posixpath.normpath(path)  # git reads rev:path literally, so src/../x.md would need a folder named ".."
    lines = blob_lines(repo, sha, path)
    if lines is None or not all(1 <= n <= len(lines) for n in (line, end_line) if n is not None):
        return {"status": "missing", **unread}
    last = max(line, end_line or line)
    block = [text.strip() for text in lines[line - 1 : last]]
    # Literal, so a path such as app/[slug]/page.tsx is not read as a glob.
    diff = git(repo, "--literal-pathspecs", "diff", "--quiet", sha, at, "--", path)
    if diff.returncode not in (0, 1):
        raise GitError(diff)
    if diff.returncode == 0:
        status, head_line = "anchored", line
    else:
        head = [text.strip() for text in blob_lines(repo, at, path) or []]
        status, head_line = "moved", find_block(head, block) or find_block(head, block[:1])
    return {
        "status": status if symbol else "incomplete",
        "line_text": block[0],
        "changed_since": diff.returncode == 1,
        "head_line": head_line,
        "symbol_found": symbol_found(symbol, lines[max(0, line - 81) : last]),
    }


def subject_pr(subject: str) -> int | None:
    m = PR_SUFFIX_RE.search(" ".join(subject.split()))
    return int(m.group(1)) if m else None


def norm_subject(subject: str) -> str:
    return PR_SUFFIX_RE.sub("", " ".join(subject.translate(CURLY_ANY).split())).casefold()


def check_history(repo: Path, sha: str, subject: str, pr: int | None) -> dict:
    if not is_commit(repo, sha):
        return {"status": "unknown-commit", "actual_subject": None, "actual_pr": None}
    log = git(repo, "log", "-1", "--no-show-signature", "--format=%s", sha, "--")
    if log.returncode:
        raise GitError(log)
    actual = log.stdout.strip()
    actual_pr = subject_pr(actual)
    named = {n for n in (pr, subject_pr(subject)) if n is not None}
    if not norm_subject(actual).startswith(norm_subject(subject)):
        status = "subject-mismatch"
    elif actual_pr is not None and named != {actual_pr}:
        status = "pr-mismatch"
    else:
        status = "anchored"
    return {"status": status, "actual_subject": actual, "actual_pr": actual_pr}


def normalize(text: str) -> str:
    """Fold en and em dash ranges and curly double quotes to ASCII, one character for one, so offsets still match."""
    return RANGE_DASH_RE.sub(r"\1-", text.translate(CURLY_DOUBLE))


def unparsed_spans(text: str, norm: str, consumed: list[tuple[int, int]]) -> list[dict]:
    """Citation-shaped spans no claim consumed, overlapping ones merged, as written in text."""
    taken = bytearray(len(text))
    for start, end in consumed:
        taken[start:end] = b"\1" * (end - start)
    spans = [m.span() for m in LOOSE_CODE_RE.finditer(norm) if "/" in m[1] or FILE_EXT_RE.search(m[1])]
    spans += [m.span() for m in SHA_TAIL_RE.finditer(norm)]
    spans += [m.span() for m in CURLY_SUBJECT_RE.finditer(text)]  # text, since norm has no curly double quotes
    merged: list[list[int]] = []
    for start, end in sorted(spans):
        if taken.find(1, start, end) >= 0:
            continue
        if merged and start < merged[-1][1]:
            merged[-1][1] = max(merged[-1][1], end)
        else:
            merged.append([start, end])
    found, line, pos = [], 1, 0
    for start, end in merged:
        line += text.count("\n", pos, start)
        pos = start
        found.append({"text": text[start:end], "line": line})
    return found


def cmd_anchors(args) -> int:
    text = read_text(args.file)
    repo = repo_root(Path(args.repo))
    if repo is None:
        print(f"error: --repo {args.repo} is not inside a git working tree", file=sys.stderr)
        return 2
    at = git(repo, "rev-parse", "--verify", "--quiet", f"{args.at}^{{commit}}").stdout.strip()
    if not at:
        print(f"error: --at {args.at} does not name a commit", file=sys.stderr)
        return 2
    norm = normalize(text)
    trees: dict[str, set[str]] = {}
    code_matches = [(m, *spaced_path(repo, m, trees)) for m in CODE_CLAIM_RE.finditer(norm)]
    code_claims = Counter(
        (path, int(m[2]), int(m[3]) if m[3] else None, m[4].strip(), m[5]) for m, path, _ in code_matches
    )
    history_matches: list[tuple[str, str, int | None, tuple[int, int]]] = []
    for m in HISTORY_CLAIM_RE.finditer(norm):
        if not history_matches or m.start() >= history_matches[-1][3][1]:  # not inside a subject run on past its quote
            history_matches.append(history_claim(repo, m))
    history_claims = Counter((sha, subject, pr) for sha, subject, pr, _ in history_matches)
    code = [
        {
            "path": path,
            "line": line,
            "end_line": end_line,
            "symbol": symbol,
            "commit": sha,
            **check_code(repo, at, sha, path, line, end_line, symbol),
            "occurrences": n,
        }
        for (path, line, end_line, symbol, sha), n in code_claims.items()
    ]
    history = [
        {"commit": sha, "subject": subject, "pr": pr, **check_history(repo, sha, subject, pr), "occurrences": n}
        for (sha, subject, pr), n in history_claims.items()
    ]
    consumed = [(start, m.end()) for m, _, start in code_matches] + [span for *_, span in history_matches]
    unparsed = unparsed_spans(text, norm, consumed)
    statuses = Counter(c["status"] for c in code + history)
    failing = ("incomplete", "missing", "unknown-commit", "subject-mismatch", "pr-mismatch")
    counts = {"code": len(code), "history": len(history)}
    for status in ("anchored", "moved", *failing):
        counts[status.replace("-", "_")] = statuses[status]
    counts["unparsed"] = len(unparsed)
    failed = sum(statuses[status] for status in failing) + len(unparsed)
    return out(
        {
            "file": args.file,
            "repo": str(repo),
            "at": at,
            "code": code,
            "history": history,
            "unparsed": unparsed,
            "counts": counts,
        },
        1 if failed else 0,
    )


# --- entry point -------------------------------------------------------------


def main(argv: list[str] | None = None) -> int:
    p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    sub = p.add_subparsers(dest="cmd", required=True)

    pc = sub.add_parser("citations", help="cross-check [n] markers vs the source appendix")
    pc.add_argument("file", help="path to the research report (or - for stdin)")
    pc.set_defaults(func=cmd_citations)

    pt = sub.add_parser("tally", help="count memlog entries by type and claims by status")
    pt.add_argument("file", help="path to .memlog.md (or - for stdin)")
    pt.set_defaults(func=cmd_tally)

    ps = sub.add_parser("staleness", help="compute re-check dates from freshness windows")
    ps.add_argument("file", help="claims JSON: [{claim, class, pub_date}] (or - for stdin)")
    ps.add_argument("--windows", required=True, help="JSON months-per-class map, e.g. '{\"pricing\": 3}'")
    ps.add_argument("--today", help="override today's date (YYYY-MM-DD)")
    ps.set_defaults(func=cmd_staleness)

    pg = sub.add_parser("slug", help="expand the run-folder pattern deterministically")
    pg.add_argument("topic", help="research topic text")
    pg.add_argument("--type", required=True, help="research type code (e.g. market)")
    pg.add_argument(
        "--pattern",
        default="research-{topic_slug}",
        help="folder pattern (default: research-{topic_slug})",
    )
    pg.add_argument("--date", help="override date (YYYY-MM-DD; default today)")
    pg.set_defaults(func=cmd_slug)

    pe = sub.add_parser("escape-sources", help="source appendix as escaped HTML with validated links")
    pe.add_argument("file", help="path to the research report (or - for stdin)")
    pe.set_defaults(func=cmd_escape_sources)

    pa = sub.add_parser(
        "anchors",
        help="re-find each code and history citation in git",
        description=(
            "moved means the file changed between the cited commit and --at (default HEAD), not that the line shifted; head_line says where the cited text now sits, or null when it is gone. "
            "incomplete means the symbol is empty; pr-mismatch means the subject ends in (#n) and the citation names no PR or another one; unparsed lists citation-shaped text no claim matched. All three fail the run. symbol_found says whether the symbol's last identifier is in the cited lines or the 80 lines above them; it never fails the run."
        ),
    )
    pa.add_argument("file", help="path to the research report (or - for stdin)")
    pa.add_argument("--repo", required=True, help="the git working tree, or any path inside it")
    pa.add_argument("--at", default="HEAD", help="revision to compare against (default: HEAD)")
    pa.set_defaults(func=cmd_anchors)

    args = p.parse_args(argv)
    try:
        return args.func(args)
    except (FileNotFoundError, GitError) as e:
        print(f"error: {e}", file=sys.stderr)
        return 2


if __name__ == "__main__":
    if sys.platform == "win32":
        # Piped output on Windows defaults to a legacy code page, not UTF-8.
        sys.stdout.reconfigure(encoding="utf-8")
        sys.stderr.reconfigure(encoding="utf-8")
    sys.exit(main())

#!/usr/bin/env python3
# /// script
# requires-python = ">=3.11"
# ///
"""Tests for recon_kit.py."""

import io
import json
import os
import shutil
import subprocess
import sys
import tempfile
import unittest
from contextlib import redirect_stderr, redirect_stdout
from datetime import date
from pathlib import Path
from unittest import mock

sys.path.insert(0, str(Path(__file__).resolve().parent.parent))

import recon_kit
from recon_kit import add_months, main, parse_date, slugify

REPORT = """---
title: 'market research: widgets'
---

# Report

The market is growing fast [1] and pricing clusters at $20 [2].
An uncited assertion sits here [4].

```
[9] inside a fence never counts
```

## Source appendix

| [n] | Supports | Publisher | Pub date | Accessed | Confidence |
| --- | --- | --- | --- | --- | --- |
| [1] | market growth | [Gartner](https://example.com/g) | 2026-01 | 2026-07-01 | high |
| [2] | pricing | [Acme](javascript:alert(1)) | 2026-05 | 2026-07-01 | medium |
| [3] | never cited | https://example.com/x | 2025-11 | 2026-07-01 | low |
"""

MEMLOG = """---
topic: widgets
updated: 2026-07-22T10:00
---

- (decision) plan approved
- (source) round 1 batch: 4 sources
- (claim) ref=[1] status=verified class=size/growth pub=2026-01 — market growing 12% CAGR
- (claim) ref=[2] status=unverified class=pricing pub=2026-05 — pricing clusters at $20
- (claim) ref=[2] status=verified class=pricing pub=2026-05 — confirmed by second source
- (claim) status=unverified class=behavior pub=2025-03 — users churn at day 8
- (event) dimension 1 complete
"""


def run(argv):
    buf = io.StringIO()
    with redirect_stdout(buf):
        code = main(argv)
    return code, json.loads(buf.getvalue())


class CitationsTest(unittest.TestCase):
    def test_cross_check(self):
        with tempfile.TemporaryDirectory() as tmp:
            report = Path(tmp) / "report.md"
            report.write_text(REPORT, encoding="utf-8")
            code, result = run(["citations", str(report)])
        self.assertEqual(result["dangling_markers"], [4])
        self.assertEqual(result["orphaned_rows"], [3])
        self.assertNotIn(9, result["markers"])  # fenced content ignored
        self.assertEqual(code, 1)


def run_text(argv, text, name="report.md"):
    with tempfile.TemporaryDirectory() as tmp:
        f = Path(tmp) / name
        f.write_text(text, encoding="utf-8")
        return run([argv[0], str(f), *argv[1:]])


class CitationsEdgeTest(unittest.TestCase):
    def test_a_bare_number_table_is_not_the_source_appendix(self):
        text = (
            "Growth is fast [1].\n\n| Rank | Vendor |\n| --- | --- |\n| 1 | Acme |\n| 2 | Beta |\n| 3 | Gamma |\n\n"
            "| [n] | Supports |\n| --- | --- |\n| [1] | growth |\n"
        )
        code, result = run_text(["citations"], text)
        self.assertEqual(result["appendix_rows"], [1])
        self.assertTrue(result["ok"])
        self.assertEqual(code, 0)

    def test_tilde_and_long_backtick_fences(self):
        text = (
            "~~~\n[7] tilde fenced\n~~~\n\n"
            "````\n```\n[8] still fenced after an inner three-backtick line\n```\n````\n\n"
            "Read after the close [1].\n\n| [n] | Supports |\n| --- | --- |\n| [1] | x |\n"
        )
        code, result = run_text(["citations"], text)
        self.assertEqual(result["markers"], [1])
        self.assertTrue(result["ok"])

    def test_a_fence_nested_in_a_list_item_hides_its_markers(self):
        text = (
            "- Example:\n\n    ```\n    [5] inside a list-nested fence\n    ```\n\n"
            "Cited [1].\n\n| [n] | Supports |\n| --- | --- |\n| [1] | x |\n"
        )
        code, result = run_text(["citations"], text)
        self.assertEqual(result["markers"], [1])
        self.assertTrue(result["ok"])


class TallyTest(unittest.TestCase):
    def test_last_status_wins_per_ref(self):
        with tempfile.TemporaryDirectory() as tmp:
            log = Path(tmp) / "memlog.md"
            log.write_text(MEMLOG, encoding="utf-8")
            code, result = run(["tally", str(log)])
        self.assertEqual(result["by_type"]["claim"], 4)
        self.assertEqual(result["claims"], {"unverified": 1, "verified": 2})
        self.assertEqual(result["claims_total"], 3)  # ref=[2] counted once
        self.assertEqual(code, 0)


class StalenessTest(unittest.TestCase):
    def test_dates(self):
        self.assertEqual(parse_date("2026-01"), date(2026, 1, 1))
        self.assertEqual(add_months(date(2026, 1, 31), 1), date(2026, 2, 28))

    def test_windows(self):
        claims = json.dumps(
            [
                {"claim": "sizing", "class": "size/growth", "pub_date": "2024-06"},
                {"claim": "pricing", "class": "pricing", "pub_date": "2026-06"},
                {"claim": "odd", "class": "unmapped", "pub_date": "2026-06"},
            ]
        )
        with tempfile.TemporaryDirectory() as tmp:
            f = Path(tmp) / "claims.json"
            f.write_text(claims, encoding="utf-8")
            code, result = run(
                [
                    "staleness",
                    str(f),
                    "--windows",
                    '{"size/growth": 18, "pricing": 3}',
                    "--today",
                    "2026-07-22",
                ]
            )
        self.assertEqual(result["stale_count"], 1)  # sizing recheck 2025-12 < today
        self.assertEqual(result["earliest_recheck"], "2025-12-01")
        self.assertEqual(result["no_window_classes"], ["unmapped"])
        self.assertEqual(code, 1)

    def test_malformed_claims_exit_2(self):
        for payload in ('{"x": 1}', '{"claims": [5]}'):
            err = io.StringIO()
            with tempfile.TemporaryDirectory() as tmp:
                f = Path(tmp) / "claims.json"
                f.write_text(payload, encoding="utf-8")
                with redirect_stderr(err), redirect_stdout(io.StringIO()):
                    code = main(["staleness", str(f), "--windows", '{"pricing": 3}'])
            self.assertEqual(code, 2, payload)
            self.assertTrue(err.getvalue().startswith("error: "), payload)

    def test_stdin_is_read_as_utf8_whatever_the_locale(self):
        claims = json.dumps([{"claim": "café prices", "class": "pricing", "pub_date": "2026-06"}], ensure_ascii=False)
        stdin = io.TextIOWrapper(io.BytesIO(claims.encode("utf-8")), encoding="latin-1")
        original = sys.stdin
        sys.stdin = stdin
        try:
            code, result = run(["staleness", "-", "--windows", '{"pricing": 3}', "--today", "2026-07-01"])
        finally:
            sys.stdin = original
        self.assertEqual(result["claims"][0]["claim"], "café prices")
        self.assertEqual(code, 0)


class SlugTest(unittest.TestCase):
    def test_deterministic_folder(self):
        self.assertEqual(slugify("Créme Brûlée: AI Tools!"), "creme-brulee-ai-tools")
        code, result = run(["slug", "SMB Accounting SaaS", "--type", "market"])
        self.assertEqual(result["folder"], "research-smb-accounting-saas")
        self.assertEqual(code, 0)
        pattern = "{research_type}-{topic_slug}-{date}"
        code, result = run(
            ["slug", "SMB Accounting SaaS", "--type", "market", "--date", "2026-07-22", "--pattern", pattern]
        )
        self.assertEqual(result["folder"], "market-smb-accounting-saas-2026-07-22")
        self.assertEqual(code, 0)


class EscapeSourcesTest(unittest.TestCase):
    def test_escaping_and_url_validation(self):
        with tempfile.TemporaryDirectory() as tmp:
            report = Path(tmp) / "report.md"
            report.write_text(REPORT, encoding="utf-8")
            code, result = run(["escape-sources", str(report)])
        self.assertEqual(result["rows"], 3)
        self.assertTrue(any(u.startswith("javascript:") for u in result["invalid_urls"]))
        self.assertNotIn("javascript:", result["html"])  # never linked
        self.assertIn('href="https://example.com/g"', result["html"])
        self.assertIn('id="src-1"', result["html"])
        self.assertEqual(code, 1)

    def test_urls_keep_balanced_parentheses(self):
        text = (
            "| [n] | Supports | Publisher |\n| --- | --- | --- |\n"
            "| [1] | a | [Rust](https://en.wikipedia.org/wiki/Rust_(programming_language)) |\n"
            "| [2] | b | https://en.wikipedia.org/wiki/Go_(programming_language) |\n"
        )
        code, result = run_text(["escape-sources"], text)
        self.assertIn('href="https://en.wikipedia.org/wiki/Rust_(programming_language)"', result["html"])
        self.assertIn('href="https://en.wikipedia.org/wiki/Go_(programming_language)"', result["html"])
        self.assertEqual(result["invalid_urls"], [])
        self.assertEqual(code, 0)


SESSION_OLD = """class SessionStore:
    def load(self, key):
        return self.cache[key]

    def evict(self, key):
        del self.cache[key]
"""

# load moves from line 2 to line 5; evict and the old return line are gone.
SESSION_NEW = """import os


class SessionStore:
    def load(self, key):
        return self.cache.get(key)
"""


def git(repo, *args):
    """git in a fixture repository, shut off from the machine's config and from a hook's GIT_DIR."""
    env = {k: v for k, v in os.environ.items() if not k.startswith("GIT_")}
    env.update(GIT_CONFIG_GLOBAL=os.devnull, GIT_CONFIG_NOSYSTEM="1")
    ident = ["-c", "user.name=test", "-c", "user.email=test@example.com", "-c", "commit.gpgsign=false"]
    proc = subprocess.run(["git", *ident, *args], cwd=repo, env=env, capture_output=True, text=True, check=True)
    return proc.stdout.strip()


@unittest.skipUnless(shutil.which("git"), "git is not on PATH")
class AnchorsTest(unittest.TestCase):
    def setUp(self):
        tmp = tempfile.TemporaryDirectory()
        self.addCleanup(tmp.cleanup)
        self.report = Path(tmp.name) / "report.md"
        self.repo = Path(tmp.name) / "repo"
        self.repo.mkdir()
        git(self.repo, "init", "-q")
        self.old = self.commit(
            "add session store",
            {
                "README.md": "# demo\n",
                "app/[id]/page.tsx": "export default Page\n",
                "app/(site)/page.tsx": "export default Page\n",
                "app/i/page.tsx": "a\n",
                "src/legacy.py": "def old():\n    pass\n",
                "src/session.py": SESSION_OLD,
            },
        )
        (self.repo / "src" / "legacy.py").unlink()
        # app/i/page.tsx changes too: read as a glob, app/[id]/page.tsx would match it.
        self.new = self.commit("Make eviction one-way (#412)", {"src/session.py": SESSION_NEW, "app/i/page.tsx": "b\n"})

    def commit(self, subject, files):
        for name, text in files.items():
            path = self.repo / name
            path.parent.mkdir(parents=True, exist_ok=True)
            path.write_text(text, encoding="utf-8", newline="\n")
        git(self.repo, "add", "-A")
        git(self.repo, "commit", "-q", "-m", subject)
        return git(self.repo, "rev-parse", "HEAD")

    def anchors(self, doc, *extra):
        self.report.write_text(doc, encoding="utf-8")
        return run(["anchors", str(self.report), "--repo", str(self.repo), *extra])

    def test_an_unchanged_file_is_anchored(self):
        old = self.old[:7]
        doc = (
            f"`README.md:1 (title) @ {old}` `app/[id]/page.tsx:1 (Page) @ {old}` (app/(site)/page.tsx:1 (Page) @ {old})"
        )
        self.report.write_text(doc, encoding="utf-8")
        code, result = run(["anchors", str(self.report), "--repo", str(self.repo / "src")])
        readme, page, group = result["code"]
        self.assertEqual(
            (readme["status"], readme["line_text"], readme["changed_since"], readme["head_line"]),
            ("anchored", "# demo", False, 1),
        )
        self.assertEqual(
            (page["status"], group["path"], group["status"]), ("anchored", "app/(site)/page.tsx", "anchored")
        )
        self.assertTrue(Path(result["repo"]).samefile(self.repo))  # found from a subfolder
        self.assertEqual(result["at"], self.new)
        self.assertEqual(code, 0)

    def test_a_moved_line_is_found_at_head(self):
        doc = f"`src/session.py:2-3 (SessionStore.load) @ {self.old[:7]}`\n"
        code, result = self.anchors(doc)
        claim = result["code"][0]
        self.assertEqual(
            (claim["status"], claim["end_line"], claim["line_text"], claim["head_line"]),
            ("moved", 3, "def load(self, key):", 5),
        )
        self.assertEqual(code, 0)
        code, result = self.anchors(doc, "--at", self.old[:7])
        self.assertEqual((result["at"], result["code"][0]["status"]), (self.old, "anchored"))

    def test_a_moved_range_is_matched_whole_before_its_first_line(self):
        models = "class A:\n    def __init__(self):\n        self.a = 1\n\n\nclass B:\n    def __init__(self):\n        self.b = 2\n"
        cited = self.commit("add models", {"src/models.py": models})[:7]
        self.commit("import os in models", {"src/models.py": "import os\n\n\n" + models})
        code, result = self.anchors(f"`src/models.py:7-8 (B.__init__) @ {cited}` `src/models.py:8-7 (B) @ {cited}`")
        self.assertEqual(
            [(c["status"], c["line_text"], c["head_line"]) for c in result["code"]],
            [("moved", "def __init__(self):", 10), ("moved", "self.b = 2", 11)],
        )
        self.assertEqual(code, 0)

    def test_head_line_is_null_when_the_text_or_the_file_is_gone(self):
        old = self.old[:7]
        code, result = self.anchors(f"`src/session.py:5 (SessionStore.evict) @ {old}` `src/legacy.py:1 (old) @ {old}`")
        self.assertEqual(
            [(c["path"], c["status"], c["head_line"]) for c in result["code"]],
            [("src/session.py", "moved", None), ("src/legacy.py", "moved", None)],
        )
        self.assertEqual(code, 0)

    def test_missing_files_lines_and_commits_exit_1(self):
        old = self.old[:7]
        code, result = self.anchors(
            f"`src/nope.py:1 (gone) @ {old}`, `src/session.py:7 (past) @ {old}`, "
            f"`src/session.py:6-9 (SessionStore.evict) @ {old}`"
        )
        self.assertEqual([c["status"] for c in result["code"]], ["missing"] * 3)
        self.assertEqual(code, 1)
        code, result = self.anchors('`README.md:1 (title) @ deadbeef` and `deadbeef "never made"`')
        self.assertEqual([c["status"] for c in result["code"] + result["history"]], ["unknown-commit"] * 2)
        self.assertEqual(result["counts"]["unknown_commit"], 2)
        self.assertEqual(code, 1)

    def test_a_cited_path_is_normalized_for_git_and_reported_as_written(self):
        old = self.old[:7]
        code, result = self.anchors(
            f"`src/../README.md:1 (title) @ {old}` `app/./[id]/page.tsx:1 (Page) @ {old}` "
            f"`src/../../repo/README.md:1 (title) @ {old}`"
        )
        self.assertEqual(
            [(c["path"], c["status"]) for c in result["code"]],
            [
                ("src/../README.md", "anchored"),
                ("app/./[id]/page.tsx", "anchored"),
                ("src/../../repo/README.md", "missing"),  # normalizes to ../repo, outside the repository
            ],
        )
        self.assertEqual(code, 1)

    def test_history_subjects_match_exactly_or_as_a_prefix(self):
        old, new = self.old[:7], self.new[:7]
        code, result = self.anchors(f'Added in `{old} "add session store"`, then `{new} "make eviction" #412`.')
        exact, prefix = result["history"]
        self.assertEqual((exact["status"], exact["actual_pr"]), ("anchored", None))
        self.assertEqual(
            (prefix["status"], prefix["pr"], prefix["actual_subject"], prefix["actual_pr"]),
            ("anchored", 412, "Make eviction one-way (#412)", 412),
        )
        self.assertEqual(code, 0)
        code, result = self.anchors(f'`{old} "add the session cache"`')
        self.assertEqual(result["history"][0]["status"], "subject-mismatch")
        self.assertEqual(code, 1)

    def test_a_quote_after_a_code_claim_sha_is_not_a_history_claim(self):
        old, new = self.old[:7], self.new[:7]
        code, result = self.anchors(
            f'README.md:1 (title) @ {old} "the title"\n{old} "add session store"\nThen `{new} "make eviction" #412`.\n'
        )
        self.assertEqual([c["path"] for c in result["code"]], ["README.md"])
        self.assertEqual(
            [(h["subject"], h["pr"]) for h in result["history"]], [("add session store", None), ("make eviction", 412)]
        )
        self.assertEqual(code, 0)

    def test_fenced_claims_count_and_duplicates_are_listed_once(self):
        old = self.old[:7]
        claim = f"README.md:1 (title) @ {old}"
        code, result = self.anchors(f'See `{claim}`.\n\n```text\n{claim}\n{old} "add session store"\n```\n')
        self.assertEqual([c["occurrences"] for c in result["code"]], [2])
        self.assertEqual(result["history"][0]["status"], "anchored")  # cited only inside the fence
        self.assertEqual(code, 0)

    def test_stdin(self):
        doc = f"`README.md:1 (title) @ {self.old[:7]}`\n"
        original = sys.stdin
        sys.stdin = io.TextIOWrapper(io.BytesIO(doc.encode("utf-8")))
        try:
            code, result = run(["anchors", "-", "--repo", str(self.repo)])
        finally:
            sys.stdin = original
        self.assertEqual((result["file"], result["code"][0]["status"]), ("-", "anchored"))
        self.assertEqual(code, 0)

    def test_no_claims_exit_0_and_usage_errors_exit_2(self):
        code, result = self.anchors("No citations here.\n")
        self.assertEqual((result["code"], result["history"], code), ([], [], 0))
        plain = self.repo.parent / "plain"
        plain.mkdir()
        for repo, at in ((plain, "HEAD"), (self.repo, "no-such-rev")):
            err = io.StringIO()
            with redirect_stderr(err), redirect_stdout(io.StringIO()):
                code = main(["anchors", str(self.report), "--repo", str(repo), "--at", at])
            self.assertEqual(code, 2, repo)
            self.assertTrue(err.getvalue().startswith("error: "), repo)

    def test_dash_ranges_and_curly_quoted_subjects_parse(self):
        old = self.old[:7]
        code, result = self.anchors(
            f"`src/session.py:2\u20133 (SessionStore.load) @ {old}`, `src/session.py:5\u20146 (SessionStore.evict) @ {old}`"
            f" and {old} \u201cadd session store\u201d.",
            "--at",
            old,
        )
        self.assertEqual(
            [(c["line"], c["end_line"], c["status"]) for c in result["code"]], [(2, 3, "anchored"), (5, 6, "anchored")]
        )
        self.assertEqual([(h["subject"], h["status"]) for h in result["history"]], [("add session store", "anchored")])
        self.assertEqual((result["unparsed"], code), ([], 0))

    def test_citation_shaped_text_no_claim_matched_is_unparsed(self):
        old = self.old[:7]
        code, result = self.anchors(
            "Not citations: https://example.com:8080/a/b.py:3, localhost:3000, 127.0.0.1:8080, 10:30\u201311:00,\n"
            f"`uses: actions/checkout@{self.old}`, dev@{old}.io.\n"
            f"No commit `src/session.py:2`, a range `config/app.yaml:4\u20136 @ {old}`, empty `src/session.py:2 ()`.\n"
            f"No path shape `Dockerfile:3 @ {old}`, `src/session.py:2 (load()) @ {old}`, `src/session.py:2:5 (load) @ {old}`.\n"
            f"Upper case `README.md:1 (title) @ ABCDEF0`, single quotes {old} \u2018add session store\u2019.\n"
        )
        self.assertEqual(
            [(u["line"], u["text"]) for u in result["unparsed"]],
            [
                (3, "src/session.py:2"),
                (3, f"config/app.yaml:4\u20136 @ {old}"),
                (3, "src/session.py:2 ()"),
                (4, f"@ {old}"),
                (4, f"src/session.py:2 (load()) @ {old}"),
                (4, f"src/session.py:2:5 (load) @ {old}"),
                (5, "README.md:1 (title) @ ABCDEF0"),
                (5, f"{old} \u2018add session store\u2019"),
            ],
        )
        self.assertEqual((result["code"], result["history"], result["counts"]["unparsed"], code), ([], [], 8, 1))

    def test_an_empty_symbol_is_incomplete(self):
        old = self.old[:7]
        code, result = self.anchors(
            f"`README.md:1 () @ {old}` `src/session.py:2 ( ) @ {old}` `README.md:1 () @ deadbeef`"
        )
        self.assertEqual(
            [(c["status"], c["line_text"]) for c in result["code"]],
            [("incomplete", "# demo"), ("incomplete", "def load(self, key):"), ("unknown-commit", None)],
        )
        self.assertEqual(code, 1)
        self.assertEqual((result["counts"]["incomplete"], result["unparsed"]), (2, []))
        self.assertEqual([c["symbol_found"] for c in result["code"]], [False, False, None])

    def test_a_path_with_spaces_parses_whole_and_words_before_it_are_left_out(self):
        spaced = self.commit("add a spaced file", {"src/My File.java": "class A {\n  void load() {}\n}\n"})[:7]
        old = self.old[:7]
        code, result = self.anchors(
            f"`src/My File.java:2 (A.load) @ {spaced}`\nThe loader in src/My File.java:2 (A.load) @ {spaced} reads it, "
            f"as does src/session.py:2 (SessionStore.load) @ {old}. Gone: src/No File.java:1 (A) @ {spaced}."
        )
        self.assertEqual(
            [(c["path"], c["status"], c["occurrences"]) for c in result["code"]],
            [("src/My File.java", "anchored", 2), ("src/session.py", "moved", 1), ("File.java", "missing", 1)],
        )
        self.assertEqual((result["unparsed"], code), ([], 1))

    def test_a_subject_with_a_quoted_phrase_parses_whole_with_its_pr(self):
        quoted = self.commit('fix: handle "cached" sessions (#412)', {"README.md": "# demo 2\n"})[:7]
        code, result = self.anchors(
            f'`{quoted} "fix: handle "cached" sessions (#412)" #412` and then {quoted} "fix: handle "cached" sessions" #412 '
            f'landed. Reverted in `{self.old[:7]} "add session store"`.'
        )
        self.assertEqual(
            [(h["subject"], h["pr"], h["status"]) for h in result["history"]],
            [
                ('fix: handle "cached" sessions (#412)', 412, "anchored"),
                ('fix: handle "cached" sessions', 412, "anchored"),
                ("add session store", None, "anchored"),
            ],
        )
        self.assertEqual((result["unparsed"], code), ([], 0))

    def test_a_subject_ending_in_a_pr_needs_the_citation_to_name_it(self):
        old, new = self.old[:7], self.new[:7]
        code, result = self.anchors(
            f'`{new} "Make eviction one-way"` `{new} "Make eviction one-way" #999` `{new} "Make eviction one-way (#412)"` '
            f'`{new} "Make eviction one-way (#412)" #999` `{old} "add session store" #7`'
        )
        self.assertEqual(
            [(h["status"], h["actual_pr"]) for h in result["history"]],
            [("pr-mismatch", 412), ("pr-mismatch", 412), ("anchored", 412), ("pr-mismatch", 412), ("anchored", None)],
        )
        self.assertEqual((result["counts"]["pr_mismatch"], code), (3, 1))

    def test_symbol_found_is_reported_and_never_fails_the_run(self):
        big = self.commit("add a long file", {"src/big.py": "def target():\n" + "    pass\n" * 90})[:7]
        old = self.old[:7]
        code, result = self.anchors(
            f"`src/big.py:81 (target) @ {big}` `src/big.py:82 (target) @ {big}` "
            f"`src/session.py:3 (SessionStore.load) @ {old}` `src/session.py:2 (SessionStore.loa) @ {old}` "
            f"`README.md:1 (title) @ {old}`"
        )
        self.assertEqual(
            [(c["status"], c["symbol_found"]) for c in result["code"]],
            [("anchored", True), ("anchored", False), ("moved", True), ("moved", False), ("anchored", False)],
        )
        self.assertEqual(code, 0)
        code, result = self.anchors(f"`src/nope.py:1 (target) @ {old}` `README.md:1 (title) @ deadbeef`")
        self.assertEqual([c["symbol_found"] for c in result["code"]], [None, None])

    def test_git_errors_other_than_a_missing_object_exit_2(self):
        real_git = recon_kit.git
        self.report.write_text(
            f'`src/session.py:2 (load) @ {self.old[:7]}` `{self.new[:7]} "Make eviction one-way" #412`',
            encoding="utf-8",
        )
        for failing in ("-e", "blob", "diff", "log"):

            def offline(repo, *args, failing=failing):
                if failing in args:
                    stderr = "fatal: could not fetch 3f2a9c1 from promisor remote\n"
                    return subprocess.CompletedProcess(["git", *args], 128, "", stderr)
                return real_git(repo, *args)

            err = io.StringIO()
            with self.subTest(failing), mock.patch.object(recon_kit, "git", offline):
                with redirect_stderr(err), redirect_stdout(io.StringIO()):
                    code = main(["anchors", str(self.report), "--repo", str(self.repo)])
                self.assertEqual(code, 2)
                self.assertTrue(err.getvalue().startswith("error: git "))
                self.assertIn("could not fetch 3f2a9c1 from promisor remote", err.getvalue())


if __name__ == "__main__":
    unittest.main()

---
name: bmad-deep-recon
description: 'Decision-grade research on a topic or this repository. Use when the user says "deep recon", "research this", "draft a research prompt", "process this research report", "market research", "domain research", "technical research", "competitor research", "what users say about X", "literature review", "help me choose between", "research the codebase", "research how X works here" or "why is X built this way". Not for a quick code question answered in chat.'
---

# BMad Deep Recon

## Overview

You are **Deep Recon** — a research director, not a search engine. Your value is choosing research worth running and turning whatever comes back into a decision-grade artifact this project consumes without reprocessing. Every engagement serves a **decision** — enter a market, pick a stack, scope a product, commit to a domain — and is shaped by it from the first question to the final artifact.

Three services, freely combined — each detailed in its reference: **Draft** a deep-research prompt the user runs in their own tool, **Process** a finished report into the succinct cited summary downstream skills read, or **Run** the research here through parallel web fan-out. Draft → run externally → Process is the natural loop; Run is fully capable on its own. A fourth, **Codebase**, researches this repository; `references/codebase.md` governs it wherever it differs from the rules here.

**Epistemics — two standing rules, inherited verbatim by every subagent you spawn:**

1. **Never conclude from training data alone.** What you already know proposes hypotheses, queries, and structure; conclusions require evidence retrieved or imported *this run*. A claim you cannot evidence is stated as an unverified belief or not at all.
2. **The research firewall.** Project context — tickets, briefs, PRDs, code, memory, `{workflow.persistent_facts}` — shapes *what to ask*, never *what is true*. It is inadmissible as evidence: every claim in a research artifact traces to a digest or import file with a source. Research subagents receive only their brief — no project files, no ambient context — unless the plan explicitly grants a named document.

## How you work

- **Nothing exists until it is a file.** Every digest, import extraction, and report section is written to the run folder the moment it lands — the conversation is a control channel, never the store. A run that dies mid-flight resumes from disk with nothing lost.
- **Extract, don't ingest.** Raw reports and search results never enter the parent context whole; subagents return relevance-filtered digests, and the parent reads digest files JIT.
- **A claim is a sentence with a source.** Publisher, publication date, access date. No naked numbers.
- **Report what is real.** Thin public data is reported as thin, absence of evidence is a finding, and freshness is part of truth — each pack sets windows per claim class; a market size from three years ago is history, not fact.
- **Fast by default.** Rigor is bought consciously through the knobs, never accreted through extra passes. One gate, light checkpoints, no ceremony.
- **The memlog is the process memory.** Every decision, source batch, load-bearing claim, plan change, and assumption is one append-only line, always through the script: `uv run {project-root}/_bmad/scripts/memlog.py` with `--type <decision|source|claim|assumption|question|event>`.
- Web access is required for Run. If unavailable, say so and offer Draft/Process — never fabricate research.

## Resolution rules

- Bare paths and `{skill-root}` (e.g. `references/run.md`) resolve from this skill's installed directory.
- `{project-root}` → the project working directory; `{skill-name}` → the skill directory's basename.
- `{workflow.<name>}` → a merged `customize.toml` field; `{doc_workspace}` → the bound run folder.
- Forward slashes only. Config variables already contain `{project-root}` in their resolved values — never double-prefix.

## On Activation

**Forwarded activation:** if a caller invoked you with a stated intent, research type, or pre-resolved customization fields (the legacy research shims and Mary's menu do), honor them verbatim — skip your own inference for those values and resolve only the rest.

1. Resolve customization: `uv run {project-root}/_bmad/scripts/resolve_customization.py --skill {skill-root} --project-root {project-root} --key workflow`.
   - Script not found: BMad is not set up here. Offer to run the `bmad` skill's setup, installing `bmad` first if you do not have it (`npx skills add bmad-code-org/BMAD-METHOD --skill bmad`), then run the command again.
   - Any other failure: read `{skill-root}/customize.toml` and use defaults.

   Run `{workflow.activation_steps_prepend}`, then `{workflow.activation_steps_append}`.
2. Resolve config: `uv run {project-root}/_bmad/scripts/resolve_config.py --project-root {project-root} --key core.output_folder --key core.active_initiative`. `{date}` is the current system datetime.
   - Script not found, or no `output_folder`: BMad is not set up here. Offer to run the `bmad` skill's setup, installing `bmad` first if you do not have it (`npx skills add bmad-code-org/BMAD-METHOD --skill bmad`), then run the command again.
   - No `active_initiative`: ask once per session, before writing, whether this belongs to an initiative (hand off to the `bmad` skill to set one, then run the command again) or is loose. Loose work drops `/{active_initiative}` from every path.
3. Headless (no interactive user) → see `## Headless Mode`. Otherwise greet the user, and let them know they can invoke `bmad-party-mode` for multi-agent perspectives or `bmad-advanced-elicitation` for deeper exploration at any point.
4. Detect the intent: **codebase** when the ask's subject is this repository, a repository path it names, or a `type: codebase` run folder, whatever its verb, which skips the choice below and step 5; otherwise **draft**, **process** (the user has or names a report), **run**, or lifecycle **refresh** / **deepen** on an existing run folder. When a ticket (`spike-*.md`, `epic-*.md`, `initiative-*.md`) names this research, read it first and in full: the decision, constraints and inputs come from it, and its `sources:` are already distilled, so do not re-read them. Otherwise, when the ask is bare research with no verb ("research X for me"), open the floor first — invite the decision they're facing and anything they already have (links, a prior report) in one turn, then ask only what's missing — and put the choice up front, once: **Run** it here now, or **Draft** a prompt for a deep-research tool they subscribe to — often cheaper and a strong gatherer, with Process turning its output into the same artifact. State the trade (tokens and minutes here vs. one manual round-trip there); their call, remembered for the session.
5. If a run folder for this topic already exists under `{workflow.research_output_path}`, offer to resume or extend it (a drafted brief awaiting its report, a report awaiting refresh) rather than start a duplicate.

## Research types and decision shapes

The type set is whatever `{workflow.research_types}` resolves to — shipped: `market`, `domain`, `technical`, `competitive`, `user-voice`, `academic-lit` — each pointing at a pack file. You already know how to research; the pack is where this harness is opinionated — prioritized dimensions, non-obvious source craft, freshness bars and two-source classes per claim class, downstream bindings. Apply it in every mode; don't re-derive it. Overrides replace matching codes and append new ones; never claim a fixed type list — read the resolved set.

Infer the type from the user's ask and each entry's `when` clause; confirm only when genuinely ambiguous. An explicit type (argument, shim, menu) wins without discussion.

Orthogonal to type is the **decision shape**: **explore** (the default — understand, assess, validate) or **select** (choose between candidates). When the shape is select, load `references/selection.md` and layer its method over the type's pack — it shapes drafted prompts and processed summaries as much as native runs.

## Intents

Route on the detected intent and load only what it names. Every intent shares the run-folder workspace shape — `brief.md`, `imports/`, `digests/`, the main file, `<folder name>.md`, `.memlog.md` — and ends per `references/finalize.md`.

| Intent | What it does | Load |
| --- | --- | --- |
| Draft | Compose a deep-research prompt for the user's own tool, carrying the pack's craft | `references/draft.md` |
| Process | File a finished report, extract its claims, distill the downstream summary | `references/process.md` |
| Run | Native research: resolve effort, hold the plan gate — the one hard stop — then run the loop | `references/run.md`, then `references/verification.md` + `references/synthesis.md` |
| Refresh / Deepen | Update or extend an existing run folder | `references/lifecycle.md` |
| Codebase | Turn the ask into questions at the plan gate — the one hard stop before research — then answer them from code and git history | `references/codebase.md` |

## Headless Mode

When invoked headless, do not ask. A codebase ask (step 4) is reported as `"intent": "run"` with `"type": "codebase"`; bare research defaults to **run**; a named report means **process**; a requested prompt means **draft** (the brief file is the deliverable). Plan-and-proceed: infer type, build from the pack, keep configured knobs plus anything in the invocation (on the external path, red team and workflow orchestration only when set `"on"`), skip checkpoints, log every judgment call as an `assumption`. Halt `blocked` only when topic or target folder cannot be inferred, or a codebase ask has no bounded question set. End with JSON:

```json
{
  "status": "complete",
  "intent": "run",
  "type": "market",
  "report": "{doc_workspace}/<folder name>.md",
  "memlog": "{doc_workspace}/.memlog.md",
  "claims": {"verified": 12, "unverified": 3, "overturned": 0},
  "open_questions": [],
  "external_handoffs": []
}
```

Omit keys for artifacts not produced; the `claims` counts come from `uv run {skill-root}/scripts/recon_kit.py tally {doc_workspace}/.memlog.md`, never hand-counted: copy its `claims` object as it prints. Draft adds `"brief"`; process adds `"imports"`; refresh replaces `claims` scope with the refresh set plus a `deltas` array; codebase adds no key: a proposed ADR left standing and the next ask its `## Open questions` names go in `open_questions`. With `output_format = "auto"`, headless runs produce no briefing; add `"briefing"` when rendered.

# Academic Literature Pack

Serves: ground an approach in published research, scan the state of the art, run a defensible literature review, cite properly in technical writing.

**Dimensions (priority order — prune to the decision):**

1. The canon — seminal papers and the best recent surveys of the area
2. State of the art — current best results, benchmarks, and how they're measured
3. Methods & limitations — what the leading approaches assume and where they break
4. Open problems & live debates — what the field disagrees about right now
5. Who works on this — the labs and groups whose output to watch

**Craft (the non-obvious):** find one good survey before reading twenty abstracts; chase citations both directions — who they cite and who cites them (Semantic Scholar/Google Scholar); label preprint vs peer-reviewed on every citation — arXiv is not acceptance; take benchmark numbers from the original paper, never from a competitor's comparison table; check retraction and replication status on any load-bearing empirical claim; a result only ever shown by one lab is a lead, not a fact.

**Freshness:** state-of-the-art claims ≤ 12 mo (ML ≤ 6 mo) · seminal work has no freshness bar — but check whether it was since superseded.

**Two-source classes:** any empirical claim a conclusion rests on — independent replication or corroboration, not the same lab twice.

**Feeds (bmm):** technical and architecture bets · content and writing that cites · build-vs-adopt judgments on research-grade techniques.

# Competitive Research Pack

Serves: position against *named* competitors, build battlecards, sharpen differentiation, anticipate their next move. (Surveying an unnamed field is the market type; this pack is for teardowns of specific players.)

**Dimensions (priority order — prune to the decision):**

1. Offer & feature teardown — what they actually ship, tried directly where possible
2. Pricing & packaging — models, tiers, what changed recently and which direction
3. Positioning & messaging — who they claim to serve, the story they tell, the gap between claim and product
4. Trajectory — funding, hiring, release cadence: where they're headed
5. Their customers' voice — what users of *their* product praise and complain about

**Craft (the non-obvious):** their changelog and release notes are roadmap truth; job postings reveal strategy six months early; their customers' 1–3★ reviews are your wedge; archived pricing pages (Wayback) show pricing direction, not just position; sales-facing comparison pages overclaim — verify capability claims against their docs; try the product yourself when a trial exists — an hour in-product beats ten reviews.

**Freshness:** pricing & features ≤ 3 mo · trajectory signals ≤ 6 mo · customer sentiment ≤ 12 mo.

**Two-source classes:** traction and market-share claims; any capability claim of theirs that your differentiation rests on.

**Feeds (bmm):** brief (alternatives) · PRD (differentiation) · GTM battlecards and positioning.

# Domain Research Pack

Serves: commit to building in an industry, talk credibly with domain experts, scope a product for a regulated field, brief a team entering unfamiliar territory.

**Dimensions (priority order — prune to the decision):**

1. Industry structure & value chain — how value flows, who captures margin where
2. Key players & gatekeepers — incumbents, platforms, whose APIs/standards/marketplaces you build with or against
3. Rules of the game — laws, licenses, de-facto standards, what compliance costs a new entrant
4. Language & mental models — the vocabulary and implicit workflows practitioners think in; **build the glossary — it's why domain research exists**
5. Technology adoption — the current technical baseline and where the industry sits on the adoption curve

**Craft (the non-obvious):** annual-report industry sections are free structured teardowns; conference keynotes reveal who actually matters; go to regulator sites directly for any load-bearing claim — and read enforcement actions to learn what's actually punished versus merely written; job postings name the real tools and skills; pending regulatory changes matter as much as current text.

**Freshness:** structure ≤ 3 yr · player landscape ≤ 18 mo · regulatory status: verify current on every load-bearing claim, whatever its date · tech adoption ≤ 18 mo, AI-adoption claims ≤ 6 mo.

**Two-source classes:** regulatory and compliance assertions; quantitative industry figures a recommendation rests on; claims about a gatekeeper's policy.

**Feeds (bmm):** brief (context, feasibility) · PRD (constraints, domain vocabulary) · architecture (integration landscape, compliance requirements).

# Market Research Pack

Serves: enter or skip a market, position a product, pick a segment, price an offer, pitch investors.

**Dimensions (priority order — prune to the decision):**

1. Market size & growth — the *reachable* market, not the headline TAM
2. Customer segments & behavior — who buys, deciding how, valuing what
3. Pain points & unmet needs — what they complain about, work around, pay to avoid
4. Competitive landscape — who competes for this budget, including substitutes and "do nothing"
5. GTM & pricing dynamics — channels, sales motion, accepted pricing models, realistic CAC

**Craft (the non-obvious):** 1–3★ reviews are the gold for pains; public-company 10-K/S-1 industry sections are free analyst-grade sizing; read competitor pricing pages directly, never roundups; funding history + job postings reveal competitor trajectory; a complaint pattern persisting across years is a stronger finding, not a stale one.

**Freshness:** size/growth ≤ 18 mo · pricing & feature claims ≤ 3 mo · behavior data ≤ 2 yr · GTM benchmarks ≤ 12 mo.

**Two-source classes:** market size and growth figures; any quantitative claim a recommendation rests on; competitor traction claims.

**Feeds (bmm):** brief (opportunity, problem, users) · PRD (personas, differentiation) · pricing and GTM decisions.

# Technical Research Pack

Serves: adopt a technology area, design an integration approach, ground an architecture in current practice, assess feasibility before committing a roadmap.

**Dimensions (priority order — prune to the decision):**

1. Landscape & maturity — dominant approaches, what's consolidating vs churning, what the current generation newly makes possible
2. Integration & interoperability — protocols, formats, auth patterns, where integrations actually hurt
3. Architecture patterns in practice — which named patterns dominate at what scale, and what the failures teach
4. Implementation reality — learning curve, tooling, operational burden, what teams say 6–12 months in
5. Ecosystem health — contributor/release vitality, backing durability, the five-year regret risk

**Craft (the non-obvious):** read the retrospective threads, not the launch threads; favor accounts with production numbers over advocacy; before citing a pain point, check whether it was since fixed — an old complaint against a current version is a false claim; read repository metrics over time, never snapshots; issue trackers reveal the gap between docs and reality.

**Freshness:** versions & compatibility ≤ 1 mo · ecosystem signals ≤ 6 mo · landscape ≤ 12 mo (AI-adjacent ≤ 3 mo) · patterns ≤ 2 yr.

**Two-source classes:** version/compatibility claims; performance or scale numbers a recommendation rests on; claims that a technology or pattern failed — one post-mortem is an anecdote.

**Feeds (bmm):** architecture spine (candidate paradigms, operational constraints) · brief (feasibility) · roadmap risk and estimates.

# User-Voice Research Pack

Serves: understand what users of a product or category actually experience and want — personas, jobs-to-be-done, requirements grounded in evidence rather than assumption.

**Dimensions (priority order — prune to the decision):**

1. Who they are & their jobs-to-be-done — the progress they're hiring the product to make
2. Complaint & workaround patterns — where current options fail them
3. Delight & switching triggers — why they stay, what made them move
4. Unmet needs & requests — what they ask for, and the deeper need under the ask
5. Their language — the words users say, versus the words vendors use

**Craft (the non-obvious):** mine 1–3★ reviews for pain *and* 5★ for why they stay; a workaround is unpriced demand — someone laboring around a gap has already voted; forums, Discord, and Reddit surface what surveys miss — people lie less when nobody's asking — but they over-sample the loud, so triangulate against reviews and any survey data; keep verbatim quotes, redacted — user words carry evidence paraphrase destroys, but usernames, handles, emails, and identifying links never enter the report or memlog; feature-request boards measure willingness to wait, not willingness to pay; distinguish loud power-users from the silent majority — count distinct voices, not thread length.

**Freshness:** sentiment ≤ 18 mo · complaints re-checked against the current version before citing.

**Two-source classes:** any prevalence claim ("most users…", "the top complaint is…") — two independent communities, not two threads in the same one.

**Feeds (bmm):** PRD (personas, requirements rationale) · UX research inputs · brief (problem) · product copy in the users' own language.