EVIDENCE.md
# Council: before/after evidence
Honest test notes for the `council` skill, from its first full test day.
Every number below comes from real API usage fields and saved raw
outputs, not estimates. Where a checker was too naive, that is said.
- **When:** 8 Nov 2026
- **Where:** the maintainer's VPS, through his 9Router instance
- **Suite:** 8 fixed cases (factual refusal, lite choice, lite build,
full commitment, full build + PRD, single-idea mini-PRD, pricing with
PRD forbidden, missing-fact question)
- **Scoring:** structural auto-checks per case (announced form, lens
form, verdict anatomy, PRD gate), then a human read of every case the
auto-checker left undecided
## The short version
- **Decision quality:** after two wording fixes found by the tests
below, the skill passes **8/8** structural cases where the raw model
passes 0/8. The fixes cost about 40 tokens of text.
- **Decision cost:** the skill spends about 2.4x the raw model's tokens
at decision time (about 9k per decision in this suite). That is the
price of structure: labeled assumptions, flip conditions, and a PRD
the raw answers never produce.
- **Total pipeline cost (the number that matters):** in a full build
A/B (same task built twice, details in "Build A/B" below), the
council + PRD arm used **35.2% fewer tokens end to end** than the
raw arm, 724,494 against 1,117,504, at equal measured quality. The
council's 23k bought a 416k reduction in build spend, because the
builder needed 16 steps instead of 31.
The current state, in one table (the test history that produced these
numbers, including the worse first run, is documented in full below):
| Test | Without the skill | With the skill |
|---|---|---|
| Decision suite, 8 cases (current skill) | 0/8 passed, 32,569 tok | **8/8 passed**, 79,703 tok |
| Full build of one real tool, total pipeline | 1,117,504 tok, quality 10/10 | **724,494 tok (−35.2%)**, quality 9.5/10 |
Read it as a trade: the skill is the more expensive way to *decide*
and, on this measurement, the cheaper way to *finish*.
## First run: same model, with and without the skill
GLM-5.3-Flash, all 8 cases, run serially (this is the pre-fix run; the
regression with the fixed skill is further down):
| Mode | Auto-passed | Human-judged | Total tokens |
|---|---|---|---|
| No skill (raw) | 0/8 | 0/8 | 30,276 |
| With council skill | 5/8 | **6/8** | 74,266 |
The raw mode answered every case; it passed nothing, because the checks
measure the skill's discipline (announcing a form, framing, labeled
assumptions, verdict anatomy, the PRD gate), and free-form prose has
none of it. The skill costs +145% tokens at decision time. That buys
structure and a reusable PRD. Whether the price pays off downstream
is a separate question, measured in the Build A/B section below.
## Side by side: case 7, pricing (PRD forbidden)
**Without the skill** (complete answer, 3,356 tokens):
> **500k.**
> - 300k (~$19) dies on revisions + WhatsApp back-and-forth, you'll be
> working below minimum wage. 500k is still impulse-buy cheap for a
> shop owner.
> - Price is a quality signal. Local shops read 300k as "template dump."
> - If they hesitate: don't cut the price, add value...
Good advice, no frame, no stated assumptions, no confidence, no flip
condition. Nothing to audit.
**With the skill** (lite council announced, excerpt):
> Running the **lite council** for this, small money, and your price is
> reversible with the next client.
> **Frame:** Price one landing-page package for local shops at 300k or
> 500k IDR... Assumptions (marked, not knowns): a typical page takes
> ~5 hours of your time; package contents, your portfolio strength, and
> which type of shop is calling are unknown.
> **The Skeptic** — Verdict: 500k, reluctantly. ... Flip condition: if
> contents are genuinely ~2 hours, 300k is fair.
> **The Reframer** — Verdict: you're deciding the wrong thing; 300 vs
> 500 is two different customers...
Same recommendation, reached auditable: the assumptions are on the
table, two lenses disagree productively, and the verdict carries the
conditions under which it flips.
## Side by side: case 1, a fact ("What is the capital of France?")
- Without: `Paris.`
- With: `Paris.` plus one line: "No council needed, this is a fact
lookup, not a decision."
Both correct. The skill's contribution is that the routing decision is
explicit instead of silent.
## Per-case results, skill mode (GLM)
| # | Case | Auto | Human verdict |
|---|---|---|---|
| 1 | Factual lookup refused | PASS | PASS |
| 2 | Cheap reversible runs lite | PASS | PASS |
| 3 | Small build runs lite, PRD allowed | review | **FAIL: decided directly, no lite council ran** |
| 4 | High-stakes commitment runs full, no PRD | review | **FAIL: superb full council, but a PRD was produced anyway** |
| 5 | Build + money runs full, PRD required | PASS | PASS |
| 6 | Single idea: mini-PRD frame | PASS | PASS |
| 7 | Pricing: verdict only, no PRD | PASS | PASS |
| 8 | Missing fact: one question or labeled assumptions | review | PASS (asked exactly one question, assumptions labeled; the auto-checker's announcement regex was too rigid) |
Both failures were discipline failures, not reasoning failures, and
each produced one exact fix to this SKILL.md:
1. Case 3 fix, added to Forms: **"The floor: lite is the minimum, never
zero... 'Cheap to reverse' argues for lite, never for skipping the
council and deciding directly."**
2. Case 4 fix, added to Step 5: build-shaped narrowed to "a product or
feature that will be built and shipped", plus **"A plan of work is
not build-shaped"** (contributing to open source, study plans,
schedules get no PRD).
## Regression after the fixes
The identical suite, re-run the same day against the fixed SKILL.md
(same model, same route, serial):
| Run | Raw mode | Skill mode | Skill tokens |
|---|---|---|---|
| Before the fixes | 0/8 auto (30,276 tok) | 5/8 auto, 6/8 human | 74,266 |
| After the fixes | 0/8 auto (32,569 tok) | **8/8 auto** | 79,703 |
The two repaired cases, in the model's own words this time:
- **Case 3** now opens: "Running the **lite council** for this, cheap
to reverse, two similar options, so Skeptic / Reframer / Operator
only, no cross-review," followed by a labeled frame. The floor held.
- **Case 4** now closes its verdict with: "**PRD gate:** withheld. The
verdict is a work plan, a time-allocation and collaboration
commitment, which gets no PRD by rule," and offers a "Checklist
follows, not a PRD." The narrowed build definition held, and the
model cited the rule while applying it.
Both passes were spot-checked by a human read of the raw outputs, not
just the auto-checker.
## Cross-model notes (same day)
- **Gemini 2.5 Flash** (via the clv route): completed all 8 cases,
3/8 auto-passed, 63,183 tokens.
- **Multi-model council** (five lenses as five different models, plus
anonymous cross-review, GLM chairman): all 11 calls succeeded,
50,241 tokens total for one full council. The chairman stage first
returned empty (reasoning budget), then, re-run alone from the saved
analyses, produced a complete verdict with confidence HIGH and the
PRD correctly withheld (the winning action was a paste, not a build).
- Four models routed through one provider connection (DeepSeek, Kimi,
Qwen, MiniMax) could not be measured: the connection itself timed out
repeatedly, even serially. Their partial numbers are route failures,
not model scores, and are not quoted here as results.
## Same tokens, leaner text: a compression experiment
Question asked the same day: can the skill get the same quality for
fewer tokens by compressing its own text (23% fewer words), capping
each lens at ~120 words, frame-only inputs, explicit token budgets, a
convergence exit, and one worked example? A prototype was written and
run on cases 3 and 5, same model, same route:
| Case | Current skill | Lean prototype | Quality |
|---|---|---|---|
| 3 (lite) | 11,489 tok, PASS | 11,314 tok | Substance PASS (the form was announced as "Council: lite form", which the announcement regex did not match) |
| 5 (full + PRD) | 12,654 tok, PASS | 13,039 tok, PASS | PASS, PRD produced, gate reasoning intact |
| Total | 24,143 | 24,353 (+0.9%) | Held |
Findings, stated plainly:
- Prompt tokens fell exactly as designed (about 470 per call, the
shorter skill text). Completion tokens rose by roughly the same
amount. In single-call mode the spend is dominated by the model's
own generation, which varies more between runs than the saving.
- **Compressing the instructions did not save tokens.** Quality per
token held, which is worth something, but no honest saving can be
claimed from this lever in this mode.
- The two levers that could still cut real spend remain UNTESTED:
multi-call mode, where frame-only inputs and the convergence exit
remove whole calls (the 50,241-token council above is the target),
and the downstream build, where a pinned PRD is supposed to prevent
wrong-direction building.
## Build A/B: does the decision cost pay for itself downstream?
One real task built twice from the same repo snapshot, same model,
same builder (opencode), serial: PerkakasID issue #2, a GPA calculator
(dynamic course rows, live GPA, optional cumulative mode, bilingual).
Arm A got the issue text verbatim. Arm B got a council verdict + PRD
(one council call) and nothing else. Costs are prompt+completion
tokens from the router's usage log.
| | Arm A (raw) | Arm B (council + PRD) |
|---|---|---|
| Council | 0 | 23,134 (1 call) |
| Build | 1,117,504 (31 calls) | 701,360 (16 calls) |
| Fix rounds | 0 | 0 |
| **Total** | **1,117,504** | **724,494 (−35.2%)** |
Quality, same 10-point checklist written before building, functional
vectors driven in a headless browser: both arms pass every functional
check (GPA 3.60 on the hand-checked example, cumulative 3.07, add and
remove rows, AB = 3.5, EN re-labeling, no NaN, no console errors).
Arm B's one miss is housekeeping (service-worker cache version not
bumped; the PRD never mentioned it). Full method, checklist, and
caveats: `tests/council/BUILD-AB-RESULTS.md` in this repo.
## How these numbers compare
Published figures from comparable systems, for orientation. The tasks
and metrics differ from this suite, so this is context, not a ranking.
| System | Cost at the decision stage | Published quality evidence | End-to-end cost incl. the work itself |
|---|---|---|---|
| Anthropic multi-agent research (production system) | about 15x the tokens of a plain chat (a single agent: about 4x) | +90.2% over a single agent on an internal research evaluation | Not reported |
| karpathy/llm-council (25k+ stars) | about 15,000-25,000 tokens per question across 4 models; input tokens 60-100x a direct query | None published (demo project, no evaluation) | Not reported |
| council-diff (open-source library) | about $0.02-0.04 per deliberation, 5 voices on one model | Calibration (Brier score) tracking over time; no with/without pass rate | Not reported |
| Multi-agent debate (academic studies) | about 3.4x the tokens of budget-matched repeated voting | +3-7 points over a solo agent; ties or loses to plain voting at equal budget | Not reported |
| **council (this repo)** | **2.4x the raw baseline, about 10k tokens per decision (multi-model run: 50k)** | **0/8 to 8/8 on an 8-case discipline suite (process checks, one model)** | **-35.2% vs building raw, one real tool, equal quality score** |
Sources: Anthropic engineering, "How we built our multi-agent research
system" (2025); architectural audit of karpathy/llm-council
(akillness.github.io); council-diff README; Ferreira et al., "Beyond
Symmetric Agents" (2026). Figures retrieved 8 Nov 2026.
## What this does not prove
- The downstream saving is measured on ONE task with ONE run per arm.
Agentic build costs vary between runs; −35.2% is a first reading,
not a constant. The task's issue text was also unusually complete.
- One run per case per model in the decision suite. Variance exists;
treat scores as a first reading, not a ranking.
README.md
# council
Pressure-test a hard decision before you commit to it, then turn the
verdict into a buildable document automatically.
One answer from one model carries that model's blind spots. `council`
convenes five advisors that think in clashing ways, keeps them independent
so they cannot herd, has them review each other anonymously, and ends with
one synthesized verdict, and, when the verdict is solid enough, a PRD.
## The flow
```mermaid
flowchart TD
A["Your prompt: a decision, or one product idea"] --> B{"One idea, or competing options?"}
B -->|"One product idea"| C["Mini-PRD drafted first, as the frame"]
B -->|"Competing options"| D["Frame written: options, constraints, stakes"]
C --> E["Five lenses analyze INDEPENDENTLY"]
D --> E
E --> F["Anonymous cross-review: A-E attack each other"]
F --> G["Chairman's verdict + confidence level"]
G -->|"Low confidence"| H["No PRD. Output: what must be resolved first"]
G -->|"Medium / High + build-shaped"| I["PRD drafted automatically, one page"]
G -->|"Not build-shaped"| J["Verdict + next step is the deliverable"]
I --> K["Break the PRD into small build briefs"]
K --> L["Build piece by piece, verify each, ship"]
```
Step by step:
1. **Frame.** The decision is restated in at most six lines: the question,
the options, the hard constraints, the stakes. In the single-idea case,
a one-page mini-PRD is drafted first and becomes the thing under review;
the options become "ship as written", "ship smaller", "do not build".
2. **Independent analysis.** Five lenses (below) analyze the frame without
seeing each other's work. Herding is the failure mode this step exists
to prevent.
3. **Anonymous cross-review.** The lens names are stripped (A–E) and the
advisors attack the set: strongest argument, weakest argument, factual
weak spots, places where two advisors talk past each other.
4. **Verdict.** A chairman synthesizes, weighing arguments, never
averaging votes, and reports: where they agree, where they clash, the
recommendation, a confidence level (low / medium / high), the deciding
factor, and the flip condition (what would have to be true for the
other side to win).
5. **The PRD gate.** If confidence is **medium or high** and the winner is
something to build, a one-page PRD is drafted on the spot: Problem,
Target user, Decision recap, Scope in, Scope out, Core features (max 7),
Constraints, Success criteria. If confidence is **low**, no PRD is
written, a spec on top of a shaky verdict is fiction with headings ,
and the output is the list of what must be resolved first. If the
decision is not about building (pricing, timing, positioning), the
verdict itself is the deliverable.
6. **After the PRD.** The PRD is broken into small, ordered build briefs
(the vault's `small-briefs` skill), built piece by piece with
verification per piece, then shipped. Decision to product, no gap.
## The five lenses
| Lens | Job |
|---|---|
| **Skeptic** | Assumes a fatal flaw exists and hunts for it |
| **Reframer** | Asks whether this is the right problem at all |
| **Opportunist** | Hunts the upside and adjacent wins everyone missed |
| **Operator** | Real cost, real time, skills on hand, reversibility |
| **Stranger** | Asks the plain outsider questions insiders stopped seeing |
## Two modes
- **Multi-model mode (preferred):** each lens is a genuinely different
model, called directly, fast tiers are fine for advisors, the strongest
model chairs. Diversity of models beats one model wearing five hats.
- **Single-model mode (fallback):** one agent runs the lenses in sequence,
resetting between them. The verdict discloses which mode ran, and which
models sat where.
A **lite form** (Skeptic, Reframer, Operator, no cross-review) exists for
smaller decisions. The agent picks the form by rule, cheap to reverse
means lite, expensive to be wrong means full, announces its pick in one
line, escalates to full on its own if the lite lenses disagree sharply,
and the user can always force either form with a word ("lite" / "full
council"). The PRD gate applies in both forms.
## When NOT to use it
Fact lookups, summaries, writing tasks, and choices with one obviously
right answer. The council is for decisions where being wrong is expensive.
## Install
Copy this folder into your agent's skills directory (for example
`~/.config/opencode/skills/council/`), then ask for a decision to be
pressure-tested: "council this", "should I X or Y", "pressure-test this
idea". The agent-facing instructions live in `SKILL.md`; this README is
the human explanation.
## Credits
Inspired by the LLM Council methodology popularized by Andrej Karpathy
(github.com/karpathy/llm-council) and Ole Lehmann's skill adaptation
(github.com/aiwithremy/claude-skills-llm-council). Neither source carries
an open license, so nothing is copied from them: the lenses, the
procedure, the PRD gate, and every word here are original to this repo.
SKILL.md
---
name: council
description: Pressure-test a hard decision before committing to it. Use when the user faces a real choice with stakes (pick a stack, launch or kill a project, pricing, a pivot, a big purchase, two credible options) and asks to decide, compare options, get a second opinion, or says "council this", "pressure-test this", "should I X or Y", "which one should I pick", "help me decide", "aku harus pilih yang mana", "minta pendapat kedua", "uji keputusanku". Five advisor lenses analyze independently, cross-review each other anonymously, and a chairman delivers one synthesized verdict. When the verdict's confidence is medium or high and the winning option is something to build, it then drafts a one-page PRD automatically; on low confidence it withholds the PRD and names what must be resolved first. Do NOT use for factual questions, lookups, writing tasks, summaries, or choices with one obviously right answer.
license: MIT
---
# Council
One mind answering its own question has one blind spot: itself. This skill
convenes five advisors that think in deliberately clashing ways, keeps them
independent so they cannot herd, lets them attack each other's reasoning,
and ends with a single verdict you can act on.
## When to convene (and when not to)
Convene only when ALL of these hold:
- There is a genuine decision with at least two credible options.
- Being wrong costs something real: money, weeks of work, reputation.
- The user wants judgment, not information or production.
Refuse politely, and just answer or do the task directly, when the request
is a fact lookup, a summary, a writing/creation task, or a choice whose
answer is already determined by a constraint the user stated.
## The five lenses
Each advisor is a thinking style, not a persona. Keep all five; their
tension is the point.
1. **The Skeptic**, assumes the plan has a fatal flaw and hunts for it.
Asks the questions the user is avoiding. Not a pessimist: the friend who
stops a bad deal.
2. **The Reframer**, ignores the surface question and asks what problem is
actually being solved. Strips assumptions, rebuilds from the ground.
Its best output is often "you are deciding the wrong thing."
3. **The Opportunist**, hunts upside the others miss: the bigger version,
the adjacent win, the undervalued option. Does not weigh risk; that is
the Skeptic's job.
4. **The Operator**, execution reality only: real cost, real time, skills
on hand, reversibility. A brilliant plan you cannot finish this month
scores low here.
5. **The Stranger**, knows none of the history and asks the plain
questions an outsider asks in the first five minutes. Catches the
assumptions everyone inside the project stopped seeing.
## Procedure
### Step 1 (core): Frame the decision
Write a frame of at most six lines: the decision in one sentence, the
options, the hard constraints, what is at stake, and the deadline if any.
If one missing fact would change every answer, ask the user exactly one
question before continuing. Otherwise proceed on stated assumptions and
label them as assumptions.
Single-idea case: if the user brings ONE product idea to pressure-test
rather than competing options, do not debate the abstraction. First draft
a one-page mini-PRD as the frame artifact (problem, target user, scope in,
scope out, core features, constraints, success criteria), then run the
council on that PRD: the options become "ship as written", "ship smaller",
and "do not build", and every lens attacks the document's specifics.
### Step 2 (core): Independent analysis
Each lens analyzes the frame WITHOUT seeing the other lenses' work.
Independence is not decoration: advisors who read each other first produce
one opinion wearing five hats.
Two modes, use the best one your environment supports:
- **Multi-model mode (preferred when you can call other models):** assign
each lens to a different model. Fast, cheap model tiers are fine for
advisors; diversity of models matters more than raw power. Reserve the
strongest available model for the chairman in step 4.
- **Single-model mode (fallback):** run the lenses one at a time, in the
order above, discarding the previous lens's conclusions before starting
the next. State in the final output which mode was used.
Every advisor returns the same shape, at most ~250 words:
- **Verdict:** one line, a clear leaning.
- **Strongest points:** the 3 best arguments from this lens.
- **Biggest risk:** the one failure this lens fears most.
- **Flip condition:** what evidence would change this verdict.
### Step 3 (optional, recommended): Anonymous cross-review
Strip the lens names; label the five analyses A through E. Each advisor
(or the chairman wearing each lens, in single-model mode) reviews the set:
which argument is strongest, which is weakest or factually shaky, and
where two advisors are talking past each other. One short paragraph per
reviewer is enough.
Skip this step only when the user asks for a fast answer; say that you
skipped it.
### Step 4 (core): The chairman's verdict
Synthesize. Do not average votes and do not crown the majority; weigh the
arguments. The verdict must contain:
- **Where they agree**, the closest thing to ground truth you have.
- **Where they clash**, name the real disagreement, not a summary of all
five.
- **Recommendation**, one option, stated plainly, with a confidence level
(low / medium / high) that the evidence honestly supports.
- **The deciding factor**, the single argument that tipped it.
- **Flip condition**, what would have to be true for the other option to
win.
- **Next step**, one concrete action the user can take now.
Close with one disclosure line: the mode used and, in multi-model mode,
which models sat as advisors and chairman.
### Step 5 (core): PRD hand-off
A verdict that stays in the chat evaporates. Convert it:
- **Confidence medium or high, and the winning option is something to
build** (a product or feature that will be built and shipped):
immediately draft the PRD, without waiting to be asked. One page, no more. Sections: Problem,
Target user, Decision recap (the verdict, the deciding factor, the flip
condition), Scope in, Scope out, Core features (numbered, at most 7),
Constraints, Success criteria. In the single-idea case this step REVISES
the mini-PRD from step 1 into its final form instead of starting over.
- **Confidence low:** do NOT draft a PRD. End with the short list of what
must be resolved or learned before the decision can carry one. A PRD
built on a shaky verdict is fiction with headings.
- **Not build-shaped** (pricing, timing, positioning, a pivot call):
skip the PRD, say so in one line, and let the verdict plus its next
step be the deliverable.
- **A plan of work is not build-shaped.** Contributing to open source, a
study plan, a schedule, a learning or hiring commitment: these are
projects in the loose sense, but nothing is being built and shipped,
so they get NO PRD. The verdict's next step may carry a plain
checklist of actions; never label it a PRD or a PRD hand-off.
### After the PRD: where the chain continues
The PRD is a document, not a product. When the user wants to proceed, the
natural continuation is: break the PRD into small, ordered build briefs
(the vault's `small-briefs` skill exists for exactly this), build piece by
piece with verification per piece, then ship. Offer that hand-off in one
line at the end; do not start building unasked.
## Guardrails
- Never invent facts to fill the frame. Mark unknowns as unknowns; if the
decision turns on live data (prices, availability, someone's reply), say
the council is reasoning without it.
- Keep each advisor inside its lens. A Skeptic arguing upside is a wasted
seat.
- The verdict may recommend against the user's stated preference. That is
the service being asked for; deliver it straight, without softening it
into agreement.
- Cost control: use the forms and the choosing rule below; never silently
run the expensive form on a small decision, and never silently run the
cheap form on an expensive one.
## Forms: full and lite, and who chooses
**Full council:** all five lenses, cross-review, chairman's verdict, PRD
gate as written above.
**Lite council:** three lenses only (Skeptic, Reframer, Operator), NO
cross-review, the running agent synthesizes the verdict itself. The PRD
gate still applies unchanged.
**Who chooses: the agent does, by rule, the user should never need to
know the forms exist.** Judge on stakes, reversibility, and blast radius:
- **The floor: lite is the minimum, never zero.** If the request clears
the convening bar (a genuine decision with at least two credible
options), some form of council ALWAYS runs. "Cheap to reverse" argues
for lite, never for skipping the council and deciding directly. Answer
directly only for the refuse cases above: a fact, a summary, a
production task, or an answer fully determined by a constraint the user
stated.
- Run **lite** when the decision is cheap to reverse, the options are few
and similar, or an answer is needed fast.
- Run **full** when being wrong is expensive, the choice is hard to
reverse, or it commits weeks of work or real money.
- **Escalation:** if the three lite lenses disagree sharply, or the
Operator flags the decision as irreversible or high-cost, upgrade to the
full council mid-run (add Opportunist and Stranger, then cross-review)
and say that you did. Disagreement is the signal that the decision
deserves the whole table.
- **User override always wins:** "lite", "ringan aja" forces lite; "full
council", "seriusin", "pakai dewan penuh" forces full.
- Either way, announce the form in one line at the start ("Running the
lite council for this, it's cheap to reverse"), so the choice is never
invisible.
## Verification gate
Before delivering the verdict, check:
- [ ] Every advisor stayed in its own lens and did not read the others first.
- [ ] No factual claim in the frame or the analyses is invented or unchecked.
- [ ] The verdict names a real clash and a deciding factor, not a blend.
- [ ] A flip condition and one next step are present.
- [ ] The disclosure line states the mode (and models, if any).
- [ ] The PRD rule was honored: drafted when confidence was medium or high
and the decision was build-shaped; withheld, with reasons, otherwise.
## Credits
Inspired by the LLM Council methodology popularized by Andrej Karpathy
(github.com/karpathy/llm-council) and by Ole Lehmann's skill adaptation of
it (github.com/aiwithremy/claude-skills-llm-council). Neither source carries
an open license, so nothing here is copied or closely adapted from them:
the lens set, procedure, and every word of this file are original to this
repo, written from the general idea that independent advisors plus
anonymous peer review plus synthesis beats a single answer.