evals/cases.yaml
cases:
# --- Positive cases ---
- id: verify_skill_output
prompt: "Verify this skill's output against its definition. Here's the SKILL.md for architecture-reviewer and the output it produced for a React app review. Generate assertions to check if the output meets the skill's stated capabilities."
fixtures: []
rubric:
- "Generates structured assertions with type, target, and weight fields"
- "Assertions cover output completeness, format compliance, and content quality"
- "Follows the assertion schema from evals/cases.yaml format"
trigger_expected: true
assertions:
- type: matches_regex
target: "(type:|type:)"
weight: 1.0
- type: matches_regex
target: "(target:|target:)"
weight: 1.0
- type: matches_regex
target: "(weight:|weight:)"
weight: 1.0
- type: matches_regex
target: "(contains|not_contains|matches_regex|output_format)"
weight: 0.8
- type: matches_regex
target: "(0\\.[0-9]|1\\.0)"
weight: 0.6
- id: generate_assertions_for_skill
prompt: "Generate eval assertions for the test-harness skill. Given these 3 task prompts, produce machine-checkable assertions that verify the skill produces correct pytest test suites."
fixtures: []
rubric:
- "Produces assertions for each task prompt separately"
- "Uses appropriate assertion types (contains for test functions, output_format for code blocks)"
- "Assigns weights based on importance (higher for core output, lower for nice-to-have)"
trigger_expected: true
assertions:
- type: matches_regex
target: "(contains|output_format|matches_regex)"
weight: 1.0
- type: matches_regex
target: "(weight:|weight:)"
weight: 1.0
- type: matches_regex
target: "(pytest|def test_|test suite)"
weight: 0.8
- type: matches_regex
target: "(1\\.0|0\\.8|0\\.6|0\\.3)"
weight: 0.6
- type: not_contains
target: "TODO"
weight: 0.3
- id: diagnose_skill_failure
prompt: "The architecture-reviewer skill failed 3 out of 7 assertions when reviewing a FastAPI app. Here are the assertion results. Diagnose why it failed and suggest specific fixes to the SKILL.md."
fixtures: []
rubric:
- "Classifies each failure into a root-cause category (missing capability, format mismatch, etc.)"
- "References specific sections of the skill definition"
- "Provides concrete remediation steps with exact wording changes"
trigger_expected: true
assertions:
- type: matches_regex
target: "(DIAGNOSTIC|diagnostic|ROOT CAUSE|root cause)"
weight: 1.0
- type: matches_regex
target: "(missing capability|format mismatch|hallucinated|incomplete)"
weight: 1.0
- type: matches_regex
target: "(REMEDIATION|remediation|fix|change)"
weight: 0.8
- type: matches_regex
target: "(SKILL\\.md|section|Phase|Step)"
weight: 0.6
- type: matches_regex
target: "(HIGH|MEDIUM|LOW)"
weight: 0.6
# --- Negative cases ---
- id: not_pr_review
prompt: "Review my pull request for code quality issues and suggest improvements"
fixtures: []
rubric:
- "Should NOT activate surrogate-verifier — this is a PR review task for pr-review"
trigger_expected: false
- id: not_package_evaluator
prompt: "Score this skill's frontmatter quality and trigger coverage across 6 dimensions"
fixtures: []
rubric:
- "Should NOT activate — this is package metadata evaluation for package-evaluator"
trigger_expected: false
- id: not_test_harness
prompt: "Write pytest unit tests for this Python function that validates email addresses"
fixtures: []
rubric:
- "Should NOT activate — this is test generation for application code, handled by test-harness"
trigger_expected: false
references/assertion-patterns.md
# Assertion Patterns by Skill Category
Catalog of assertion patterns organized by skill category. Each pattern includes the
assertion type, typical targets, recommended weight, and false-positive avoidance notes.
---
## Development Skills (code generation, testing, building)
| Pattern | Type | Target Example | Weight | Notes |
| ----------------------------- | --------------- | -------------------------------- | ------ | ---------------------------------------- |
| Code block present | `output_format` | `code_block` | 0.8 | Most dev skills produce code |
| Language-specific syntax | `matches_regex` | `def \w+\(` (Python function) | 0.6 | Match language the skill targets |
| Import statements | `contains` | `import` or `from ... import` | 0.4 | Only if skill generates complete files |
| Test function naming | `matches_regex` | `def test_\w+` | 0.8 | For test-generation skills |
| No placeholder code | `not_contains` | `TODO`, `FIXME`, `pass # ` | 0.3 | Generated code should be complete |
| Tool usage for file reading | `calls_tool` | `Read` | 0.5 | Skills that analyze existing code |
## Review Skills (code review, architecture, security)
| Pattern | Type | Target Example | Weight | Notes |
| ----------------------------- | --------------- | -------------------------------- | ------ | ---------------------------------------- |
| Severity classification | `matches_regex` | `CRITICAL\|HIGH\|MEDIUM\|LOW` | 1.0 | Reviews must classify findings |
| File:line references | `matches_regex` | `\w+\.\w+:\d+` | 0.8 | Findings should cite locations |
| Recommendation present | `contains` | `Recommendation` or `Fix` | 0.6 | Reviews should suggest fixes |
| Structured output | `output_format` | `markdown_table` | 0.8 | Findings tables are standard |
| No false praise | `not_contains` | `looks great`, `no issues found` | 0.3 | Unless genuinely clean |
## Research Skills (analysis, literature review, investigation)
| Pattern | Type | Target Example | Weight | Notes |
| ----------------------------- | --------------- | -------------------------------- | ------ | ---------------------------------------- |
| Citation present | `matches_regex` | `\[[\d]+\]` or `http` | 0.8 | Research should cite sources |
| Multi-section output | `matches_regex` | `^## .+` | 0.6 | Structured analysis with headings |
| Evidence-based claims | `contains` | `according to`, `the data shows` | 0.4 | Claims should reference evidence |
| Comparison structure | `output_format` | `markdown_table` | 0.6 | Comparative analysis needs tables |
| Web search tool usage | `calls_tool` | `WebSearch` | 0.5 | Research skills should search |
## Business Skills (proposals, market analysis, feasibility)
| Pattern | Type | Target Example | Weight | Notes |
| ----------------------------- | --------------- | -------------------------------- | ------ | ---------------------------------------- |
| Quantitative data | `matches_regex` | `\$[\d,.]+[KMB]?` or `\d+%` | 0.8 | Business analysis needs numbers |
| Risk identification | `contains` | `risk`, `assumption` | 0.6 | Should identify uncertainties |
| Recommendation section | `matches_regex` | `^## .*(Recommend|Next Step)` | 1.0 | Must conclude with actionable guidance |
| Framework application | `contains` | `SWOT`, `TAM`, `Lean Canvas` | 0.4 | Domain frameworks demonstrate depth |
| Numbered list structure | `output_format` | `numbered_list` | 0.6 | Actionable steps should be ordered |
## Visualization Skills (diagrams, images, presentations)
| Pattern | Type | Target Example | Weight | Notes |
| ----------------------------- | --------------- | -------------------------------- | ------ | ---------------------------------------- |
| Output file created | `matches_regex` | `\.(html\|svg\|png\|pdf)` | 1.0 | Visualization must produce files |
| File write tool usage | `calls_tool` | `Write` | 0.8 | Must write the output file |
| Structural markup | `contains` | `<svg` or `<div` or `<html` | 0.6 | HTML/SVG output expected |
| No broken references | `not_contains` | `undefined`, `NaN`, `null` | 0.3 | Output should be complete |
---
## Weight Calibration Guide
| Weight Range | Meaning | Use When |
| ------------ | --------------------------- | -------------------------------------------------- |
| 0.9 - 1.0 | Must-have | Core output of the skill; failure means broken |
| 0.6 - 0.8 | Expected | Standard quality signal; absence is concerning |
| 0.3 - 0.5 | Nice-to-have | Quality indicator but not essential |
| 0.1 - 0.2 | Weak signal | Informational; high false-positive risk |
Prefer fewer high-weight assertions over many low-weight ones. 5-7 assertions with
clear signals produce better diagnostics than 15 weak checks.
references/diagnostic-templates.md
# Failure Diagnostic Templates
Structured templates for producing actionable failure diagnostics. Each template
covers a root-cause category with severity, explanation pattern, and remediation format.
---
## Root-Cause Categories
### 1. Missing Capability (HIGH)
The skill definition promises a feature that the output does not deliver.
**Detection signal:** `contains` assertion failed for a target derived from a stated capability.
**Diagnostic template:**
```
[HIGH] Missing capability: "{target}"
Skill definition claims: {quoted section from SKILL.md where capability is stated}
Output analysis: The output does not contain {target} or equivalent content.
Root cause: {hypothesis — missing workflow step, unclear instruction, or dependency gap}
Remediation: Add explicit workflow step: "In Phase {N}, generate {target} by {method}"
```
**Worked example:**
```
[HIGH] Missing capability: "## Error Handling"
Skill definition claims: "Workflow Phase 3: Document error handling patterns"
Output analysis: The output contains 4 sections but none addressing error handling.
Root cause: Phase 3 instruction says "document error handling" but does not specify
the output heading format or what constitutes error handling content.
Remediation: Change Phase 3 instruction to: "Generate a section titled '## Error
Handling' with a table of error scenarios, their causes, and recovery steps."
```
---
### 2. Format Mismatch (HIGH)
The output structure does not match the declared format.
**Detection signal:** `output_format` assertion failed.
**Diagnostic template:**
```
[HIGH] Format mismatch: expected "{target}" format
Skill definition specifies: {output format instruction from SKILL.md}
Actual output: {brief description of what the output looks like}
Root cause: {ambiguous format instruction, conflicting instructions, or missing format constraint}
Remediation: Replace "{vague instruction}" with "Output as a {target} with columns: [A, B, C]"
```
---
### 3. Incomplete Output (MEDIUM)
Multiple assertions in the same logical group failed, indicating a section is partially present.
**Detection signal:** 2+ `contains` assertions failed that relate to the same skill phase or section.
**Diagnostic template:**
```
[MEDIUM] Incomplete output: {section/phase name}
Expected items: {list of targets that should be present}
Present items: {list of targets found}
Missing items: {list of targets not found}
Root cause: {workflow step produces partial output, context cap reached, or scope ambiguity}
Remediation: {add explicit enumeration of required items in the workflow instruction}
```
---
### 4. Hallucinated Content (HIGH)
The output contains content that should not be present.
**Detection signal:** `not_contains` assertion failed (forbidden pattern IS present).
**Diagnostic template:**
```
[HIGH] Hallucinated content: "{target}" found in output
Constraint: The skill should not produce {target}
Location: Found at approximately {position description}
Root cause: {missing negative constraint, or positive instruction that inadvertently encourages it}
Remediation: Add explicit constraint: "Do NOT include {target} unless {specific condition}"
```
---
### 5. Wrong Tool Usage (MEDIUM)
The skill used unexpected tools or failed to use expected ones.
**Detection signal:** `calls_tool` assertion failed.
**Diagnostic template:**
```
[MEDIUM] Wrong tool usage: expected "{target}" tool
Skill workflow specifies: {tool usage instruction from SKILL.md}
Actual tools used: {list of tools detected in output, if identifiable}
Root cause: {ambiguous tool instruction, or skill defaulted to a different approach}
Remediation: Change instruction to explicitly name the tool: "Use the {target} tool to {action}"
```
---
### 6. Partial Success (LOW)
Some assertions in a logical group pass while others fail, indicating the skill is close
but not fully aligned.
**Diagnostic template:**
```
[LOW] Partial success: {N}/{M} assertions passed in {group}
Passed: {list of passed targets}
Failed: {list of failed targets}
Gap analysis: {what the passing assertions tell us about the output's strengths}
Remediation: {targeted fix for the specific failed assertions without disrupting passing ones}
```
---
## Diagnostic Output Format
Combine all findings into a single structured diagnostic:
```
DIAGNOSTIC: [{skill-name}] failed on "{task-prompt-summary}"
FAILED ASSERTIONS ({N}/{M} total):
1. [SEVERITY] type={type} target="{target}" — {Category}: {one-line explanation}
2. [SEVERITY] type={type} target="{target}" — {Category}: {one-line explanation}
ROOT CAUSES:
- {Category}: {Detailed explanation with SKILL.md section reference}
- {Category}: {Detailed explanation}
REMEDIATION (ordered by impact):
1. {Highest-impact fix with exact SKILL.md change}
2. {Second fix}
3. {Third fix if applicable}
CONFIDENCE: {high|medium|low} — {why: e.g., "high: clear format mismatch with explicit fix"}
```
## Escalation Signals
When these patterns appear, the verifier should recommend **test escalation** (harder assertions
for the next round):
- All format assertions pass but content assertions fail → skill produces correct structure but wrong content
- Tool assertions pass but output assertions fail → skill invokes correct tools but misinterprets results
- All assertions pass at low weights but fail at high weights → skill handles easy cases but misses core requirements
SKILL.md
---
name: surrogate-verifier
description: 'Generates structured test assertions and failure diagnostics for skill packages from a definition and task prompt. Triggers on: "verify this skill", "generate assertions", "surrogate verification", "diagnose skill failure". NOT for code review, use pr-review.'
metadata:
version: 1.0.1
category: review
tags: [verification, assertions, testing, co-evolution, diagnostics, eval]
difficulty: advanced
phase: verify
---
# Surrogate Verifier
Generate structured test assertions and failure diagnostics for skill packages through
information-isolated verification. The verifier operates without access to the skill
generator's reasoning — it sees only the skill definition, a task prompt, and the
output artifacts. This isolation prevents confirmation bias and is the single largest
contributor to skill quality in co-evolutionary generation (+30pp per EvoSkills).
## Reference Files
| File | Contents | Load When |
| ----------------------------------- | ---------------------------------------------------------------- | ------------------------------ |
| `references/assertion-patterns.md` | Assertion catalog by skill category with weight guidance | Always |
| `references/diagnostic-templates.md`| Failure diagnostic templates with root-cause categories | When producing failure reports |
## Information Isolation Protocol
This is the most critical constraint. Violating isolation degrades verification quality.
**The verifier MUST NOT access:**
- The generator's conversation history or reasoning chain
- Prior evolution iterations or refinement context
- The generator's internal notes or decision rationale
- Any context beyond what is explicitly listed below
**The verifier receives ONLY:**
1. The skill's `SKILL.md` content (the definition file)
2. One or more task prompts representing intended use
3. The skill's output (when diagnosing failures)
4. The assertion results from `scripts/eval_assertions.py` (when diagnosing)
**Implementation:** When invoked by the `test-engineer` agent, this skill MUST be loaded
into a **separate Agent spawn** using `isolation: "worktree"` or at minimum a fresh session
with no shared context. The invoking agent passes artifacts as explicit text, not as
conversation references.
## Workflow
### Mode 1: Assertion Generation
Generate assertions for a skill given its definition and task prompts.
#### Phase 1: Skill Analysis
Read the `SKILL.md` definition and extract:
1. **Stated capabilities** — what the skill claims to do (from description + workflow sections)
2. **Output format** — expected structure of the skill's output (markdown, JSON, tables, etc.)
3. **Error handling** — documented failure modes and recovery paths
4. **Prerequisites** — required tools, dependencies, or context
5. **Trigger boundaries** — what the skill does NOT handle (negative scope)
#### Phase 2: Assertion Design
For each task prompt, generate 5-10 assertions covering these dimensions:
| Dimension | Assertion Types to Use | Purpose |
| --------------------- | ------------------------------- | ------------------------------------------ |
| Output completeness | `contains`, `matches_regex` | All claimed sections/components present |
| Format compliance | `output_format`, `contains` | Output matches declared structure |
| Factual signals | `contains`, `not_contains` | Key domain terms present, hallmarks absent |
| Tool usage | `calls_tool` | Expected tools were invoked |
| Negative constraints | `not_contains` | Forbidden patterns absent |
**Weight assignment:**
- Output completeness assertions: weight 1.0 (must have)
- Format compliance: weight 0.8 (structural correctness)
- Factual signals: weight 0.6 (content quality)
- Tool usage: weight 0.5 (method verification)
- Negative constraints: weight 0.3 (absence checks are weaker signals)
See `references/assertion-patterns.md` for category-specific assertion catalogs.
#### Phase 3: Output
Produce assertions in the `evals/cases.yaml` schema format:
```yaml
assertions:
- type: contains
target: "## Scalability"
weight: 1.0
- type: output_format
target: markdown_table
weight: 0.8
- type: not_contains
target: "TODO"
weight: 0.3
- type: calls_tool
target: Read
weight: 0.5
```
**Context cap:** Do not consume more than 70% of the available context window. If the
skill definition is very long, focus assertion generation on the workflow phases and
output format sections. Summarize rather than quote verbatim.
### Mode 2: Failure Diagnostics
When an oracle returns `fail`, produce a structured diagnostic explaining why.
#### Input
- The skill's `SKILL.md` (same as Mode 1)
- The task prompt that was executed
- The output that failed
- The assertion results: which passed, which failed, with details
#### Phase 1: Failure Classification
Categorize each failed assertion into a root-cause category:
| Category | Signal | Severity |
| --------------------- | -------------------------------------------------------------- | ---------- |
| Missing capability | `contains` assertion failed for a claimed feature | HIGH |
| Format mismatch | `output_format` assertion failed | HIGH |
| Incomplete output | Multiple `contains` assertions failed in the same section | MEDIUM |
| Hallucinated content | `not_contains` assertion failed (forbidden pattern present) | HIGH |
| Wrong tool usage | `calls_tool` assertion failed | MEDIUM |
| Partial success | Some assertions in a group pass, others fail | LOW |
#### Phase 2: Root-Cause Analysis
For each failed assertion:
1. Identify the specific section of `SKILL.md` that promises the missing capability
2. Compare what the skill definition instructs vs. what the output actually contains
3. Hypothesize why the gap exists (missing workflow step, ambiguous instruction, wrong tool choice)
#### Phase 3: Remediation Suggestions
For each root cause, produce a concrete, actionable fix:
- **Missing capability:** "Add a workflow step between Phase 2 and Phase 3 that explicitly generates [X]"
- **Format mismatch:** "Change the output format instruction from 'produce a summary' to 'produce a markdown table with columns: [A, B, C]'"
- **Hallucinated content:** "Add a negative constraint in the workflow: 'Do NOT include [X] unless [condition]'"
- **Wrong tool usage:** "Replace 'use Bash to read the file' with 'use the Read tool for file contents'"
#### Phase 4: Diagnostic Output
Produce a structured diagnostic string:
```
DIAGNOSTIC: [skill-name] failed on [task-prompt-summary]
FAILED ASSERTIONS (N/M):
1. [SEVERITY] type=contains target="..." — Missing capability: [explanation]
2. [SEVERITY] type=output_format target="..." — Format mismatch: [explanation]
ROOT CAUSES:
- [category]: [specific explanation with SKILL.md section reference]
REMEDIATION:
1. [Concrete change to SKILL.md with exact section and wording]
2. [Concrete change to workflow with step numbers]
```
See `references/diagnostic-templates.md` for worked examples per root-cause category.
## Budget Parameters
Per EvoSkills Algorithm 1:
- **Context cap:** 0.7 (70% of available context window)
- **Max surrogate retries:** 15 per oracle round
- **Max oracle rounds:** 5 (enforced by the orchestrating agent, not the verifier)
The verifier does not track its own budget — the `test-engineer` agent manages iteration limits.
## Limitations
- **No execution capability:** The verifier generates assertions but does not execute them. Execution
is handled by `scripts/run_evals.py` (the oracle).
- **Text-only verification:** Cannot verify visual outputs, interactive behaviors, or side effects.
Assertions operate on the textual output only.
- **Single-turn scope:** Each verification is independent. The verifier does not remember prior
rounds (the orchestrating agent feeds context as needed).
- **Assertion granularity:** The 5 assertion types cover common patterns but not all possible
verification needs. Custom assertion types require extending `scripts/eval_assertions.py`.
## Error Handling
| Error | Resolution |
| ----------------------------- | --------------------------------------------------------------- |
| Skill definition too large | Summarize to workflow phases + output format sections only |
| No assertions generatable | Return empty assertions list with warning; skill may be too vague |
| Ambiguous output format | Default to `contains` assertions; avoid `output_format` checks |
| Context cap exceeded | Truncate diagnostic detail; preserve failed assertion list |