references/llm_review_schema.md
# LLM Review JSON Schema
Use this schema when requesting the LLM-axis review.
## Required JSON Shape
```json
{
"score": 0,
"summary": "one-paragraph assessment",
"findings": [
{
"severity": "high|medium|low",
"path": "skills/<skill-name>/file.ext",
"line": 0,
"message": "problem statement",
"improvement": "concrete fix"
}
]
}
```
## Constraints
- Return JSON only (no markdown wrapper).
- Set `score` in range `0..100`.
- Set `line` to `null` when unknown.
- Keep `severity` to `high`, `medium`, or `low`.
- Make `message` specific and testable.
- Make `improvement` actionable, preferably with file-level targets.
## Review Focus
- Validate logic and behavior in scripts (`scripts/*.py`).
- Check mismatch between `SKILL.md` instructions and actual script behavior.
- Check missing tests and risk of regressions.
- Identify ambiguous instructions that reduce execution reliability.
references/scoring_rubric.md
# Scoring Rubric Rationale
This document explains why each auto-axis component has its current weight.
## Weights
- `metadata_use_case` (20)
Why: bad frontmatter and unclear trigger conditions make a skill hard to invoke correctly.
- `workflow_coverage` (25)
Why: operators need executable flow; missing core sections creates ambiguity in real use.
- `execution_safety_reproducibility` (25)
Why: command examples and path hygiene determine whether results are repeatable and safe.
- `supporting_artifacts` (10)
Why: scripts/references/tests provide baseline maintainability, but should not dominate scoring.
- `test_health` (20)
Why: runtime confidence is critical; passing tests strongly increases trust in automation.
## Threshold Policy
- `90+`: production-ready baseline
- `80-89`: usable with targeted improvements
- `70-79`: notable gaps; strengthen before regular use
- `<70`: high risk; treat as draft and prioritize fixes
## Improvement Trigger
When final score is `< 90`, improvement items are mandatory in report output.
## Declared Verification Is Not Scored
If the target project declares verification axes in `skills-index.yaml`, the reviewer surfaces them
separately. They do not change component weights, findings, improvement items, or the final score.
The declarations are metadata supplied by the target project, not an independent reviewer finding.
## Knowledge-Only Skill Handling
For skills with no executable scripts (`scripts/*.py` absent) but with reference docs:
- Classify as `knowledge_only`.
- Do not penalize missing bash command examples.
- Treat script/test artifacts as mostly not-applicable (`supporting_artifacts` and `test_health` adjusted).
- Still require clear `When to Use`, `Prerequisites`, and workflow structure.
requirements.txt
# Runtime dependencies for standalone .skill installs (issue #330).
# Verified by scripts/check_skill_deps.py.
PyYAML>=6.0
scripts/run_dual_axis_review.py
#!/usr/bin/env python3
# /// script
# requires-python = ">=3.9"
# dependencies = ["pyyaml>=6.0"]
# ///
"""Dual-axis skill reviewer: auto checks + optional LLM score merge.
Cross-project usage:
uv run ~/.claude/skills/dual-axis-skill-reviewer/scripts/run_dual_axis_review.py \\
--project-root /path/to/target/project
"""
from __future__ import annotations
import argparse
import importlib.util
import json
import os
import random
import re
import subprocess
import sys
from dataclasses import dataclass
from datetime import datetime
from pathlib import Path
import yaml
VERIFICATION_AXES = (
"instruction_contract",
"unit_tests",
"workflow_contract",
"end_to_end_replay",
"data_provenance",
"financial_logic_review",
"empirical_validation",
"security_review",
)
VALID_VERIFICATION_VALUES = frozenset({"passed", "not_verified", "not_applicable"})
def _is_valid_verification_value(value: object) -> bool:
return isinstance(value, str) and value in VALID_VERIFICATION_VALUES
@dataclass
class Finding:
severity: str
path: str
line: int | None
message: str
improvement: str
def parse_args() -> argparse.Namespace:
parser = argparse.ArgumentParser(
description=(
"Pick one random skill (skills/*/SKILL.md), run rule-based review, and optionally "
"merge an LLM review score."
)
)
parser.add_argument(
"--project-root",
default=".",
help="Project root (default: current directory)",
)
parser.add_argument(
"--skill",
help="Optional skill name (e.g. vcp-screener) or SKILL.md path",
)
parser.add_argument(
"--seed",
type=int,
help="Random seed (for reproducible skill selection)",
)
parser.add_argument(
"--output-dir",
default="reports",
help="Directory for markdown/json review output (default: reports)",
)
parser.add_argument(
"--skip-tests",
action="store_true",
help="Skip running skill-level tests during review",
)
parser.add_argument(
"--emit-llm-prompt",
action="store_true",
help="Write an LLM review request prompt file alongside the report",
)
parser.add_argument(
"--llm-review-json",
help=(
"Path to LLM review JSON. Expected shape: "
"{score, summary, findings:[{severity,path,line,message,improvement}]}"
),
)
parser.add_argument(
"--all",
action="store_true",
help="Review all skills sequentially and output a summary table",
)
parser.add_argument(
"--auto-weight",
type=float,
default=0.5,
help="Weight for auto score when LLM score is provided (default: 0.5)",
)
parser.add_argument(
"--llm-weight",
type=float,
default=0.5,
help="Weight for LLM score when LLM score is provided (default: 0.5)",
)
return parser.parse_args()
def discover_skills(project_root: Path) -> list[Path]:
"""Find candidate skill definition files."""
return sorted(project_root.glob("skills/*/SKILL.md"))
def pick_skill(skills: list[Path], user_value: str | None, seed: int | None) -> Path:
"""Select a skill by explicit value or random choice."""
if user_value:
requested = Path(user_value)
if requested.name == "SKILL.md" and requested.exists():
return requested.resolve()
if "/" in user_value or user_value.endswith(".md"):
candidate = (Path.cwd() / requested).resolve()
if candidate.exists():
return candidate
by_name = [p for p in skills if p.parent.name == user_value]
if by_name:
return by_name[0]
raise ValueError(f"Skill not found: {user_value}")
rng = random.Random(seed)
return rng.choice(skills)
def find_line(lines: list[str], pattern: str) -> int | None:
"""Find the first line matching a regex pattern."""
for idx, line in enumerate(lines, start=1):
if re.search(pattern, line):
return idx
return None
def has_heading(text: str, patterns: list[str]) -> bool:
"""Check whether markdown includes at least one heading pattern."""
return any(re.search(pattern, text, flags=re.IGNORECASE | re.MULTILINE) for pattern in patterns)
def parse_frontmatter(lines: list[str]) -> dict[str, str]:
"""Parse YAML frontmatter block from a markdown file."""
if not lines or lines[0].strip() != "---":
return {}
end = None
for idx, line in enumerate(lines[1:], start=1):
if line.strip() == "---":
end = idx
break
if end is None:
return {}
block = "".join(lines[1:end]).strip()
if not block:
return {}
try:
parsed = yaml.safe_load(block)
except yaml.YAMLError:
return {}
if not isinstance(parsed, dict):
return {}
result: dict[str, str] = {}
for key, value in parsed.items():
if value is None:
continue
result[str(key)] = str(value)
return result
def _verification_unavailable(status: str) -> dict:
return {
"source": "skills-index.yaml",
"declaration_status": status,
"declaration_complete": False,
"axes": {},
"not_verified_axes": [],
"not_applicable_axes": [],
"all_applicable_axes_passed": False,
"note": (
"Declared metadata only; excluded from auto, LLM, and final scores. "
"A live high-severity issue check is still required for a production decision."
),
}
def load_verification_summary(project_root: Path, skill_name: str) -> dict:
"""Load non-scoring verification declarations from the target project's index."""
index_path = project_root / "skills-index.yaml"
if not index_path.is_file():
return _verification_unavailable("missing_index")
try:
index = yaml.safe_load(index_path.read_text(encoding="utf-8"))
except (OSError, yaml.YAMLError):
return _verification_unavailable("malformed_index")
if not isinstance(index, dict) or not isinstance(index.get("skills"), list):
return _verification_unavailable("malformed_index")
entry = next(
(
item
for item in index["skills"]
if isinstance(item, dict) and item.get("id") == skill_name
),
None,
)
if entry is None:
return _verification_unavailable("missing_entry")
if "verification" not in entry:
return _verification_unavailable("missing_block")
verification = entry["verification"]
if not isinstance(verification, dict):
return _verification_unavailable("malformed")
if set(verification) != set(VERIFICATION_AXES):
summary = _verification_unavailable("partial")
summary["axes"] = {
axis: verification[axis]
for axis in VERIFICATION_AXES
if axis in verification and _is_valid_verification_value(verification[axis])
}
return summary
if any(not _is_valid_verification_value(value) for value in verification.values()):
return _verification_unavailable("malformed")
axes = {axis: verification[axis] for axis in VERIFICATION_AXES}
not_verified = sorted(axis for axis, value in axes.items() if value == "not_verified")
not_applicable = sorted(axis for axis, value in axes.items() if value == "not_applicable")
return {
"source": "skills-index.yaml",
"declaration_status": "complete",
"declaration_complete": True,
"axes": axes,
"not_verified_axes": not_verified,
"not_applicable_axes": not_applicable,
"all_applicable_axes_passed": not not_verified,
"note": (
"Declared metadata only; excluded from auto, LLM, and final scores. "
"A live high-severity issue check is still required for a production decision."
),
}
def extract_bash_blocks(text: str) -> list[str]:
blocks = re.findall(r"```bash\s*(.*?)```", text, flags=re.DOTALL)
return [block.strip() for block in blocks if block.strip()]
def collect_output_dir_values(bash_blocks: list[str]) -> list[str]:
"""Extract output-dir argument values from bash snippets."""
values: list[str] = []
pattern = re.compile(r"--output-dir\s+([^\s]+)")
for block in bash_blocks:
for cmd_line in block.splitlines():
match = pattern.search(cmd_line)
if match:
values.append(match.group(1))
return values
def discover_test_dirs(skill_dir: Path) -> list[Path]:
"""Find test directories supported by this repository."""
candidates = [skill_dir / "scripts" / "tests", skill_dir / "tests"]
test_dirs = []
for tests_dir in candidates:
if tests_dir.exists() and list(tests_dir.glob("test_*.py")):
test_dirs.append(tests_dir)
return test_dirs
def run_tests(project_root: Path, skill_dir: Path) -> tuple[str, str | None, str]:
"""Run discovered tests and return status, command, and output."""
test_dirs = discover_test_dirs(skill_dir)
if not test_dirs:
return "not_found", None, ""
test_targets = [path.as_posix() for path in test_dirs]
uv_command = [
"uv",
"run",
"--extra",
"dev",
"pytest",
*test_targets,
"-q",
]
uv_command_text = " ".join(uv_command)
py_command = [sys.executable, "-m", "pytest", *test_targets, "-q"]
py_command_text = " ".join(py_command)
# Prefer the already-synchronized interpreter's pytest. Running `uv run` here
# would resolve/mutate dependencies on a cold environment and leak setup output
# ("Installed N packages ...") into test_output, making test evidence
# non-deterministic across warm/cold runs. Fall back to `uv run` only when the
# current interpreter has no pytest (e.g. a bare ~/.claude launcher).
env = dict(os.environ)
env["UV_CACHE_DIR"] = str(project_root / ".uv-cache")
if importlib.util.find_spec("pytest") is not None:
try:
proc = subprocess.run(
py_command,
cwd=project_root,
capture_output=True,
text=True,
timeout=180,
check=False,
)
command_text = py_command_text
except subprocess.TimeoutExpired:
return "timeout", py_command_text, "pytest timeout (>180s)"
else:
try:
proc = subprocess.run(
uv_command,
cwd=project_root,
capture_output=True,
text=True,
timeout=180,
check=False,
env=env,
)
command_text = uv_command_text
except FileNotFoundError:
return "tool_missing", None, "Neither `uv` nor `python -m pytest` is available."
except subprocess.TimeoutExpired:
return "timeout", uv_command_text, "pytest timeout (>180s)"
output = f"{proc.stdout}\n{proc.stderr}".strip()
if proc.returncode == 0:
return "passed", command_text, output
return "failed", command_text, output
def normalize_severity(value: str) -> str:
"""Normalize severity labels to high/medium/low."""
candidate = (value or "").strip().lower()
if candidate in {"high", "medium", "low"}:
return candidate
return "medium"
def load_llm_review(llm_review_json: str | None, project_root: Path) -> dict | None:
"""Load and validate LLM review JSON."""
if not llm_review_json:
return None
path = (project_root / llm_review_json).resolve()
if not path.exists():
raise ValueError(f"LLM review JSON not found: {llm_review_json}")
data = json.loads(path.read_text(encoding="utf-8"))
score = data.get("score")
if not isinstance(score, (int, float)):
raise ValueError("LLM review JSON must include numeric `score`.")
findings_raw = data.get("findings", [])
if findings_raw is None:
findings_raw = []
if not isinstance(findings_raw, list):
raise ValueError("LLM review JSON `findings` must be a list.")
findings = []
for raw in findings_raw:
if not isinstance(raw, dict):
continue
findings.append(
{
"severity": normalize_severity(str(raw.get("severity", "medium"))),
"path": str(raw.get("path", "")),
"line": raw.get("line"),
"message": str(raw.get("message", "")).strip(),
"improvement": str(raw.get("improvement", "")).strip(),
}
)
return {
"provided": True,
"source_file": str(path),
"score": max(0, min(100, int(round(float(score))))),
"summary": str(data.get("summary", "")).strip(),
"findings": findings,
}
def collect_skill_inventory(project_root: Path, skill_dir: Path) -> dict:
"""List key files so the LLM can review concrete artifacts."""
def rel(items: list[Path]) -> list[str]:
return [item.relative_to(project_root).as_posix() for item in sorted(items)]
test_files = []
for tests_dir in discover_test_dirs(skill_dir):
test_files.extend(list(tests_dir.glob("test_*.py")))
return {
"skill_md": (skill_dir / "SKILL.md").relative_to(project_root).as_posix(),
"scripts": rel(list((skill_dir / "scripts").glob("*.py"))),
"tests": rel(test_files),
"references": rel(list((skill_dir / "references").glob("*.md"))),
}
def build_llm_prompt(
project_root: Path,
skill_dir: Path,
auto_review: dict,
) -> str:
"""Create a strict prompt for LLM-axis review output."""
inventory = collect_skill_inventory(project_root, skill_dir)
lines = [
"# LLM Skill Review Request",
"",
"Please perform a deep review of the following skill, including:",
"- SKILL.md instruction quality and consistency",
"- helper scripts for correctness/risk",
"- tests and coverage gaps",
"- practical usability for operators",
"",
"## Skill",
f"- Name: `{auto_review['skill_name']}`",
f"- SKILL.md: `{inventory['skill_md']}`",
"",
"## Files to inspect",
"- Scripts:",
]
lines.extend([f" - `{path}`" for path in inventory["scripts"]] or [" - (none)"])
lines.append("- Tests:")
lines.extend([f" - `{path}`" for path in inventory["tests"]] or [" - (none)"])
lines.append("- References:")
lines.extend([f" - `{path}`" for path in inventory["references"]] or [" - (none)"])
lines.extend(
[
"",
"## Auto-axis baseline (for reference only; verify independently)",
f"- Auto score: {auto_review['score']}/100",
"- Auto findings:",
]
)
if auto_review["findings"]:
for finding in auto_review["findings"]:
lines.append(f" - [{finding['severity']}] {finding['message']}")
else:
lines.append(" - none")
lines.extend(
[
"",
"## Output format (strict JSON only)",
"```json",
"{",
' "score": 0,',
' "summary": "one-paragraph overall assessment",',
' "findings": [',
" {",
' "severity": "high|medium|low",',
' "path": "skills/.../file.ext",',
' "line": 0,',
' "message": "problem statement",',
' "improvement": "concrete fix"',
" }",
" ]",
"}",
"```",
"",
"Scoring rule: 0-100. If score < 90, include concrete improvements.",
]
)
return "\n".join(lines) + "\n"
def combine_reviews(
auto_review: dict,
llm_review: dict | None,
auto_weight: float,
llm_weight: float,
) -> dict:
"""Merge auto and LLM scores into a final weighted assessment."""
if llm_review and llm_review.get("provided"):
aw = max(0.0, auto_weight)
lw = max(0.0, llm_weight)
if aw + lw == 0:
aw = 0.5
lw = 0.5
aw_norm = aw / (aw + lw)
lw_norm = lw / (aw + lw)
final_score = int(round(auto_review["score"] * aw_norm + llm_review["score"] * lw_norm))
weights = {"auto_weight": aw_norm, "llm_weight": lw_norm}
else:
final_score = int(auto_review["score"])
weights = {"auto_weight": 1.0, "llm_weight": 0.0}
combined_findings = []
for finding in auto_review.get("findings", []):
combined_findings.append({**finding, "axis": "auto"})
if llm_review and llm_review.get("provided"):
for finding in llm_review.get("findings", []):
combined_findings.append({**finding, "axis": "llm"})
severity_rank = {"high": 0, "medium": 1, "low": 2}
combined_findings.sort(key=lambda item: severity_rank.get(item.get("severity", "medium"), 1))
improvements = []
if final_score < 90:
for finding in combined_findings:
message = finding.get("message", "").strip()
improvement = finding.get("improvement", "").strip()
if not message or not improvement:
continue
text_item = f"{message} -> {improvement}"
if text_item not in improvements:
improvements.append(text_item)
return {
"score": final_score,
"weights": weights,
"improvements_required": final_score < 90,
"improvement_items": improvements,
"findings": combined_findings,
}
def score_skill(
project_root: Path,
skill_file: Path,
skip_tests: bool,
) -> dict:
"""Run deterministic checks and scoring for one skill."""
skill_dir = skill_file.parent
skill_name = skill_dir.name
rel_skill_file = skill_file.relative_to(project_root).as_posix()
lines = skill_file.read_text(encoding="utf-8").splitlines(keepends=True)
text = "".join(lines)
frontmatter = parse_frontmatter(lines)
findings: list[Finding] = []
ref_count = len(list((skill_dir / "references").glob("*.md")))
script_count = len(list((skill_dir / "scripts").glob("*.py")))
test_count = 0
for tests_dir in discover_test_dirs(skill_dir):
test_count += len(list(tests_dir.glob("test_*.py")))
skill_type = "knowledge_only" if script_count == 0 and ref_count > 0 else "execution_or_hybrid"
# 1) Metadata & use-case clarity (max 20)
metadata_score = 0
if frontmatter:
metadata_score += 8
else:
findings.append(
Finding(
severity="high",
path=rel_skill_file,
line=1,
message="Frontmatter is missing or malformed.",
improvement="Add valid YAML frontmatter with `name` and `description`.",
)
)
declared_name = frontmatter.get("name")
if declared_name:
metadata_score += 6
if declared_name != skill_name:
findings.append(
Finding(
severity="medium",
path=rel_skill_file,
line=find_line(lines, r"^name:\s"),
message=f"`name` does not match directory name (`{declared_name}` vs `{skill_name}`).",
improvement="Align frontmatter `name` with the skill folder name.",
)
)
else:
metadata_score += 6
else:
findings.append(
Finding(
severity="high",
path=rel_skill_file,
line=find_line(lines, r"^---"),
message="Frontmatter `name` is missing.",
improvement="Add `name: <skill-folder-name>` to frontmatter.",
)
)
# 2) Workflow coverage (max 25)
workflow_score = 0
required_sections = {
"When to Use": [r"^##\s*When to Use", r"^##\s*When to Use This Skill"],
"Prerequisites": [r"^##\s*Prerequisites", r"^##\s*Input Requirements"],
"Workflow": [
r"^##\s*Workflow",
r"^##\s*Execution Workflow",
r"^##\s*Evaluation Process",
r"^##\s*.+\s+Workflow$",
],
"Output": [r"^##\s*Output", r"^##\s*Output Files"],
"Resources": [
r"^##\s*Resources",
r"^##\s*Reference Documents",
r"^##\s*Data Sources",
r"^##\s*Available Reference Documentation",
r"^##\s*.+\s+Reference Documentation$",
],
}
for section_name, heading_patterns in required_sections.items():
if has_heading(text, heading_patterns):
workflow_score += 5
elif section_name == "Output" and skill_type == "knowledge_only":
workflow_score += 3
findings.append(
Finding(
severity="low",
path=rel_skill_file,
line=None,
message="No explicit `## Output` section for advisory/knowledge skill.",
improvement="Add `## Output` and state that output is conversational guidance (no file generation).",
)
)
else:
findings.append(
Finding(
severity="medium",
path=rel_skill_file,
line=None,
message=f"Missing section: `## {section_name}`.",
improvement=f"Add `## {section_name}` to improve operator guidance.",
)
)
# 3) Execution safety & reproducibility (max 25)
exec_score = 0
bash_blocks = extract_bash_blocks(text)
if bash_blocks:
exec_score += 6
# Relative paths (e.g. `scripts/run.py`) are preferred because SKILL.md
# is read in the context of the skill directory. Full paths like
# `skills/{skill}/scripts/...` are redundant and break portability.
if any("scripts/" in block for block in bash_blocks):
exec_score += 5
else:
findings.append(
Finding(
severity="low",
path=rel_skill_file,
line=None,
message="No script path references found in bash examples.",
improvement="Use relative paths like `scripts/...` in commands.",
)
)
output_dir_values = collect_output_dir_values(bash_blocks)
unsafe_output_dir = any(
"/scripts" in value or value.endswith("scripts") for value in output_dir_values
)
if output_dir_values and not unsafe_output_dir:
exec_score += 5
elif output_dir_values and unsafe_output_dir:
findings.append(
Finding(
severity="medium",
path=rel_skill_file,
line=find_line(lines, r"--output-dir"),
message="Sample commands write generated reports into source `scripts/` directory.",
improvement="Use a dedicated output path like `reports/` to avoid source tree pollution.",
)
)
else:
exec_score += 3
elif skill_type == "knowledge_only":
# Knowledge-centric skills may be intentionally non-executable.
exec_score += 8
else:
findings.append(
Finding(
severity="medium",
path=rel_skill_file,
line=None,
message="No runnable bash examples found.",
improvement="Add executable command examples in a `bash` code block.",
)
)
# API key handling: exempt skills that don't use API keys
api_key_patterns = [
r"FMP_API_KEY",
r"FINVIZ_API_KEY",
r"ALPACA_API_KEY",
r"--api-key",
]
requires_api = frontmatter.get("requires_api_key")
if requires_api is not None:
skill_uses_api = requires_api.lower() not in ("false", "no", "0")
else:
# Check SKILL.md and scripts for API key patterns
scripts_text = ""
scripts_dir = skill_dir / "scripts"
if scripts_dir.is_dir():
for script_file in sorted(scripts_dir.glob("*.py")):
try:
scripts_text += script_file.read_text(encoding="utf-8")
except OSError:
pass
combined_text = text + scripts_text
skill_uses_api = any(re.search(pat, combined_text) for pat in api_key_patterns)
if not skill_uses_api:
exec_score += 4
elif re.search(r"export\s+(FMP_API_KEY|FINVIZ_API_KEY|ALPACA_API_KEY)=", text) or re.search(
r"--api-key\s+\S+", text
):
exec_score += 4
elif any(re.search(pat, text) for pat in api_key_patterns):
exec_score += 2
findings.append(
Finding(
severity="low",
path=rel_skill_file,
line=find_line(lines, r"(FMP|FINVIZ|ALPACA)_API_KEY|--api-key"),
message="API key is documented, but no copy-paste setup command is shown.",
improvement="Add an explicit example like `export FMP_API_KEY=...`.",
)
)
ref_paths = re.findall(r"`([^`]*references/[^`]*)`", text)
if not ref_paths:
exec_score += 3
else:
# Relative paths (e.g. `references/foo.md`) are preferred because SKILL.md
# is read in the context of the skill directory. Full paths like
# `skills/{skill}/references/...` are redundant and break portability.
relative_paths = [p for p in ref_paths if p.startswith("references/")]
if len(relative_paths) == len(ref_paths):
exec_score += 5
else:
exec_score += 2
findings.append(
Finding(
severity="low",
path=rel_skill_file,
line=find_line(lines, r"references/"),
message="Reference paths use full project paths instead of relative paths.",
improvement=(
"Use relative paths like `references/...` instead of "
f"`skills/{skill_name}/references/...` — SKILL.md is loaded "
"in the skill directory context."
),
)
)
# PII / hardcoded path check (no score impact, finding only)
pii_pattern = re.compile(r"/Users/[A-Za-z0-9._-]+/")
all_skill_files: list[tuple[str, list[str]]] = [(rel_skill_file, lines)]
for subdir_name in ("scripts", "references"):
subdir = skill_dir / subdir_name
if subdir.is_dir():
for child in sorted(subdir.rglob("*")):
if child.is_file() and child.suffix in (".py", ".md", ".yaml", ".yml", ".sh"):
try:
child_lines = child.read_text(encoding="utf-8").splitlines()
except OSError:
continue
rel_child = child.relative_to(project_root).as_posix()
all_skill_files.append((rel_child, child_lines))
for file_rel, file_lines in all_skill_files:
for line_num, line_text in enumerate(file_lines, start=1):
if pii_pattern.search(line_text):
findings.append(
Finding(
severity="high",
path=file_rel,
line=line_num,
message="Hardcoded absolute user path detected (potential PII leak).",
improvement=(
"Replace with a relative path or dynamic resolution "
"like `Path(__file__).resolve().parents[N]`."
),
)
)
break # one finding per file is sufficient
# 4) Supporting artifacts (max 10)
artifact_score = 0
if ref_count > 0:
artifact_score += 4
else:
findings.append(
Finding(
severity="medium",
path=rel_skill_file,
line=None,
message="No markdown reference files found in `references/`.",
improvement="Add methodology/reference docs to support consistent interpretation.",
)
)
if script_count > 0:
artifact_score += 3
elif skill_type == "knowledge_only":
artifact_score += 3
else:
findings.append(
Finding(
severity="medium",
path=rel_skill_file,
line=None,
message="No executable helper scripts found in `scripts/`.",
improvement="Add automation scripts for repeatable execution.",
)
)
if test_count > 0:
artifact_score += 3
elif skill_type == "knowledge_only":
artifact_score += 2
else:
findings.append(
Finding(
severity="medium",
path=rel_skill_file,
line=None,
message="No `test_*.py` tests found for skill scripts.",
improvement="Add skill-level tests under `scripts/tests/` or `tests/`.",
)
)
# 5) Test health (max 20)
test_score = 0
test_status = "skipped"
test_command: str | None = None
test_output = ""
if skip_tests:
if skill_type == "knowledge_only" and test_count == 0:
test_status = "not_applicable"
test_score = 12
else:
test_status = "skipped"
test_score = 8 if test_count > 0 else 0
else:
if skill_type == "knowledge_only" and script_count == 0:
test_status = "not_applicable"
test_score = 12
else:
test_status, test_command, test_output = run_tests(project_root, skill_dir)
if test_status == "passed":
test_score = 20
elif test_status in {"not_found", "tool_missing"}:
test_score = 6 if test_count > 0 else 0
elif test_status == "timeout":
test_score = 2
findings.append(
Finding(
severity="medium",
path=rel_skill_file,
line=None,
message="Skill tests timed out; health could not be verified.",
improvement="Optimize tests or split slower integration tests.",
)
)
else:
findings.append(
Finding(
severity="high",
path=rel_skill_file,
line=None,
message="Skill tests failed.",
improvement="Fix failing tests before relying on this skill in automation.",
)
)
total_score = metadata_score + workflow_score + exec_score + artifact_score + test_score
total_score = max(0, min(100, total_score))
severity_rank = {"high": 0, "medium": 1, "low": 2}
findings.sort(key=lambda item: severity_rank[item.severity])
improvements = []
if total_score < 90:
for finding in findings:
text_item = f"{finding.message} -> {finding.improvement}"
if text_item not in improvements:
improvements.append(text_item)
return {
"skill_name": skill_name,
"skill_file": rel_skill_file,
"skill_type": skill_type,
"score": total_score,
"score_breakdown": {
"metadata_use_case": metadata_score,
"workflow_coverage": workflow_score,
"execution_safety_reproducibility": exec_score,
"supporting_artifacts": artifact_score,
"test_health": test_score,
},
"findings": [
{
"severity": f.severity,
"path": f.path,
"line": f.line,
"message": f.message,
"improvement": f.improvement,
}
for f in findings
],
"improvements_required": total_score < 90,
"improvement_items": improvements,
"test_status": test_status,
"test_command": test_command,
"test_output": test_output,
}
def to_markdown(report: dict) -> str:
auto_review = report["auto_review"]
llm_review = report["llm_review"]
final_review = report["final_review"]
lines = [
"# Dual-Axis Skill Review",
"",
f"- Generated at: {report['generated_at']}",
f"- Selected skill: `{report['skill_name']}`",
f"- Skill file: `{report['skill_file']}`",
f"- Skill type: `{auto_review.get('skill_type', 'unknown')}`",
f"- Selection mode: `{report['selection_mode']}`",
f"- Seed: `{report['seed']}`",
f"- Auto score: **{auto_review['score']} / 100**",
f"- LLM score: **{llm_review['score']} / 100**"
if llm_review["provided"]
else "- LLM score: `not provided`",
f"- Final score: **{final_review['score']} / 100**",
"",
"## Auto Score Breakdown",
]
for key, value in auto_review["score_breakdown"].items():
lines.append(f"- {key}: {value}")
lines.extend(
[
"",
"## Score Weights",
f"- auto_weight: {final_review['weights']['auto_weight']:.2f}",
f"- llm_weight: {final_review['weights']['llm_weight']:.2f}",
]
)
verification = report.get("declared_verification") or _verification_unavailable("unavailable")
lines.extend(
[
"",
"## Declared Verification (Non-Scoring)",
f"- Declaration status: `{verification['declaration_status']}`",
f"- Declaration complete: `{str(verification['declaration_complete']).lower()}`",
"- All applicable axes passed: "
f"`{str(verification['all_applicable_axes_passed']).lower()}`",
f"- Not verified: `{', '.join(verification['not_verified_axes']) or 'none'}`",
f"- Not applicable: `{', '.join(verification['not_applicable_axes']) or 'none'}`",
f"- Note: {verification['note']}",
]
)
if verification["axes"]:
lines.append("")
for axis, value in verification["axes"].items():
lines.append(f"- `{axis}`: `{value}`")
lines.extend(["", "## Findings (Combined)"])
if final_review["findings"]:
for idx, finding in enumerate(final_review["findings"], start=1):
loc = finding.get("path", "")
if finding.get("line"):
loc = f"{loc}:{finding['line']}"
lines.append(
f"{idx}. [{finding.get('axis', 'unknown').upper()}|"
f"{finding.get('severity', 'medium').upper()}] `{loc}` - "
f"{finding.get('message', '').strip()}"
)
else:
lines.append("- No notable findings.")
lines.extend(
[
"",
"## Test Verification (Auto Axis)",
f"- Status: `{auto_review['test_status']}`",
f"- Command: `{auto_review['test_command']}`",
]
)
if report.get("llm_prompt_file"):
lines.extend(["", "## LLM Prompt File", f"- `{report['llm_prompt_file']}`"])
if final_review["improvements_required"]:
lines.extend(["", "## Improvement Items (Final Score < 90)"])
for idx, item in enumerate(final_review["improvement_items"], start=1):
lines.append(f"{idx}. {item}")
return "\n".join(lines) + "\n"
def review_single_skill(
args: argparse.Namespace,
project_root: Path,
skill_file: Path,
) -> dict:
"""Run review for a single skill and write output files. Returns the report dict."""
auto_review = score_skill(
project_root=project_root,
skill_file=skill_file,
skip_tests=args.skip_tests,
)
try:
llm_review = load_llm_review(args.llm_review_json, project_root)
except (json.JSONDecodeError, ValueError) as exc:
print(f"Invalid --llm-review-json: {exc}", file=sys.stderr)
return {}
if llm_review is None:
llm_review = {
"provided": False,
"source_file": None,
"score": None,
"summary": "",
"findings": [],
}
final_review = combine_reviews(
auto_review=auto_review,
llm_review=llm_review if llm_review["provided"] else None,
auto_weight=args.auto_weight,
llm_weight=args.llm_weight,
)
now = datetime.now()
generated_at = now.strftime("%Y-%m-%d %H:%M:%S")
output_dir = (project_root / args.output_dir).resolve()
output_dir.mkdir(parents=True, exist_ok=True)
timestamp = now.strftime("%Y-%m-%d_%H%M%S")
stem = f"skill_review_{auto_review['skill_name']}_{timestamp}"
json_path = output_dir / f"{stem}.json"
md_path = output_dir / f"{stem}.md"
llm_prompt_file = None
if args.emit_llm_prompt:
prompt_file = output_dir / f"skill_review_prompt_{auto_review['skill_name']}_{timestamp}.md"
prompt_file.write_text(
build_llm_prompt(project_root, skill_file.parent, auto_review),
encoding="utf-8",
)
llm_prompt_file = str(prompt_file)
report = {
"generated_at": generated_at,
"seed": args.seed,
"selection_mode": "manual" if args.skill else "random",
"skill_name": auto_review["skill_name"],
"skill_file": auto_review["skill_file"],
"auto_review": auto_review,
"llm_review": llm_review,
"final_review": final_review,
"declared_verification": load_verification_summary(project_root, auto_review["skill_name"]),
"llm_prompt_file": llm_prompt_file,
}
json_path.write_text(json.dumps(report, ensure_ascii=False, indent=2), encoding="utf-8")
md_path.write_text(to_markdown(report), encoding="utf-8")
return report
def main() -> int:
args = parse_args()
project_root = Path(args.project_root).resolve()
skills = discover_skills(project_root)
if not skills:
print("No skills found at skills/*/SKILL.md", file=sys.stderr)
return 1
if args.all:
return _run_all(args, project_root, skills)
try:
selected = pick_skill(skills, args.skill, args.seed)
except ValueError as exc:
print(str(exc), file=sys.stderr)
return 1
report = review_single_skill(args, project_root, selected)
if not report:
return 1
auto_review = report["auto_review"]
llm_review = report["llm_review"]
final_review = report["final_review"]
print(f"Selected skill: {auto_review['skill_name']}")
print(f"Auto score: {auto_review['score']}/100")
if llm_review["provided"]:
print(f"LLM score: {llm_review['score']}/100")
print(f"Final score: {final_review['score']}/100")
print(
"Declared verification: "
f"{report['declared_verification']['declaration_status']} (non-scoring)"
)
if report.get("llm_prompt_file"):
print(f"LLM prompt: {report['llm_prompt_file']}")
if auto_review["test_status"] == "passed":
summary_line = next(
(line for line in auto_review["test_output"].splitlines() if " passed" in line),
"",
)
if summary_line:
print(f"Tests: {summary_line.strip()}")
return 0
def _run_all(
args: argparse.Namespace,
project_root: Path,
skills: list[Path],
) -> int:
"""Review all skills sequentially and output a summary table."""
rows: list[dict] = []
for skill_file in skills:
report = review_single_skill(args, project_root, skill_file)
if not report:
continue
auto_review = report["auto_review"]
final_review = report["final_review"]
high_count = sum(1 for f in final_review.get("findings", []) if f.get("severity") == "high")
verification = report["declared_verification"]
rows.append(
{
"skill": auto_review["skill_name"],
"score": final_review["score"],
"pass": final_review["score"] >= 90,
"high_findings": high_count,
"verification_status": verification["declaration_status"],
"declaration_complete": verification["declaration_complete"],
"verification_gaps": len(verification["not_verified_axes"]),
"not_verified_axes": verification["not_verified_axes"],
"not_applicable_axes": verification["not_applicable_axes"],
"all_applicable_axes_passed": verification["all_applicable_axes_passed"],
}
)
if not rows:
print("No skills reviewed.", file=sys.stderr)
return 1
# Print summary table
header = f"{'Skill':<40} {'Score':>5} {'Pass':>4} {'High':>4} {'Verify':>10} {'Gaps':>4}"
print(header)
print("-" * len(header))
for row in rows:
status = "YES" if row["pass"] else "NO"
print(
f"{row['skill']:<40} {row['score']:>5} {status:>4} "
f"{row['high_findings']:>4} {row['verification_status']:>10} "
f"{row['verification_gaps']:>4}"
)
passed = sum(1 for r in rows if r["pass"])
print(f"\nTotal: {len(rows)} skills, {passed} passed, {len(rows) - passed} failed")
# Write summary JSON
output_dir = (project_root / args.output_dir).resolve()
output_dir.mkdir(parents=True, exist_ok=True)
now = datetime.now()
summary_path = output_dir / f"skill_review_all_{now.strftime('%Y-%m-%d_%H%M%S')}.json"
summary_path.write_text(
json.dumps(
{"generated_at": now.strftime("%Y-%m-%d %H:%M:%S"), "results": rows},
ensure_ascii=False,
indent=2,
),
encoding="utf-8",
)
print(f"Summary JSON: {summary_path}")
return 0 if all(r["pass"] for r in rows) else 1
if __name__ == "__main__":
raise SystemExit(main())
scripts/tests/conftest.py
"""Test fixtures for dual-axis reviewer."""
from __future__ import annotations
import importlib.util
import sys
from pathlib import Path
import pytest
@pytest.fixture(scope="module")
def reviewer_module():
"""Load run_dual_axis_review.py as a module for unit tests."""
script_path = Path(__file__).resolve().parents[1] / "run_dual_axis_review.py"
spec = importlib.util.spec_from_file_location("run_dual_axis_review", script_path)
if spec is None or spec.loader is None:
raise RuntimeError("Failed to load run_dual_axis_review.py")
module = importlib.util.module_from_spec(spec)
sys.modules[spec.name] = module
spec.loader.exec_module(module)
return module
scripts/tests/test_run_dual_axis_review.py
from __future__ import annotations
import json
import subprocess
import sys
from pathlib import Path
from types import SimpleNamespace
import pytest
def write_text(path: Path, text: str) -> None:
path.parent.mkdir(parents=True, exist_ok=True)
path.write_text(text, encoding="utf-8")
def test_parse_frontmatter_supports_yaml(reviewer_module):
lines = [
"---\n",
"name: sample-skill\n",
"description: >-\n",
" line one: value\n",
" line two\n",
"---\n",
"# body\n",
]
frontmatter = reviewer_module.parse_frontmatter(lines)
assert frontmatter["name"] == "sample-skill"
assert "line one: value" in frontmatter["description"]
assert "line two" in frontmatter["description"]
def test_parse_frontmatter_rejects_invalid_yaml(reviewer_module):
lines = [
"---\n",
"name: sample\n",
"description: [unterminated\n",
"---\n",
]
assert reviewer_module.parse_frontmatter(lines) == {}
def test_pick_skill_by_name_and_missing(reviewer_module, tmp_path: Path):
skill_a = tmp_path / "skills" / "a-skill" / "SKILL.md"
skill_b = tmp_path / "skills" / "b-skill" / "SKILL.md"
write_text(skill_a, "---\nname: a-skill\ndescription: x\n---\n")
write_text(skill_b, "---\nname: b-skill\ndescription: x\n---\n")
skills = [skill_a, skill_b]
picked = reviewer_module.pick_skill(skills, "b-skill", None)
assert picked == skill_b
with pytest.raises(ValueError):
reviewer_module.pick_skill(skills, "missing-skill", None)
def test_discover_test_dirs_supports_two_layouts(reviewer_module, tmp_path: Path):
skill_dir = tmp_path / "skills" / "sample"
write_text(skill_dir / "scripts" / "tests" / "test_a.py", "def test_a():\n assert True\n")
write_text(skill_dir / "tests" / "test_b.py", "def test_b():\n assert True\n")
dirs = reviewer_module.discover_test_dirs(skill_dir)
assert skill_dir / "scripts" / "tests" in dirs
assert skill_dir / "tests" in dirs
def test_run_tests_prefers_synchronized_interpreter(reviewer_module, tmp_path: Path, monkeypatch):
skill_dir = tmp_path / "skills" / "sample"
write_text(skill_dir / "tests" / "test_x.py", "def test_x():\n assert True\n")
calls: list[list[str]] = []
def fake_run(cmd, **kwargs):
calls.append(cmd)
return subprocess.CompletedProcess(cmd, 0, "ok\n1 passed\n", "")
monkeypatch.setattr(reviewer_module.subprocess, "run", fake_run)
monkeypatch.setattr(
reviewer_module.importlib.util,
"find_spec",
lambda name: SimpleNamespace() if name == "pytest" else None,
)
status, command, output = reviewer_module.run_tests(tmp_path, skill_dir)
assert status == "passed"
assert command is not None and sys.executable in command
assert "1 passed" in output
assert calls[0][0] == sys.executable
assert not any(call[0] == "uv" for call in calls)
def test_run_tests_falls_back_to_uv_without_pytest_installed(
reviewer_module, tmp_path: Path, monkeypatch
):
skill_dir = tmp_path / "skills" / "sample"
write_text(skill_dir / "tests" / "test_x.py", "def test_x():\n assert True\n")
calls: list[list[str]] = []
def fake_run(cmd, **kwargs):
calls.append(cmd)
if cmd[0] == "uv":
return subprocess.CompletedProcess(cmd, 0, "ok\n1 passed\n", "")
raise AssertionError("python pytest should not be tried when pytest is unavailable")
monkeypatch.setattr(reviewer_module.subprocess, "run", fake_run)
monkeypatch.setattr(reviewer_module.importlib.util, "find_spec", lambda name: None)
status, command, output = reviewer_module.run_tests(tmp_path, skill_dir)
assert status == "passed"
assert command is not None and command.startswith("uv")
assert "1 passed" in output
assert calls[0][0] == "uv"
def test_run_tests_python_timeout_is_captured(reviewer_module, tmp_path: Path, monkeypatch):
skill_dir = tmp_path / "skills" / "sample"
write_text(skill_dir / "tests" / "test_x.py", "def test_x():\n assert True\n")
calls: list[list[str]] = []
def fake_run(cmd, **kwargs):
calls.append(cmd)
raise subprocess.TimeoutExpired(cmd=cmd, timeout=180)
monkeypatch.setattr(reviewer_module.subprocess, "run", fake_run)
monkeypatch.setattr(
reviewer_module.importlib.util,
"find_spec",
lambda name: SimpleNamespace() if name == "pytest" else None,
)
status, command, output = reviewer_module.run_tests(tmp_path, skill_dir)
assert status == "timeout"
assert command is not None and sys.executable in command
assert calls[0][0] == sys.executable
assert "timeout" in output.lower()
def test_load_llm_review_validation(reviewer_module, tmp_path: Path):
bad = tmp_path / "bad.json"
bad.write_text(json.dumps({"score": "bad"}), encoding="utf-8")
with pytest.raises(ValueError):
reviewer_module.load_llm_review(str(bad), tmp_path)
good = tmp_path / "good.json"
good.write_text(
json.dumps(
{
"score": 85,
"summary": "ok",
"findings": [
{
"severity": "HIGH",
"path": "skills/x/SKILL.md",
"line": 5,
"message": "issue",
"improvement": "fix",
}
],
}
),
encoding="utf-8",
)
loaded = reviewer_module.load_llm_review(str(good), tmp_path)
assert loaded["score"] == 85
assert loaded["findings"][0]["severity"] == "high"
def test_combine_reviews_handles_zero_weights(reviewer_module):
auto = {"score": 80, "findings": []}
llm = {"provided": True, "score": 60, "findings": []}
merged = reviewer_module.combine_reviews(auto, llm, auto_weight=0.0, llm_weight=0.0)
assert merged["score"] == 70
assert merged["weights"]["auto_weight"] == 0.5
assert merged["weights"]["llm_weight"] == 0.5
def test_load_verification_summary_complete_and_non_scoring(reviewer_module, tmp_path: Path):
skill_dir = tmp_path / "skills" / "sample-skill"
skill_md = skill_dir / "SKILL.md"
write_text(
skill_md,
"---\nname: sample-skill\ndescription: sample\n---\n"
"## When to Use\nx\n## Prerequisites\nx\n## Workflow\nx\n"
"## Output\nx\n## Resources\nx\n",
)
before = reviewer_module.score_skill(tmp_path, skill_md, skip_tests=True)
write_text(
tmp_path / "skills-index.yaml",
"skills:\n"
"- id: sample-skill\n"
" verification:\n"
" instruction_contract: passed\n"
" unit_tests: not_verified\n"
" workflow_contract: not_applicable\n"
" end_to_end_replay: not_applicable\n"
" data_provenance: not_verified\n"
" financial_logic_review: not_applicable\n"
" empirical_validation: not_applicable\n"
" security_review: not_verified\n",
)
summary = reviewer_module.load_verification_summary(tmp_path, "sample-skill")
after = reviewer_module.score_skill(tmp_path, skill_md, skip_tests=True)
assert summary["declaration_status"] == "complete"
assert summary["declaration_complete"] is True
assert summary["not_verified_axes"] == [
"data_provenance",
"security_review",
"unit_tests",
]
assert summary["not_applicable_axes"] == [
"empirical_validation",
"end_to_end_replay",
"financial_logic_review",
"workflow_contract",
]
assert summary["all_applicable_axes_passed"] is False
assert before == after
def test_verification_metadata_does_not_change_any_review_axis(reviewer_module, tmp_path: Path):
skill_dir = tmp_path / "skills" / "sample-skill"
skill_md = skill_dir / "SKILL.md"
write_text(
skill_md,
"---\nname: sample-skill\ndescription: sample\n---\n"
"## When to Use\nx\n## Prerequisites\nx\n## Workflow\nx\n"
"## Output\nx\n## Resources\nx\n",
)
llm_path = tmp_path / "llm.json"
write_text(
llm_path,
json.dumps(
{
"score": 77,
"summary": "independent review",
"findings": [
{
"severity": "low",
"path": "skills/sample-skill/SKILL.md",
"line": 1,
"message": "same finding",
"improvement": "same fix",
}
],
}
),
)
args = SimpleNamespace(
skip_tests=True,
llm_review_json="llm.json",
auto_weight=0.4,
llm_weight=0.6,
output_dir="before",
emit_llm_prompt=False,
skill="sample-skill",
seed=None,
)
before = reviewer_module.review_single_skill(args, tmp_path, skill_md)
write_text(
tmp_path / "skills-index.yaml",
"skills:\n- id: sample-skill\n verification:\n"
+ "".join(f" {axis}: not_verified\n" for axis in reviewer_module.VERIFICATION_AXES),
)
args.output_dir = "after"
after = reviewer_module.review_single_skill(args, tmp_path, skill_md)
assert before["auto_review"] == after["auto_review"]
assert before["llm_review"] == after["llm_review"]
assert before["final_review"] == after["final_review"]
assert before["declared_verification"]["declaration_status"] == "missing_index"
assert after["declared_verification"]["declaration_status"] == "complete"
@pytest.mark.parametrize(
("index_text", "status"),
[
(None, "missing_index"),
("skills: []\n", "missing_entry"),
("skills:\n- id: sample-skill\n", "missing_block"),
("skills:\n- id: sample-skill\n verification: []\n", "malformed"),
(
"skills:\n- id: sample-skill\n verification:\n"
" instruction_contract: [passed]\n"
" unit_tests: not_verified\n"
" workflow_contract: not_applicable\n"
" end_to_end_replay: not_applicable\n"
" data_provenance: not_verified\n"
" financial_logic_review: not_applicable\n"
" empirical_validation: not_applicable\n"
" security_review: not_verified\n",
"malformed",
),
(
"skills:\n- id: sample-skill\n verification:\n instruction_contract: passed\n",
"partial",
),
],
)
def test_load_verification_summary_unavailable_states(
reviewer_module, tmp_path: Path, index_text: str | None, status: str
):
if index_text is not None:
write_text(tmp_path / "skills-index.yaml", index_text)
summary = reviewer_module.load_verification_summary(tmp_path, "sample-skill")
assert summary["declaration_status"] == status
assert summary["declaration_complete"] is False
assert summary["all_applicable_axes_passed"] is False
def test_score_skill_counts_tests_in_root_tests_dir(reviewer_module, tmp_path: Path):
project_root = tmp_path
skill_dir = project_root / "skills" / "sample-skill"
skill_md = skill_dir / "SKILL.md"
write_text(
skill_md,
"\n".join(
[
"---",
"name: sample-skill",
"description: sample",
"---",
"",
"## When to Use",
"x",
"## Prerequisites",
"x",
"## Workflow",
"```bash",
"python3 scripts/run.py --output-dir reports/",
"```",
"## Output",
"x",
"## Resources",
"- `references/ref.md`",
"",
]
),
)
write_text(skill_dir / "scripts" / "run.py", "print('ok')\n")
write_text(skill_dir / "references" / "ref.md", "# ref\n")
write_text(skill_dir / "tests" / "test_sample.py", "def test_sample():\n assert True\n")
review = reviewer_module.score_skill(project_root, skill_md, skip_tests=True)
assert review["score_breakdown"]["supporting_artifacts"] == 10
assert review["score_breakdown"]["test_health"] == 8
messages = [finding["message"] for finding in review["findings"]]
assert "No `test_*.py` tests found for skill scripts." not in messages
def test_normalize_severity_direct(reviewer_module):
assert reviewer_module.normalize_severity("HIGH") == "high"
assert reviewer_module.normalize_severity(" medium ") == "medium"
assert reviewer_module.normalize_severity("unknown") == "medium"
def test_extract_bash_blocks_handles_empty_and_multiple(reviewer_module):
text = """
before
```bash
echo one
```
middle
```bash
```
after
```bash
echo two
```
"""
blocks = reviewer_module.extract_bash_blocks(text)
assert blocks == ["echo one", "echo two"]
def test_build_llm_prompt_includes_inventory(reviewer_module, tmp_path: Path):
project_root = tmp_path
skill_dir = project_root / "skills" / "prompt-skill"
write_text(skill_dir / "SKILL.md", "---\nname: prompt-skill\ndescription: d\n---\n")
write_text(skill_dir / "scripts" / "main.py", "print('x')\n")
write_text(skill_dir / "tests" / "test_main.py", "def test_main():\n assert True\n")
write_text(skill_dir / "references" / "note.md", "# note\n")
prompt = reviewer_module.build_llm_prompt(
project_root=project_root,
skill_dir=skill_dir,
auto_review={"skill_name": "prompt-skill", "score": 88, "findings": []},
)
assert "LLM Skill Review Request" in prompt
assert "skills/prompt-skill/scripts/main.py" in prompt
assert "skills/prompt-skill/tests/test_main.py" in prompt
assert "strict JSON only" in prompt
def test_to_markdown_contains_combined_sections(reviewer_module):
report = {
"generated_at": "2026-02-20 00:00:00",
"skill_name": "x-skill",
"skill_file": "skills/x-skill/SKILL.md",
"selection_mode": "manual",
"seed": 1,
"auto_review": {
"score": 80,
"score_breakdown": {"a": 1},
"test_status": "passed",
"test_command": "pytest x",
},
"llm_review": {"provided": True, "score": 70},
"final_review": {
"score": 75,
"weights": {"auto_weight": 0.5, "llm_weight": 0.5},
"findings": [
{
"axis": "auto",
"severity": "medium",
"path": "skills/x-skill/SKILL.md",
"line": 3,
"message": "m",
}
],
"improvements_required": True,
"improvement_items": ["m -> fix"],
},
"llm_prompt_file": "reports/prompt.md",
}
md = reviewer_module.to_markdown(report)
assert "# Dual-Axis Skill Review" in md
assert "Final score: **75 / 100**" in md
assert "## Findings (Combined)" in md
assert "## Improvement Items (Final Score < 90)" in md
assert "## Declared Verification (Non-Scoring)" in md
def test_execution_safety_max_25(reviewer_module, tmp_path: Path):
"""Verify execution_safety_reproducibility never exceeds 25."""
project_root = tmp_path
skill_dir = project_root / "skills" / "perfect-skill"
skill_md = skill_dir / "SKILL.md"
write_text(
skill_md,
"\n".join(
[
"---",
"name: perfect-skill",
"description: test",
"requires_api_key: true",
"---",
"",
"## When to Use",
"x",
"## Prerequisites",
"x",
"## Workflow",
"```bash",
"export FMP_API_KEY=your_key_here",
"python3 scripts/run.py --output-dir reports/ --api-key $FMP_API_KEY",
"```",
"## Output",
"x",
"## Resources",
"- `references/ref.md`",
"",
]
),
)
write_text(skill_dir / "scripts" / "run.py", "print('ok')\n")
write_text(skill_dir / "references" / "ref.md", "# ref\n")
write_text(skill_dir / "tests" / "test_sample.py", "def test():\n assert True\n")
review = reviewer_module.score_skill(project_root, skill_md, skip_tests=True)
exec_safety = review["score_breakdown"]["execution_safety_reproducibility"]
assert exec_safety <= 25, f"execution_safety_reproducibility={exec_safety} exceeds max 25"
def test_api_key_score_exempt_with_frontmatter_flag(reviewer_module, tmp_path: Path):
"""Skills with requires_api_key: false should get full API key points."""
project_root = tmp_path
skill_dir = project_root / "skills" / "no-api-skill"
skill_md = skill_dir / "SKILL.md"
write_text(
skill_md,
"\n".join(
[
"---",
"name: no-api-skill",
"description: test",
"requires_api_key: false",
"---",
"",
"## When to Use",
"x",
"## Prerequisites",
"x",
"## Workflow",
"```bash",
"python3 scripts/run.py --output-dir reports/",
"```",
"## Output",
"x",
"## Resources",
"- `references/ref.md`",
"",
]
),
)
write_text(skill_dir / "scripts" / "run.py", "print('ok')\n")
write_text(skill_dir / "references" / "ref.md", "# ref\n")
review = reviewer_module.score_skill(project_root, skill_md, skip_tests=True)
# API key exempt: should not have API key finding
messages = [f["message"] for f in review["findings"]]
assert not any("API key" in m for m in messages), f"Unexpected API key finding: {messages}"
def test_api_key_score_exempt_when_no_api_reference(reviewer_module, tmp_path: Path):
"""Skills with no API key references should get full API key points."""
project_root = tmp_path
skill_dir = project_root / "skills" / "chart-skill"
skill_md = skill_dir / "SKILL.md"
write_text(
skill_md,
"\n".join(
[
"---",
"name: chart-skill",
"description: Chart analysis skill",
"---",
"",
"## When to Use",
"x",
"## Prerequisites",
"x",
"## Workflow",
"```bash",
"python3 scripts/analyze.py --output-dir reports/",
"```",
"## Output",
"x",
"## Resources",
"- `references/ref.md`",
"",
]
),
)
write_text(skill_dir / "scripts" / "analyze.py", "print('ok')\n")
write_text(skill_dir / "references" / "ref.md", "# ref\n")
review = reviewer_module.score_skill(project_root, skill_md, skip_tests=True)
messages = [f["message"] for f in review["findings"]]
assert not any("API key" in m for m in messages), f"Unexpected API key finding: {messages}"
def test_domain_specific_workflow_and_reference_headings_are_recognized(
reviewer_module, tmp_path: Path
):
project_root = tmp_path
skill_dir = project_root / "skills" / "knowledge-workflow"
skill_md = skill_dir / "SKILL.md"
write_text(
skill_md,
"\n".join(
[
"---",
"name: knowledge-workflow",
"description: guide",
"---",
"",
"## When to Use This Skill",
"x",
"## Prerequisites",
"x",
"## Backtesting Workflow",
"x",
"## Available Reference Documentation",
"- `references/ref.md`",
"",
]
),
)
write_text(skill_dir / "references" / "ref.md", "# ref\n")
review = reviewer_module.score_skill(project_root, skill_md, skip_tests=True)
assert review["score_breakdown"]["workflow_coverage"] >= 18
messages = [f["message"] for f in review["findings"]]
assert "Missing section: `## Workflow`." not in messages
assert "Missing section: `## Resources`." not in messages
def test_knowledge_only_skill_not_penalized_for_no_scripts_or_tests(
reviewer_module, tmp_path: Path
):
project_root = tmp_path
skill_dir = project_root / "skills" / "knowledge-only"
skill_md = skill_dir / "SKILL.md"
write_text(
skill_md,
"\n".join(
[
"---",
"name: knowledge-only",
"description: guide",
"---",
"",
"## When to Use",
"x",
"## Prerequisites",
"x",
"## Backtesting Workflow",
"x",
"## Output",
"Conversational guidance only",
"## Available Reference Documentation",
"- `references/ref.md`",
"",
]
),
)
write_text(skill_dir / "references" / "ref.md", "# ref\n")
review = reviewer_module.score_skill(project_root, skill_md, skip_tests=False)
assert review["skill_type"] == "knowledge_only"
assert review["score_breakdown"]["supporting_artifacts"] == 9
assert review["score_breakdown"]["test_health"] == 12
assert review["test_status"] == "not_applicable"
messages = [f["message"] for f in review["findings"]]
assert "No runnable bash examples found." not in messages
assert "No executable helper scripts found in `scripts/`." not in messages
assert "No `test_*.py` tests found for skill scripts." not in messages
def test_combine_reviews_boundary_at_90_no_improvement(reviewer_module):
auto = {
"score": 90,
"findings": [
{"severity": "low", "message": "x", "improvement": "y", "path": "a", "line": None}
],
}
merged = reviewer_module.combine_reviews(auto, None, auto_weight=1.0, llm_weight=0.0)
assert merged["score"] == 90
assert merged["improvements_required"] is False
assert merged["improvement_items"] == []
def test_main_e2e_generates_report_files(tmp_path: Path):
project_root = tmp_path
skill_dir = project_root / "skills" / "e2e-skill"
write_text(
skill_dir / "SKILL.md",
"\n".join(
[
"---",
"name: e2e-skill",
"description: test",
"---",
"",
"## When to Use",
"x",
"## Prerequisites",
"x",
"## Workflow",
"```bash",
"python3 scripts/run.py --output-dir reports/",
"```",
"## Output",
"x",
"## Resources",
"- `references/ref.md`",
"",
]
),
)
write_text(skill_dir / "scripts" / "run.py", "print('ok')\n")
write_text(skill_dir / "references" / "ref.md", "# ref\n")
script_path = Path(__file__).resolve().parents[1] / "run_dual_axis_review.py"
proc = subprocess.run(
[
sys.executable,
str(script_path),
"--project-root",
str(project_root),
"--skill",
"e2e-skill",
"--skip-tests",
],
capture_output=True,
text=True,
check=False,
)
assert proc.returncode == 0
report_files = list((project_root / "reports").glob("skill_review_e2e-skill_*.json"))
assert report_files
report = json.loads(report_files[0].read_text(encoding="utf-8"))
assert report["declared_verification"]["declaration_status"] == "missing_index"
def test_all_summary_surfaces_verification_without_changing_score_gate(tmp_path: Path):
project_root = tmp_path
skill_dir = project_root / "skills" / "all-skill"
write_text(
skill_dir / "SKILL.md",
"---\nname: all-skill\ndescription: test\n---\n"
"## When to Use\nx\n## Prerequisites\nx\n## Workflow\nx\n"
"## Output\nx\n## Resources\nx\n",
)
write_text(
project_root / "skills-index.yaml",
"skills:\n- id: all-skill\n verification:\n"
+ "".join(
f" {axis}: not_verified\n"
for axis in (
"instruction_contract",
"unit_tests",
"workflow_contract",
"end_to_end_replay",
"data_provenance",
"financial_logic_review",
"empirical_validation",
"security_review",
)
),
)
script_path = Path(__file__).resolve().parents[1] / "run_dual_axis_review.py"
proc = subprocess.run(
[
sys.executable,
str(script_path),
"--project-root",
str(project_root),
"--all",
"--skip-tests",
],
capture_output=True,
text=True,
check=False,
)
summary_path = next((project_root / "reports").glob("skill_review_all_*.json"))
row = json.loads(summary_path.read_text(encoding="utf-8"))["results"][0]
assert row["verification_status"] == "complete"
assert row["declaration_complete"] is True
assert row["verification_gaps"] == 8
assert row["not_verified_axes"] == [
"data_provenance",
"empirical_validation",
"end_to_end_replay",
"financial_logic_review",
"instruction_contract",
"security_review",
"unit_tests",
"workflow_contract",
]
assert row["not_applicable_axes"] == []
assert row["all_applicable_axes_passed"] is False
assert row["pass"] == (row["score"] >= 90)
assert "Verify" in proc.stdout
def test_api_key_detected_from_scripts(reviewer_module, tmp_path: Path):
"""API key reference in scripts (not SKILL.md) should be detected."""
project_root = tmp_path
skill_dir = project_root / "skills" / "api-in-scripts"
skill_md = skill_dir / "SKILL.md"
# SKILL.md has no API key references
write_text(
skill_md,
"\n".join(
[
"---",
"name: api-in-scripts",
"description: test",
"---",
"",
"## When to Use",
"x",
"## Prerequisites",
"x",
"## Workflow",
"```bash",
"python3 scripts/run.py --output-dir reports/",
"```",
"## Output",
"x",
"## Resources",
"- `references/ref.md`",
"",
]
),
)
# Script uses FMP_API_KEY
write_text(
skill_dir / "scripts" / "run.py",
'import os\napi_key = os.environ.get("FMP_API_KEY")\nprint(api_key)\n',
)
write_text(skill_dir / "references" / "ref.md", "# ref\n")
review = reviewer_module.score_skill(project_root, skill_md, skip_tests=True)
# Compare with a skill that truly doesn't use APIs
no_api_dir = project_root / "skills" / "no-api-skill2"
no_api_md = no_api_dir / "SKILL.md"
write_text(
no_api_md,
"\n".join(
[
"---",
"name: no-api-skill2",
"description: test",
"---",
"",
"## When to Use",
"x",
"## Prerequisites",
"x",
"## Workflow",
"```bash",
"python3 scripts/run.py --output-dir reports/",
"```",
"## Output",
"x",
"## Resources",
"- `references/ref.md`",
"",
]
),
)
write_text(no_api_dir / "scripts" / "run.py", "print('ok')\n")
write_text(no_api_dir / "references" / "ref.md", "# ref\n")
no_api_review = reviewer_module.score_skill(project_root, no_api_md, skip_tests=True)
# API-using skill should have lower exec score (0 for API handling)
# vs non-API skill (4 for API exempt)
api_exec = review["score_breakdown"]["execution_safety_reproducibility"]
no_api_exec = no_api_review["score_breakdown"]["execution_safety_reproducibility"]
assert api_exec < no_api_exec, (
f"API skill exec_score ({api_exec}) should be lower than non-API skill ({no_api_exec})"
)
def test_pii_hardcoded_path_detected(reviewer_module, tmp_path):
"""Test that hardcoded /Users/username/ paths produce a high finding."""
project_root = tmp_path
skill_dir = project_root / "skills" / "pii-skill"
skill_md = skill_dir / "SKILL.md"
write_text(
skill_md,
"\n".join(
[
"---",
"name: pii-skill",
"description: test pii detection",
"---",
"## When to Use",
"x",
"## Prerequisites",
"x",
"## Workflow",
"```bash",
"python3 scripts/run.py",
"```",
"## Output",
"x",
"## Resources",
"x",
"",
]
),
)
write_text(
skill_dir / "scripts" / "run.py",
'PATH = "/Users/johndoe/Projects/my-project/data"\nprint(PATH)\n',
)
write_text(skill_dir / "references" / "ref.md", "# ref\n")
review = reviewer_module.score_skill(project_root, skill_md, skip_tests=True)
pii_findings = [f for f in review["findings"] if "Hardcoded absolute user path" in f["message"]]
assert len(pii_findings) == 1
assert pii_findings[0]["severity"] == "high"
assert "scripts/run.py" in pii_findings[0]["path"]
def test_pii_clean_skill_no_finding(reviewer_module, tmp_path):
"""Test that a clean skill produces no PII finding."""
project_root = tmp_path
skill_dir = project_root / "skills" / "clean-skill"
skill_md = skill_dir / "SKILL.md"
write_text(
skill_md,
"\n".join(
[
"---",
"name: clean-skill",
"description: no pii here",
"---",
"## When to Use",
"x",
"## Prerequisites",
"x",
"## Workflow",
"```bash",
"python3 scripts/run.py --output-dir reports/",
"```",
"## Output",
"x",
"## Resources",
"x",
"",
]
),
)
write_text(skill_dir / "scripts" / "run.py", "print('clean')\n")
write_text(skill_dir / "references" / "ref.md", "# ref\n")
review = reviewer_module.score_skill(project_root, skill_md, skip_tests=True)
pii_findings = [f for f in review["findings"] if "Hardcoded absolute user path" in f["message"]]
assert len(pii_findings) == 0
SKILL.md
---
name: dual-axis-skill-reviewer
description: "Review skills in any project using a dual-axis method: (1) deterministic code-based checks (structure, scripts, tests, execution safety) and (2) LLM deep review findings. Use when you need reproducible quality scoring for `skills/*/SKILL.md`, want to gate merges with a score threshold (for example 90+), or need concrete improvement items for low-scoring skills. Works across projects via --project-root."
---
# Dual Axis Skill Reviewer
Run the dual-axis reviewer script and save reports to `reports/`.
The script supports:
- Random or fixed skill selection
- Auto-axis scoring with optional test execution
- LLM prompt generation
- LLM JSON review merge with weighted final score
- Cross-project review via `--project-root`
- Non-scoring display of `skills-index.yaml` production verification declarations
## When to Use
- Need reproducible scoring for one skill in `skills/*/SKILL.md`.
- Need improvement items when final score is below 90.
- Need both deterministic checks and qualitative LLM code/content review.
- Need to review skills in a **different project** from the command line.
## Prerequisites
- Python 3.9+
- `uv` (recommended — auto-resolves `pyyaml` dependency via inline metadata)
- For tests: `uv sync --extra dev` or equivalent in the target project
- For LLM-axis merge: JSON file that follows the LLM review schema (see Resources)
## Workflow
Determine the correct script path based on your context:
- **Same project**: `skills/dual-axis-skill-reviewer/scripts/run_dual_axis_review.py`
- **Global install**: `~/.claude/skills/dual-axis-skill-reviewer/scripts/run_dual_axis_review.py`
The examples below use `REVIEWER` as a placeholder. Set it once:
```bash
# If reviewing from the same project:
REVIEWER=skills/dual-axis-skill-reviewer/scripts/run_dual_axis_review.py
# If reviewing another project (global install):
REVIEWER=~/.claude/skills/dual-axis-skill-reviewer/scripts/run_dual_axis_review.py
```
### Step 1: Run Auto Axis + Generate LLM Prompt
```bash
uv run "$REVIEWER" \
--project-root . \
--emit-llm-prompt \
--output-dir reports/
```
When reviewing a different project, point `--project-root` to it:
```bash
uv run "$REVIEWER" \
--project-root /path/to/other/project \
--emit-llm-prompt \
--output-dir reports/
```
### Step 2: Run LLM Review
- Use the generated prompt file in `reports/skill_review_prompt_<skill>_<timestamp>.md`.
- Ask the LLM to return strict JSON output.
- When running inside Claude Code, let Claude act as orchestrator: read the generated prompt, produce the LLM review JSON, and save it for the merge step.
### Step 3: Merge Auto + LLM Axes
```bash
uv run "$REVIEWER" \
--project-root . \
--skill <skill-name> \
--llm-review-json <path-to-llm-review.json> \
--auto-weight 0.5 \
--llm-weight 0.5 \
--output-dir reports/
```
### Step 4: Optional Controls
- Fix selection for reproducibility: `--skill <name>` or `--seed <int>`
- Review all skills at once: `--all`
- Skip tests for quick triage: `--skip-tests`
- Change report location: `--output-dir <dir>`
- Increase `--auto-weight` for stricter deterministic gating.
- Increase `--llm-weight` when qualitative/code-review depth is prioritized.
## Output
- `reports/skill_review_<skill>_<timestamp>.json`
- `reports/skill_review_<skill>_<timestamp>.md`
- `reports/skill_review_prompt_<skill>_<timestamp>.md` (when `--emit-llm-prompt` is enabled)
When the target project has a `skills-index.yaml` entry with a complete `verification` block, JSON
and Markdown reports also show its declared axes, `not_verified` gaps, non-applicable axes, and
`all_applicable_axes_passed`. This section is informational only. It is excluded from auto/LLM/final
scores and does not replace a live high-severity issue check.
## Installation (Global)
To use this skill from any project, symlink it into `~/.claude/skills/`:
```bash
ln -sfn /path/to/claude-trading-skills/skills/dual-axis-skill-reviewer \
~/.claude/skills/dual-axis-skill-reviewer
```
After this, Claude Code will discover the skill in all projects, and the script is accessible at `~/.claude/skills/dual-axis-skill-reviewer/scripts/run_dual_axis_review.py`.
## Resources
- Auto axis scores metadata, workflow coverage, execution safety, artifact presence, and test health.
- Auto axis detects `knowledge_only` skills and adjusts script/test expectations to avoid unfair penalties.
- LLM axis scores deep content quality (correctness, risk, missing logic, maintainability).
- Final score is weighted average.
- If final score is below 90, improvement items are required and listed in the markdown report.
- Script: `skills/dual-axis-skill-reviewer/scripts/run_dual_axis_review.py`
- LLM schema: `references/llm_review_schema.md`
- Rubric detail: `references/scoring_rubric.md`