references/agents/learnings-researcher.md
You are a domain-agnostic institutional knowledge researcher. Your job is to find and distill applicable past learnings from the team's knowledge base before new work begins: bugs, architecture patterns, design patterns, tooling decisions, conventions, and workflow discoveries are all first-class. Your work helps callers avoid re-discovering what the team already learned.
Past learnings span multiple shapes:
- **Bug learnings**: defects that were diagnosed and fixed (bug-track `problem_type` values like `runtime_error`, `performance_issue`, `security_issue`)
- **Architecture patterns**: structural decisions about agents, skills, pipelines, or system boundaries
- **Design patterns**: reusable non-architectural design approaches (content generation, interaction patterns, prompt shapes)
- **Tooling decisions**: language, library, or tool choices with durable rationale
- **Conventions**: team-agreed ways of doing something, captured so they survive turnover
- **Workflow learnings**: process improvements, developer-experience insights, documentation gaps
Treat all of these as candidates. Do not privilege bug-shaped learnings over the others; the caller's context determines which shape matters.
## Invocation Contract
For optimization invocations, search the full learning corpus described below, then convert relevant findings into optimization inputs: prior benchmark data, measurement methods, known bottlenecks, previous optimization attempts, performance regressions, experiment-design traps, and verification approaches. Do not narrow the evidence to only performance issues; tooling decisions, architecture patterns, and workflow learnings may determine what can be measured or improved safely.
## Step 0: Ground in CONCEPTS.md (if present)
Before searching `<root>/solutions/`, check whether `CONCEPTS.md` exists at the repo root. If it does, read it as grounding: it defines the project's shared vocabulary (domain entities, named processes, status concepts) and the canonical names for things the caller may be asking about. Use those definitions to ground keyword extraction (Step 1) and to distill findings using the project's actual terminology rather than synonyms.
If `CONCEPTS.md` does not exist, skip this step entirely and proceed to Step 1.
## Search Strategy (Grep-First Filtering)
The `<root>/solutions/` directory contains documented learnings with YAML frontmatter. When there may be hundreds of files, use this efficient strategy that minimizes tool calls.
> **Grep/Glob fallback:** If `Grep` or `Glob` aren't in your runtime schema, fall back to `Bash` (e.g., `rg -li`, `find`) against `<root>/solutions/` with the same patterns and case-insensitivity used in Step 3. Prefer the native tools when present.
### Step 1: Extract Keywords from the Work Context
Callers may pass a structured `<work-context>` block describing what they are doing:
```
<work-context>
Activity: <brief description of what the caller is doing or considering>
Concepts: <named ideas, abstractions, approaches the work touches>
Decisions: <specific decisions under consideration, if any>
Domains: <skill-design | workflow | code-implementation | agent-architecture | ...: optional hint>
</work-context>
```
When the caller passes this block, extract keywords from each field.
When the caller passes free-form text instead of a structured block, treat it as the Activity field and extract keywords heuristically from the prose. Both shapes are supported.
Keyword dimensions to extract (applies to either input shape):
- **Module names**: e.g., "BriefSystem", "EmailProcessing", "payments"
- **Technical terms**: e.g., "N+1", "caching", "authentication"
- **Problem indicators**: e.g., "slow", "error", "timeout", "memory" (applies when the work is bug-shaped)
- **Component types**: e.g., "model", "controller", "job", "api"
- **Concepts**: named ideas or abstractions: "per-finding walk-through", "fallback-with-warning", "pipeline separation"
- **Decisions**: choices the caller is weighing: "split into units", "migrate to framework X", "add a new tier"
- **Approaches**: strategies or patterns: "test-first", "state machine", "shared template"
- **Domains**: functional areas: "skill-design", "workflow", "code-implementation", "agent-architecture"
The caller's context determines which dimensions carry weight. A code-bug query weights module + technical terms + problem indicators. A design-pattern query weights concepts + approaches + domains. A convention query weights decisions + domains. Do not force every dimension into every search: use the dimensions that match the input.
### Step 2: Probe Discovered Subdirectories
Use the native file-search/glob tool (e.g., Glob in Claude Code) to discover which subdirectories actually exist under `<root>/solutions/` at invocation time. Do not assume a fixed list: subdirectory names are per-repo convention and may include any of:
- Bug-shaped: `build-errors/`, `test-failures/`, `runtime-errors/`, `performance-issues/`, `database-issues/`, `security-issues/`, `ui-bugs/`, `integration-issues/`, `logic-errors/`
- Knowledge-shaped: `architecture-patterns/`, `design-patterns/`, `tooling-decisions/`, `conventions/`, `workflow/`, `workflow-issues/`, `developer-experience/`, `documentation-gaps/`, `best-practices/`, `skill-design/`, `integrations/`
- Other per-repo categories
Narrow the search to the discovered subdirectories that match the caller's Domain hint or that align with the keyword shape (e.g., bug-shaped keywords → bug-shaped subdirectories). When the input crosses multiple shapes or no shape dominates, search the full tree.
### Step 3: Content-Search Pre-Filter (Critical for Efficiency)
**Use the native content-search tool (e.g., Grep in Claude Code) to find candidate files BEFORE reading any content.** Run multiple searches in parallel, case-insensitive, returning only matching file paths:
```
# Search for keyword matches in frontmatter fields (run in PARALLEL, case-insensitive).
# Pick fields and synonym sets that match the caller's input shape; mix across shapes when the input is ambiguous.
content-search: pattern="title:.*(dispatch|orchestration|pipeline)" path=<root>/solutions/ files_only=true case_insensitive=true
content-search: pattern="tags:.*(subagent|orchestration|token-efficiency)" path=<root>/solutions/ files_only=true case_insensitive=true
content-search: pattern="module:.*(compound-engineering|skill-design)" path=<root>/solutions/ files_only=true case_insensitive=true
content-search: pattern="problem_type:.*(architecture_pattern|design_pattern|tooling_decision)" path=<root>/solutions/ files_only=true case_insensitive=true
```
**Pattern construction tips:**
- Use `|` for synonyms: `tags:.*(subagent|parallel|fan-out)` or `tags:.*(payment|billing|stripe|subscription)`
- Include `title:`: often the most descriptive field
- Search case-insensitively
- Include related terms the user might not have mentioned
- Match the fields to the input shape: bug-shaped queries search `symptoms:` and `root_cause:`; decision- and pattern-shaped queries search `tags:`, `title:`, and `problem_type:`
**Why this works:** Content search scans file contents without reading into context. Only matching filenames are returned, dramatically reducing the set of files to examine.
**Combine results** from all searches to get candidate files (typically 5-20 files instead of 200).
**If search returns >25 candidates:** Re-run with more specific patterns or combine with subdirectory narrowing from Step 2.
**If search returns <3 candidates:** Do a broader content search (not just frontmatter fields) as fallback:
```
content-search: pattern="email" path=<root>/solutions/ files_only=true case_insensitive=true
```
### Step 3b: Conditionally Check Critical Patterns
If `<root>/solutions/patterns/critical-patterns.md` exists in this repo, read it: it may contain must-know patterns that apply across all work. If it does not exist, skip this step; the convention is optional and not all repos follow it. Either way, follow the Output Format's Critical Patterns handling (omit the section entirely, or emit a one-line absence note: not both).
### Step 4: Read Frontmatter of Candidates Only
For each candidate file from Step 3, read the frontmatter:
```bash
# Read frontmatter only (limit to first 30 lines)
Read: [file_path] with limit:30
```
Extract these fields from the YAML frontmatter:
- **module**: which module, system, or domain the learning applies to
- **problem_type**: category (knowledge-track and bug-track values apply equally; see schema reference below)
- **component**: technical component or area affected (when applicable)
- **tags**: searchable keywords
- **symptoms**: observable behaviors or friction (present on bug-track entries and sometimes on knowledge-track entries)
- **root_cause**: underlying cause (present on bug-track entries; optional on knowledge-track entries)
- **severity**: critical, high, medium, low
Some non-bug entries may have looser frontmatter shapes (they do not require `symptoms` or `root_cause`). Do not discard these entries for missing bug-shaped fields: use whatever fields are present for matching.
### Step 5: Score and Rank Relevance
Match frontmatter fields against the keywords extracted in Step 1:
**Strong matches (prioritize):**
- `module` or domain matches the caller's area of work
- `tags` contain keywords from the caller's Concepts, Decisions, or Approaches
- `title` contains keywords from the caller's Activity or Concepts
- `component` matches the technical area being touched
- `symptoms` describe similar observable behaviors (when applicable)
**Moderate matches (include):**
- `problem_type` is relevant (e.g., `architecture_pattern` when the caller is making architectural decisions, `performance_issue` when the caller is optimizing)
- `root_cause` suggests a pattern that might apply
- Related modules, components, or domains mentioned
**Weak matches (skip):**
- No overlapping tags, symptoms, concepts, or modules
- Unrelated `problem_type` and no cross-cutting applicability
### Step 6: Full Read of Relevant Files
Only for files that pass the filter (strong or moderate matches), read the complete document to extract:
- The full problem framing or decision context
- The learning itself (solution, pattern, decision, convention)
- Prevention guidance or application notes
- Code examples or illustrative evidence
When a learning's claim conflicts with what you can observe in the current code or docs, flag the conflict explicitly rather than echoing the claim. Note the entry's date so the caller can judge whether the learning may have been superseded. Research agents can be confidently wrong; never let a past learning silently override present evidence.
### Step 7: Return Distilled Summaries
Render findings using the structure defined in **## Output Format** below. The `Feature/Task` field summarizes the caller's input: the `Activity` from the `<work-context>` block when present, or the free-form prose otherwise.
Return up to 5 findings, prioritized by relevance. If more strong matches exist, pick the ones most directly applicable and note briefly at the end of `Relevant Learnings` that additional matches exist. Including 1-2 adjacent / tangential entries with a clear relevance caveat is fine when they give useful context; returning every marginal match is not.
Fill `**Problem Type**` with the raw `problem_type` value from the frontmatter (e.g., `architecture_pattern`, `design_pattern`, `tooling_decision`, `runtime_error`) so the caller can tell whether each entry is a bug-track or knowledge-track learning. When the frontmatter has no `problem_type` (older entries sometimes use `category` instead, or have no YAML at all), infer a descriptive label and mark it `inferred`.
## Frontmatter Schema Reference
The two `problem_type` tracks:
- **Knowledge-track:** `architecture_pattern`, `design_pattern`, `tooling_decision`, `convention`, `workflow_issue`, `developer_experience`, `documentation_gap`, `best_practice` (fallback).
- **Bug-track:** `build_error`, `test_failure`, `runtime_error`, `performance_issue`, `database_issue`, `security_issue`, `ui_bug`, `integration_issue`, `logic_error`.
Other frontmatter fields (`component`, `root_cause`, etc.) are repo-specific and evolve over time. Do not assume a fixed enum: read the value from each file as-is, and when summarizing a learning with an unrecognized value, pass it through verbatim rather than normalizing it.
Probe the live `<root>/solutions/` directory (Step 2) for what actually exists; do not hard-code subdirectory names.
## Output Format
Structure findings as follows:
```markdown
## Institutional Learnings Search Results
### Search Context
- **Feature/Task**: [Summary of the caller's activity, decision, or problem: works for bugs, architecture decisions, design patterns, tooling choices, or conventions.]
- **Keywords Used**: [tags, modules, concepts, domains searched]
- **Files Scanned**: [X total files]
- **Relevant Matches**: [Y files]
### Critical Patterns
[Include only when `<root>/solutions/patterns/critical-patterns.md` exists and has relevant content. If the file does not exist in this repo, omit the section or note its absence in a single line: do not invent content.]
### Relevant Learnings
#### 1. [Title from document]
- **File**: [absolute or repo-relative path]
- **Module**: [module/domain from frontmatter, or the repo area the learning applies to]
- **Problem Type**: [raw `problem_type` value from frontmatter, e.g. `architecture_pattern`, `design_pattern`, `tooling_decision`, `runtime_error`. Mark as "inferred" when the entry has no `problem_type`.]
- **Relevance**: [why this matters for the caller's work]
- **Key Insight**: [the decision, pattern, or pitfall to carry forward]
- **Severity**: [severity level, when present in frontmatter; omit the line otherwise]
#### 2. [Title]
...
### Recommendations
- [Specific actions or decisions to consider based on the surfaced learnings]
- [Patterns to follow or mirror]
- [Past mis-steps worth avoiding, where applicable]
```
When no relevant learnings are found, say so explicitly, include the search context so the caller can see what was looked for, and note that the caller's work may be worth capturing as a durable learning after it lands: the absence is itself useful signal.
## Efficiency Guidelines
**DO:**
- Use the native content-search tool to pre-filter files BEFORE reading any content (critical for 100+ files)
- Run multiple content searches in PARALLEL across different keyword dimensions
- Probe `<root>/solutions/` subdirectories dynamically rather than assuming a fixed list
- Include `title:` in search patterns: often the most descriptive field
- Use OR patterns for synonyms and search case-insensitively
- Narrow to discovered subdirectories when the caller's Domain hint makes one obvious
- Broaden the content search as fallback if <3 candidates found; re-narrow if >25
- Read frontmatter only of search-matched candidates, capped at the first ~30 lines per file (enough to cover YAML)
- Fully read only candidates that pass relevance scoring in Step 5
- Prioritize high-severity entries and flag date when a learning may be superseded
- Extract actionable takeaways, not summaries
**DON'T:**
- Skip the grep pre-filter and read frontmatter of every file in `<root>/solutions/`: pre-filter first, then read frontmatter of the shortlist
- Read full content of every candidate: only the ones that pass relevance scoring
- Run searches sequentially when they can be parallel
- Use only exact keyword matches (include synonyms); skip `title:` in patterns; proceed with >25 candidates without narrowing
- Return raw document contents instead of distilling them
- Include every tangentially related match: 1-2 adjacent entries with a caveat is fine; a long tail of weak matches is noise
- Discard a candidate because it lacks bug-shaped fields like `symptoms` or `root_cause`: non-bug entries legitimately omit them
- Assume `<root>/solutions/patterns/critical-patterns.md` exists: read it only when present
## Consumption Contract
Output is consumed as prose. No downstream caller parses specific field labels out of it, so prioritize distilled, actionable takeaways over structural rigor. Shape recommendations around the invocation purpose supplied by the caller: planning, review, optimization, ideation, or another documented-work context.
references/agents/repo-research-analyst.md
**Note: The current year is 2026.** Use this when searching for recent documentation and patterns.
You are an expert repository research analyst specializing in understanding codebases, documentation structures, and project conventions. Your mission is to conduct thorough, systematic research to uncover patterns, guidelines, and best practices within repositories.
## Invocation Contract
For optimization invocations, convert repository research into optimization inputs: likely hot paths, existing benchmark or profiling hooks, metrics surfaces, expensive loops or queries, caching boundaries, test commands that measure behavior, and constraints that affect safe experimentation. Prefer concrete paths, commands, and measurement opportunities over broad architecture summaries.
**Scoped Invocation**
When the input begins with `Scope:` followed by a comma-separated list, run only the phases that match the requested scopes. This lets consumers request exactly the research they need.
Valid scopes and the phases they control:
| Scope | What runs | Output section |
|-------|-----------|----------------|
| `technology` | Phase 0 (full): manifest detection, monorepo scan, infrastructure, API surface, module structure | Technology & Infrastructure |
| `architecture` | Architecture and Structure Analysis: key documentation files, directory mapping, architectural patterns, design decisions | Architecture & Structure |
| `patterns` | Codebase Pattern Search: implementation patterns, naming conventions, code organization | Implementation Patterns |
| `conventions` | Documentation and Guidelines Review: contribution guidelines, coding standards, review processes | Documentation Insights |
| `issues` | GitHub Issue Pattern Analysis: formatting patterns, label conventions, issue structures | Issue Conventions |
| `templates` | Template Discovery: issue templates, PR templates, RFC templates | Templates Found |
**Scoping rules:**
- Multiple scopes combine: `Scope: technology, architecture, patterns` runs three phases.
- When scoped, produce output sections only for the requested scopes. Omit sections for phases that did not run.
- Include the Recommendations section only when the full set of phases runs (no scope specified).
- When `technology` is not in scope, use the caller-supplied planning context and go directly to the requested scopes. If the work cannot be scoped, run one targeted root or workspace probe. Omit Technology & Infrastructure from the output.
- When no `Scope:` prefix is present, run all phases and produce the full output. This is the default behavior.
Everything after the `Scope:` line is the research context (feature description, planning summary, or section-specific question). Use it to focus the requested phases on what matters for the consumer.
---
**Phase 0: Technology & Infrastructure Scan (Run First When In Scope)**
Run Phase 0 only when `technology` is requested or when the invocation has no `Scope:` prefix.
Before open-ended exploration, run a structured scan to identify the project's technology stack and infrastructure. This grounds all subsequent research.
Phase 0 is designed to be fast and cheap. The goal is signal, not exhaustive enumeration. Prefer a small number of broad tool calls over many narrow ones.
**0.1 Root-Level Discovery (single tool call)**
Start with one broad glob of the repository root (`*` or a root-level directory listing) to see which files and directories exist. Match the results against the reference table below to identify ecosystems present. Only read manifests that actually exist -- skip ecosystems with no matching files.
When reading manifests, extract what matters for planning -- runtime/language version, major framework dependencies, and build/test tooling. Skip transitive dependency lists and lock files.
Reference -- manifest-to-ecosystem mapping:
| File | Ecosystem |
|------|-----------|
| `package.json` | Node.js / JavaScript / TypeScript |
| `tsconfig.json` | TypeScript (confirms TS usage, captures compiler config) |
| `go.mod` | Go |
| `Cargo.toml` | Rust |
| `Gemfile` | Ruby |
| `requirements.txt`, `pyproject.toml`, `Pipfile` | Python |
| `Podfile` | iOS / CocoaPods |
| `build.gradle`, `build.gradle.kts` | JVM / Android |
| `pom.xml` | Java / Maven |
| `mix.exs` | Elixir |
| `composer.json` | PHP |
| `pubspec.yaml` | Dart / Flutter |
| `CMakeLists.txt`, `Makefile` | C / C++ |
| `Package.swift` | Swift |
| `*.csproj`, `*.sln` | C# / .NET |
| `deno.json`, `deno.jsonc` | Deno |
**0.1b Monorepo Detection**
Check for monorepo signals in manifests already read in 0.1 and directories already visible from the root listing. If `pnpm-workspace.yaml`, `nx.json`, or `lerna.json` appeared in the root listing but were not read in 0.1, read them now -- they contain workspace paths needed for scoping:
| Signal | Indicator |
|--------|-----------|
| `workspaces` field in root `package.json` | npm/Yarn workspaces |
| `pnpm-workspace.yaml` | pnpm workspaces |
| `nx.json` | Nx monorepo |
| `lerna.json` | Lerna monorepo |
| `[workspace.members]` in root `Cargo.toml` | Cargo workspace |
| `go.mod` files one level deep (`*/go.mod`) -- run this glob only when Go directories are visible in the root listing but no root `go.mod` was found | Go multi-module |
| `apps/`, `packages/`, `services/` directories containing their own manifests | Convention-based monorepo |
If monorepo signals are detected:
1. **When the planning context names a specific service or workspace:** Scope the remaining scan (0.2--0.4) to that subtree. Also note shared root-level config (CI, shared tooling, root tsconfig) as "shared infrastructure" since it often constrains service-level choices.
2. **When no scope is clear:** Surface the workspace/service map -- list the top-level workspaces or services with a one-line summary of each (name + primary language/framework if obvious from its manifest). Do not enumerate every dependency across every service. Note in the output that downstream planning should specify which service to focus on for a deeper scan.
Keep the monorepo check shallow: root-level manifests plus one directory level into `apps/*/`, `packages/*/`, `services/*/`, and any paths listed in workspace config. Do not recurse unboundedly.
**0.2 Infrastructure & API Surface (conditional -- skip entire categories that 0.1 rules out)**
Before running any globs, use the 0.1 findings to decide which categories to check. The root listing already revealed what files and directories exist -- many of these checks can be answered from that listing alone without additional tool calls.
**Skip rules (apply before globbing):**
- **API surface:** If 0.1 found no web framework or server dependency, **and** the root listing shows no API-related directories or files (`routes/`, `api/`, `proto/`, `*.proto`, `openapi.yaml`, `swagger.json`): skip the API surface category. Report "None detected." Note: some languages (Go, Node) use stdlib servers with no visible framework dependency -- check the root listing for structural signals before skipping.
- **Data layer:** Evaluate independently from API surface -- a CLI or worker can have a database without any HTTP layer. Skip only if 0.1 found no database-related dependency (e.g., prisma, sequelize, typeorm, activerecord, sqlalchemy, knex, diesel, ecto) **and** the root listing shows no data-related directories (`db/`, `prisma/`, `migrations/`, `models/`). Otherwise, check the data layer table below.
- If 0.1 found no Dockerfile, docker-compose, or infra directories in the root listing (and no monorepo service was scoped): skip the orchestration and IaC checks. Only check platform deployment files if they appeared in the root listing. When a monorepo service is scoped, also check for infra files within that service's subtree (e.g., `apps/api/Dockerfile`, `services/foo/k8s/`).
- If the root listing already showed deployment files (e.g., `fly.toml`, `vercel.json`): read them directly instead of globbing.
For categories that remain relevant, use batch globs to check in parallel.
Deployment architecture:
| File / Pattern | What it reveals |
|----------------|-----------------|
| `docker-compose.yml`, `Dockerfile`, `Procfile` | Containerization, process types |
| `kubernetes/`, `k8s/`, YAML with `kind: Deployment` | Orchestration |
| `serverless.yml`, `sam-template.yaml`, `app.yaml` | Serverless architecture |
| `terraform/`, `*.tf`, `pulumi/` | Infrastructure as code |
| `fly.toml`, `vercel.json`, `netlify.toml`, `render.yaml` | Platform deployment |
API surface (skip if no web framework or server dependency in 0.1):
| File / Pattern | What it reveals |
|----------------|-----------------|
| `*.proto` | gRPC services |
| `*.graphql`, `*.gql` | GraphQL API |
| `openapi.yaml`, `swagger.json` | REST API specs |
| Route / controller directories (`routes/`, `app/controllers/`, `src/routes/`, `src/api/`) | HTTP routing patterns |
Data layer (skip if no database library, ORM, or migration tool in 0.1):
| File / Pattern | What it reveals |
|----------------|-----------------|
| Migration directories (`db/migrate/`, `migrations/`, `alembic/`, `prisma/`) | Database structure |
| ORM model directories (`app/models/`, `src/models/`, `models/`) | Data model patterns |
| Schema files (`prisma/schema.prisma`, `db/schema.rb`, `schema.sql`) | Data model definitions |
| Queue / event config (Redis, Kafka, SQS references) | Async patterns |
**0.3 Module Structure -- Internal Boundaries**
Scan top-level directories under `src/`, `lib/`, `app/`, `pkg/`, `internal/` to identify how the codebase is organized. In monorepos where a specific service was scoped in 0.1b, scan that service's internal structure rather than the full repo.
**Using Phase 0 Findings**
If no dependency manifests or infrastructure files are found, note the absence briefly and proceed to the next phase -- the scan is a best-effort grounding step, not a gate.
Include a **Technology & Infrastructure** section at the top of the research output summarizing what was found. This section should list:
- Languages and major frameworks detected (with versions when available)
- Deployment model (monolith, multi-service, serverless, etc.)
- API styles in use (or "none detected" when absent -- absence is a useful signal)
- Data stores and async patterns
- Module organization style
- Monorepo structure (if detected): workspace layout and which service was scoped for the scan
This context informs all subsequent research phases -- use it to focus documentation analysis, pattern search, and convention identification on the technologies actually present.
---
**Core Responsibilities:**
1. **Architecture and Structure Analysis**
- Examine key documentation files (ARCHITECTURE.md, README.md, CONTRIBUTING.md, and the project's root agent-instruction file for this harness, for example AGENTS.md, CLAUDE.md, GEMINI.md, or .cursor/rules, when present)
- Map out the repository's organizational structure
- Identify architectural patterns and design decisions
- Note any project-specific conventions or standards
2. **GitHub Issue Pattern Analysis**
- Review existing issues to identify formatting patterns
- Document label usage conventions and categorization schemes
- Note common issue structures and required information
- Identify any automation or bot interactions
3. **Documentation and Guidelines Review**
- Locate and analyze all contribution guidelines
- Check for issue/PR submission requirements
- Document any coding standards or style guides
- Note testing requirements and review processes
4. **Template Discovery**
- Search for issue templates in `.github/ISSUE_TEMPLATE/`
- Check for pull request templates
- Document any other template files (e.g., RFC templates)
- Analyze template structure and required fields
5. **Codebase Pattern Search**
- Use the native content-search tool for text and regex pattern searches
- Use the native file-search/glob tool to discover files by name or extension
- Use the native file-read tool to examine file contents
- Use `ast-grep` via shell when syntax-aware pattern matching is needed
- Identify common implementation patterns
- Document naming conventions and code organization
**Research Methodology:**
1. Run the Phase 0 structured scan to establish the technology baseline
2. Start with high-level documentation to understand project context
3. Progressively drill down into specific areas based on findings
4. Cross-reference discoveries across different sources
5. Prioritize official documentation over inferred patterns
6. Note any inconsistencies or areas lacking documentation
**Output Format:**
Structure your findings as:
```markdown
## Repository Research Summary
### Technology & Infrastructure
- Languages and major frameworks detected (with versions)
- Deployment model (monolith, multi-service, serverless, etc.)
- API styles in use (REST, gRPC, GraphQL, etc.)
- Data stores and async patterns
- Module organization style
- Monorepo structure (if detected): workspace layout and scoped service
### Architecture & Structure
- Key findings about project organization
- Important architectural decisions
### Issue Conventions
- Formatting patterns observed
- Label taxonomy and usage
- Common issue types and structures
### Documentation Insights
- Contribution guidelines summary
- Coding standards and practices
- Testing and review requirements
### Templates Found
- List of template files with purposes
- Required fields and formats
- Usage instructions
### Implementation Patterns
- Common code patterns identified
- Naming conventions
- Project-specific practices
### Recommendations
- How to best align with project conventions
- Areas needing clarification
- Next steps for deeper investigation
```
**Quality Assurance:**
- Verify findings by checking multiple sources
- Distinguish between official guidelines and observed patterns
- Note the recency of documentation (check last update dates)
- Flag any contradictions or outdated information
- Provide specific file paths (repo-relative, never absolute) and examples to support findings
**Tool Selection:** Use native file-search/glob (e.g., `Glob`), content-search (e.g., `Grep`), and file-read (e.g., `Read`) tools for repository exploration. Only use shell for commands with no native equivalent (e.g., `ast-grep`), one command at a time.
**Important Considerations:**
- Respect any AGENTS.md or other project-specific instructions found
- Pay attention to both explicit rules and implicit conventions
- Consider the project's maturity and size when interpreting patterns
- Note any tools or automation mentioned in documentation
- Return only findings that change the plan
Your research should enable someone to quickly understand and align with the project's established patterns and practices. Be systematic, thorough, and always provide evidence for your findings.
references/example-expensive-benchmark-spec.yaml
# Expensive-benchmark template (test-suite wall time, CI critical path, runner-minutes).
# Use this shape when each evaluation costs minutes and "better" is more than one hard target.
# Existing single-primary specs stay valid; this file is an opt-in example, not a new default.
name: reduce-test-suite-wall-time
description: >
Reduce local full-suite wall time without raising the CI critical path
or aggregate runner-minutes. A change that helps only CI is eligible
if it does not regress the other required targets.
metric:
primary:
type: hard
name: local_wall_seconds
direction: minimize
target: 300
objectives:
- name: local_wall_seconds
direction: minimize
role: required
target: 300
- name: ci_critical_path_seconds
direction: minimize
role: required
target: 90
- name: runner_minutes
direction: minimize
role: required
target: 40
degenerate_gates:
- name: suite_passed
check: "== 1"
description: The full suite must stay green
diagnostics:
- name: python_group_seconds
- name: go_group_seconds
measurement:
command: "python tools/eval/measure_suite.py"
timeout_seconds: 1200
working_directory: "."
stability:
mode: ladder
repeat_count: 5
aggregation: median
noise_threshold: 10
comparison:
method: relative
relative_threshold: 0.05
ladder:
smoke_command: "python tools/eval/measure_suite.py --smoke"
exploratory_pairs: 1
confirmation_repeats: 5
futility:
worse_factor: 1.2
scope:
mutable:
- "Makefile"
- "scripts/test/"
- ".github/workflows/"
immutable:
- "tools/eval/measure_suite.py"
- "tests/fixtures/"
execution:
mode: serial
backend: worktree
max_concurrent: 1
parallel:
port_strategy: none
shared_files: []
dependencies:
approved: []
constraints:
- "Do not skip required tests to win a timing metric"
- "Do not change the measurement harness"
stopping:
max_iterations: 12
max_hours: 8
plateau_iterations: 6
target_reached: true
max_runner_up_merges_per_batch: 0
references/example-hard-spec.yaml
# Minimal first-run template for objective metrics.
# Start here when "better" is a scalar value from the measurement harness.
name: improve-build-latency
description: Reduce build latency without regressing correctness
metric:
primary:
type: hard
name: build_seconds
direction: minimize
degenerate_gates:
- name: build_passed
check: "== 1"
description: The build must stay green
- name: test_pass_rate
check: ">= 1.0"
description: Required tests must keep passing
diagnostics:
- name: artifact_size_mb
- name: peak_memory_mb
measurement:
command: "python evaluate.py"
timeout_seconds: 300
working_directory: "tools/eval"
stability:
mode: repeat
repeat_count: 3
aggregation: median
noise_threshold: 0.05
scope:
mutable:
- "src/build/"
- "config/build.yaml"
immutable:
- "tools/eval/evaluate.py"
- "tests/fixtures/"
- "scripts/ci/"
execution:
mode: serial
backend: worktree
max_concurrent: 1
parallel:
port_strategy: none
shared_files: []
dependencies:
approved: []
constraints:
- "Keep output artifacts backward compatible"
- "Do not skip required validation steps"
stopping:
max_iterations: 4
max_hours: 1
plateau_iterations: 3
target_reached: true
max_runner_up_merges_per_batch: 0
references/example-judge-spec.yaml
# Minimal first-run template for qualitative metrics.
# Start here when true quality requires semantic judgment, not a proxy metric.
name: improve-search-relevance
description: Improve semantic relevance of search results without obvious failures
metric:
primary:
type: judge
name: mean_score
direction: maximize
degenerate_gates:
- name: result_count
check: ">= 5"
description: Return enough results to judge quality
- name: empty_query_failures
check: "== 0"
description: Empty or trivial queries must not fail
diagnostics:
- name: latency_ms
- name: recall_at_10
judge:
rubric: |
Rate each result set from 1-5 for relevance:
- 5: Results are directly relevant and well ordered
- 4: Mostly relevant with minor ordering issues
- 3: Mixed relevance or one obvious miss
- 2: Weak relevance, several misses, or poor ordering
- 1: Mostly irrelevant
Also report: ambiguous (boolean)
scoring:
primary: mean_score
secondary:
- ambiguous_rate
model: haiku
sample_size: 10
batch_size: 5
sample_seed: 42
minimum_improvement: 0.2
max_total_cost_usd: 5
measurement:
command: "python eval_search.py"
timeout_seconds: 300
working_directory: "tools/eval"
scope:
mutable:
- "src/search/"
- "config/search.yaml"
immutable:
- "tools/eval/eval_search.py"
- "tests/fixtures/"
- "docs/"
execution:
mode: serial
backend: worktree
max_concurrent: 1
parallel:
port_strategy: none
shared_files: []
dependencies:
approved: []
constraints:
- "Preserve the existing search response shape"
- "Do not add new dependencies on the first run"
stopping:
max_iterations: 4
max_hours: 1
plateau_iterations: 3
target_reached: true
max_runner_up_merges_per_batch: 0
references/experiment-log-schema.yaml
# Experiment Log Schema
# This is the canonical schema for the experiment log file that accumulates
# across an optimization run.
#
# Location: .context/compound-engineering/ce-optimize/<spec-name>/experiment-log.yaml
#
# PERSISTENCE MODEL:
# The experiment log on disk is the SINGLE SOURCE OF TRUTH. The agent's
# in-memory context is expendable and will be compacted during long runs.
#
# Write discipline:
# - Each experiment gets one log entry, appended on its first measurement
# (SKILL.md step 3.3), before batch evaluation
# - Later ladder samples for that experiment update the same entry in place
# - Outcome fields may also be updated in-place after batch evaluation (step 3.5)
# - The `best` section is updated after each batch if a new best is found
# - The `hypothesis_backlog` is updated after each batch
# - The agent re-reads this file from disk at every phase boundary
#
# The orchestrator does NOT read the full log each iteration -- it uses a
# rolling window (last 10 experiments) + a strategy digest file for
# hypothesis generation. But the full log exists on disk for resume,
# crash recovery, and post-run analysis.
# ============================================================================
# TOP-LEVEL STRUCTURE
# ============================================================================
structure:
spec:
type: string
required: true
description: "Name of the optimization spec this log belongs to"
run_id:
type: string
required: true
description: "Unique identifier for this optimization run (timestamp-based). Distinguishes resumed runs from fresh starts."
started_at:
type: string
format: "ISO 8601 timestamp"
required: true
baseline:
type: object
required: true
description: "Metrics measured on the original code before any optimization"
children:
timestamp:
type: string
format: "ISO 8601 timestamp"
gates:
type: object
description: "Key-value pairs of gate metric names to their baseline values"
metrics:
type: object
description: "Required hard-objective snapshots (aggregate plus samples) that decide.mjs loads for later comparisons"
diagnostics:
type: object
description: "Key-value pairs of diagnostic metric names to their baseline values"
judge:
type: object
description: "Judge scores on the baseline (only when primary type is 'judge')"
children:
# All fields from the scoring config appear here
# Plus:
sample_seed:
type: integer
judge_cost_usd:
type: number
experiments:
type: array
required: true
description: "Ordered list of all experiments, including kept, reverted, errored, and deferred"
items:
type: object
# See EXPERIMENT ENTRY below
best:
type: object
required: true
description: "Summary of the current best result"
children:
iteration:
type: integer
description: "Iteration number of the best experiment (use 0 for the baseline snapshot before any experiment is kept)"
metrics:
type: object
description: "All metric values from the current best state (seed with baseline metrics during CP-1)"
judge:
type: object
description: "Judge scores from the best experiment (only when primary type is 'judge')"
total_judge_cost_usd:
type: number
description: "Running total of all judge costs across all experiments"
hypothesis_backlog:
type: array
description: "Remaining hypotheses not yet tested"
items:
type: object
children:
description:
type: string
category:
type: string
priority:
type: string
enum: [high, medium, low]
dep_status:
type: string
enum: [approved, needs_approval, not_applicable]
required_deps:
type: array
items:
type: string
opportunity:
type: object
description: "Pre-implementation opportunity record; see opportunity_record below. Optional for legacy logs."
opportunity_record:
children:
workload:
type: string
description: "Representative workload and input identity"
baseline:
type: string
description: "Revision or recorded snapshot against which the benefit is estimated"
evidence:
type: string
description: "Source location and observed cost with units or workload share; rubric evidence for qualitative work"
expected_benefit:
type: string
description: "Target metric, units, expected reduction/increase range or upper bound, and assumptions; unknown with the missing evidence and cheapest resolving measurement when not estimable"
confidence:
type: string
description: "Confidence and the evidence or uncertainty that justifies it"
cost_and_risk:
type: string
description: "Estimated implementation and measurement effort, behavioral risk, and required correctness checks"
# ============================================================================
# EXPERIMENT ENTRY
# ============================================================================
experiment_entry:
required_children:
iteration:
type: integer
description: "Sequential experiment number (1-indexed, monotonically increasing)"
batch:
type: integer
description: "Batch number this experiment was part of. Multiple experiments in the same batch ran in parallel."
hypothesis:
type: string
description: "Human-readable description of what this experiment tried"
category:
type: string
description: "Category for grouping and diversity selection (e.g., signal-extraction, graph-signals, embedding, algorithm, preprocessing)"
outcome:
type: enum
values:
- measured # measurement finished and metrics were persisted, awaiting batch evaluation / integration
- promising # eligible on current samples but the ladder still needs confirmation before keep
- kept # eligible, confirmed, and integrated onto the optimization branch
- not_selected # eligible after comparison but not integrated (not the winner, overlapping, or past the runner-up cap)
- reverted # compared and not eligible (regressed a required objective, or none improved)
- inconclusive # delta inside the comparison threshold; not a keep and not a demonstrated regression
- censored # aborted as noncompetitive under the predeclared futility bound
- degenerate # degenerate gate or smoke test failed -> immediately reverted, no judge evaluation
- error # measurement command crashed, timed out, or produced malformed output
- deferred_needs_approval # experiment needs an unapproved dependency -> set aside for batch approval
- timeout # measurement command exceeded timeout_seconds
- runner_up_kept # file-disjoint runner-up that was cherry-picked and re-measured successfully
- runner_up_reverted # file-disjoint runner-up that was cherry-picked but combined measurement was not better
description: >
The loop branches on this value.
'measured' and 'promising' are non-terminal: CP-3 persists raw metrics
before batch-level comparison, and 'promising' means the ladder still
needs confirmation samples. An eligible decide `keep` stays `measured`
until its diff is on the optimization branch. 'kept' and 'runner_up_kept'
mean that integration happened. 'not_selected' is terminal for an
eligible candidate that was not integrated. 'deferred_needs_approval'
items are re-presented at wrap-up. All other states are terminal for
that experiment.
optional_children:
opportunity:
type: object
description: "Copy of opportunity_record made before this experiment's implementation; never reconstructed from its result. Missing in legacy logs means unrecorded."
comparisons:
type: array
description: "One record per distinct reference/candidate/workload pairing used in a decision. Later in-place updates must not replace a previously persisted distinct pairing. Absent legacy evidence is unknown, not inferred from the current best."
items:
type: object
children:
kind:
type: string
enum: [standalone, integrated]
reference_revision:
type: string
description: "Identity that uniquely identifies the measured reference bytes. A revision is enough when it names those bytes; a HEAD shared by different uncommitted trees is not."
candidate_revision:
type: string
description: "Identity that uniquely identifies the measured candidate bytes. A revision is enough when it names those bytes; a HEAD shared by different uncommitted trees is not."
workload:
type: string
reference:
type: object
description: "Existing snapshot shape: metrics and judge, with aggregates and samples where available"
candidate:
type: object
description: "Existing snapshot shape: metrics and judge, with aggregates and samples where available"
uncertainty:
type: string
description: "Configured comparison method, observed variability, confirmation status, and any missing uncertainty evidence"
correctness:
type: string
description: "Checks performed and results or exact unverified constraints; passing a timing comparison alone is not correctness evidence"
changes:
type: array
description: "Files modified by this experiment"
items:
type: object
children:
file:
type: string
summary:
type: string
gates:
type: object
description: "Gate metric values from the measurement command"
gates_passed:
type: boolean
description: "Whether all degenerate gates passed"
diagnostics:
type: object
description: "Diagnostic metric values from the measurement command"
metrics:
type: object
description: "Required hard-objective snapshots (aggregate plus samples) that decide.mjs loads"
judge:
type: object
description: "Judge evaluation scores (only when primary type is 'judge' and gates passed)"
children:
# All fields from scoring.primary and scoring.secondary appear here
# Plus:
judge_cost_usd:
type: number
description: "Cost of judge calls for this experiment"
primary_delta:
type: string
description: "Change in primary metric from current best (e.g., '+0.7', '-0.3')"
objective_results:
type: object
description: "Per-required-objective comparison from decide.mjs (verdict, delta, relative)"
sample_count:
type: integer
description: "How many harness samples were spent on this experiment"
next_measurement:
type: string
enum: [none, smoke, exploratory, add_sample, confirm]
description: "Ladder next step from decide.mjs; none when the decision is terminal"
learnings:
type: string
description: "What was learned from this experiment. The agent reads these to avoid re-trying similar approaches and to inform new hypothesis generation."
commit:
type: string
description: "Git commit SHA on the optimization branch (only for 'kept' and 'runner_up_kept' outcomes)"
deferred_reason:
type: string
description: "Why this experiment was deferred (only for 'deferred_needs_approval' outcome)"
error_message:
type: string
description: "Error details (only for 'error' and 'timeout' outcomes)"
merged_with:
type: integer
description: "Iteration number of the experiment this was merged with (only for 'runner_up_kept' and 'runner_up_reverted')"
# ============================================================================
# OUTCOME STATE TRANSITIONS
# ============================================================================
#
# proposed (in hypothesis_backlog)
# -> selected for batch
# -> experiment dispatched
# -> measurement completed
# -> gates failed -> outcome: degenerate
# -> measurement error -> outcome: error
# -> measurement timeout -> outcome: timeout
# -> smoke failed -> outcome: degenerate
# -> futile / censored -> outcome: censored
# -> gates passed
# -> persist raw metrics -> outcome: measured or promising
# -> judge evaluated (if type: judge)
# -> decide.mjs eligible, next_measurement none -> stay measured until integration
# -> diff on optimization branch -> outcome: kept
# -> eligible leftover -> outcome: not_selected
# -> runner-up, file-disjoint -> cherry-pick + re-measure
# -> combined eligible and integrated -> outcome: runner_up_kept
# -> combined not kept -> outcome: runner_up_reverted
# -> inconclusive -> outcome: inconclusive
# -> not eligible -> outcome: reverted
# -> needs unapproved dep -> outcome: deferred_needs_approval
#
# Only 'kept' and 'runner_up_kept' produce a commit on the optimization branch.
# Only 'deferred_needs_approval' items are re-presented at wrap-up for approval.
# ============================================================================
# STRATEGY DIGEST (separate file)
# ============================================================================
#
# Written after each batch to:
# .context/compound-engineering/ce-optimize/<spec-name>/strategy-digest.md
#
# Contains a compressed summary of:
# - What hypothesis categories have been tried
# - Which approaches succeeded (kept) and which failed (reverted)
# - The exploration frontier: what hasn't been tried yet
# - Key learnings that should inform next hypotheses
#
# The orchestrator reads the strategy digest (not the full experiment log)
# when generating new hypotheses between batches.
references/experiment-prompt-template.md
# Experiment Worker Prompt Template
This template is used by the orchestrator to dispatch each experiment to a subagent or Codex. Variable substitution slots are filled at spawn time.
---
## Template
```
You are an optimization experiment worker.
Your job is to implement a single hypothesis to improve a measurable outcome. You will modify code within a defined scope, then stop. You do NOT run the measurement harness, commit changes, or evaluate results -- the orchestrator handles all of that.
<experiment-context>
Experiment: #{iteration} for optimization target: {spec_name}
Hypothesis: {hypothesis_description}
Category: {hypothesis_category}
Current best metrics:
{current_best_metrics}
Baseline metrics (before any optimization):
{baseline_metrics}
</experiment-context>
<scope-rules>
You MAY modify files in these paths:
{scope_mutable}
You MUST NOT modify files in these paths:
{scope_immutable}
CRITICAL: Do not modify any file outside the mutable scope. The measurement harness and evaluation data are immutable by design -- the agent cannot game the metric by changing how it is measured.
</scope-rules>
<constraints>
{constraints}
</constraints>
<approved-dependencies>
You may add or use these dependencies without further approval:
{approved_dependencies}
If your implementation requires a dependency NOT in this list, STOP and note it in your output. Do not install unapproved dependencies.
</approved-dependencies>
<previous-experiments>
Recent experiments and their outcomes (for context -- avoid re-trying approaches that already failed):
{recent_experiment_summaries}
</previous-experiments>
<instructions>
1. Read and understand the relevant code in the mutable scope
2. Implement the hypothesis described above
3. Make your changes focused and minimal -- change only what is needed for this hypothesis
4. Do NOT run the measurement harness (the orchestrator handles this)
5. Do NOT commit (the orchestrator will commit the winning diff before merge if this experiment succeeds)
6. Do NOT modify files outside the mutable scope
7. When done, run `git diff --stat` so the orchestrator can see your changes
8. If you discover you need an unapproved dependency, note it and stop
Focus on implementing the hypothesis well. The orchestrator will measure and evaluate the results.
</instructions>
```
## Variable Reference
| Variable | Source | Description |
|----------|--------|-------------|
| `{iteration}` | Experiment counter | Sequential experiment number |
| `{spec_name}` | Spec file `name` field | Optimization target identifier |
| `{hypothesis_description}` | Hypothesis backlog | What this experiment should try |
| `{hypothesis_category}` | Hypothesis backlog | Category (signal-extraction, algorithm, etc.) |
| `{current_best_metrics}` | Experiment log `best` section | Current best metric values (compact YAML or key: value pairs) |
| `{baseline_metrics}` | Experiment log `baseline` section | Original baseline before any optimization |
| `{scope_mutable}` | Spec `scope.mutable` | List of files/dirs the worker may modify |
| `{scope_immutable}` | Spec `scope.immutable` | List of files/dirs the worker must not touch |
| `{constraints}` | Spec `constraints` | Free-text constraints to follow |
| `{approved_dependencies}` | Spec `dependencies.approved` | Dependencies approved for use |
| `{recent_experiment_summaries}` | Rolling window (last 10) from experiment log | Compact summaries: hypothesis, outcome, learnings |
## Notes
- This template works for both subagent and Codex dispatch. No platform-specific assumptions.
- For Codex dispatch: write the filled template to a temp file and pipe via stdin (`cat /tmp/optimize-exp-XXXXX.txt | codex exec --skip-git-repo-check - 2>&1`).
- For subagent dispatch: pass the filled template as the subagent prompt.
- Keep `{recent_experiment_summaries}` concise -- 2-3 lines per experiment, last 10 only. Do not include the full experiment log.
- The worker should NOT read the full experiment log or strategy digest. It receives only what the orchestrator provides.
references/judge-prompt-template.md
# Judge Evaluation Prompt Template
This template is used by the orchestrator to dispatch batched LLM-as-judge evaluation calls. Each judge sub-agent evaluates a batch of sampled output items and returns structured JSON scores.
The orchestrator:
1. Reads the experiment's output
2. Selects samples per the stratification config (using fixed seed)
3. Groups samples into batches of `judge.batch_size`
4. Dispatches `ceil(sample_size / batch_size)` parallel sub-agents using this template
5. Aggregates returned JSON scores
---
## Item Evaluation Template
```
You are a quality judge evaluating output items for an optimization experiment.
Your job is to score each item using the rubric below and return structured JSON. Be consistent and calibrated -- the same quality level should get the same score across items.
<rubric>
{rubric}
</rubric>
<items>
{items_json}
</items>
<output-contract>
Return ONLY a valid JSON array. No prose, no markdown, no explanation outside the JSON.
Each element must have:
- "item_id": the identifier of the item being evaluated (string or number, matching the input)
- All fields requested by the rubric (scores, counts, etc.)
- "ambiguous": true if you cannot confidently score this item (e.g., insufficient context, borderline case). When ambiguous, still provide your best-guess score but flag it.
Example output format (adapt field names to match the rubric):
[
{"item_id": "cluster-42", "score": 4, "distinct_topics": 1, "outlier_count": 0, "ambiguous": false},
{"item_id": "cluster-17", "score": 2, "distinct_topics": 3, "outlier_count": 2, "ambiguous": false},
{"item_id": "cluster-99", "score": 3, "distinct_topics": 2, "outlier_count": 1, "ambiguous": true}
]
Rules:
- Evaluate each item independently
- Score based on the rubric, not on how other items in this batch scored
- If an item is empty or has only 1 element when it should have more, score it based on what is present
- For very large items (many elements), focus on a representative subset and note if quality varies across the item
- Every item in the batch MUST appear in your output
</output-contract>
```
## Singleton Evaluation Template
```
You are a quality judge evaluating singleton items -- items that are currently NOT in any group/cluster.
Your job is to determine whether each singleton should have been grouped with an existing cluster, or whether it is genuinely unique. Return structured JSON.
<rubric>
{singleton_rubric}
</rubric>
<singletons>
{singletons_json}
</singletons>
<existing-clusters>
A summary of existing clusters for reference (titles/themes only, not full contents):
{cluster_summaries}
</existing-clusters>
<output-contract>
Return ONLY a valid JSON array. No prose, no markdown, no explanation outside the JSON.
Each element must have:
- "item_id": the identifier of the singleton
- All fields requested by the singleton rubric (should_cluster, best_cluster_id, confidence, etc.)
Example output format (adapt field names to match the rubric):
[
{"item_id": "issue-1234", "should_cluster": true, "best_cluster_id": "cluster-42", "confidence": 4},
{"item_id": "issue-5678", "should_cluster": false, "best_cluster_id": null, "confidence": 5}
]
Rules:
- A singleton that genuinely has no match in existing clusters should get should_cluster: false
- A singleton that clearly belongs in an existing cluster should get should_cluster: true with the cluster ID
- High confidence (4-5) means you are very sure. Low confidence (1-2) means the item is borderline.
- Every singleton in the batch MUST appear in your output
</output-contract>
```
## Variable Reference
| Variable | Source | Description |
|----------|--------|-------------|
| `{rubric}` | Spec `metric.judge.rubric` | User-defined scoring rubric |
| `{items_json}` | Sampled output items | JSON array of items to evaluate (one batch worth) |
| `{singleton_rubric}` | Spec `metric.judge.singleton_rubric` | User-defined rubric for singleton evaluation |
| `{singletons_json}` | Sampled singleton items | JSON array of singleton items to evaluate |
| `{cluster_summaries}` | Experiment output | Summary of existing clusters (titles/themes) for singleton reference |
## Notes
- Designed for Haiku by default -- prompts are concise and well-structured for smaller models
- The rubric is part of the immutable measurement harness -- the experiment agent cannot modify it
- The `ambiguous` flag on items helps the orchestrator identify noisy evaluations without forcing bad scores
- For singleton evaluation, the orchestrator provides cluster summaries (not full contents) to keep judge context lean
- Each sub-agent evaluates one batch independently -- sub-agents do not see each other's results
- **That independence is required, not merely preferred.** These scores gate accept/revert, so a judge must be a separate context from the one that authored the hypothesis and ran the experiment. Where no dispatch is available, block the judge pass rather than scoring inline: an orchestrator grading its own experiment is not a measurement.
references/loop.md
# Phases 2-3: hypotheses and the optimization loop
Read this before generating hypotheses and follow it for the whole loop. The body owns the dependency pre-approval gate and the stopping criteria; this file carries hypothesis generation, batch selection, experiment dispatch, result collection and persistence, batch evaluation, the state update, and the cross-cutting concerns.
## Phase 2: Hypothesis Generation
### 2.1 Analyze Current Approach
Read the code within `scope.mutable` to understand:
- The current implementation approach
- Obvious improvement opportunities
- Constraints and dependencies between components
The next action is the cheapest executable step that would change what gets implemented. A locating measurement belongs in this phase when it is cheaper than an implementation experiment, would change keep or skip, and can be taken. Named-workload cost needs attributed shares before implementation when a Phase 1 baseline total cannot decide keep or skip; that total is the scoring reference, not those shares. A scored variant space does not require a performance profile.
Do not treat the implementation backlog as empty, and do not proceed to wrap-up, while a cheaper locating measurement can still be taken and would change keep or skip. If locating would change keep or skip but cannot be obtained, wrap-up and say what blocked it. Do not implement without that measurement.
Optionally read `references/agents/repo-research-analyst.md` and dispatch a generic subagent seeded with that local prompt for deeper codebase analysis if the scope is large or unfamiliar. Do not dispatch a standalone agent by type/name. Pass the active project and optimization context, request only question-specific scopes such as `patterns`, and go directly to current owning code. If the optimization cannot be scoped, allow one targeted root or workspace probe.
### 2.2 Generate Hypothesis List
Generate an initial set of hypotheses. Each hypothesis should have:
- **Description**: what to try
- **Category**: one of the standard categories (signal-extraction, graph-signals, embedding, algorithm, preprocessing, parameter-tuning, architecture, data-handling) or a domain-specific category
- **Priority**: high, medium, or low as a summary label
- **Required dependencies**: any new packages or tools needed
- **Opportunity**: the log schema's `opportunity` record (estimate or explicit unknown)
Include user-provided hypotheses if any were given as input.
Record an `opportunity` on every hypothesis using the log schema before implementation. Connect whatever observed cost or rubric evidence exists to the expected change in the target metric, with units, a comparison baseline, and the assumptions behind the estimate. Prefer a supported range or upper bound over a point estimate. If the benefit or the cost share cannot be estimated, record it as unknown and name the cheapest measurement that would resolve the uncertainty. A subjective priority score is not a measured benefit. The `priority` field does not rank the backlog.
An unknown opportunity may sit on the backlog. It is not a runnable implementation experiment while a cheaper locating measurement would change keep or skip. A scored variant space may leave numerical benefit unknown and does not require a performance profile.
The backlog contains the credible opportunities supported by current evidence, not a required number of ideas. Rank by expected target benefit, confidence, implementation and measurement cost, and behavioral risk. Present the ranked opportunities and their estimates when recording CP-2, after the disk write and verification.
### 2.3 Dependency Pre-Approval
The body owns this gate. Record its outcome on each hypothesis as `dep_status: approved` or `needs_approval`, which is what batch selection reads.
### 2.4 Record Hypothesis Backlog (CP-2)
**MANDATORY CHECKPOINT.** Write the initial backlog to the experiment log file and verify. CP-2 is incomplete until each hypothesis in that write carries its opportunity record.
```yaml
hypothesis_backlog:
- description: "Remove template boilerplate before embedding"
category: "signal-extraction"
priority: high
dep_status: approved
required_deps: []
opportunity:
workload: "notification-clustering fixture"
baseline: "CP-1 baseline"
evidence: "judge rubric: boilerplate dilutes embeddings; no profile"
expected_benefit: "unknown; cheapest resolve: one judged stripped-vs-current sample"
confidence: "low: unmeasured"
cost_and_risk: "small edit; judge sample; no new deps"
- description: "Try HDBSCAN clustering algorithm"
category: "algorithm"
priority: medium
dep_status: needs_approval
required_deps: ["scikit-learn"]
opportunity:
workload: "notification-clustering fixture"
baseline: "CP-1 baseline"
evidence: "algorithm family untried on this fixture"
expected_benefit: "unknown; cheapest resolve: one exploratory run after dep approval"
confidence: "low: unmeasured"
cost_and_risk: "new dependency scikit-learn; judge sample"
```
---
## Phase 3: Optimization Loop
This phase repeats in batches until a stopping criterion is met.
### 3.1 Batch Selection
Select hypotheses for this batch:
- Build a runnable backlog by excluding hypotheses with `dep_status: needs_approval`
- A hypothesis is not runnable while a cheaper locating measurement would still change keep or skip
- If `execution.mode` is `serial`, or the current decision needs to attribute a cost change to one lever, force `batch_size = 1`
- Otherwise, `batch_size = min(runnable_backlog_size, execution.max_concurrent)`
- Select by the ranked expected benefit, confidence, cost, and risk above; the priority label does not decide order. Category diversity breaks remaining ties.
When a cheaper locating measurement can be taken and would still change keep or skip, take that measurement and update the backlog before selecting a batch. Do not treat that state as an empty backlog.
When no executable next action remains, proceed to Phase 4 (wrap-up). An action is executable only if it can be taken now. Locating that would change keep or skip but cannot be obtained is a blocker, not a reason to keep the loop open; wrap-up and say what blocked it. Deferred dependencies are presented there instead of the loop spinning forever.
### 3.2 Dispatch Experiments
The experiment's forecast is the backlog `opportunity` as of dispatch. Do not reconstruct it from later results. Copy that backlog value into the experiment entry at its first CP-3 write, including when that write recovers a `result.yaml` marker that has no forecast. Revised estimates for later experiments must not overwrite an earlier experiment's forecast. Missing forecasts in resumed legacy runs stay unrecorded.
For each hypothesis in the batch, dispatch according to `execution.mode`. In `serial` mode, run exactly one experiment to completion before selecting the next hypothesis. In `parallel` mode, dispatch the batch concurrently.
**Bounded dispatch.** Do not assume the host will accept all concurrent subagents at once; the active-subagent cap varies by host and profile and is independent of `execution.max_concurrent` (which caps worktrees, a separate budget). Queue the selected experiments, dispatch only as many as the host accepts, and when a capacity or active-agent-limit error appears, treat it as backpressure: retry the queued experiment after a slot frees rather than marking it failed. Mark an experiment failed only when dispatch fails for a non-capacity reason that survives correcting the invocation, or a successfully dispatched experiment errors/times out.
The Phase 3 blocks below each set `SKILL_DIR` inline as well (the loaded `ce-optimize` skill directory; see the Bundled scripts note in Phase 1): shell state does not persist from Phase 1, so each block carries its own assignment.
**Worktree backend:**
1. Create experiment worktree:
```bash
SKILL_DIR="<absolute path of the directory containing this SKILL.md>";
WORKTREE_PATH=$(bash "$SKILL_DIR/scripts/experiment-worktree.sh" create "<spec_name>" <exp_index> "optimize/<spec_name>" <shared_files...>) # creates optimize-exp/<spec_name>/exp-<NNN>
```
2. Apply port parameterization if configured (set env vars for the measurement script)
3. Fill the experiment prompt template (`references/experiment-prompt-template.md`) with:
- Iteration number, spec name
- Hypothesis description and category
- Current best and baseline metrics
- Mutable and immutable scope
- Constraints and approved dependencies
- Rolling window of last 10 experiments (concise summaries)
4. Dispatch a subagent with the filled prompt, working in the experiment worktree
**Codex backend:**
1. Check environment guard -- do NOT delegate if already inside a Codex sandbox:
```bash
# If these exist, we're already in Codex -- fall back to subagent
test -n "${CODEX_SANDBOX:-}" || test -n "${CODEX_SESSION_ID:-}" || test ! -w .git
```
2. Fill the experiment prompt template
3. Write the filled prompt to a temp file
4. Dispatch via Codex:
```bash
cat /tmp/optimize-exp-XXXXX.txt | codex exec --skip-git-repo-check - 2>&1
```
5. Security posture: use the user's selection (ask once per session if not set in spec)
### 3.3 Collect and Persist Results
Persist a `comparisons` record for each distinct reference, candidate, and workload pairing used in a decision. Each side's identity must uniquely identify the bytes that were measured; a shared HEAD is not enough when the candidate is uncommitted. Record the workload, both snapshots, and the decision's uncertainty and correctness evidence. Standalone and integrated pairings stay distinct in this array; a later in-place update must not replace a previously persisted distinct pairing. A runner-up's contribution is its confirmed change against the branch it was added to, not its standalone gain. These records explain results; `decide.mjs` still owns acceptance, using the existing snapshot fields.
Process experiments as they complete: do NOT wait for the entire batch to finish before writing results.
For each completed experiment, **immediately**:
1. **Run measurement** in the experiment's worktree. Spend only the measurement the current decision needs (see Phase 1). When `stability.mode` is `ladder` and a smoke command is set, run that smoke check first: failure is terminally `degenerate`, and success proceeds to the first exploratory sample of `measurement.command` before comparison. Otherwise start with one exploratory sample. Pass `CE_OPTIMIZE_CENSOR_AFTER` to `measure.sh` only when elapsed wall time itself proves the candidate cannot become eligible: every required objective is already hopeless, not merely the primary. Otherwise let measurement finish so other required objectives can still win, and let `decide.mjs` assess futility after the payload is complete.
```bash
SKILL_DIR="<absolute path of the directory containing this SKILL.md>";
bash "$SKILL_DIR/scripts/measure.sh" "<measurement.command>" <timeout_seconds> "<worktree_path>/<measurement.working_directory or .>" <env_vars...>
```
When mode is `repeat`, keep running `repeat_count` times and aggregating as in Phase 1. When mode is `stable`, run once.
2. **Write crash-recovery marker**: immediately after measurement, write `result.yaml` in the experiment worktree containing the raw metrics. This ensures the measurement is recoverable even if the agent crashes before updating the main log.
3. **Read raw JSON output** from the measurement script
4. **Evaluate degenerate gates**:
- For each gate in `metric.degenerate_gates`, parse the operator and threshold
- Compare the metric value against the threshold
- If ANY gate fails: mark outcome as `degenerate`, skip judge evaluation, save money
5. **If gates pass AND primary type is `judge`**:
- **Independence gate: check before dispatching.** A judge must not have authored the hypothesis or run the experiment it is scoring, and must not see other judges' results; that independence is what makes these scores usable as an accept/revert gate. If the host exposes no way to dispatch judges as separate agents, do **not** score inline: mark the experiment's outcome `error` with the reason (judges undispatchable), skip judge evaluation exactly as a failed degenerate gate does, and continue to the log-and-append step so the entry is still written to disk. An experiment stopped here never carries judge metrics, so it is not eligible to become `best` and does not enter the accept/revert comparison: it is unmeasured, not poor-scoring. Report the blocker to the user at the batch summary.
- Read the experiment's output (cluster assignments, search results, etc.)
- Apply stratified sampling per `metric.judge.stratification` config (using `sample_seed`)
- Group samples into batches of `metric.judge.batch_size`
- Fill the judge prompt template (`references/judge-prompt-template.md`) for each batch
- Dispatch the `ceil(sample_size / batch_size)` judge sub-agents using the same bounded dispatch as Phase 3.2: queue them, dispatch to whatever concurrency the host accepts, and treat a capacity error as backpressure (retry the queued batch after a slot frees) rather than a scoring failure. These judge sub-agents are a separate budget from the experiment worktrees.
- Each sub-agent returns structured JSON scores
- Aggregate scores: compute the configured primary judge field from `metric.judge.scoring.primary` (which should match `metric.primary.name`) plus any `scoring.secondary` values
- If `singleton_sample > 0`: also dispatch singleton evaluation sub-agents
6. **Compare with `decide.mjs`.** Invoke it only after gates pass and the payload holds every required objective value: hard metrics from measurement, and judge scores when those were collected. The payload is the spec as loaded plus the baseline and candidate snapshots. The script reads the nested spec (`metric`, `measurement.stability`) and owns eligibility, noise, and the ladder next step. Do not reconstruct a flattened payload, and do not re-derive the threshold in prose.
```bash
SKILL_DIR="<absolute path of the directory containing this SKILL.md>";
NODE="$(for c in node nodejs; do command -v "$c" >/dev/null 2>&1 && "$c" -e '' >/dev/null 2>&1 && { echo "$c"; break; }; done)";
[ -n "$NODE" ] || { echo "no working Node runtime on PATH (tried node, nodejs)" >&2; exit 1; };
"$NODE" "$SKILL_DIR/scripts/decide.mjs" "<payload.json>"
```
If that probe finds no runtime, do not invoke an empty command: mark the experiment `error` with that reason and continue the batch. Use `decision` and `next_measurement`. Collect the requested measurement and repeat this sequence whenever `next_measurement` is not `none`. Do not keep a candidate until `next_measurement` is `none`. Record `inconclusive` and `censored` as those outcomes, not as `reverted`. Each extra sample belongs to this same experiment: write it onto the existing entry at CP-3, then decide again.
7. **IMMEDIATELY persist this experiment on disk (CP-3)**: do not defer this to batch evaluation. The durable unit is one log entry per experiment at `.context/compound-engineering/ce-optimize/<spec-name>/experiment-log.yaml`. After the first measurement, append that entry. After every later ladder sample for the same experiment, write the accumulated metrics and current outcome onto that same entry. Do not append a second entry for the same hypothesis, and do not rewrite a different experiment's samples. Write a decide terminal only when `next_measurement` is `none`. Until then the entry stays nonterminal: `promising` while the keep path still needs samples, `measured` otherwise (including an inconclusive result that still wants samples). When `next_measurement` is `none`, an eligible result stays `measured` until its diff is on the optimization branch; a non-eligible result gets the decide terminal (`reverted`, `inconclusive`, `censored`, `degenerate`). `kept` and `runner_up_kept` wait until that integration. The raw metrics are on disk and safe from context compaction.
8. **VERIFY the write (CP-3 verification)**: read the experiment log back from disk and confirm the entry just written is present. If verification fails, retry the write. Do NOT proceed to the next experiment until this entry is confirmed on disk.
**Why immediately + verify?** The agent's context window is NOT a durable store. Context compaction, session crashes, and restarts are expected during long runs: results that exist only in the agent's memory are lost. The verification step catches silent write failures that would otherwise lose data.
### 3.4 Evaluate Batch
After all experiments in the batch have been measured:
1. **Decide eligibility from `decide.mjs`, not from the primary metric alone.** An experiment is eligible when it improves at least one required objective beyond the configured comparison threshold and does not violate any other required objective. When `metric.objectives` is absent, the primary is the only required objective. `inconclusive` is not a keep.
2. **Rank** the eligible experiments in the batch by the script's `rank_score` (primary relative gain when the primary moved; otherwise the strongest required-objective relative gain). Identify that winner as the experiment to keep. An eligible experiment may be kept even if the ranking primary did not move.
3. **If `decide.mjs` returns `keep` for that winner: KEEP**
- Commit the experiment branch first so the winning diff exists as a real commit before any merge or cherry-pick
- Include only mutable-scope changes in that commit; if no eligible diff remains, treat the experiment as non-improving and revert it
- Merge the committed experiment branch into the optimization branch
- Use the message `optimize(<spec-name>): <hypothesis description>` for the experiment commit
- After the merge succeeds, clean up the winner's experiment worktree and branch; the integrated commit on the optimization branch is the durable artifact
- This is now the new baseline for subsequent batches
4. **Check file-disjoint runners-up** (up to `max_runner_up_merges_per_batch`):
- For each runner-up that also improved, check file-level disjointness with the kept experiment
- **File-level disjointness**: two experiments are disjoint if they modified completely different files. Same file = overlapping, even if different lines.
- If disjoint: cherry-pick the runner-up onto the new baseline and run the same decide loop as step 3.3 against a fresh sample set for that combined snapshot: do not reuse the standalone experiment's accumulated samples, whose meaning is against the previous baseline. Collect further measurement whenever `next_measurement` is not `none`. Persist the combined pairing as `kind: integrated` on that same log entry without replacing the standalone comparison. Keep the original standalone log entry for audit.
- Keep the cherry-pick only when that result is eligible and `next_measurement` is `none` (outcome: `runner_up_kept`); then clean up that runner-up's experiment worktree and branch
- Otherwise: revert the cherry-pick, log as "promising alone but neutral/harmful in combination" (outcome: `runner_up_reverted`), then clean up the runner-up's experiment worktree and branch
- Stop after first failed combination
5. **Handle deferred deps**: experiments that need unapproved dependencies get outcome `deferred_needs_approval`
6. **Close the rest.** Cleanup worktrees. `kept` and `runner_up_kept` are only for diffs on the optimization branch. Eligible candidates that were not integrated become `not_selected`. Leave `inconclusive`, `censored`, and `degenerate` as `decide.mjs` returned them.
### 3.5 Update State (CP-4)
**MANDATORY CHECKPOINT.** By this point, individual experiment results are already on disk (written in step 3.3). This step updates aggregate state and verifies.
1. **Re-read the experiment log from disk**: do not trust in-memory state. The log is the source of truth.
2. **Finalize outcomes**: update experiment entries from step 3.4 evaluation (mark `kept`, `reverted`, `runner_up_kept`, etc.). Write these outcome updates to disk immediately.
3. **Update the `best` section** in the experiment log if a new best was found. Write to disk.
4. **Write strategy digest** to `.context/compound-engineering/ce-optimize/<spec-name>/strategy-digest.md`:
- Categories tried so far (with success/failure counts)
- Key learnings from this batch and overall
- Remaining opportunities, their supporting evidence, and whether current measurements still support their estimates; mark stale estimates for reassessment before selecting them
- Current best metrics and improvement from baseline
5. **Generate new hypotheses** based on learnings:
- Re-read the strategy digest from disk (not from memory)
- Read the rolling window (last 10 experiments from the log on disk)
- Do NOT read the full experiment log -- use the digest for broad context
- After a keep on a cost target, re-attribute before adding implementation hypotheses only when the keep leaves the current cost shares unable to decide keep or skip
- Add new hypotheses to the backlog and write the updated backlog to disk
6. **Write updated hypothesis backlog to disk**: the backlog section of the experiment log must reflect newly added hypotheses and removed (tested) ones.
**CP-4 Verification:** Read the experiment log back from disk. Confirm: (a) all experiment outcomes from this batch are finalized, (b) the `best` section reflects the current best, (c) the hypothesis backlog is updated. Read `strategy-digest.md` back and confirm it exists. Only THEN proceed to the next batch or stopping criteria check.
**Checkpoint: at this point, all state for this batch is on disk. If the agent crashes and restarts, it can resume from the experiment log without loss.**
### 3.6 Check Stopping Criteria
Stop the loop as soon as any one of these holds:
- **Target reached**: `stopping.target_reached` is true and the current best meets every declared required target (`decide.mjs` `target_reached` on the current-best snapshot). When `metric.objectives` is absent, that is the single `metric.primary.target` if set. Do not stop for a primary-only hit while another required target is still unmet.
- **Max iterations**: total experiments run >= `stopping.max_iterations`
- **Max hours**: wall-clock time since Phase 3 started (not since the invocation) >= `stopping.max_hours`
- **Judge budget exhausted**: `metric.judge.max_total_cost_usd` is set and cumulative judge spend has reached it
- **Plateau**: no improvement for `stopping.plateau_iterations` **consecutive** experiments
- **Manual stop**: the user interrupts. Save state, then go to Phase 4.
- **No runnable hypothesis left**: no executable next action remains
If none is met, proceed to the next batch (3.1).
### 3.7 Cross-Cutting Concerns
**Codex failure cascade**: Track consecutive Codex delegation failures. After 3 consecutive failures, auto-disable Codex for remaining experiments and fall back to subagent dispatch. Log the switch.
**Error handling**: Classify a failed measurement from what `measure.sh` actually signaled. The censored stderr marker (with exit 125) is `censored`. Exit 124 is `timeout`. Any other non-zero exit (including 125 without that marker) is `error`. Log that outcome with the error message, revert the experiment, and continue the batch.
**Progress reporting**: After each batch, report:
- Batch N of estimated M (based on backlog size)
- Experiments run this batch and total
- Current best metric and improvement from baseline
- Cumulative judge cost (if applicable)
**Crash recovery**: See Persistence Discipline section. Per-experiment `result.yaml` markers are written in step 3.3. Individual experiment results are appended to the log immediately in step 3.3. Batch-level state (outcomes, best, digest) is written in step 3.5. On resume (Phase 0.4), the log on disk is the ground truth: scan for any `result.yaml` markers not yet reflected in the log.
---
references/measurement.md
# Phase 0.3-1.7: prior learnings, identity, and measurement scaffolding
Read this after the spec is saved and follow it through the approval gate. The body owns the two gates in here that stop the run (the clean-tree gate and the user approval gate) and this file carries the procedure around them: prior-learnings search, run identity and resume detection, the branch and scratch space, the measurement harness, the baseline, the parallelism probe, and the worktree budget.
### 0.3 Search Prior Learnings
Read `references/agents/learnings-researcher.md` and dispatch a generic subagent seeded with that local prompt to search for prior optimization work on similar topics. Do not dispatch a standalone agent by type/name. If relevant learnings exist, incorporate them into the approach.
### 0.4 Run Identity Detection
Check if `optimize/<spec-name>` branch already exists:
```bash
git rev-parse --verify "optimize/<spec-name>" 2>/dev/null
```
**If branch exists**, check for an existing experiment log at `.context/compound-engineering/ce-optimize/<spec-name>/experiment-log.yaml`.
Present the user with a choice via the platform question tool:
- **Resume**: read ALL state from the experiment log on disk (do not rely on any in-memory context from a prior session). Recover any measured-but-unlogged experiments by scanning worktree directories for `result.yaml` markers. Then apply the body's resume rule to decide what is skipped and which gates are re-entered.
- **Fresh start**: archive the old branch to `optimize-archive/<spec-name>/archived-<timestamp>`, clear the experiment log, start from scratch
### 0.5 Create Optimization Branch and Scratch Space
```bash
git checkout -b "optimize/<spec-name>" # or switch to existing if resuming
```
Create scratch directory:
```bash
mkdir -p .context/compound-engineering/ce-optimize/<spec-name>/
```
---
## Phase 1: Measurement Scaffolding
**This phase is a HARD GATE. The user must approve baseline and parallel readiness before Phase 2.**
**Bundled scripts.** Phases 1 and 3 call helper scripts that ship in this skill's `scripts/` directory (`measure.sh`, `decide.mjs`, `parallel-probe.sh`, `experiment-worktree.sh`). The Bash tool's working directory is the user's project, not the skill directory, so a bare `scripts/<name>` path will not resolve: invoke each by the skill's own absolute path. Every runnable block below already sets `SKILL_DIR` inline (shell state does not persist between Bash tool calls, so each block must carry it); just replace the `<absolute path …>` placeholder with the directory you loaded this `ce-optimize` SKILL.md from before running. The shape:
```bash
SKILL_DIR="<absolute path of the directory containing this SKILL.md>";
bash "$SKILL_DIR/scripts/<name>"
```
### 1.1 Clean-Tree Gate
The body owns this gate. Run `git status --porcelain`, filter the output against `scope.mutable` and `scope.immutable`, and apply the body's rule to the result: name the dirty in-scope files and ask the user to commit or stash them, and do not continue until they are clean.
### 1.2 Build or Validate Measurement Harness
**If user provides a measurement harness** (the `measurement.command` already exists):
1. Run it once via the measurement script:
```bash
SKILL_DIR="<absolute path of the directory containing this SKILL.md>";
bash "$SKILL_DIR/scripts/measure.sh" "<measurement.command>" <timeout_seconds> "<measurement.working_directory or .>"
```
2. Validate the JSON output:
- Contains keys for all degenerate gate metric names
- Contains keys for all diagnostic metric names
- Contains keys for every required hard objective (`metric.primary` when it is hard, plus every `metric.objectives` entry)
- Values are numeric or boolean as expected
3. If validation fails, report what is missing and ask the user to fix the harness
**If agent must build the harness:**
1. Analyze the codebase to understand the current approach and what should be measured
2. Build an evaluation script (e.g., `evaluate.py`, `evaluate.sh`, or equivalent)
3. Add the evaluation script path to `scope.immutable` -- the experiment agent must not modify it
4. Run it once and validate the output
5. Present the harness and its output to the user for review
### 1.3 Establish Baseline
Run the measurement harness on the current code. Baseline and final confirmation always use the full configured protocol (`repeat_count` samples when mode is `repeat` or `ladder`; one run when mode is `stable`). Exploratory experiments later may spend less; the baseline must not.
**If stability mode is `repeat` or `ladder`:**
Do not start this protocol until the counts that mode uses are coherent. Repeat needs a positive `repeat_count`. Ladder needs positive `exploratory_pairs` and `confirmation_repeats` (falling back to `repeat_count`) with confirmation at least the exploratory count: the same rule `scripts/decide.mjs` uses. A repeat-mode spec does not need ladder fields.
1. Run the harness that many times (`repeat_count` in repeat mode; the coherent confirmation count in ladder mode)
2. Aggregate results using the configured aggregation method (median, mean, min, max)
3. Calculate variance across runs
4. If variance exceeds the configured comparison threshold, warn the user and suggest increasing `repeat_count`
**Spend only the measurement the current decision needs.** After Phase 1, a smoke failure is degenerate; one paired exploratory sample can reject a clearly worse candidate or mark it inconclusive; add samples only while the result is promising or inconclusive; run the full configured protocol only before keeping a candidate and for the run's final confirmation. `scripts/decide.mjs` returns that next step. When mode is `stable` or `repeat`, keep the existing full-protocol behavior.
The Phase 1 baseline total is the scoring reference for later comparisons. It is not the cost shares of a named workload. Attribution, when a cost target needs it, is Phase 2 locating work, not a second Phase 1 baseline.
Record the baseline in the experiment log. Persist every required hard objective under `metrics` (or `judge` when the primary is a judge score) so `decide.mjs` can load the same snapshot shape later experiments use. Gates and diagnostics stay in their own containers.
```yaml
baseline:
timestamp: "<current ISO 8601 timestamp>"
gates:
<gate_name>: <value>
...
metrics:
<required_hard_objective>: { aggregate: <value>, samples: [<value>, ...] }
...
diagnostics:
<diagnostic_name>: <value>
...
```
If primary type is `judge`, also run the judge evaluation on baseline output to establish the starting judge score.
### 1.4 Parallelism Readiness Probe
Run the parallelism probe script:
```bash
SKILL_DIR="<absolute path of the directory containing this SKILL.md>";
bash "$SKILL_DIR/scripts/parallel-probe.sh" "<project_directory>" "<measurement.command>" "<measurement.working_directory>" <shared_files...>
```
Read the JSON output. Present any blockers to the user with suggested mitigations. Treat the probe as intentionally narrow: it should inspect the measurement command, the measurement working directory, and explicitly declared shared files, not the entire repository.
### 1.5 Worktree Budget Check
Count existing worktrees:
```bash
SKILL_DIR="<absolute path of the directory containing this SKILL.md>";
bash "$SKILL_DIR/scripts/experiment-worktree.sh" count
```
If count + `execution.max_concurrent` would exceed 12:
- Warn the user
- Suggest cleaning up existing worktrees or reducing `max_concurrent`
- Do NOT block -- the user may proceed at their own risk
### 1.6 Write Baseline to Disk (CP-1)
**MANDATORY CHECKPOINT.** Before presenting results to the user, write the initial experiment log with baseline metrics to disk:
1. Create the experiment log file at `.context/compound-engineering/ce-optimize/<spec-name>/experiment-log.yaml`
2. Include all required top-level sections from `references/experiment-log-schema.yaml`: `spec`, `run_id`, `started_at`, `baseline`, `experiments`, and `best`
3. Seed `experiments` as an empty array and seed `best` from the baseline snapshot (use `iteration: 0`, baseline metrics, and baseline judge scores if present) so later phases have a valid current-best state to compare against
4. Optionally seed `hypothesis_backlog: []` here as well so the log shape is stable before Phase 2 populates it
5. **Verify**: read the file back and confirm the required sections are present and the baseline values match
6. Only THEN present results to the user
### 1.7 User Approval Gate
The body owns this gate: what is presented, the options and the condition on adjusting the spec, the uncapped-spend disclosure, and the rule that Phase 2 does not start without explicit approval. A resume that cannot prove the user cleared this gate runs it again, so this phase supplies the same payload then. What this phase supplies to it: the baseline's gate values, diagnostic values, and judge scores; the experiment log path; the probe results with any blockers and mitigations; the clean-tree confirmation; the worktree count and projection; and the estimated per-experiment judge cost against the configured cap.
---
references/optimize-spec-schema.yaml
# Optimization Spec Schema
# This is the canonical schema for optimization spec files created by users
# to configure a /ce-optimize run. The orchestrating agent validates specs
# against this schema before proceeding.
#
# Usage: Create a YAML file matching this schema and pass it to /ce-optimize.
# The agent reads this spec, validates required fields, and uses it to
# configure the entire optimization run.
# ============================================================================
# REQUIRED FIELDS
# ============================================================================
required_fields:
name:
type: string
pattern: "^[a-z0-9]+(?:-[a-z0-9]+)*$"
description: "Unique identifier for this optimization run (lowercase kebab-case, safe for git refs and worktree paths)"
example: "improve-issue-clustering"
description:
type: string
description: "Human-readable description of the optimization goal"
example: "Improve coherence and coverage of issue/PR clusters"
metric:
type: object
description: "Three-tier metric configuration"
required_children:
primary:
type: object
description: "The metric the loop optimizes against"
required_children:
type:
type: enum
values:
- hard # scalar metric from measurement command (e.g., build time, test pass rate)
- judge # LLM-as-judge quality score from sampled outputs
description: "Whether the primary metric comes from the measurement command directly or from LLM-as-judge evaluation"
name:
type: string
description: "Metric name: must match a key in the measurement command's JSON output (for hard type) or a scoring field (for judge type)"
example: "cluster_coherence"
direction:
type: enum
values:
- maximize
- minimize
description: "Whether higher or lower is better"
optional_children:
baseline:
type: number
default: null
description: "Filled automatically during Phase 1 baseline measurement. Do not set manually."
target:
type: number
default: null
description: "Optional target value. Loop stops when this is reached. When metric.objectives is set, target_reached instead requires every required objective that declares a target."
example: 4.2
degenerate_gates:
type: array
description: "Fast boolean checks that reject obviously broken solutions before expensive evaluation. Run first, before the primary metric or judge."
required: true
items:
type: object
required_children:
name:
type: string
description: "Metric name: must match a key in the measurement command's JSON output"
check:
type: string
description: "Comparison operator and threshold. Supported operators: >=, <=, >, <, ==, !="
example: "<= 0.10"
optional_children:
description:
type: string
description: "Human-readable explanation of what this gate catches"
optional_children:
objectives:
type: array
default: []
description: >
Additional required or secondary hard targets. The primary is always
a required comparison, even when this list omits it. An experiment is
eligible when it improves at least one required objective beyond the
comparison threshold and does not violate any other required
objective. Completion via target_reached requires every declared
required target. A spec without metric.objectives keeps single-primary
acceptance: the primary is the only required objective.
items:
type: object
required_children:
name:
type: string
description: "Metric name: must match a key in the measurement command's JSON output"
direction:
type: enum
values:
- maximize
- minimize
optional_children:
role:
type: enum
values:
- required
- secondary
default: required
description: "required objectives participate in eligibility and completion; secondary objectives are logged only"
type:
type: enum
values:
- hard
default: hard
description: "Always hard. Additional objectives come from the measurement command."
target:
type: number
default: null
description: "Declared success value for this objective. target_reached requires every required objective that sets this."
max_regression:
type: object
description: "Optional explicit violation bound. When set, violation uses this bound rather than the comparison threshold, including when the bound is looser. When unset, a comparison-threshold regression is a violation."
required_children:
type:
type: enum
values:
- relative
- absolute
value:
type: number
diagnostics:
type: array
default: []
description: "Metrics logged for understanding but never gated on. Useful for understanding WHY a primary metric changed."
items:
type: object
required_children:
name:
type: string
description: "Metric name: must match a key in the measurement command's JSON output"
judge:
type: object
description: "LLM-as-judge configuration. Required when metric.primary.type is 'judge'. Ignored when type is 'hard'."
required_when: "metric.primary.type == 'judge'"
required_children:
rubric:
type: string
description: "Multi-line rubric text sent to the judge model. Must instruct the judge to return JSON."
example: |
Rate this cluster 1-5:
- 5: All items clearly about the same issue/feature
- 4: Strong theme, minor outliers
- 3: Related but covers 2-3 sub-topics
- 2: Weak connection
- 1: Unrelated items grouped together
scoring:
type: object
required_children:
primary:
type: string
description: "Field name from judge JSON output to use as the primary optimization target"
example: "mean_score"
optional_children:
secondary:
type: array
default: []
description: "Additional scoring fields to log (not optimized against)"
optional_children:
model:
type: enum
values:
- haiku
- sonnet
default: haiku
description: "Model to use for judge evaluation. Haiku is cheaper and faster; Sonnet is more nuanced."
sample_size:
type: integer
default: 10
description: "Total number of output items to sample for judge evaluation per experiment"
stratification:
type: array
default: null
description: "Stratified sampling buckets. If null, uses uniform random sampling."
items:
type: object
required_children:
bucket:
type: string
description: "Bucket name for this stratum"
count:
type: integer
description: "Number of items to sample from this bucket"
singleton_sample:
type: integer
default: 0
description: "Number of singleton items to sample for false-negative evaluation"
singleton_rubric:
type: string
default: null
description: "Rubric for evaluating sampled singletons. Required if singleton_sample > 0."
sample_seed:
type: integer
default: 42
description: "Fixed seed for reproducible sampling across experiments"
batch_size:
type: integer
default: 5
description: "Number of samples per judge sub-agent batch. Controls parallelism vs overhead."
minimum_improvement:
type: number
default: 0.3
description: "Minimum judge score improvement required to accept an experiment as 'better'. Accounts for sample-composition variance when output structure changes between experiments. Distinct from measurement.stability.noise_threshold which handles run-to-run flakiness."
max_total_cost_usd:
type: number
default: 5
description: "Stop judge evaluation when cumulative judge spend reaches this cap. This is a first-run safety default; raise it only after the rubric and harness are trustworthy. Set to null only with explicit user approval."
measurement:
type: object
description: "How to run the measurement harness"
required_children:
command:
type: string
description: "Shell command that runs the evaluation and outputs JSON to stdout. The JSON must contain keys matching all gate names and diagnostic names."
example: "python evaluate.py"
optional_children:
timeout_seconds:
type: integer
default: 600
description: "Maximum seconds for the measurement command to run before being killed"
output_format:
type: enum
values:
- json
default: json
description: "Format of the measurement command's stdout. Currently only JSON is supported."
working_directory:
type: string
default: "."
description: "Working directory for the measurement command, relative to the repo root"
stability:
type: object
default: { mode: "stable" }
description: "How to handle metric variance across runs"
required_children:
mode:
type: enum
values:
- stable # run once, trust the result
- repeat # run N times, aggregate
- ladder # smoke, one paired exploratory sample, adaptive extras, full protocol only before keep
default: stable
optional_children:
repeat_count:
type: integer
default: 5
description: "Number of times to run the harness when mode is 'repeat', and the confirmation sample count when mode is 'ladder'"
aggregation:
type: enum
values:
- median
- mean
- min
- max
default: median
description: "How to combine repeated measurements into a single value"
noise_threshold:
type: number
default: 0.02
description: "Absolute improvement that must be exceeded when comparison.method is absolute (the default). Applied to hard metrics only."
comparison:
type: object
default: { method: "absolute" }
description: "How to decide keep / revert / inconclusive. Unset method preserves the legacy absolute noise_threshold rule."
required_children:
method:
type: enum
values:
- absolute
- relative
- paired
default: absolute
description: "absolute uses noise_threshold; relative uses relative_threshold; paired uses sample ranges / paired diffs and may return inconclusive"
optional_children:
relative_threshold:
type: number
default: 0.05
description: "Fraction of the current-best value that counts as a real change when method is relative or paired"
ladder:
type: object
description: "Cost-aware measurement stages. Used when mode is ladder; ignored otherwise."
optional_children:
smoke_command:
type: string
default: null
description: "Cheap correctness check. Failure is degenerate and skips timed measurement."
exploratory_pairs:
type: integer
default: 1
description: "Paired exploratory samples before deciding whether to spend more. Must be a positive integer."
confirmation_repeats:
type: integer
default: null
description: "Full protocol sample count before keep. Defaults to repeat_count. Must be a positive integer at least as large as exploratory_pairs."
futility:
type: object
description: "Predeclared abort bound for a clearly noncompetitive live run"
optional_children:
worse_factor:
type: number
default: 1.2
description: "Censor after the first sample if the candidate is this multiple worse than the current best (minimize: candidate >= best * factor)"
after_elapsed_seconds:
type: number
default: null
description: "Censor a still-noncompetitive live run once elapsed time reaches this bound. When set, must be a positive number."
scope:
type: object
description: "What the experiment agent is allowed to modify"
required_children:
mutable:
type: array
description: "Files and directories the agent MAY modify during experiments"
items:
type: string
description: "File path or directory (relative to repo root). Directories match all files within."
example:
- "src/clustering/"
- "src/preprocessing/"
- "config/clustering.yaml"
immutable:
type: array
description: "Files and directories the agent MUST NOT modify. The measurement harness should always be listed here."
items:
type: string
example:
- "evaluate.py"
- "tests/fixtures/"
- "data/"
# ============================================================================
# OPTIONAL FIELDS
# ============================================================================
optional_fields:
execution:
type: object
default: { mode: "parallel", backend: "worktree", max_concurrent: 4 }
description: "How experiments are executed"
optional_children:
mode:
type: enum
values:
- parallel # run experiments simultaneously (default)
- serial # run one at a time
default: parallel
backend:
type: enum
values:
- worktree # git worktrees for isolation (default)
- codex # Codex sandboxes for isolation
default: worktree
max_concurrent:
type: integer
default: 4
minimum: 1
description: "Maximum experiments to run in parallel. Capped at 6 for worktree backend. 8+ only valid for Codex backend."
codex_security:
type: enum
values:
- full-auto # --full-auto (workspace write)
- yolo # --dangerously-bypass-approvals-and-sandbox
default: null
description: "Codex security posture. If null, user is asked once per session."
parallel:
type: object
default: {}
description: "Parallelism configuration discovered or set during Phase 1"
optional_children:
port_strategy:
type: enum
values:
- parameterized # use env var for port
- none # no port parameterization needed
default: null
description: "If null, auto-detected during Phase 1 parallelism probe"
port_env_var:
type: string
default: null
description: "Environment variable name for port parameterization (e.g., EVAL_PORT)"
port_base:
type: integer
default: null
description: "Base port number. Each experiment gets port_base + experiment_index."
shared_files:
type: array
default: []
description: "Files that must be copied into each experiment worktree (e.g., SQLite databases)"
items:
type: string
exclusive_resources:
type: array
default: []
description: "Resources requiring exclusive access (e.g., 'gpu'). If non-empty, forces serial mode."
items:
type: string
dependencies:
type: object
default: { approved: [] }
description: "Dependency management for experiments"
optional_children:
approved:
type: array
default: []
description: "Pre-approved new dependencies that experiments may add"
items:
type: string
constraints:
type: array
default: []
description: "Free-text constraints that experiment agents must follow"
items:
type: string
example:
- "Do not change the output format of clusters"
- "Preserve backward compatibility with existing cluster consumers"
stopping:
type: object
default: { max_iterations: 100, max_hours: 8, plateau_iterations: 10, target_reached: true }
description: "When the optimization loop should stop. Any criterion can trigger a stop."
optional_children:
max_iterations:
type: integer
default: 100
description: "Stop after this many total experiments"
max_hours:
type: number
default: 8
description: "Stop after this many hours of wall-clock time"
plateau_iterations:
type: integer
default: 10
description: "Stop if no improvement for this many consecutive experiments"
target_reached:
type: boolean
default: true
description: "Stop when the primary metric reaches the target value (if set)"
max_runner_up_merges_per_batch:
type: integer
default: 1
description: "Maximum number of file-disjoint runner-up experiments to attempt merging per batch after keeping the best experiment"
# ============================================================================
# VALIDATION RULES
# ============================================================================
validation_rules:
- "All required fields must be present"
- "name must be lowercase kebab-case (`^[a-z0-9]+(?:-[a-z0-9]+)*$`)"
- "metric.primary.type must be 'hard' or 'judge'"
- "If metric.primary.type is 'judge', metric.judge must be present with rubric and scoring"
- "metric.degenerate_gates must have at least one entry"
- "measurement.command must be a non-empty string"
- "scope.mutable must have at least one entry"
- "scope.immutable must have at least one entry"
- "Gate check operators must be one of: >=, <=, >, <, ==, !="
- "execution.max_concurrent must be >= 1"
- "execution.max_concurrent must not exceed 6 when execution.backend is 'worktree'"
- "If parallel.exclusive_resources is non-empty, execution.mode should be 'serial'"
- "If metric.judge.singleton_sample > 0, metric.judge.singleton_rubric must be present"
- "If metric.primary.type is 'judge' and metric.judge.max_total_cost_usd is null, the user should explicitly approve uncapped spend"
- "stopping must have at least one non-default criterion or use defaults"
- "If metric.objectives is set, each additional objective is a hard metric whose name is unique and matches a measurement JSON key"
- "The primary is always a required comparison, including when metric.objectives omits it"
- "If measurement.stability.mode is 'ladder', comparison.method should be relative or paired unless the user explicitly keeps absolute"
- "A spec without metric.objectives keeps single-primary acceptance"
- "comparison.method, when set, must be one of: absolute, relative, paired"
- "If measurement.stability.mode is 'ladder', exploratory_pairs and confirmation_repeats (or repeat_count) must be positive integers, and confirmation must be at least exploratory_pairs"
- "futility.after_elapsed_seconds, when set, must be a positive number"
references/persistence.md
# Persistence: the rules, the checkpoints, and resume
Read this before Phase 0 and follow it for the whole run. The body states the invariant and names the six checkpoints; this file carries the rules that implement them, the checkpoint table, the file layout, and the resume procedure.
### Core Rules
1. **Write each experiment result to disk IMMEDIATELY after measurement**: not after the batch, not after evaluation, IMMEDIATELY. Append the experiment entry to the experiment log file the moment its metrics are known, before evaluating the next experiment. This is the #1 crash-safety rule.
2. **VERIFY every critical write**: after writing the experiment log, read the file back and confirm the entry is present. This catches silent write failures. Do not proceed to the next experiment until verification passes.
3. **Re-read from disk at every phase boundary and before every decision**: never trust in-memory state across phase transitions, batch boundaries, or after any operation that might have taken significant time. Re-read the experiment log and strategy digest from disk.
4. **One experiment, one log entry.** Append a new experiment entry on its first measurement. Later ladder samples for that same experiment update that entry's metrics and outcome in place so a crash can resume the ladder without losing samples or duplicating the hypothesis. Distinct `comparisons` pairings accumulate on that same entry; in-place updates must not replace a previously persisted distinct pairing. Never rewrite a different experiment's samples or gate values. Outcome, `best`, and `hypothesis_backlog` are also updated in place at batch evaluation (CP-4). Do not rebuild the file from memory.
5. **Per-experiment result markers for crash recovery**: each experiment writes a `result.yaml` marker in its worktree immediately after measurement. On resume, scan for these markers to recover experiments that were measured but not yet logged.
6. **Strategy digest is written after every batch, before generating new hypotheses**: the agent reads the digest (not its memory) when deciding what to try next.
7. **Never present results to the user without writing them to disk first**: the pattern is: measure -> write to disk -> verify -> THEN show the user. Not the reverse.
### Mandatory Disk Checkpoints
These are non-negotiable write-then-verify steps. At each checkpoint, the agent MUST write the specified file and then read it back to confirm the write succeeded.
| Checkpoint | File Written | Phase |
|---|---|---|
| CP-0: Spec saved | `spec.yaml` | Phase 0, after user approval |
| CP-1: Baseline recorded | `experiment-log.yaml` (initial with baseline) | Phase 1, after baseline measurement |
| CP-2: Hypothesis backlog saved | `experiment-log.yaml` (hypothesis_backlog section) | Phase 2, after hypothesis generation |
| CP-3: Each experiment result | `experiment-log.yaml` (append on first measurement; update that entry on later samples) | Phase 3.3, immediately after each measurement |
| CP-4: Batch summary | `experiment-log.yaml` (outcomes + best) + `strategy-digest.md` | Phase 3.5, after batch evaluation |
| CP-5: Final summary | `experiment-log.yaml` (final state) | Phase 4, at wrap-up |
**Format of a verification step:**
1. Write the file using the native file-write tool
2. Read the file back using the native file-read tool
3. Confirm the expected content is present
4. If verification fails, retry the write. If it fails twice, alert the user.
### File Locations (all under `.context/compound-engineering/ce-optimize/<spec-name>/`)
The scratch space under `.context/` is gitignored: it survives a local resume but does not travel with the branch, so anything needed durably must be exported to a tracked path.
| File | Purpose | Written When |
|------|---------|-------------|
| `spec.yaml` | Optimization spec (fixed once the Phase 1 approval gate is cleared) | Phase 0 (CP-0) |
| `experiment-log.yaml` | Full history of all experiments | Initialized at CP-1, appended at first CP-3, updated on later samples and at CP-4 |
| `strategy-digest.md` | Compressed learnings for hypothesis generation | Written at CP-4 after each batch |
| `<worktree>/result.yaml` | Per-experiment crash-recovery marker | Immediately after measurement, before CP-3 |
### On Resume
When Phase 0.4 detects an existing run:
1. Read the experiment log from disk: this is the ground truth
2. Scan worktree directories for `result.yaml` markers not yet in the log
3. Recover any measured-but-unlogged experiments. The recovered first CP-3 entry copies `opportunity` from the hypothesis backlog as of dispatch; `result.yaml` holds metrics only, so a missing forecast stays unrecorded rather than being reconstructed from the result
4. Continue as the body's resume rule directs: skip the work the log proves finished, and re-enter any gate the log does not prove was cleared
---
references/spec.md
# Phase 0: input type and the optimization spec
Read this at the start of Phase 0. It carries how to tell a spec path from a goal description, and how to load, build, or validate the spec before CP-0. The schemas it validates against are `references/optimize-spec-schema.yaml` and `references/experiment-log-schema.yaml`.
### 0.1 Determine Input Type
Check whether the input is:
- **A spec file path** (ends in `.yaml` or `.yml`): read and validate it
- **A description of the optimization goal**: help the user create a spec interactively
### 0.2 Load or Create Spec
**If spec file provided:**
1. Read the YAML spec file. The orchestrating agent parses YAML natively -- no shell script parsing.
2. Validate the spec against **every** rule in the `validation_rules` section of `references/optimize-spec-schema.yaml` (that section is the single source of truth for what a valid spec requires: do not rely on a remembered subset; conditional rules such as the singleton-rubric and exclusive-resources requirements live only there).
3. If any rule fails, report the specific failures and ask the user to fix them before proceeding
**If description provided:**
1. Analyze the project to understand what can be measured. `references/usage-guide.md` has longer kickoff prompt shapes if the interview needs them.
2. **Detect whether the optimization target is qualitative or quantitative**: this determines `type: hard` vs `type: judge` and is the single most important spec decision:
**Use `type: hard`** when:
- The metric is a scalar number with a clear "better" direction
- The metric is objectively measurable (build time, test pass rate, latency, memory usage)
- No human judgment is needed to evaluate "is this result actually good?"
- Examples: reduce build time, increase test coverage, reduce API latency, decrease bundle size
If the user names more than one hard success condition that must all hold (local wall time and CI critical path and runner-minutes, for example), put them in `metric.objectives` as `role: required` and keep `metric.primary` as the ranking key. A spec without `metric.objectives` keeps single-primary acceptance. If each evaluation costs minutes, set `measurement.stability.mode: ladder` with a relative or paired comparison and a futility bound; do not spend the full confirmation protocol on every exploratory experiment. Start from `references/example-expensive-benchmark-spec.yaml` for that shape.
**Use `type: judge`** when:
- The quality of the output requires semantic understanding to evaluate
- A human reviewer would need to look at the results to say "this is better"
- Proxy metrics exist but can mislead (e.g., "more clusters" does not mean "better clusters")
- The optimization could produce degenerate solutions that look good on paper
- Examples: clustering quality, search relevance, summarization quality, code readability, UX copy, recommendation relevance
**IMPORTANT**: If the target is qualitative, **strongly recommend `type: judge`**. Explain that hard metrics alone will optimize proxy numbers without checking actual quality. Show the user the three-tier approach:
- **Degenerate gates** (hard, cheap, fast): catch obviously broken solutions: e.g., "all items in 1 cluster" or "0% coverage". Run first. If gates fail, skip the expensive judge step.
- **LLM-as-judge** (the actual optimization target): sample outputs, score them against a rubric, aggregate. This is what the loop optimizes.
- **Diagnostics** (logged, not gated): distribution stats, counts, timing: useful for understanding WHY a judge score changed.
If the user insists on `type: hard` for a qualitative target, proceed but warn that the results may optimize a misleading proxy.
3. **Design the sampling strategy** (for `type: judge`):
Guide the user through defining stratified sampling. The key question is: "What parts of the output space do you need to check quality on?"
Walk through these questions:
- **What does one "item" look like?** (a cluster, a search result page, a summary, etc.)
- **What are the natural size/quality strata?** (e.g., large clusters vs small clusters vs singletons)
- **Where are quality failures most likely?** (e.g., very large clusters may be degenerate merges; singletons may be missed groupings)
- **What total sample size balances cost vs signal?** (default: 30 items, adjust based on output volume)
Example stratified sampling for clustering:
```yaml
stratification:
- bucket: "top_by_size" # largest clusters: check for degenerate mega-clusters
count: 10
- bucket: "mid_range" # middle of non-solo cluster size range: representative quality
count: 10
- bucket: "small_clusters" # clusters with 2-3 items: check if connections are real
count: 10
singleton_sample: 15 # singletons: check for false negatives (items that should cluster)
```
The sampling strategy is domain-specific. For search relevance, strata might be "top-3 results", "results 4-10", "tail results". For summarization, strata might be "short documents", "long documents", "multi-topic documents".
**Singleton evaluation is critical when the goal involves coverage**: sampling singletons with the singleton rubric checks whether the system is missing obvious groupings.
4. **Design the rubric** (for `type: judge`):
Help the user define the scoring rubric. A good rubric:
- Has a 1-5 scale (or similar) with concrete descriptions for each level
- Includes supplementary fields that help diagnose issues (e.g., `distinct_topics`, `outlier_count`)
- Is specific enough that two judges would give similar scores
- Does NOT assume bigger/more is better: "3 items per cluster average" is not inherently good or bad
Example for clustering:
```yaml
rubric: |
Rate this cluster 1-5:
- 5: All items clearly about the same issue/feature
- 4: Strong theme, minor outliers
- 3: Related but covers 2-3 sub-topics that could reasonably be split
- 2: Weak connection: items share superficial similarity only
- 1: Unrelated items grouped together
Also report: distinct_topics (integer), outlier_count (integer)
```
5. Guide the user through the remaining spec fields:
- What degenerate cases should be rejected? (gates: e.g., "solo_pct <= 0.95" catches all-singletons, "max_cluster_size <= 500" catches mega-clusters)
- What command runs the measurement?
- What files can be modified? What is immutable?
- Any constraints or dependencies?
- If this is the first run: recommend `execution.mode: serial`, `execution.max_concurrent: 1`, `stopping.max_iterations: 4`, and `stopping.max_hours: 1`
- If the user named multiple required hard targets or an expensive harness: recommend `metric.objectives` plus `stability.mode: ladder` as above, and show `references/example-expensive-benchmark-spec.yaml`
- If `type: judge`: recommend `sample_size: 10`, `batch_size: 5`, and `max_total_cost_usd: 5` until the rubric and harness are trusted
6. Write the spec to `.context/compound-engineering/ce-optimize/<spec-name>/spec.yaml`
7. Present the spec to the user for approval before proceeding
references/usage-guide.md
# `ce-optimize` Usage Guide
## What This Skill Is For
The `ce-optimize` skill is for hard engineering problems where:
1. You can measure the same target twice.
2. You can either attribute a named-workload cost or try multiple scored variants.
3. You want the skill to keep confirmed improvements and reject the rest.
On a cost target, the first useful action is often a locating measurement, not a batch of implementation experiments. On a scored variant space, the skill searches and keeps. It is not one-shot implementation of a change you already know.
## When To Use It
Reach for `ce-optimize` when the problem looks like:
- "Find the smallest memory limit that stops OOM crashes without wasting RAM."
- "Tune clustering parameters without collapsing everything into one garbage cluster."
- "Find a prompt that is cheaper but still produces summaries good enough for downstream clustering."
- "Compare several ranking, retrieval, batching, or threshold strategies against the same harness."
Choose `type: hard` when success is objective and cheap to measure:
- Memory usage
- Latency
- Throughput
- Test pass rate
- Build time
Choose `type: judge` when a numeric metric can be gamed or when human usefulness matters:
- Cluster coherence
- Search relevance
- Summary quality
- Prompt quality
- Classification quality with semantic edge cases
## When Not To Use It
`ce-optimize` is usually the wrong tool when:
- The change is already known: make it, or use `ce-work`
- The job is diagnosing failing or slow behavior: that is `ce-debug`
- There is no repeatable measurement harness
- The search space is fake and only has one plausible answer
- The cost of evaluating variants is too high to justify multiple runs
## How To Think About It
The pattern is:
1. Define the target.
2. Build or validate the measurement harness first.
3. Take the cheapest next action that would change what gets implemented: a locating measurement on a cost target, or a scored variant on a search target.
4. Keep confirmed improvements and reject the rest.
The core rule is simple:
- If a hard metric captures "better," optimize the hard metric.
- If a hard metric can be gamed, add LLM-as-judge.
Example: lowering a clustering threshold may increase cluster coverage. That sounds good until everything ends up in one giant cluster. Hard metrics may say "improved"; an LLM judge sampling real clusters can say "this is trash."
## First-Run Advice
For the first run:
- Prefer `execution.mode: serial`
- Set `execution.max_concurrent: 1`
- Keep `stopping.max_iterations` small
- Keep `stopping.max_hours` small
- Avoid new dependencies until the baseline is trustworthy
- In judge mode, use a small sample and a low cost cap
The goal of the first run is to validate the harness, not to win the optimization immediately.
## Example Prompts
### 1. Memory Tuning
```text
Run the `ce-optimize` skill to find the smallest memory setting that keeps this service stable under our load test.
The current container limit is 512 MB and the app sometimes OOM-crashes. Do not just jump to 8 GB. Try a small set of realistic memory limits, run the same load test for each one, and score the results using:
- did the process OOM
- did tail latency spike badly
- did GC pauses become excessive
Prefer the smallest memory limit that passes the guard rails.
```
### 2. Clustering Quality
```text
Run the `ce-optimize` skill to improve issue and PR clustering quality.
We have about 18k open issues and PRs. We want to test changes that improve clustering quality, reduce singleton clusters, and improve match quality within each cluster.
Do not mutate the shared default database. Copy it for the run, then use per-experiment copies when needed.
Do not optimize only for coverage. Use LLM-as-judge to sample clusters and confirm they still preserve real semantic similarity instead of collapsing into giant low-quality clusters.
```
### 3. Expensive Test Suite
```text
Run the `ce-optimize` skill to reduce this repository's full test-suite wall time without making CI slower or spending more runner-minutes.
Local warm median is currently about six minutes and the range is wide. Treat local wall time, CI critical path, and aggregate runner-minutes as required targets: a change that helps only CI may be kept if it does not regress the others, and the run is not done until every declared target is met.
Do not spend a five-run cold/warm protocol on every exploratory experiment. Smoke for correctness, take one paired sample, abort anything already far worse than the current best, and reserve the full protocol for a candidate you are about to keep and for final confirmation.
```
### 4. Prompt Optimization
```text
Run the `ce-optimize` skill to create a summarization prompt for issues and PRs that minimizes token spend while still producing summaries that are good enough for downstream clustering.
I want the loop to compare prompt variants, measure token cost, and judge whether the summaries preserve the distinctions needed to cluster related issues together without merging unrelated ones.
```
## Choosing Between Hard Metrics And Judge Mode
Use hard metrics alone when:
- "Better" is obvious from the numbers.
Add judge mode when:
- The numbers can improve while the real output gets worse.
Common pattern:
- Hard gates reject broken outputs.
- Judge mode scores the surviving candidates for actual usefulness.
That hybrid setup is often the best default for ranking, clustering, and prompt work.
## First-run defaults
A first run optimizes for signal and safety, not throughput:
- Start from `references/example-hard-spec.yaml` when the metric is objective and cheap to measure; use `references/example-judge-spec.yaml` only when quality genuinely requires semantic judgment; use `references/example-expensive-benchmark-spec.yaml` when each run costs minutes or several hard targets must all hold.
- Prefer `execution.mode: serial` with `execution.max_concurrent: 1`.
- Cap the run with `stopping.max_iterations: 4` and `stopping.max_hours: 1`.
- Add no new dependencies until the baseline and measurement harness are trusted.
- For judge mode, start at `sample_size: 10`, `batch_size: 5`, and `max_total_cost_usd: 5`.
references/wrap-up.md
# Phase 4: wrap-up
Read this at wrap-up. The body owns the post-completion options the user chooses from; this file carries what each one needs: the deferred-hypothesis presentation, the results summary, what is preserved and what is not, the mechanical-apply bar for review findings, and the cleanup rules.
## Phase 4: Wrap-Up
### 4.1 Present Deferred Hypotheses
If any hypotheses were deferred due to unapproved dependencies:
1. List them with their dependency requirements
2. Ask the user whether to approve, skip, or save for a future run
3. If approved: add to backlog and offer to re-enter Phase 3 for one more round
### 4.2 Summarize Results
Report from the persisted forecasts and measurements, with the final state confirmed using the configured measurement protocol. If confirmation is unavailable, label the final values unconfirmed and state why. A legacy log remains reportable: missing forecasts, comparison baselines, or uncertainty stay unrecorded rather than being reconstructed from the final result.
The summary must contain:
- **Overall result:** original baseline -> final for every required objective (the primary when no objectives are declared), with units, absolute change, target status, and percentage change where meaningful. A zero baseline has no defined percentage change; an ordinal judge score is reported in score points, not as a percentage improvement.
- **Opportunity -> result:** each retained change's original expected benefit beside its measured before/after result, the comparison identity and workload, and whether the evidence supports the estimate. Identify standalone versus integrated results. If the forecast and result use different baselines or workloads, label them non-comparable instead of declaring that the forecast was met or missed.
- **Evidence quality:** measurement uncertainty and confirmation status, correctness checks and their results, and any unverified constraints. Report measured incremental contributions only when the corresponding reference measurements exist. Do not add percentages from successive changes or count a standalone runner-up gain as its integrated contribution; the overall gain comes from original-to-final measurement.
- **Remaining opportunity:** what still costs time or resources, with current evidence and whether further work appears worthwhile. Without fresh evidence, label remaining estimates stale or unknown; do not claim a new bottleneck from the old profile alone.
- **Run accounting:** stopping reason, duration, outcome counts, judge cost when applicable, and the log path. Preserve the existing outcome distinctions, including `Not selected: <count>`, inconclusive, censored, deferred, errors, and timeouts. Short reports may omit individual rejected experiments, but retain required-objective results and evidence limitations.
### 4.3 Preserve and Offer Next Steps
The optimization branch (`optimize/<spec-name>`) is preserved with all commits from kept experiments.
The experiment log and strategy digest remain in local `.context/...` scratch space for resume and audit on this machine only; they do not travel with the branch because `.context/` is gitignored.
Present these options after the summary:
1. **Run `ce-code-review`** on the cumulative diff (baseline to final), on the optimization branch. Do not commit or push from this step.
2. **Run `ce-compound`** to document the winning strategy as an institutional learning.
3. **Create PR** from the optimization branch to the default branch.
4. **Continue**: re-enter Phase 3, state re-read first.
5. **Done**: leave the branch for manual review.
For option 1, load `ce-code-review` on the optimization branch, interactive or `mode:agent`, and land eligible fixes under the bar below before moving to the next option.
**Mechanical-apply bar:** apply any finding with a concrete `suggested_fix` that is a clear, reversible improvement; push back (keep, don't apply) when the reviewer is wrong, noting why. Defer anything whose right fix needs a design or product decision (architecture direction, contract shape, behavior change needing sign-off) and any finding with no concrete fix to act on: surface what was deferred. Confirm evidence still matches at `file:line` before editing. After applying, run tests (at least targeted tests for what changed; broader suite for multi-file edits). Do not commit or push from this step: leave the diff on the optimization branch for the Create PR option.
Option 4 (continue) re-enters Phase 3 with the current state, state re-read from disk first.
### 4.4 Cleanup
Clean up scratch space:
```bash
# Keep the experiment log for local resume/audit on this machine
# Remove temporary batch artifacts
rm -f .context/compound-engineering/ce-optimize/<spec-name>/strategy-digest.md
```
Do NOT delete the experiment log if the user may resume locally or wants a local audit trail. If they need a durable shared artifact, summarize or export the results into a tracked path before cleanup.
Do NOT delete experiment worktrees that are still being referenced.
scripts/decide.mjs
#!/usr/bin/env node
// Decide keep / revert / inconclusive / censored / degenerate for one
// candidate against the current best. Owns multi-objective eligibility,
// noise-aware comparison, and the measurement-ladder next step.
//
// Usage:
// node decide.mjs # JSON on stdin
// node decide.mjs <input.json>
//
// A spec with no `objectives` and no `ladder` reproduces the legacy
// single-primary + absolute noise_threshold rule, except that a delta
// inside the threshold is `inconclusive` rather than a silent revert.
export function median(values) {
if (!values.length) return null
const sorted = [...values].sort((a, b) => a - b)
const mid = Math.floor(sorted.length / 2)
return sorted.length % 2 === 0 ? (sorted[mid - 1] + sorted[mid]) / 2 : sorted[mid]
}
function parseCheck(check) {
const match = String(check).trim().match(/^(>=|<=|==|!=|>|<)\s*(.+)$/)
if (!match) {
throw new Error(`invalid gate check: ${check}`)
}
return { op: match[1], threshold: Number(match[2]) }
}
export function gatePasses(value, check) {
const { op, threshold } = parseCheck(check)
const n = finiteNumber(value)
if (n == null || !Number.isFinite(threshold)) return false
switch (op) {
case ">=":
return n >= threshold
case "<=":
return n <= threshold
case ">":
return n > threshold
case "<":
return n < threshold
case "==":
return n === threshold
case "!=":
return n !== threshold
default:
return false
}
}
function signedDelta(baseline, candidate, direction) {
return direction === "minimize" ? baseline - candidate : candidate - baseline
}
function finiteNumber(value) {
if (value == null || value === "") return null
const n = typeof value === "number" ? value : Number(value)
return Number.isFinite(n) ? n : null
}
function firstNonNegativeNumber(values, fallback) {
for (const value of values) {
const n = finiteNumber(value)
if (n != null && n >= 0) return n
}
return fallback
}
function positiveInteger(value, fallback) {
const n = typeof value === "number" ? value : Number(value)
if (!Number.isInteger(n) || n < 1) return fallback
return n
}
function verdictFromSigned(delta, threshold) {
if (delta > threshold) return "improved"
if (delta < -threshold) return "regressed"
return "inconclusive"
}
function closedResult(fields) {
return {
eligible: false,
next_measurement: "none",
target_reached: false,
improved_objectives: [],
violated_objectives: [],
comparisons: {},
primary_delta: null,
rank_score: 0,
...fields,
}
}
function configuredAggregation(value) {
return value === "mean" || value === "min" || value === "max" ? value : "median"
}
function aggregateSamples(samples, method) {
if (!samples.length) return null
if (method === "mean") return samples.reduce((sum, value) => sum + value, 0) / samples.length
if (method === "min") return Math.min(...samples)
if (method === "max") return Math.max(...samples)
return median(samples)
}
function valueBundle(raw, aggregation = "median") {
if (raw && typeof raw === "object" && !Array.isArray(raw)) {
const samples = Array.isArray(raw.samples) ? raw.samples.map(finiteNumber) : []
if (samples.some((n) => n == null)) return { aggregate: null, samples: [] }
const aggregate = samples.length
? aggregateSamples(samples, aggregation)
: finiteNumber(raw.aggregate)
return { aggregate: aggregate ?? null, samples }
}
if (raw != null && typeof raw !== "object") {
const n = finiteNumber(raw)
return n == null ? { aggregate: null, samples: [] } : { aggregate: n, samples: [] }
}
return null
}
function metricBundle(source, name, aggregation, type) {
if (!source) return null
const containers =
type === "judge"
? [source.judge?.[name], source.metrics?.[name], source.diagnostics?.[name], source.gates?.[name]]
: [source.metrics?.[name], source.judge?.[name], source.diagnostics?.[name], source.gates?.[name]]
for (const raw of containers) {
if (raw === undefined) continue
return valueBundle(raw, aggregation)
}
return null
}
function normalizeSpec(spec) {
const metric = spec.metric ?? {}
const measurement = spec.measurement ?? {}
const stability = spec.stability ?? measurement.stability ?? {}
const primary = spec.primary ?? metric.primary ?? {}
const judge = spec.judge ?? metric.judge
const comparison = spec.comparison ?? stability.comparison
return {
...spec,
primary,
objectives: spec.objectives ?? metric.objectives,
degenerate_gates: spec.degenerate_gates ?? metric.degenerate_gates,
judge,
comparison,
ladder: spec.ladder ?? stability.ladder ?? {},
stability_mode: spec.stability_mode ?? stability.mode,
aggregation: configuredAggregation(spec.aggregation ?? stability.aggregation),
repeat_count: spec.repeat_count ?? stability.repeat_count,
noise_threshold: spec.noise_threshold ?? stability.noise_threshold,
minimum_improvement:
spec.minimum_improvement ?? comparison?.minimum_improvement ?? judge?.minimum_improvement,
measurement,
stability,
}
}
function comparisonDefaults(spec) {
const stability = spec.stability ?? spec.measurement?.stability ?? {}
const comparison = spec.comparison ?? stability.comparison ?? {}
const usesJudge = spec.primary?.type === "judge" || spec.judge != null
return {
method: comparison.method ?? "absolute",
noise_threshold: firstNonNegativeNumber(
[comparison.noise_threshold, spec.noise_threshold, stability.noise_threshold],
0.02,
),
relative_threshold: firstNonNegativeNumber([comparison.relative_threshold], 0.05),
minimum_improvement: firstNonNegativeNumber(
[comparison.minimum_improvement, spec.minimum_improvement, spec.judge?.minimum_improvement],
usesJudge ? 0.3 : null,
),
}
}
function requiredObjectives(spec) {
const primary = spec.primary ?? {}
const listed = Array.isArray(spec.objectives) ? spec.objectives : []
const extras = listed
.map((objective) => ({
name: objective.name,
direction: objective.direction ?? primary.direction ?? "maximize",
role: objective.role ?? "required",
type: objective.type ?? "hard",
target: objective.target ?? null,
max_regression: objective.max_regression ?? null,
}))
.filter((objective) => objective.role !== "secondary")
if (!primary.name) return extras
const listedPrimary = extras.find((objective) => objective.name === primary.name)
return [
{
name: primary.name,
direction: primary.direction ?? listedPrimary?.direction ?? "maximize",
role: "required",
type: primary.type ?? listedPrimary?.type ?? "hard",
target: primary.target ?? listedPrimary?.target ?? null,
max_regression: listedPrimary?.max_regression ?? primary.max_regression ?? null,
},
...extras.filter((objective) => objective.name !== primary.name),
]
}
export function compareObjective({
baselineValue,
candidateValue,
baselineSamples,
candidateSamples,
direction,
type,
comparison,
maxRegression,
}) {
const delta = signedDelta(baselineValue, candidateValue, direction)
const denom = Math.abs(baselineValue)
const relative = denom > 0 ? delta / denom : 0
const absThreshold =
type === "judge" && comparison.minimum_improvement != null
? comparison.minimum_improvement
: comparison.noise_threshold
const relativeThreshold = comparison.relative_threshold
let verdict
if ((comparison.method === "relative" || comparison.method === "paired") && denom <= 0) {
verdict = "inconclusive"
} else if (comparison.method === "relative") {
verdict = verdictFromSigned(relative, relativeThreshold)
} else if (comparison.method === "paired") {
const baseSamples = baselineSamples?.filter((n) => n != null) ?? []
const candSamples = candidateSamples?.filter((n) => n != null) ?? []
const threshold = relativeThreshold * denom
if (!baseSamples.length || !candSamples.length) {
verdict =
type === "judge"
? verdictFromSigned(delta, absThreshold)
: verdictFromSigned(delta, threshold) === "regressed"
? "regressed"
: "inconclusive"
} else {
const diffs = candSamples.map((value, index) =>
signedDelta(baseSamples[Math.min(index, baseSamples.length - 1)], value, direction),
)
const lo = Math.min(...diffs)
const hi = Math.max(...diffs)
if (verdictFromSigned(lo, threshold) === "improved") verdict = "improved"
else if (verdictFromSigned(hi, threshold) === "regressed") verdict = "regressed"
else verdict = "inconclusive"
}
} else {
verdict = verdictFromSigned(delta, absThreshold)
}
if (
type === "judge" &&
comparison.minimum_improvement != null &&
verdict === "improved" &&
delta <= comparison.minimum_improvement
) {
verdict = "inconclusive"
}
let violated = verdict === "regressed"
if (maxRegression && verdict !== "improved") {
const rawBound = Number(maxRegression.value)
if (Number.isFinite(rawBound) && rawBound >= 0) {
const bound = maxRegression.type === "relative" ? rawBound * denom : rawBound
violated = -delta > bound
if (violated) verdict = "regressed"
}
}
return { verdict, delta, relative, violated }
}
function evaluateGates(spec, candidate) {
const gates = Array.isArray(spec.degenerate_gates) ? spec.degenerate_gates : []
const values = candidate?.gates ?? {}
const failures = []
for (const gate of gates) {
if (!gatePasses(values[gate.name], gate.check)) {
failures.push(gate.name)
}
}
return failures
}
function futilityBound(futility, baselineValue, direction) {
const factor = finiteNumber(futility.worse_factor ?? 1.2)
const baseline = finiteNumber(baselineValue)
if (factor == null || factor <= 1 || baseline == null || baseline <= 0) return null
return direction === "minimize" ? baseline * factor : baseline / factor
}
function isFutile({
ladder,
direction,
baselineValue,
candidateValue,
elapsedSeconds,
sampleCount,
enabled,
}) {
const futility = ladder.futility
if (!enabled || futility == null || typeof futility !== "object") return false
const afterElapsed = finiteNumber(futility.after_elapsed_seconds)
if (
afterElapsed != null &&
afterElapsed > 0 &&
elapsedSeconds != null &&
Number(elapsedSeconds) >= afterElapsed &&
signedDelta(baselineValue, candidateValue, direction) <= 0
) {
return true
}
const bound = futilityBound(futility, baselineValue, direction)
if (bound == null || candidateValue == null) return false
const worse = signedDelta(bound, candidateValue, direction) <= 0
return worse && (sampleCount ?? 1) <= (ladder.exploratory_pairs ?? 1)
}
function rankScore(primaryComparison, improved) {
if (primaryComparison?.verdict === "improved") return primaryComparison.relative ?? 0
if (!improved.length) return primaryComparison?.relative ?? 0
return Math.max(...improved.map((item) => item.relative ?? 0))
}
export function decide(input) {
const spec = normalizeSpec(input.spec ?? {})
const baseline = input.baseline ?? {}
const candidate = input.candidate ?? {}
const primary = spec.primary ?? {}
if (!primary.name) {
return closedResult({ decision: "error", reason: "missing primary metric" })
}
const comparison = comparisonDefaults(spec)
const required = requiredObjectives(spec)
const ladder = spec.ladder ?? {}
const ladderEnabled = Boolean(ladder.enabled || spec.stability_mode === "ladder")
const exploratoryPairs = positiveInteger(ladder.exploratory_pairs, 1)
let confirmationRepeats = positiveInteger(ladder.confirmation_repeats ?? spec.repeat_count, 5)
if (confirmationRepeats < exploratoryPairs) confirmationRepeats = exploratoryPairs
if (candidate.smoke_passed === false) {
return closedResult({ decision: "degenerate", reason: "smoke test failed" })
}
const gateFailures = evaluateGates(spec, candidate)
if (gateFailures.length) {
return closedResult({
decision: "degenerate",
violated_objectives: gateFailures,
reason: `degenerate gate failed: ${gateFailures.join(", ")}`,
})
}
const comparisons = {}
const improved = []
const violated = []
const missing = []
const candidateBundles = {}
const baselineBundles = {}
const aggregation = spec.aggregation ?? "median"
for (const objective of required) {
const base = metricBundle(baseline, objective.name, aggregation, objective.type)
const cand = metricBundle(candidate, objective.name, aggregation, objective.type)
baselineBundles[objective.name] = base
candidateBundles[objective.name] = cand
if (!base || base.aggregate == null || !cand || cand.aggregate == null) {
missing.push(objective.name)
continue
}
const result = compareObjective({
baselineValue: base.aggregate,
candidateValue: cand.aggregate,
baselineSamples: base.samples,
candidateSamples: cand.samples,
direction: objective.direction,
type: objective.type,
comparison,
maxRegression: objective.max_regression,
})
comparisons[objective.name] = result
if (result.verdict === "improved") improved.push({ name: objective.name, ...result })
if (result.violated) violated.push(objective.name)
}
if (missing.length) {
return closedResult({
decision: "error",
comparisons,
reason: `missing required metric: ${missing.join(", ")}`,
})
}
const primaryBundle =
candidateBundles[primary.name] ?? metricBundle(candidate, primary.name, aggregation, primary.type)
const baselinePrimary =
baselineBundles[primary.name] ?? metricBundle(baseline, primary.name, aggregation, primary.type)
const primaryComparison = comparisons[primary.name] ?? null
const eligible = improved.length > 0 && violated.length === 0
const stillContending = required.some(
(objective) => comparisons[objective.name]?.verdict === "inconclusive",
)
const sampleCount = Math.min(
...required.map((objective) => {
const fromSamples = candidateBundles[objective.name]?.samples?.length
if (Number.isInteger(fromSamples) && fromSamples > 0) return fromSamples
return positiveInteger(candidate.sample_count, 1)
}),
)
if (
!eligible &&
!stillContending &&
isFutile({
ladder,
direction: primary.direction,
baselineValue: baselinePrimary?.aggregate,
candidateValue: primaryBundle?.aggregate,
elapsedSeconds: candidate.elapsed_seconds,
sampleCount,
enabled: ladderEnabled,
})
) {
return closedResult({
decision: "censored",
improved_objectives: improved.map((item) => item.name),
violated_objectives: violated,
comparisons,
primary_delta: primaryComparison?.delta ?? null,
reason: "noncompetitive under the predeclared futility bound",
})
}
const withTargets = required.filter((objective) => objective.target != null)
const targetReached =
withTargets.length > 0 &&
withTargets.every((objective) => {
const value = candidateBundles[objective.name]?.aggregate
if (value == null) return false
return signedDelta(objective.target, value, objective.direction) >= 0
})
let decision
if (eligible) decision = "keep"
else if (violated.length) decision = "revert"
else if (stillContending) decision = "inconclusive"
else decision = "revert"
let nextMeasurement = "none"
if (ladderEnabled && ladder.smoke_command && candidate.smoke_passed == null) {
nextMeasurement = "smoke"
} else if (ladderEnabled && (decision === "keep" || decision === "inconclusive")) {
const confirming = decision === "keep"
const sampleBudget = confirming
? confirmationRepeats
: Math.min(exploratoryPairs + 1, confirmationRepeats)
if (sampleCount < sampleBudget) {
if (confirming) decision = "promising"
if (sampleCount < exploratoryPairs) nextMeasurement = "exploratory"
else nextMeasurement = confirming ? "confirm" : "add_sample"
}
}
let reason = "no required objective improved"
if (eligible) {
reason = `improved ${improved.map((item) => item.name).join(", ")} without violating other required objectives`
} else if (decision === "inconclusive") {
reason = "delta inside the comparison threshold"
} else if (violated.length) {
reason = `violated ${violated.join(", ")}`
}
return {
decision,
eligible,
next_measurement: nextMeasurement,
target_reached: targetReached,
improved_objectives: improved.map((item) => item.name),
violated_objectives: violated,
comparisons,
primary_delta: primaryComparison?.delta ?? null,
rank_score: rankScore(primaryComparison, improved),
reason,
}
}
function isCli(argv = process.argv) {
const entry = argv[1] ?? ""
return entry.endsWith("decide.mjs")
}
async function runCli(argv = process.argv, io = process) {
const { readFileSync } = await import("node:fs")
const source = argv[2] && argv[2] !== "-" ? argv[2] : 0
const input = JSON.parse(readFileSync(source, "utf8"))
io.stdout.write(`${JSON.stringify(decide(input), null, 2)}\n`)
}
if (isCli()) {
await runCli()
}
scripts/experiment-worktree.sh
#!/bin/bash
# Experiment Worktree Manager
# Creates, cleans up, and manages worktrees for optimization experiments.
# Each experiment gets an isolated worktree with copied shared resources.
#
# Usage:
# experiment-worktree.sh create <spec_name> <exp_index> <base_branch> [shared_file ...]
# experiment-worktree.sh cleanup <spec_name> <exp_index>
# experiment-worktree.sh cleanup-all <spec_name>
# experiment-worktree.sh count
#
# Worktrees are created at: .worktrees/optimize-<spec>-exp-<NNN>/
# Branches are named: optimize-exp/<spec>/exp-<NNN>
set -euo pipefail
RED='\033[0;31m'
GREEN='\033[0;32m'
YELLOW='\033[1;33m'
BLUE='\033[0;34m'
NC='\033[0m'
GIT_ROOT=$(git rev-parse --show-toplevel 2>/dev/null) || {
echo -e "${RED}Error: Not in a git repository${NC}" >&2
exit 1
}
WORKTREE_DIR="$GIT_ROOT/.worktrees"
experiment_branch_name() {
local spec_name="${1:?Error: spec_name required}"
local padded_index="${2:?Error: padded_index required}"
# Keep experiment refs outside optimize/<spec> so they do not collide
# with the long-lived optimization branch namespace.
echo "optimize-exp/${spec_name}/exp-${padded_index}"
}
ensure_worktree_exclude() {
local exclude_file
exclude_file=$(git rev-parse --git-path info/exclude)
mkdir -p "$(dirname "$exclude_file")"
if ! grep -q "^\.worktrees$" "$exclude_file" 2>/dev/null; then
echo ".worktrees" >> "$exclude_file"
fi
}
is_registered_worktree() {
local worktree_path="${1:?Error: worktree_path required}"
git worktree list --porcelain | awk -v target="$worktree_path" '
$1 == "worktree" && $2 == target { found = 1 }
END { exit(found ? 0 : 1) }
'
}
is_branch_checked_out() {
local branch_name="${1:?Error: branch_name required}"
local branch_ref="refs/heads/$branch_name"
git worktree list --porcelain | awk -v target="$branch_ref" '
$1 == "branch" && $2 == target { found = 1 }
END { exit(found ? 0 : 1) }
'
}
reset_worktree_to_base() {
local worktree_path="${1:?Error: worktree_path required}"
local branch_name="${2:?Error: branch_name required}"
local base_branch="${3:?Error: base_branch required}"
local current_branch
current_branch=$(git -C "$worktree_path" symbolic-ref --quiet --short HEAD 2>/dev/null || true)
if [[ "$current_branch" != "$branch_name" ]]; then
echo -e "${RED}Error: Existing worktree is on unexpected branch: ${current_branch:-detached} (expected $branch_name)${NC}" >&2
echo -e "${RED}Clean up the stale worktree before rerunning this experiment.${NC}" >&2
return 1
fi
echo -e "${YELLOW}Resetting existing experiment worktree to base: $branch_name -> $base_branch${NC}" >&2
git -C "$worktree_path" reset --hard "$base_branch" >/dev/null
git -C "$worktree_path" clean -fdx >/dev/null
}
# Create an experiment worktree
create_worktree() {
local spec_name="${1:?Error: spec_name required}"
local exp_index="${2:?Error: exp_index required}"
local base_branch="${3:?Error: base_branch required}"
shift 3
local padded_index
padded_index=$(printf "%03d" "$exp_index")
local worktree_name="optimize-${spec_name}-exp-${padded_index}"
local branch_name
branch_name=$(experiment_branch_name "$spec_name" "$padded_index")
local worktree_path="$WORKTREE_DIR/$worktree_name"
# Check if worktree already exists
if [[ -d "$worktree_path" ]]; then
if ! git -C "$worktree_path" rev-parse --is-inside-work-tree >/dev/null 2>&1 || \
! is_registered_worktree "$worktree_path"; then
echo -e "${RED}Error: Existing path is not a valid registered git worktree: $worktree_path${NC}" >&2
echo -e "${RED}Remove or repair that directory before rerunning the experiment.${NC}" >&2
return 1
fi
echo -e "${YELLOW}Worktree already exists: $worktree_path${NC}" >&2
reset_worktree_to_base "$worktree_path" "$branch_name" "$base_branch"
else
mkdir -p "$WORKTREE_DIR"
ensure_worktree_exclude
# Create worktree from the base branch
if ! git worktree add -b "$branch_name" "$worktree_path" "$base_branch" --quiet 2>/dev/null; then
if git show-ref --verify --quiet "refs/heads/$branch_name"; then
if is_branch_checked_out "$branch_name"; then
echo -e "${RED}Error: Existing experiment branch is already checked out: $branch_name${NC}" >&2
echo -e "${RED}Clean up the stale worktree before rerunning this experiment.${NC}" >&2
return 1
fi
echo -e "${YELLOW}Resetting existing experiment branch to base: $branch_name -> $base_branch${NC}" >&2
git branch -f "$branch_name" "$base_branch" >/dev/null
git worktree add "$worktree_path" "$branch_name" --quiet
else
echo -e "${RED}Error: Failed to create worktree for $branch_name from $base_branch${NC}" >&2
return 1
fi
fi
fi
# Copy .env files from main repo
for f in "$GIT_ROOT"/.env*; do
if [[ -f "$f" ]]; then
local basename
basename=$(basename "$f")
if [[ "$basename" != ".env.example" ]]; then
cp "$f" "$worktree_path/$basename"
fi
fi
done
# Copy shared files
for shared_file in "$@"; do
if [[ -f "$GIT_ROOT/$shared_file" ]]; then
local dir
dir=$(dirname "$worktree_path/$shared_file")
mkdir -p "$dir"
cp "$GIT_ROOT/$shared_file" "$worktree_path/$shared_file"
elif [[ -d "$GIT_ROOT/$shared_file" ]]; then
local dir
dir=$(dirname "$worktree_path/$shared_file")
mkdir -p "$dir"
rm -rf "$worktree_path/$shared_file"
cp -R "$GIT_ROOT/$shared_file" "$worktree_path/$shared_file"
fi
done
echo "$worktree_path"
}
# Clean up a single experiment worktree
cleanup_worktree() {
local spec_name="${1:?Error: spec_name required}"
local exp_index="${2:?Error: exp_index required}"
local padded_index
padded_index=$(printf "%03d" "$exp_index")
local worktree_name="optimize-${spec_name}-exp-${padded_index}"
local branch_name
branch_name=$(experiment_branch_name "$spec_name" "$padded_index")
local worktree_path="$WORKTREE_DIR/$worktree_name"
if [[ -d "$worktree_path" ]]; then
git worktree remove "$worktree_path" --force 2>/dev/null || {
# If worktree remove fails, try manual cleanup
rm -rf "$worktree_path" 2>/dev/null || true
git worktree prune 2>/dev/null || true
}
fi
# Delete the experiment branch
git branch -D "$branch_name" 2>/dev/null || true
echo -e "${GREEN}Cleaned up: $worktree_name${NC}" >&2
}
# Clean up all experiment worktrees for a spec
cleanup_all() {
local spec_name="${1:?Error: spec_name required}"
local prefix="optimize-${spec_name}-exp-"
local count=0
if [[ ! -d "$WORKTREE_DIR" ]]; then
echo -e "${YELLOW}No worktrees directory found${NC}" >&2
return 0
fi
for worktree_path in "$WORKTREE_DIR"/${prefix}*; do
if [[ -d "$worktree_path" ]]; then
local worktree_name
worktree_name=$(basename "$worktree_path")
# Extract index from name
local index_str="${worktree_name#$prefix}"
git worktree remove "$worktree_path" --force 2>/dev/null || {
rm -rf "$worktree_path" 2>/dev/null || true
}
# Delete the branch
local branch_name
branch_name=$(experiment_branch_name "$spec_name" "$index_str")
git branch -D "$branch_name" 2>/dev/null || true
count=$((count + 1))
fi
done
git worktree prune 2>/dev/null || true
# Clean up empty worktree directory
if [[ -d "$WORKTREE_DIR" ]] && [[ -z "$(ls -A "$WORKTREE_DIR" 2>/dev/null)" ]]; then
rmdir "$WORKTREE_DIR" 2>/dev/null || true
fi
echo -e "${GREEN}Cleaned up $count experiment worktree(s) for $spec_name${NC}" >&2
}
# Count total worktrees (for budget check)
count_worktrees() {
local count=0
if [[ -d "$WORKTREE_DIR" ]]; then
for worktree_path in "$WORKTREE_DIR"/*; do
if [[ -d "$worktree_path" ]] && [[ -e "$worktree_path/.git" ]]; then
count=$((count + 1))
fi
done
fi
echo "$count"
}
# Main
main() {
local command="${1:-help}"
case "$command" in
create)
shift
create_worktree "$@"
;;
cleanup)
shift
cleanup_worktree "$@"
;;
cleanup-all)
shift
cleanup_all "$@"
;;
count)
count_worktrees
;;
help)
cat << 'EOF'
Experiment Worktree Manager
Usage:
experiment-worktree.sh create <spec_name> <exp_index> <base_branch> [shared_file ...]
experiment-worktree.sh cleanup <spec_name> <exp_index>
experiment-worktree.sh cleanup-all <spec_name>
experiment-worktree.sh count
Commands:
create Create an experiment worktree with copied shared files
cleanup Remove a single experiment worktree and its branch
cleanup-all Remove all experiment worktrees for a spec
count Count total active worktrees (for budget checking)
Worktrees: .worktrees/optimize-<spec>-exp-<NNN>/
Branches: optimize-exp/<spec>/exp-<NNN>
EOF
;;
*)
echo -e "${RED}Unknown command: $command${NC}" >&2
exit 1
;;
esac
}
main "$@"
scripts/measure.sh
#!/bin/bash
# Measurement Runner
# Runs a measurement command, captures JSON output, and handles timeouts.
# The orchestrating agent (not this script) evaluates gates and handles
# stability repeats.
#
# Usage: measure.sh <command> <timeout_seconds> [working_directory] [KEY=VALUE ...]
#
# Arguments:
# command - Shell command to run (e.g., "python evaluate.py")
# timeout_seconds - Maximum seconds before killing the command
# working_directory - Directory to run the command in (default: .)
# KEY=VALUE - Optional environment variables to set before running
#
# Output:
# stdout: Raw JSON output from the measurement command
# stderr: Passed through from the measurement command
# exit code: Same as the measurement command (124 for timeout, 125 when
# CE_OPTIMIZE_CENSOR_AFTER fires before timeout_seconds)
set -euo pipefail
# Parse arguments
COMMAND="${1:?Error: command argument required}"
TIMEOUT="${2:?Error: timeout_seconds argument required}"
shift 2
WORKDIR="."
if [[ $# -gt 0 ]] && [[ "$1" != *=* ]]; then
WORKDIR="$1"
shift
fi
# Set any KEY=VALUE environment variables
for arg in "$@"; do
if [[ "$arg" == *=* ]]; then
export "$arg"
fi
done
# Change to working directory
cd "$WORKDIR" || {
echo "Error: cannot cd to $WORKDIR" >&2
exit 1
}
run_timed_command() {
local timeout_bin="$1"
if [[ -n "${CENSOR_STATUS_FILE:-}" ]]; then
"$timeout_bin" "$TIMEOUT" bash -c 'bash -c "$1"; printf "%s\n" "$?" > "$2"; exit 0' _ "$COMMAND" "$CENSOR_STATUS_FILE"
return
fi
"$timeout_bin" "$TIMEOUT" bash -c "$COMMAND"
}
run_with_timeout() {
if command -v timeout >/dev/null 2>&1; then
run_timed_command timeout
return
fi
if command -v gtimeout >/dev/null 2>&1; then
run_timed_command gtimeout
return
fi
PY=""
for c in python3 python py; do
if command -v "$c" >/dev/null 2>&1 && "$c" -c '' >/dev/null 2>&1; then
PY="$c"
break
fi
done
if [ -n "$PY" ]; then
"$PY" - "$TIMEOUT" "$COMMAND" "${CENSOR_STATUS_FILE:-}" <<'PY'
import os
import signal
import subprocess
import sys
timeout_seconds = float(sys.argv[1])
command = sys.argv[2]
status_file = sys.argv[3] if len(sys.argv) > 3 and sys.argv[3] else ""
proc = subprocess.Popen(["bash", "-c", command], start_new_session=True)
try:
rc = proc.wait(timeout=timeout_seconds)
if status_file:
with open(status_file, "w", encoding="utf-8") as fh:
fh.write(f"{rc}\n")
sys.exit(0)
sys.exit(rc)
except subprocess.TimeoutExpired:
os.killpg(proc.pid, signal.SIGTERM)
try:
proc.wait(timeout=5)
except subprocess.TimeoutExpired:
os.killpg(proc.pid, signal.SIGKILL)
proc.wait()
sys.exit(124)
PY
return
fi
echo "Error: no timeout implementation available (tried timeout, gtimeout, and a working Python 3 interpreter)" >&2
exit 1
}
# Optional futility bound: CE_OPTIMIZE_CENSOR_AFTER=<seconds> kills a live
# run that has already exceeded a predeclared noncompetitive bound. Distinct
# from timeout_seconds (the spec's hard cap). Exit 125 means censored; 124
# still means the configured timeout fired.
CENSOR_AFTER="${CE_OPTIMIZE_CENSOR_AFTER:-}"
CENSORING=0
CENSOR_STATUS_FILE=""
if [[ -n "$CENSOR_AFTER" ]] && awk -v a="$CENSOR_AFTER" -v t="$TIMEOUT" 'BEGIN { exit !(a ~ /^[0-9]+(\.[0-9]+)?$/ && t+0 == t && a+0 > 0 && a+0 < t+0) }'; then
TIMEOUT="$CENSOR_AFTER"
CENSORING=1
CENSOR_STATUS_FILE=$(mktemp "${TMPDIR:-/tmp}/ce-optimize-censor-XXXXXX")
fi
# Run the measurement command with timeout
# timeout returns 124 if the command times out
# We pass stdout and stderr through directly
set +e
run_with_timeout
status=$?
set -e
if [[ $CENSORING -eq 1 ]]; then
if [[ -s "$CENSOR_STATUS_FILE" ]]; then
status=$(cat "$CENSOR_STATUS_FILE")
elif [[ $status -eq 124 ]]; then
echo "Error: measurement censored after ${CENSOR_AFTER}s (noncompetitive bound)" >&2
rm -f "$CENSOR_STATUS_FILE"
exit 125
fi
rm -f "$CENSOR_STATUS_FILE"
fi
exit "$status"
scripts/parallel-probe.sh
#!/bin/bash
# Parallelism Probe
# Detects common parallelism blockers in the target project.
# Output is advisory -- the skill presents results to the user for approval.
#
# Usage: parallel-probe.sh <project_directory> [measurement_command] [measurement_workdir] [shared_file ...]
#
# Arguments:
# project_directory - Root directory of the project to probe
# measurement_command - The measurement command from the spec (optional, for port detection)
# measurement_workdir - Measurement working directory relative to project root (default: .)
# shared_file - Explicitly declared shared files that parallel runs depend on
#
# Output:
# JSON to stdout with:
# mode: "parallel" | "serial" | "user-decision"
# blockers: [ { type, description, suggestion } ]
set -euo pipefail
PROJECT_DIR="${1:?Error: project_directory argument required}"
MEASUREMENT_CMD="${2:-}"
MEASUREMENT_WORKDIR="${3:-.}"
shift 3 2>/dev/null || shift $# 2>/dev/null || true
SHARED_FILES=()
if [[ $# -gt 0 ]]; then
SHARED_FILES=("$@")
fi
cd "$PROJECT_DIR" || {
echo '{"mode":"serial","blockers":[{"type":"error","description":"Cannot access project directory","suggestion":"Check path"}]}'
exit 0
}
PY=""
for c in python3 python py; do
if command -v "$c" >/dev/null 2>&1 && "$c" -c '' >/dev/null 2>&1; then
PY="$c"
break
fi
done
if [ -z "$PY" ]; then
echo '{"mode":"serial","blockers":[{"type":"missing_dependency","description":"A working Python 3 interpreter is required for structured probe output","suggestion":"Install Python 3 (python3, python, or py on PATH) or skip the probe and review parallel-readiness manually"}],"blocker_count":1}'
exit 0
fi
BLOCKERS="[]"
SCAN_PATHS=()
add_blocker() {
local type="$1"
local desc="$2"
local suggestion="$3"
BLOCKERS=$(echo "$BLOCKERS" | "$PY" -c "
import json, sys
b = json.load(sys.stdin)
b.append({'type': '$type', 'description': '''$desc''', 'suggestion': '''$suggestion'''})
print(json.dumps(b))
" 2>/dev/null || echo "$BLOCKERS")
}
add_scan_path() {
local candidate="$1"
if [[ -z "$candidate" ]]; then
return
fi
if [[ -e "$candidate" ]]; then
SCAN_PATHS+=("$candidate")
fi
}
add_scan_path "$MEASUREMENT_WORKDIR"
if [[ ${#SHARED_FILES[@]} -gt 0 ]]; then
for shared_file in "${SHARED_FILES[@]}"; do
add_scan_path "$shared_file"
done
fi
if [[ ${#SCAN_PATHS[@]} -eq 0 ]]; then
SCAN_PATHS=(".")
fi
# Check 1: Hardcoded ports in measurement command
if [[ -n "$MEASUREMENT_CMD" ]]; then
# Look for common port patterns in the command itself
if echo "$MEASUREMENT_CMD" | grep -qE '(--port(?:\s+|=)[0-9]+|:\s*[0-9]{4,5}|PORT=[0-9]+|localhost:[0-9]+)'; then
add_blocker "port" "Measurement command contains hardcoded port reference" "Parameterize port via environment variable (e.g., PORT=\$EVAL_PORT)"
fi
fi
# Check 2: SQLite databases in the measurement workdir or declared shared files
SQLITE_FILES=$(find "${SCAN_PATHS[@]}" -maxdepth 4 -type f \( -name '*.db' -o -name '*.sqlite' -o -name '*.sqlite3' \) ! -path '*/.git/*' ! -path '*/node_modules/*' ! -path '*/.claude/*' ! -path '*/.context/*' ! -path '*/.worktrees/*' 2>/dev/null | head -10 || true)
if [[ -n "$SQLITE_FILES" ]]; then
FILE_COUNT=$(echo "$SQLITE_FILES" | wc -l | tr -d ' ')
add_blocker "shared_file" "Found $FILE_COUNT SQLite database file(s)" "Copy database files into each experiment worktree"
fi
# Check 3: Lock/PID files in the measurement workdir or declared shared files
LOCK_FILES=$(find "${SCAN_PATHS[@]}" -maxdepth 4 -type f \( -name '*.lock' -o -name '*.pid' \) ! -path '*/.git/*' ! -path '*/node_modules/*' ! -path '*/.claude/*' ! -path '*/.context/*' ! -path '*/.worktrees/*' ! -name 'package-lock.json' ! -name 'yarn.lock' ! -name 'bun.lock' ! -name 'bun.lockb' ! -name 'Gemfile.lock' ! -name 'poetry.lock' ! -name 'Cargo.lock' 2>/dev/null | head -10 || true)
if [[ -n "$LOCK_FILES" ]]; then
FILE_COUNT=$(echo "$LOCK_FILES" | wc -l | tr -d ' ')
add_blocker "lock_file" "Found $FILE_COUNT lock/PID file(s) that may cause contention" "Ensure measurement command cleans up lock files, or run in serial mode"
fi
# Check 4: Exclusive resource hints in the measurement command
if [[ -n "$MEASUREMENT_CMD" ]] && echo "$MEASUREMENT_CMD" | grep -qiE '(cuda|gpu|tensorflow|torch|nvidia-smi|CUDA_VISIBLE_DEVICES)'; then
add_blocker "exclusive_resource" "Measurement command appears to use GPU or another exclusive accelerator" "GPU is typically an exclusive resource -- consider serial mode or device parameterization"
fi
# Determine mode
BLOCKER_COUNT=$(echo "$BLOCKERS" | "$PY" -c "import json,sys; print(len(json.load(sys.stdin)))" 2>/dev/null || echo "0")
if [[ "$BLOCKER_COUNT" == "0" ]]; then
MODE="parallel"
elif echo "$BLOCKERS" | "$PY" -c "import json,sys; b=json.load(sys.stdin); exit(0 if any(x['type']=='exclusive_resource' for x in b) else 1)" 2>/dev/null; then
MODE="serial"
else
MODE="user-decision"
fi
# Output JSON result
"$PY" -c "
import json
print(json.dumps({
'mode': '$MODE',
'blockers': $BLOCKERS,
'blocker_count': $BLOCKER_COUNT
}, indent=2))
"
SKILL.md
---
name: ce-optimize
description: "Optimize a named target with a measured loop: attribute a workload's cost, or score variants and keep winners. Use when a working system's metric should move and the winning change is not already known. Use ce-debug when the job is diagnosis; use ce-work when the change is already known."
argument-hint: "[path to optimization spec YAML, or describe the optimization goal]"
---
# Optimize a measurable target
**Outcome:** confirmed improvements to the named target live on an `optimize/<spec-name>` branch, with a disk log. The next consumer is the user at wrap-up.
**Intent:** the next action is the cheapest step that would change what gets implemented. Attribute the cost of a named workload before searching implementations. Search and keep a scored variant space without requiring a profile.
**Horizon:** this is a long-running loop, not a one-shot edit. A first run stays short and serial until the harness is trusted. Spec first-run defaults are typically a few experiments and about an hour. Stop as soon as a stopping criterion holds. Do not grind to the iteration cap after the target is met or locating would not change keep or skip.
**Done when:** a stopping criterion fired, every declared required target is met or another stop fired first, the final state is written and verified on disk, and the user has been given the post-completion options. If the run instead stopped at a gate it could not clear, say what blocked it.
Invoking this skill authorizes reading the repo, building the harness, and (after the Phase 1 approval gate) isolated experiments and keep/revert commits on `optimize/<spec-name>`. Ask when spend is uncapped, when a new dependency appears, when wrap-up would push or open a PR, or when only the user can choose among the post-completion options. Do not ask again to run the next in-envelope experiment.
Independent calls and dispatches that do not depend on each other go in one response. Serialize only real dependencies.
Before each phase, say what it should produce. After each batch, say current best, this-batch and total counts, judge cost when it applies, and the next action. Wrap-up is the closing recap from disk.
A step is done only after it ran. Describing a measurement, dispatch, or checkpoint is not doing it. Do not end a turn while in-scope work remains merely described.
## Interaction Method
Use the host's blocking question tool already in the current tool list (match by capability, not by a host-specific name). Presence in the current tool list is proof the tool exists; never call a user-facing question tool to discover whether it exists. If a matching tool is listed but unloaded, use the host's tool-discovery primitive to load that capability: do not search for another host's tool name. Fall back to numbered options on the host's chat surface only when no such tool is in the list or a real question call errors. Never skip the question silently.
## Artifact Root
Resolve `<root>` the first time you compose a path under it. Reading learnings under `<root>/solutions/` counts as composing one. Give any subagent the resolved path, not the config.
<!-- ce-docs-root:start -->
**Resolve the CE artifact root `<root>` before composing any artifact path.**
- **Read** `docs_root` from `<repo-root>/.compound-engineering/config.yaml` only (`<repo-root>` = `git rev-parse --show-toplevel`). Do not read it from `config.local.yaml`. Unset -> `<root>` is `docs`, exactly as before.
- **Validate** a set value: a repo-relative directory whose real, symlink-resolved path stays inside the repo and is neither the repo root nor under `.git/`. Otherwise stop with an error naming `docs_root` and the value -- never fall back to `docs`.
- **Use** `<root>` as the sole artifact location: create it if absent, compose each path as `<root>/<subdir>` with this skill's own subdirectory, and never also read `docs`.
<!-- ce-docs-root:end -->
## Persistence Discipline
The experiment log on disk is the source of truth. Write order is measure, write, verify, then show the user. **Read `references/persistence.md` now** for checkpoints CP-0 through CP-5, the file layout, and resume. The phases below mark where each checkpoint falls.
## The phases
Four phases run in order. Each one names the reference it cannot start without. A fresh run skips none of them: a harder optimization spends longer in a phase, it does not run fewer phases.
**A resume is not a fresh run.** On a resume, re-enter Phase 0 only far enough to detect the run and to recover any `result.yaml` markers the log is missing. Then continue from the phase the log records: skip the work the log proves finished, and re-enter any gate it does not. A checkpoint proves the work that produced it, never a user decision: the log holds no record of approval, so a resume that has not seen the user approve presents the Phase 1 gate again.
**Phase 0: Setup.** The input is a goal, or a path to a spec YAML. It comes from the user or from a calling skill. If neither supplied one, ask: "What would you like to optimize? Describe the goal, or provide a path to an optimization spec YAML file." Load or build the spec and save it (CP-0): **read `references/spec.md`**. Then search prior learnings, detect run identity, and create the branch and scratch space. **Read `references/measurement.md`** for the rest of Phase 0 and Phase 1.
**Phase 1: Measurement scaffolding.** Build or validate the harness, write the baseline (CP-1), probe parallelism, check the worktree budget. Two gates stop the run:
- **Clean-tree gate.** Do not continue while any file in `scope.mutable` or `scope.immutable` has uncommitted changes. The reference owns the check and what to ask for.
- **User approval gate.** Present what Phase 1 assembled; the reference lists what to include. If the primary type is `judge` and `max_total_cost_usd` is unset, say plainly that spend is uncapped. Offer proceed, fix issues, and adjust spec. Adjusting the spec is only available while the log holds nothing derived from it (no hypothesis backlog and no experiments) and it sends the run back through Phase 1 so the baseline matches the new spec. Once anything derived from the spec is on file, the spec is fixed for the run. **Do not enter Phase 2 until the user explicitly approves.** Then re-read the spec and baseline from disk.
**Phase 2: Hypothesis generation.** Analyze the current approach, rank the hypotheses, record the backlog (CP-2). Do not dispatch an implementation experiment while a cheaper locating measurement would change keep or skip. **Read `references/loop.md`** for this phase and Phase 3. One gate: **dependency pre-approval.** Collect every new dependency across all hypotheses and present the full list for bulk approval. A dependency the user does not approve stays in the backlog, is skipped in batch selection, and comes back at wrap-up.
**Phase 3: Optimization loop.** Select a batch, dispatch experiments, persist each result as it lands (CP-3), evaluate with `scripts/decide.mjs`, update state and the digest (CP-4), then check whether to stop. Stop as soon as any one of seven criteria holds: every declared required target is met, max iterations, max hours, judge budget exhausted, plateau, a user interrupt, or no runnable hypothesis left. `references/loop.md` states each one exactly. Otherwise start the next batch.
**Phase 4: Wrap-up.** **Read `references/wrap-up.md`** for the deferred hypotheses, the summary, what is preserved, cleanup, and the post-completion options to present. CP-5 marks the log final. **Write it only after the user picks an option that does not return to Phase 3.** Two options do return: Continue, and approving a deferred dependency.