evals/cases.yaml
cases:
# --- Positive cases ---
- id: improve_existing_prompt
prompt: "Improve this prompt: 'Summarize the document.' It's for Claude to summarize technical documents into 3-5 bullet points."
fixtures: []
rubric:
- "Identifies weaknesses in the current prompt (no output format, no length constraint, no audience specification)"
- "Generates 2-4 variants each testing a different hypothesis"
- "Each variant changes ONE variable from baseline"
- "Includes complete prompt text for each variant (not just descriptions)"
- "Designs an evaluation rubric with weighted criteria"
trigger_expected: true
assertions:
- type: matches_regex
target: "(?i)variant|version|v\\d"
weight: 1.0
- type: matches_regex
target: "(?i)rubric|criteria|evaluat"
weight: 0.8
- type: matches_regex
target: "(?i)weakness|failure|issue"
weight: 0.8
- type: matches_regex
target: "(?i)weight|score"
weight: 0.6
- type: matches_regex
target: "(?i)few.shot|chain.of.thought|direct"
weight: 0.6
- id: design_classification_prompt
prompt: "Design a prompt for GPT-4 to classify customer support tickets into categories: billing, technical, account, feature-request, other."
fixtures: []
rubric:
- "Generates variants including at minimum a direct instruction and a few-shot approach"
- "Includes concrete examples in the few-shot variant"
- "Designs test cases with at least one edge case (ambiguous ticket)"
- "Specifies success criteria (classification accuracy)"
- "Notes target model (GPT-4) in recommendations"
trigger_expected: true
assertions:
- type: matches_regex
target: "(?i)few.shot"
weight: 1.0
- type: matches_regex
target: "(?i)billing|technical|account|feature.request"
weight: 0.8
- type: matches_regex
target: "(?i)test case|edge case"
weight: 0.8
- type: matches_regex
target: "(?i)accuracy|success.criteria"
weight: 0.6
- type: matches_regex
target: "(?i)GPT-4|gpt.4"
weight: 0.6
- id: failure_mode_analysis
prompt: "Why does this prompt sometimes produce hallucinated facts? 'You are an expert. Answer the user's question about our product based on your knowledge.'"
fixtures: []
rubric:
- "Identifies missing grounding/context as the core failure mode"
- "Identifies 'based on your knowledge' as hallucination-inducing"
- "Recommends providing source documents or RAG context"
- "Generates improved variants that constrain the model to provided context"
- "Includes test cases specifically designed to trigger hallucination"
trigger_expected: true
assertions:
- type: matches_regex
target: "(?i)hallucination|grounding|context"
weight: 1.0
- type: matches_regex
target: "(?i)based on your knowledge"
weight: 0.8
- type: matches_regex
target: "(?i)RAG|retrieval|source document"
weight: 0.8
- type: matches_regex
target: "(?i)variant|improved"
weight: 0.6
- type: matches_regex
target: "(?i)test case"
weight: 0.6
# --- Negative cases ---
- id: skill_evaluation_request
prompt: "Evaluate this SKILL.md file for quality and trigger coverage."
fixtures: []
rubric:
- "Does NOT trigger prompt-lab skill (skill-evaluator territory, not prompt engineering)"
trigger_expected: false
- id: code_generation_request
prompt: "Write a Python script that processes CSV files."
fixtures: []
rubric:
- "Does NOT trigger prompt-lab skill (general coding task, not prompt engineering)"
trigger_expected: false
references/evaluation-metrics.md
# Evaluation Metrics
Metrics for measuring prompt quality, rubric design, and scoring methodology.
---
## Core Metrics
### Correctness
Does the output contain the right answer/information?
| Score | Meaning |
| ----- | ------------------------------- |
| 0 | Completely wrong or irrelevant |
| 1 | Partially correct, major errors |
| 2 | Mostly correct, minor errors |
| 3 | Fully correct |
### Format Compliance
Does the output follow the specified format?
| Score | Meaning |
| ----- | ------------------------------------ |
| 0 | Wrong format entirely |
| 1 | Right format, significant deviations |
| 2 | Right format, minor deviations |
| 3 | Perfect format compliance |
### Completeness
Does the output include all required elements?
| Score | Meaning |
| ----- | ------------------------------- |
| 0 | Missing most required elements |
| 1 | Includes some required elements |
| 2 | Includes most required elements |
| 3 | All required elements present |
### Conciseness
Is the output free of unnecessary content?
| Score | Meaning |
| ----- | ---------------------------------- |
| 0 | Extremely verbose, mostly filler |
| 1 | Significant unnecessary content |
| 2 | Slightly verbose |
| 3 | Concise, every sentence adds value |
### Groundedness
Are all claims supported by provided context? (For RAG/grounded tasks)
| Score | Meaning |
| ----- | ----------------------------------------------- |
| 0 | Mostly hallucinated |
| 1 | Mix of grounded and hallucinated claims |
| 2 | Almost fully grounded, minor unsupported claims |
| 3 | Every claim traceable to context |
---
## Rubric Design
### Weighting by Task Type
| Task Type | Correctness | Format | Completeness | Conciseness |
| -------------- | ----------- | ------ | ------------ | ----------- |
| Classification | 50% | 20% | 10% | 20% |
| Extraction | 40% | 25% | 25% | 10% |
| Analysis | 35% | 15% | 30% | 20% |
| Summarization | 30% | 15% | 25% | 30% |
| Generation | 30% | 20% | 25% | 25% |
### Custom Criteria
Add task-specific criteria when core metrics are insufficient:
| Custom Criterion | When to Use |
| ---------------- | ------------------------------------------ |
| Tone/voice | Brand-specific or audience-specific output |
| Specificity | Answers should be concrete, not generic |
| Actionability | Recommendations should be actionable |
| Safety | Output must not contain harmful content |
| Creativity | Output should be novel or engaging |
---
## Scoring Methodology
### Per-Query Scoring
For each test query × variant:
1. Score each criterion (0-3)
2. Apply weights
3. Calculate weighted sum
4. Normalize to 0-100%
```text
Score = Σ(criterion_score × weight) / (3 × Σ weights) × 100
```
### Aggregate Scoring
For each variant across all test queries:
- **Mean score** — Overall quality
- **Min score** — Worst-case performance (important for reliability)
- **Std deviation** — Consistency
- **Pass rate** — % of queries scoring above threshold
### Comparison
| Metric | Variant A | Variant B | Winner |
| --------------------- | --------- | --------- | ------ |
| Mean score | 85% | 78% | A |
| Min score | 60% | 45% | A |
| Consistency (std dev) | 8% | 15% | A |
| Pass rate (>70%) | 95% | 80% | A |
---
## Automated Evaluation
### LLM-as-Judge
Use a separate LLM call to evaluate outputs:
```text
You are an evaluation judge. Score the following output on a 0-3 scale for each criterion.
Criteria:
- Correctness: Does the output contain the right answer?
- Format: Does it follow the specified format?
- Completeness: Are all required elements present?
Expected output: {expected}
Actual output: {actual}
Provide scores as JSON:
{"correctness": N, "format": N, "completeness": N, "reasoning": "..."}
```
**Limitations:** LLM judges have their own biases. Use as a signal, not ground truth.
Human evaluation is more reliable for subjective criteria.
### Exact Match
For tasks with unambiguous correct answers:
```python
score = 1.0 if output.strip() == expected.strip() else 0.0
```
### Partial Match (Recall)
For extraction tasks:
```python
expected_items = set(expected)
output_items = set(output)
recall = len(expected_items & output_items) / len(expected_items)
```
references/failure-modes.md
# Failure Modes
Common prompt failure taxonomy with detection strategies and mitigations.
---
## Failure Taxonomy
### 1. Instruction Misinterpretation
**Symptom:** Model does something different from what was intended.
| Cause | Example | Mitigation |
| ---------------------- | ----------------------------------------------------------- | -------------------------------------------- |
| Ambiguous instruction | "Summarize this" — model writes 3 sentences vs 3 paragraphs | Specify length: "Summarize in 2-3 sentences" |
| Overloaded instruction | "Analyze and fix this code" — model does both poorly | Split into two prompts |
| Implicit assumption | "Fix the bug" without specifying which bug | Be explicit about the target |
### 2. Format Violation
**Symptom:** Output is correct but in the wrong format.
| Cause | Example | Mitigation |
| --------------------------- | ---------------------------------------------------- | ------------------------------------------------ |
| No format specified | Asked for JSON, got prose | Specify format explicitly with example |
| Format specified but buried | Format instruction in the middle of a long prompt | Put format instruction at the end (recency bias) |
| Conflicting format signals | Examples in one format, instruction asks for another | Ensure examples match requested format |
### 3. Hallucination
**Symptom:** Model states false information as fact.
| Type | Example | Mitigation |
| ----------------------- | -------------------------------------------- | -------------------------------------------------------- |
| Fabricated facts | Invents statistics, citations | Add "Only state facts supported by the provided context" |
| Fabricated reasoning | Plausible-sounding but incorrect logic | Use CoT to make reasoning visible and checkable |
| Confident wrong answers | States incorrect answer with high confidence | Add "If unsure, say so explicitly" |
### 4. Refusal / Over-Caution
**Symptom:** Model refuses to answer or adds excessive caveats.
| Cause | Mitigation |
| ------------------------- | --------------------------------------------------------- |
| Safety filter triggered | Reframe the task to be clearly benign |
| Task seems risky to model | Provide context explaining the legitimate use |
| Model uncertain | Explicitly allow uncertainty: "It's OK to be approximate" |
### 5. Repetition / Verbosity
**Symptom:** Model repeats itself or generates unnecessary content.
| Cause | Mitigation |
| ---------------------------------- | ---------------------------------------- |
| No length constraint | Add: "Maximum 200 words" or "Be concise" |
| Instruction encourages elaboration | Remove "explain in detail" if not needed |
| Few-shot examples are verbose | Use concise examples |
### 6. Anchoring to Examples
**Symptom:** Model copies patterns from examples too literally.
| Cause | Mitigation |
| ------------------------------------ | ----------------------------------------------------- |
| Examples too similar | Diversify examples across different cases |
| Examples contain irrelevant patterns | Model copies format details, not the underlying logic |
| Too many examples | Reduce to 2-3 diverse examples |
### 7. Ignoring Context
**Symptom:** Model doesn't use provided context/information.
| Cause | Mitigation |
| ------------------------------------- | ---------------------------------------------------------- |
| Context too long | Highlight relevant sections: "Pay special attention to..." |
| Context placement | Move context closer to the question (recency effect) |
| Instruction doesn't reference context | Explicitly: "Using ONLY the information above, answer..." |
### 8. Sycophancy / Agreement Bias
**Symptom:** Model agrees with the user's stated opinion instead of giving an honest answer.
| Cause | Mitigation |
| ---------------------------------- | ------------------------------------------- |
| User states opinion before asking | Ask the question before revealing your view |
| Prompt frames one option favorably | Present options neutrally |
| "Do you agree?" framing | Ask "Evaluate the pros and cons" instead |
---
## Detection Strategies
| Failure Mode | Detection Method |
| ----------------- | ------------------------------------------------ |
| Misinterpretation | Compare output structure to expected structure |
| Format violation | Parse output with expected format parser |
| Hallucination | Cross-reference claims against source material |
| Refusal | Check for "I cannot", "I'm sorry", "As an AI" |
| Verbosity | Word count comparison against target length |
| Example anchoring | Run without examples, compare output diversity |
| Context ignoring | Check if key facts from context appear in output |
| Sycophancy | Ask the same question with opposite framing |
---
## Failure Mode by Task Type
| Task Type | Most Common Failures |
| ----------------- | ------------------------------------------ |
| Classification | Misinterpretation, anchoring to examples |
| Extraction | Incomplete extraction, hallucinated fields |
| Summarization | Verbosity, missing key points |
| Analysis | Hallucination, sycophancy |
| Code generation | Format violation, subtle logic errors |
| Creative writing | Verbosity, generic output |
| RAG / grounded QA | Hallucination, context ignoring |
references/output-constraints.md
# Output Constraints
Techniques for constraining LLM output format, enforcing structure, and ensuring
parseable responses.
---
## Prompt-Level Constraints
### Length Constraints
```text
Respond in exactly 3 bullet points.
Maximum 100 words.
One sentence only.
Between 2 and 5 paragraphs.
```
### Format Constraints
```text
Respond in valid JSON matching this schema:
{"category": "string", "confidence": "number 0-1", "reasoning": "string"}
Respond as a markdown table with columns: Feature, Pros, Cons
Respond as a numbered list. Each item must start with an action verb.
```
### Content Constraints
```text
ONLY use information from the provided context. Do not add external knowledge.
If the answer is not in the context, respond with "Not found in context."
Do not include opinions or recommendations. State facts only.
Do not use technical jargon. Write for a non-technical audience.
```
---
## API-Level Constraints
### JSON Mode (OpenAI)
```python
response = client.chat.completions.create(
model="gpt-4",
messages=[...],
response_format={"type": "json_object"},
)
```
Guarantees valid JSON output. Still need to specify the schema in the prompt.
### Tool Use / Function Calling (Claude, OpenAI)
```python
# Define the expected output schema as a tool
tools = [{
"name": "classify_ticket",
"description": "Classify a support ticket",
"input_schema": {
"type": "object",
"properties": {
"category": {"type": "string", "enum": ["billing", "technical", "general"]},
"priority": {"type": "string", "enum": ["low", "medium", "high"]},
"summary": {"type": "string"},
},
"required": ["category", "priority", "summary"],
},
}]
```
Forces the model to respond with a structured object matching the schema.
### Structured Output (OpenAI)
```python
from pydantic import BaseModel
class Classification(BaseModel):
category: str
confidence: float
reasoning: str
response = client.beta.chat.completions.parse(
model="gpt-4o",
messages=[...],
response_format=Classification,
)
```
---
## Constraint Placement
### Position Matters
Models have recency bias — instructions at the end of the prompt are followed more
reliably than those at the beginning.
```text
{Context / input data}
{Main instruction}
{Format constraint — place last for best compliance}
```
### Repetition Reinforces
For critical constraints, state them twice:
```text
Important: Respond ONLY in valid JSON.
{Task instruction}
{Input}
Remember: Your response must be valid JSON. No text outside the JSON object.
```
---
## Common Constraint Failures
| Constraint | Failure Mode | Fix |
| --------------------------- | ---------------------------------------- | ---------------------------------------------------------------------- |
| "Respond in JSON" | Model wraps JSON in markdown code blocks | "Respond with raw JSON only. No markdown, no code blocks." |
| "Maximum 3 sentences" | Model writes 3 long sentences | "Maximum 3 sentences, each under 30 words" |
| "Only use provided context" | Model adds common knowledge | "If you add ANY information not in the context, mark it as [inferred]" |
| "No opinions" | Model hedges with "some might say" | "State each point as a factual observation" |
| "Use this template" | Model modifies the template structure | Provide the template with clear markers: `{FILL THIS}` |
---
## Validation After Generation
Even with constraints, validate the output programmatically:
```python
import json
def validate_output(text: str, expected_keys: list[str]) -> bool:
"""Validate that output is valid JSON with expected keys."""
try:
data = json.loads(text)
except json.JSONDecodeError:
return False
return all(key in data for key in expected_keys)
```
### Retry Strategy
If validation fails:
1. Parse the error
2. Include the error in a follow-up prompt
3. Ask the model to fix its output
4. Maximum 2 retries before failing
```python
for attempt in range(3):
response = generate(prompt)
if validate_output(response):
return response
prompt = f"Your previous response was invalid: {error}. Please fix it.\n\n{original_prompt}"
raise ValueError("Failed to generate valid output after 3 attempts")
```
references/prompt-patterns.md
# Prompt Patterns
Catalog of prompt structures organized by strategy, with templates and usage guidance.
---
## Zero-Shot
No examples. Direct instruction only.
```text
{Role/persona statement — optional}
{Task instruction}
{Input format specification}
{Output format specification}
{Constraints}
```
**When to use:** Simple tasks, capable models (GPT-4, Claude), well-defined output format.
**Risk:** Model may interpret the task differently than intended without examples.
---
## Few-Shot
Provide examples of input → output pairs before the actual task.
```text
{Task instruction}
Example 1:
Input: {example input}
Output: {example output}
Example 2:
Input: {example input}
Output: {example output}
Now do the same for:
Input: {actual input}
Output:
```
**When to use:** Pattern-following tasks, classification, formatting, extraction.
**Guidelines:**
- 2-5 examples is typical. More is not always better.
- Examples should cover the range of expected inputs (not all similar).
- Include at least one edge case example.
- Keep examples consistent in format — the model mirrors what it sees.
---
## Chain-of-Thought (CoT)
Ask the model to reason step by step before giving the final answer.
```text
{Task instruction}
Think through this step by step:
1. First, consider...
2. Then, analyze...
3. Finally, conclude...
{Input}
```
**When to use:** Multi-step reasoning, math, logical deduction, complex analysis.
**Variants:**
- **Explicit CoT:** "Think step by step" in the instruction
- **Few-shot CoT:** Examples include the reasoning steps
- **Zero-shot CoT:** Just append "Let's think step by step" (surprisingly effective)
---
## Persona/Role
Frame the model as an expert in a specific domain.
```text
You are a {role} with expertise in {domain}. You have {years} of experience
with {specific skills}.
Your task is to {instruction}.
{Input}
```
**When to use:** Domain-specific tasks where expertise framing improves output quality.
**Caution:** Personas should be specific and relevant, not generic.
"You are a senior PostgreSQL DBA" > "You are a helpful assistant."
---
## Structured Output
Specify the exact output format the model must follow.
````
{Task instruction}
Respond in the following JSON format:
```json
{
"field1": "description of what goes here",
"field2": ["array", "of", "items"],
"field3": {
"nested": "object"
}
}
````
{Input}
```text
**When to use:** When output must be machine-parseable (JSON, CSV, YAML).
**Enhancement:** Use JSON mode / structured output API features when available
(OpenAI `response_format`, Anthropic tool use).
---
## Decomposition
Break a complex task into explicit subtasks within the prompt.
```text
I need you to complete the following task in steps:
Step 1: {subtask 1}
Step 2: Using the result of Step 1, {subtask 2}
Step 3: Based on Steps 1 and 2, {subtask 3}
Present each step's result before moving to the next.
{Input}
```text
**When to use:** Complex tasks that benefit from intermediate checkpoints.
---
## Constraint-Based
Define what the output must and must not contain.
```text
{Task instruction}
Rules:
- MUST: {requirement 1}
- MUST: {requirement 2}
- MUST NOT: {prohibition 1}
- MUST NOT: {prohibition 2}
- IF {condition} THEN {behavior}
{Input}
```text
**When to use:** Tasks with strict requirements or common failure modes to prevent.
---
## Comparison / Selection Pattern
```text
Compare the following options and recommend the best one:
Option A: {description}
Option B: {description}
Evaluation criteria (in order of importance):
1. {criterion 1}
2. {criterion 2}
3. {criterion 3}
For each option, evaluate against each criterion. Then provide your recommendation
with justification.
```text
---
## Pattern Selection Guide
| Task Type | Recommended Pattern | Fallback |
|-----------|-------------------|----------|
| Classification | Few-shot | Zero-shot with examples in description |
| Extraction | Few-shot + Structured output | Zero-shot with JSON mode |
| Analysis | Chain-of-thought | Decomposition |
| Generation (creative) | Persona + Constraints | Zero-shot with tone guidance |
| Generation (technical) | Persona + Structured output | Few-shot + Template |
| Summarization | Zero-shot with length constraint | Few-shot with length examples |
| Translation/formatting | Few-shot | Zero-shot with format specification |
| Decision/recommendation | Comparison pattern | Chain-of-thought |
```
SKILL.md
---
name: prompt-lab
description: 'LLM prompt engineering: analyzes failure modes, generates variants (direct, few-shot, CoT), designs rubrics, produces test suites. Triggers on: "prompt engineering", "generate prompt variants", "A/B test prompts", "optimize prompt", "improve this prompt". NOT for SKILL.md files, use skill-evaluator.'
metadata:
version: 1.1.1
category: development
tags: [prompt-engineering, evaluation, few-shot, chain-of-thought]
difficulty: intermediate
phase: build
---
# Prompt Lab
Replaces trial-and-error prompt engineering with structured methodology: objective
definition, current prompt analysis, variant generation (instruction clarity, example
strategies, output format specification), evaluation rubric design, test case creation,
and failure mode identification.
## Reference Files
| File | Contents | Load When |
| ---------------------------------- | ------------------------------------------------------------------------------ | -------------------------- |
| `references/prompt-patterns.md` | Prompt structure catalog: zero-shot, few-shot, CoT, persona, structured output | Always |
| `references/evaluation-metrics.md` | Quality metrics (accuracy, format compliance, completeness), rubric design | Evaluation needed |
| `references/failure-modes.md` | Common prompt failure taxonomy, detection strategies, mitigations | Failure analysis requested |
| `references/output-constraints.md` | Techniques for constraining LLM output format, JSON mode, schema enforcement | Format control needed |
## Prerequisites
- Clear objective: what should the prompt accomplish?
- Target model (GPT-4, Claude, open-source) — prompting techniques vary by model
- Current prompt (if improving) or task description (if creating)
## Workflow
### Phase 1: Define Objective
1. **Task specification** — What should the LLM produce? Be specific: "Classify customer
support tickets into 5 categories" not "Handle support tickets."
2. **Success criteria** — How do you know the output is correct? Define measurable criteria
before writing any prompt.
3. **Failure modes** — What does a bad output look like? Missing information? Wrong format?
Hallucinated content? Refusal to answer?
### Phase 2: Analyze Current Prompt
If an existing prompt is provided:
1. **Structure assessment** — Is the instruction clear? Are examples provided? Is the
output format specified?
2. **Ambiguity detection** — Where could the model misinterpret the instruction?
3. **Missing components** — What's not specified that should be? (output format, tone,
length constraints, edge case handling)
4. **Failure mode mapping** — Which known failure patterns (see `references/failure-modes.md`)
apply to this prompt?
### Phase 3: Generate Variants
Create 2-4 prompt variants, each testing a different hypothesis:
| Variant Type | Hypothesis | When to Use |
| ------------------ | ------------------------------------ | -------------------------------- |
| Direct instruction | Clear instruction is sufficient | Simple tasks, capable models |
| Few-shot | Examples improve output consistency | Pattern-following tasks |
| Chain-of-thought | Reasoning improves accuracy | Multi-step logic, math, analysis |
| Persona/role | Role framing improves tone/expertise | Domain-specific tasks |
| Structured output | Format specification prevents errors | JSON, CSV, specific templates |
For each variant:
- State the hypothesis (why this variant might work)
- Identify the risk (what could go wrong)
- Provide the complete prompt text
### Phase 4: Design Evaluation
1. **Rubric** — Define weighted criteria:
| Criterion | What It Measures | Typical Weight |
| ----------------- | ------------------------------ | -------------- |
| Correctness | Output matches expected answer | 30-50% |
| Format compliance | Follows specified structure | 15-25% |
| Completeness | All required elements present | 15-25% |
| Conciseness | No unnecessary content | 5-15% |
| Tone/style | Matches requested voice | 5-10% |
2. **Test cases** — Minimum 5 cases covering:
- Happy path (standard input)
- Edge cases (unusual but valid input)
- Adversarial cases (inputs designed to confuse)
- Boundary cases (minimum/maximum input)
### Phase 5: Output
Present variants, rubric, and test cases in a structured format ready for execution.
## Output Format
```text
## Prompt Lab: {Task Name}
### Objective
{What the prompt should achieve — specific and measurable}
### Success Criteria
- [ ] {Criterion 1 — measurable}
- [ ] {Criterion 2 — measurable}
### Current Prompt Analysis
{If existing prompt provided}
- **Strengths:** {what works}
- **Weaknesses:** {what fails or is ambiguous}
- **Missing:** {what's not specified}
### Variants
#### Variant A: {Strategy Name}
```
{Complete prompt text}
```text
**Hypothesis:** {Why this approach might work}
**Risk:** {What could go wrong}
#### Variant B: {Strategy Name}
```
{Complete prompt text}
```text
**Hypothesis:** {Why this approach might work}
**Risk:** {What could go wrong}
#### Variant C: {Strategy Name}
```
{Complete prompt text}
```text
**Hypothesis:** {Why this approach might work}
**Risk:** {What could go wrong}
### Evaluation Rubric
| Criterion | Weight | Scoring |
|-----------|--------|---------|
| {criterion} | {%} | {how to score: 0-3 scale or pass/fail} |
### Test Cases
| # | Input | Expected Output | Tests Criteria |
|---|-------|-----------------|---------------|
| 1 | {standard input} | {expected} | Correctness, Format |
| 2 | {edge case} | {expected} | Completeness |
| 3 | {adversarial} | {expected} | Robustness |
### Failure Modes to Monitor
- {Failure mode 1}: {detection method}
- {Failure mode 2}: {detection method}
### Recommended Next Steps
1. Run all variants against the test suite
2. Score using the rubric
3. Select the highest-scoring variant
4. Iterate on the winner with targeted improvements
```
## Calibration Rules
1. **One variable per variant.** Each variant should change ONE thing from the baseline.
Changing instruction style AND examples AND format simultaneously makes results
uninterpretable.
2. **Test before declaring success.** A prompt that works on 3 examples may fail on the
4th. Minimum 5 diverse test cases before concluding a variant works.
3. **Failure modes are more valuable than successes.** Understanding WHY a prompt fails
guides improvement more than confirming it works.
4. **Model-specific optimization.** A prompt optimized for GPT-4 may not work for Claude
or Llama. Always note the target model.
5. **Simplest effective prompt wins.** If a zero-shot prompt scores as well as a few-shot
prompt, use the zero-shot. Fewer tokens = lower cost + latency.
## Error Handling
| Problem | Resolution |
| ----------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| No clear objective | Ask the user to define what "good output" looks like with 2-3 examples. |
| Prompt is for a task LLMs are bad at (math, counting) | Flag the limitation. Suggest tool-augmented approaches or pre/post-processing. |
| Too many variables to test | Focus on the highest-impact variable first. Iterative refinement beats combinatorial testing. |
| No existing prompt to analyze | Start with the simplest possible prompt. The first variant IS the baseline. |
| Output format requirements are strict | Use structured output mode (JSON mode, function calling) instead of prompt-only constraints. |
## When NOT to Use
Push back if:
- The task doesn't need an LLM (deterministic rules, regex, SQL) — use the right tool
- The user wants prompt execution, not design — this skill designs and evaluates, it doesn't run prompts
- The prompt is for safety-critical decisions without human review — LLM output should not be the sole input